Hi Pythonistas!
Your system has:
- multiple app servers.
- load balancer.
- Redis cache.
- CDN.
- Sharded database.
But there's a problem we haven't solved.Reads vs Writes.
In most applications:
Reads → 90% of traffic
Writes → 10% of traffic
Social media:
Post a photo → 1 write
Photo viewed → 1,000,000 reads
E-commerce:
Place an order → 1 write
Product viewed → 10,000 reads
Your single database handles both.All reads AND writes hit the same machine.What if you could separate them?
Writes → one machine.
Reads → many machines.
That's replication.
What Is Replication?
Replication = keeping multiple copies of your dataon multiple machines.One machine receives all writes.Other machines copy those writes automatically.All machines can serve reads.
┌─→ Replica 1 → handles reads
Primary (writes) ───┼─→ Replica 2 → handles reads
└─→ Replica 3 → handles reads
Primary = receives all writes.
Replica = copies from primary, serves reads.
Also called:
- Primary / Replica
- Leader / Follower
- Master / Slave (older term, being phased out)
How Replication Works
Step 1 - Write to Primary
INSERT INTO users (name) VALUES ('Afsal')
→ Primary executes the write
→ Data committed to primary's disk
Step 2 - Write recorded in replication log
Primary keeps a log of every change.
PostgreSQL → WAL (Write-Ahead Log)
MySQL → Binary Log (binlog)
MongoDB → Oplog (Operations Log)
Every INSERT, UPDATE, DELETE.
In order.
Step 3 - Replicas pull from log
Replicas connect to primary.
Continuously pull new entries.
Apply them to their own data.
Replica reads WAL:
"INSERT INTO users (name) VALUES ('Afsal')"
→ executes same INSERT
→ now has same data as primary
Step 4 - Replica serves reads
SELECT * FROM users WHERE id = 123
→ goes to any replica
→ same data as primary (mostly)
Synchronous vs Asynchronous
The biggest decision in replication.
Synchronous
Primary waits for replica to confirm before acknowledging write to app.
App writes
↓
Primary writes to disk
↓
Sends to Replica
↓
Replica confirms
↓
Primary confirms to App ← only now
Good:
- zero data loss.
- replica always up to date.
Bad:
- slow writes.
- if replica is down → writes fail.
Asynchronous
Primary confirms write immediately. Sends to replica in background.
App writes
↓
Primary writes to disk
↓
Primary confirms to App ← immediately
↓ (background)
Sends to Replica
Good:
- fast writes.
- primary not blocked by replica.
Bad:
- replica slightly behind primary.
- primary crashes before sync → data loss.
Semi-synchronous
Middle ground.
Primary waits for at least ONE replica.Not all.
Primary writes
↓
Waits for ONE replica to confirm
↓
Confirms to App
Good balance.Most common production setup.
The Data Loss Problem
Here's the uncomfortable truth.Primary has 100 writes.Replica synced only 97.
Primary crashes.
Primary (crashed):
Write 98 ✅
Write 99 ✅
Write 100 ✅
Replica (promoted):
Write 98 ❌
Write 99 ❌
Write 100 ❌
Replica promoted to new primary.Writes 98, 99, 100 are gone forever.This is called replication lag data loss.
RPO - How Much Loss Can You Afford?
This potential data loss has a name:
RPO - Recovery Point Objective.
How much data you can afford to lose.
RPO = 0 → zero data loss acceptable
RPO = 1 second → can lose 1 second of writes
RPO = 1 minute → can lose 1 minute of writes
Different businesses. Different RPOs:
Bank transaction → RPO = 0 (zero tolerance)
Social media like → RPO = seconds (acceptable)
Analytics event → RPO = minutes (fine)
Solutions to Data Loss
- Synchronous Replication
- Semi-Synchronous
Replication Lag
With async replication:replica is always slightly behind primary.This delay = replication lag.Usually milliseconds.Can be seconds during high write load.
Problems it causes:
Read-your-own-writes:
User updates profile photo.
Write → Primary.
User refreshes immediately.
Read → Replica (not synced yet).
Old photo shows.
User thinks update failed.
Monotonic reads:
Read from Replica 1 → sees 1000 rows
Read from Replica 2 → sees 998 rows
Data appears to go backwards.
Confusing.
Solutions:
Read-your-own-writes:
After a write → route that user's reads to primary
For 5-10 secondsThen back to replicas
Monotonic reads:
Route each user's reads to same replica always(sticky reads)
Read/Write Splitting
Application sends:
Writes → Primary
Reads → Replicas
Primary Failure What Happens?
Primary goes down.Writes failing.System broken.
Manual failover:
Admin promotes a replica.
Old Primary (down)
Replica 1 → promoted to new Primary
Replica 2 → now syncs from new Primary
Update DNS or proxy to point to new primary.
Downtime: minutes typically.
Automatic failover:
Tools detect failure automatically.
Elect new primary.
Redirect traffic.
No human needed.
PostgreSQL → Patroni, Repmgr
MySQL → MySQL Router, Orchestrator
Redis → Redis Sentinel
MongoDB → built-in replica sets
Primary goes down
↓
Sentinel detects (10-30 seconds)
↓
Most up-to-date replica elected
↓
Promoted to primary
↓
DNS updated
↓
App reconnects
Downtime: 10-30 seconds typically.
Split Brain Problem
Primary goes down.Network partition primary alive but unreachable.Replica thinks primary is dead.
Gets promoted to new primary.Now TWO primaries.Both accepting writes.Data diverges.When network recovers conflict.
Who is right?
Solution: quorum.Need majority of nodes to agree before electing new primary.
With 3 nodes:
need 2 out of 3.
If only 1 node can see others no election.
Prevents split brain.This is why minimum 3 nodes for high availability.Not 2.
Replication Topologies
Single Primary - Multiple Replicas
Most common.
Primary
├── Replica 1
├── Replica 2
└── Replica 3
Cascade Replication
Replica replicates from another replica.
Primary → Replica 1 → Replica 2 → Replica 3
Reduces load on primary.
Increases lag down the chain.
Multi-Primary
Multiple nodes accept writes.
Primary 1 ←→ Primary 2
Good:
write availability if one goes down.
geographic distribution.
Bad:
conflict resolution is hard.
Primary 1: UPDATE users SET name='Afsal' WHERE id=1
Primary 2: UPDATE users SET name='Afsan' WHERE id=1
Which wins? Used by CockroachDB, Cassandra, DynamoDB.
Replication vs Sharding
Two different solutions.Two different problems.
Replication → scale READS
data redundancy
high availability
same data on multiple machines
Sharding → scale WRITES and STORAGE
data split across machines
different data on each machine
Used together:
Shard 1: Primary + 2 Replicas
Shard 2: Primary + 2 Replicas
Shard 3: Primary + 2 Replicas
Each shard has its own primary.Each primary has replicas.Scales both reads and writes.High availability on each shard.This is how Instagram, Twitter work.
The CAP Theorem in Action Remember CAP theorem?
Consistency → all nodes have same data
Availability → system keeps working
Partition → network splits happen
Replication forces you to choose:
Sync replication:
Consistency ✅ → replica always up to date
Availability ❌ → writes fail if replica down
Async replication:
Availability ✅ → writes succeed even if replica down
Consistency ❌ → replica can be behind
No perfect answer.
Choose based on your RPO.
RPO = 0 → synchronous
RPO > 0 → asynchronous with monitoring
Real World
Instagram (early days):
PostgreSQL
1 Primary → all writes
2 Replicas → all reads
90% of traffic = reads.
Replicas handled 90% of database load.Primary only handled 10%.
System handled millions of users.Only sharded much later.
The lesson:
Replication before sharding.
Much simpler.
Often enough for a long time.
Mental Model
Replication → multiple copies of data
Primary → receives all writes
Replica → copies from primary, serves reads
WAL/Binlog → log of all changes
Sync → zero loss, slower, less available
Async → fast, available, possible loss
Semi-sync → one replica confirms, balance
Replication lag → replica behind primary
RPO → how much data loss is acceptable
Read/write split → writes to primary, reads to replicas
Failover → replica promoted when primary fails
Split brain → two primaries accepting writes
Quorum → majority prevents split brain
STONITH → fence old primary before electing new
Rollback → old primary discards diverged writes
Replication + Sharding → scalable + available system
What's Coming Next
The concept that ties everything together.CAP Theorem and PACELC.The fundamental tradeoffs of every distributed system.
Why you can never have everything.And how to make the right choice.
