DesignFlix 9: Replication, Stale Reads and the Next Leader

You save a profile change, refresh the page, and the old text comes back. Refresh again a moment later and the new text appears.

The write may have succeeded both times. The first read could simply have reached a database replica that hadn't caught up yet. Replication gives us more than one copy of data, but it also makes us decide which copy a request is allowed to trust.

Let's follow that profile update, then see what changes when the writer fails.

A leader accepts the change and sends it onward

In a single-leader arrangement, writes go to the leader. It records the change and sends the information needed for replicas to apply it.

Those replicas can provide extra read capacity or serve as failover candidates, depending on the database and configuration. They aren't automatically additional writers, and keeping another copy doesn't by itself configure a safe promotion process.

PostgreSQL's standby documentation provides a concrete example of replication and standby roles. Other databases have different mechanisms, so check what your chosen system actually promises.

The profile can be newer than the replica

Suppose the leader accepts the new bio at time A. The page's next request reaches a replica before it applies that change. The replica returns its older value.

That is replication lag showing up as a stale read. The replica can be healthy and doing useful work while still being the wrong source for this particular request.

For a profile-edit flow, we might return the accepted state directly and route immediate follow-up reads to a source that meets our read-after-write requirement. Some systems offer ways to wait for or identify a replicated position. Choose the mechanism your database supports rather than hoping the lag is always short.

The same issue matters more when the value controls permission, payment status or available stock. Decide which reads can tolerate age and which must validate against authoritative current state.

Synchronous replication changes the wait

With asynchronous replication, the leader can acknowledge without waiting for every replica to apply the change. That keeps the write path responsive, but a lagging failover candidate might not contain the latest acknowledged data.

Synchronous configurations can require acknowledgement from selected replicas before reporting success. The exact durability or replay condition depends on the database and settings. "Synchronous" isn't a universal promise that every reader everywhere immediately sees the write.

Waiting for replicas also affects latency and availability when they are unreachable. The right setting depends on the loss and wait your feature can accept. Put those requirements in writing before treating a default as a business guarantee.

Failure needs a safe new writer

Now the leader stops responding. A replica may be eligible for promotion, but someone must determine which node can take over and prevent conflicting writers.

The old leader might be alive behind a broken network path. If both sides accept writes independently, their histories can diverge. A safe failover mechanism needs coordination and a way to block an old writer when it no longer owns that role.

The application also needs to reconnect to the new writer. Promotion doesn't automatically repair stale connections or settle the outcome of requests interrupted during failure.

Did the last write fail, or did its successful reply disappear when the connection broke?

Handle that uncertain outcome using the operation's identity and database contract. A connection error isn't proof that nothing committed.

Failover is coming.

Other replication models move the conflict

Multi-leader designs allow writes at more than one leader. That can suit geographically separated or intermittently connected workloads, but conflicting changes need a resolution rule. Two offline edits to the same profile field won't resolve themselves because both were accepted locally.

Leaderless designs send reads and writes to replicas under a different coordination model. Cassandra documents its Dynamo-style architecture, including the role of consistency levels. Avoid reducing that model to "there is no leader, so every copy is always current."

These approaches change how writes are coordinated and what readers can observe. Choose around the actual operations and failure requirements, not the appeal of having more writable machines.

Copies aren't a substitute for recoverable history

Replication can copy an accidental deletion just as faithfully as an intended update. If every replica applies it, another live copy won't recover yesterday's value.

Backups and point-in-time recovery address a different requirement. Test them separately from failover. Also monitor replication lag, storage capacity and failed replication processes; a replica that silently stopped receiving updates can look like reassuring redundancy until we need it.

Test the user journey across copies

For our profile edit, test an immediate read from a delayed replica. For failover, test a write interrupted at the moment the leader disappears. For recovery, test a mistaken change that reaches every copy.

Replication is valuable because it can improve read capacity and failure tolerance. It becomes dependable when we know which copy can answer each request, what acknowledgement means, and how a new writer takes over safely. The number of copies is only the beginning of that design.