Adding a read replica is often the first instinct when a database is under load — and it often makes things worse in ways that take weeks to surface. Replicas do increase read throughput, but they do not reduce write pressure on the primary, do not guarantee consistent data, and the operational burden of managing lag, failover, and session consistency accumulates quietly until something breaks.

Situation

Read replicas are standard infrastructure in most relational deployments. AWS RDS, Aurora, Cloud SQL, and self-managed PostgreSQL and MySQL all support them. The pitch is straightforward: offload read traffic to replica nodes, keep the primary free for writes, scale horizontally without sharding.

That pitch is accurate as far as it goes. The problem is what it leaves out.

Engineers reach for replicas when they see high CPU or query latency on the primary. What this misses: replication is not free. Replicas consume resources on the primary for log shipping, introduce lag between writes and reads, and create an eventual-consistency model that most application code is not written to handle.

The Problem

The silent failure mode: your application writes a record, then immediately reads it back, but the read lands on a replica that has not yet applied the write. No error is returned. The user sees stale data. This is the documented behavior of asynchronous replication — the bug is routing the read to a replica without accounting for the replication window.

Under normal conditions, lag is milliseconds and rarely surfaces. Under a write burst — a batch import, a traffic spike, a schema migration — lag climbs to seconds or minutes. During that window, every read routed to a replica is potentially wrong.

The core question: which reads are safe to serve from a replica, and how do you verify that the replica is current enough to answer them?

Core Concept

flowchart TD
    App[Application Client] -->|1. Write Record| Primary[Primary Database Node]
    Primary -->|2. Ship WAL Asynchronously| Replica[Read Replica Node]
    App -->|3. Immediate Read| Replica
    Replica -->|4. Returns Stale Data| App

Replication lag is the delay between a commit on the primary and that commit being visible on a replica. How large the window gets — and what you can do about it — depends on the model.

PostgreSQL streaming replication is asynchronous by default. The primary commits before the replica confirms receipt or apply. pg_stat_replication exposes write_lag, flush_lag, and replay_lag. Under write load, replay lag dominates; the WAL apply process is fundamentally single-threaded for physical streaming replication.

MySQL Group Replication offers synchronous and semi-synchronous modes. Semi-synchronous (the default) confirms receipt but not apply — lag persists at the relay log. Fully synchronous mode blocks the primary commit until a replica confirms receipt, which reduces read lag at the cost of write latency (MySQL 8.0 Reference Manual, Group Replication).

Aurora uses shared distributed storage rather than WAL shipping, so replicas observe page mutations directly. AWS documentation cites typical lag below 10 ms. Faster than streaming replication, but the session consistency problem remains: reads routed to the Aurora reader endpoint immediately after a write can still miss it.

Replication modelLag driverSession consistency risk
PostgreSQL streaming (async)WAL ship and replayYes — read can land before write applies
MySQL semi-synchronousBinlog receipt confirmed; apply asyncYes — same apply lag pattern
MySQL Group Replication (sync)Commit blocked until majority confirms receiptReduced but not eliminated
Aurora read replicasStorage page propagation — sub-10 msYes — writer endpoint required for read-after-write

In Practice

PostgreSQL’s pg_stat_replication.replay_lag can grow unbounded under write load — including during heavy COPY operations — because the WAL apply process cannot keep pace with the primary (PostgreSQL documentation, “Monitoring Replication”). The application has no visibility into this metric unless explicitly instrumented.

AWS documentation on Aurora Replicas explicitly recommends the writer endpoint for read-after-write consistency. Even sub-10 ms storage propagation creates a window where the reader endpoint can miss the most recent write. The shared storage architecture changes the lag mechanism but not the session consistency constraint.

Where It Breaks

ScenarioWhat breaksWhy
Write burstReads return stale data silentlyReplica apply process falls behind; no error surfaces to the client
Replica promotion during failoverWrites fail for 30–120 seconds in streaming replication setupsPrimary must be confirmed, DNS or proxy updated, and applications reconnected
Session consistency violationUser writes then immediately reads stale dataConnection pooler routes the read to a replica before replication applies the write

What to Do Next

  • Problem: Routing reads to replicas without accounting for lag means applications silently return wrong answers during write bursts — no error, just stale data.
  • Solution: Classify reads by consistency requirement before routing. Reads that must see the latest write go to the primary; reads that tolerate bounded staleness go to replicas, with lag monitored against that bound.
  • Proof: Query pg_stat_replication.replay_lag on the primary (or Seconds_Behind_Source in MySQL) during a write spike. If it exceeds your application’s staleness tolerance, replica routing is already producing silent correctness errors.
  • Action: Audit your connection pooler or load balancer this week to confirm which queries reach replicas, then add a lag threshold alert — reject or redirect replica reads when lag exceeds your application’s tolerance.

The cost of replicas shows up in consistency, failover latency, and operational complexity — not on a throughput graph. That mismatch is why replica failures are hard to catch until they surface as user-visible data errors.