Aurora can promote a reader while the application remains unavailable: database failover, endpoint convergence, connection recovery, cache warming, and workload stabilization are different events.

Technology and product capabilities in this series are evaluated as of June 30, 2026.

Situation

Aurora PostgreSQL preserves PostgreSQL behavior but changes the infrastructure boundary. A writer and as many as 15 readers attach to a distributed cluster volume. Each instance still has its own compute, memory, buffer cache, network path, and local temporary storage.

That creates three scopes:

  • instance: writer or one reader has CPU, memory, connection, cache, or local-temp pressure;
  • cluster: workload, shared-volume activity, WAL retention, topology, or control-plane events affect several instances;
  • application: endpoint use, DNS caching, pool behavior, retries, and read-after-write assumptions determine whether recovery is visible to users.

The LLM must preserve these scopes. Combining every signal into “Aurora storage is slow” destroys diagnostic context.

The Problem

Aurora’s managed design invites shortcuts that would be wrong on EC2 and are still wrong here:

  • treating VolumeReadIOPs as a provisioned-IOPS saturation gauge;
  • treating every IO:XactSync wait as an Aurora storage incident;
  • averaging replica lag across readers and hiding one unhealthy instance;
  • assuming shared data storage means shared buffer caches or shared temporary storage;
  • confusing Aurora reader lag with logical-replication slot retention;
  • declaring recovery when RDS reports promotion complete.

Aurora removes EBS administration, not I/O attribution. Cluster counters, instance waits, local storage, WAL, readers, and service events answer different questions.

Did the incident originate in PostgreSQL execution, writer commits, shared-volume reads, local temporary work, a reader, retained WAL, failover selection, or application reconnection?

Build an Aurora-Specific Evidence Pack

flowchart TD
    A[application latency errors retries and pool state] --> H[time-aligned Aurora incident]
    B[cluster endpoint reader endpoint and instance topology] --> H
    C[writer load waits commits and WAL] --> H
    D[per-reader load lag cache and pending reads] --> H
    E[cluster-volume read write and storage metrics] --> H
    F[instance-local temp storage and memory] --> H
    G[RDS events failover tiers and parameter changes] --> H
    H --> I[deterministic scope classification]
    I --> J[LLM evidence correlation]
    J --> K[ranked hypotheses contradictions and gaps]
    K --> L[DBA and application-owner verification]

Keep instance and cluster dimensions. A cluster counter cannot identify responsible SQL; a writer wait cannot prove a service-wide fault.

Symptoms

Evidence patternLeading classificationWhat it does not prove
Writer DB load and IO:XactSync rise with commit throughputCommit path or workload shapeAurora storage is degraded
Cluster VolumeReadIOPs rises with one reader’s IO:DataFileReadReader cache misses or scan workloadA provisioned IOPS limit was reached
FreeLocalStorage falls with temp-file waitsLocal sort, hash, or index-build pressureCluster volume is full
One reader’s lag and pending reads riseReader-specific apply, resource, or workload pressureEvery reader is unhealthy
Logical slot lag and transaction-log usage riseConsumer or slot retention problemAurora replica promotion is impaired
RDS failover event completes but errors continueEndpoint, pool, DNS, retry, or cache recoveryPromotion failed
Post-failover reads and latency spikeCold cache or workload relocationData was copied during promotion

First Five Checks

1. Freeze topology, identity, and service events

Record cluster ARN, version, writer, readers, instance classes, Availability Zones, promotion tiers, parameter groups, storage, endpoints, and type. Preserve contemporaneous RDS events and changes.

Confirm where each application connects. The cluster endpoint follows the writer; the reader endpoint distributes new read connections across eligible readers but does not rebalance sessions already open. Instance endpoints pin callers to one DB instance. During failover, the reader endpoint can briefly direct connections to the newly promoted writer, so endpoint name alone is not proof of role.

This decision tree targets a standard single-Region cluster. Serverless v2, Optimized Reads, Limitless Database, and Global Database add different metrics; label them rather than mixing baselines.

2. Classify DB load and waits per instance

Use Database Insights or an equivalent collector to decompose DB load by instance, wait, SQL, host, and user. Keep writer and readers separate.

Interpret Aurora waits using Aurora documentation:

  • IO:XactSync means a backend is awaiting an Aurora storage acknowledgement for transaction completion; AWS also lists CPU pressure, network pressure, and excessive commit frequency as possible contributors.
  • IO:DataFileRead means a required page was not in that instance’s shared memory and had to be read.
  • IO:BufFileRead and IO:BufFileWrite identify temporary-file work.
  • locks, client waits, CPU, and IPC can still dominate without any Aurora storage problem.

At the technology freeze, AWS was moving the console workflow toward CloudWatch Database Insights. Record the mode, retention, and enabled features. DB load visibility does not imply plan history exists.

3. Separate shared volume, local temporary storage, and retained WAL

The cluster volume stores durable data across Availability Zones. VolumeReadIOPs and VolumeWriteIOPs are cluster activity counters reported in five-minute intervals; AWS describes the read metric as billed reads, not an EBS saturation percentage. Correlate rates with waits, SQL, cache state, and instance load.

Temporary sort, hash, and index-build files use instance-local storage. Track FreeLocalStorage, temp-file waits, and PostgreSQL temp-block deltas per instance. A reader can exhaust local space while the shared cluster volume remains healthy.

Logical replication and DMS introduce a third storage question. Inspect pg_replication_slots, consumer health, OldestReplicationSlotLag, and TransactionLogsDiskUsage. Aurora readers use shared storage; a stalled logical consumer retains WAL through a slot. These require different remediations.

For ordinary writer pressure, collect reset-aware pg_stat_wal deltas with commit throughput and IO:XactSync. This separates more changed data from the same work fragmented into more commits.

4. Diagnose every reader independently

Do not stop at cluster lag extrema. Pair each reader’s AuroraReplicaLag with CPU, memory, connections, DB load, waits, local storage, instance class, and traffic.

Aurora PostgreSQL also exposes cluster status through a read-only function:

SELECT server_id,
       CASE WHEN session_id = 'MASTER_SESSION_ID' THEN 'writer' ELSE 'reader' END AS role,
       replica_lag_in_msec,
       pending_read_ios,
       log_stream_speed_in_kib_per_second
FROM aurora_replica_status();

Timestamp the sample. AWS documents AuroraReplicaLag in terms of the reader page cache, not self-managed physical-standby byte lag. Correlate it with writer changes, reader size, traffic, memory, long queries, and events.

5. Reconstruct failover as a multi-stage timeline

Build timestamps for:

failure detected
promotion started
target selected
new writer available
cluster endpoint changed
old connections failed
new connections succeeded
write health restored
cache and latency returned to baseline

Verify that promotion tiers select an adequately sized reader in another Availability Zone. Inspect DNS caching, TCP keepalive, pool validation, retry backoff, transaction safety, and endpoint use. Promotion does not move existing connections.

Post-failover latency needs proof because each instance owns its cache. Cluster cache management can warm one designated tier-0 reader, but AWS requires matching writer and reader instance classes and topology. Verify it.

Decision Tree

flowchart TD
    A[Aurora PostgreSQL latency or availability incident] --> B{RDS event or topology change}
    B -->|yes| C[reconstruct promotion endpoint pool and cache timeline]
    B -->|no| D{one instance or whole cluster}
    D -->|one instance| E{reader lag or local temp pressure}
    E -->|lag| F[correlate reader resources workload and pending reads]
    E -->|temp| G[identify spilling SQL or index work]
    D -->|cluster| H{commit waits and write rate elevated}
    H -->|yes| I[separate commit frequency CPU network and service evidence]
    H -->|no| J{logical slot retention rising}
    J -->|yes| K[verify consumer progress and slot ownership]
    J -->|no| L[return to SQL locks client waits and upstream services]

LLM Correlation Contract

Require claims at the correct scope:

OBSERVATION
Reader r2 lag and pending reads increased while r1 remained at baseline [replica-08].

HYPOTHESIS
r2 has instance-specific apply or read-workload pressure; cluster-wide storage failure is contradicted.

MISSING EVIDENCE
r2 DB load by wait and its read-traffic distribution for the same interval.

ACTION
Collect the approved per-instance Database Insights slice and reader-endpoint connection distribution.

The model may recommend a collector. It may not fail over, delete a slot, modify tiers, resize instances, or change durability.

In Practice

AWS documents Aurora storage as one cluster volume copied across three Availability Zones, independent of instance count, with separate local storage for temporary work. Shared durability therefore does not imply shared compute, cache, or temporary capacity.

The Aurora metric reference defines volume I/O at cluster scope and AuroraReplicaLag at reader scope. The incident evidence pack must retain those dimensions.

AWS’s Aurora PostgreSQL wait guide warns that some waits differ from upstream PostgreSQL. Its IO:XactSync guidance names workload, CPU, and network contributors. A wait locates time; it does not identify the failed component alone.

AWS documents aurora_replica_status() as exposing lag, LSN, log speed, and pending reads. Its failover guidance separately covers endpoints, DNS, TCP, and application testing. Database and application recovery therefore need different timestamps.

Remediation Options

Proven boundaryImmediate containmentDurable directionValidate with
Excess commit frequencyReduce retry or commit amplificationBatch transactions without weakening correctnessCommit rate, IO:XactSync, latency, errors
Reader-specific pressureRemove or redirect read traffic from that readerRight-size and control reader workload placementPer-reader lag, DB load, pending reads
Local temp exhaustionCancel approved runaway work; defer index buildsQuery reduction, scoped memory, larger or optimized local storageFreeLocalStorage, temp blocks, memory, latency
Logical-slot retentionRestore the owned consumer when safeSlot ownership, lag budgets, consumer monitoringSlot lag, transaction-log usage, consumer offset
Bad failover targetUse only an approved recovery pathCorrect promotion tiers and capacity symmetryPromotion choice, recovery timeline, peak load
Cold-cache recoveryProtect the new writer from immediate overloadValidate cluster cache management where supportedRead waits, cache state, post-failover latency
Application reconnectionShed retries and recycle stale connections safelyEndpoint, DNS, keepalive, pool, and retry designConnection success, write probe, error rate

Rollback Plan

Observability changes can add cost and overhead. If Advanced Database Insights, detailed plan capture, Enhanced Monitoring, or log export was enabled for the investigation, record the previous mode and retention before change; revert after the approved evidence window if that was the plan.

For topology changes, preserve the former promotion tiers and instance configuration. Validate a resized or replacement reader before returning traffic. Deleting a logical slot can discard a consumer’s required WAL and is not a reversible cleanup; require consumer ownership and a rebuild plan. A failover is also not a parameter rollback—applications see broken sessions and in-flight transactions may need safe retry. Define abort criteria before testing it.

Where It Breaks

Failure modeMisleading conclusionCorrection
Cluster metrics assigned to one SQLThe top query caused every volume I/OCorrelate query deltas, waits, and cache misses
Billed I/O treated as a saturation gaugeAurora reached an EBS IOPS ceilingUse the metric as activity and cost evidence
Shared volume treated as shared cacheReader promotion must be warmMeasure cache recovery or validate CCM
Lag maximum used without instance evidenceAll replicas need resizingDiagnose each reader and its traffic
Aurora reader and logical slot conflatedDeleting a slot repairs reader lagKeep physical visibility and logical retention separate
Promotion completion equals recoveryThe outage ended at the RDS eventMeasure endpoint, connection, write, and latency recovery
LLM sees an event after a spikeFailover caused the original incidentReconstruct ordering and preserve contradictions

What the LLM Cannot Do

Explicit operational boundaries govern the LLM:

  • No Raw Customer Data: The LLM must not receive bind values, EXPLAIN ANALYZE output containing literal filter constants, pg_stat_activity.query text with embedded parameters, or table contents. Send normalized pg_stat_statements queryid and digest text, plan shapes, and timings.
  • No Autonomous Production Execution: The LLM cannot run VACUUM FULL, REINDEX, ALTER SYSTEM, pg_terminate_backend, index DDL, or parameter changes. It proposes; a human authorizes, applies, validates, and holds the rollback condition.
  • No Causal Conclusion Without Contradicting Evidence: Every hypothesis must cite both the telemetry that supports it and the telemetry that would falsify it. A ranked hypothesis with no disconfirming test attached is an assertion, not a diagnosis.
  • No Silent Gap-Filling: Where the evidence pack lacks a required signal, the LLM must name it explicitly under missing evidence rather than inferring a plausible value. “Not collected” and “collected and normal” are different findings and must never be merged.
  • No Change Without a Human Gate: Production changes require human authorization, a stated validation signal, and a defined rollback condition agreed before the change is applied.

What to Do Next

  • Problem: Aurora PostgreSQL separates compute from durable storage, while replica visibility, local temporary work, WAL retention, and application recovery remain distinct failure paths.
  • Solution: Preserve cluster and instance scope, correlate Aurora-specific waits with correctly interpreted metrics, and model failover as a staged timeline.
  • Proof: Rehearse a reader overload, logical-consumer stall, temp-space incident, and controlled failover. The workflow should route each to a different owner and reject destructive action without approval.
  • Action: Add topology snapshots, per-reader lag, local-storage evidence, logical-slot retention, RDS events, and application reconnection milestones to the incident evidence pack.

The next article adds the PostgreSQL security boundary: roles, SQL-text exposure, audit evidence, log controls, collector permissions, model redaction, approval gates, and actions the LLM must never perform.

Sources