Aurora MySQL is not MySQL running on a host you cannot access; its writer, readers, endpoints, local temporary storage, and shared cluster volume create different evidence boundaries and different failure modes.

Technology and product capabilities in this series are evaluated as of June 30, 2026.

Situation

The prior MySQL articles classified EC2 failures and traced workload regression from digest to execution behavior. Aurora retains those fundamentals but moves storage durability, instance replacement, and endpoint routing behind a cluster control plane.

An Aurora cluster has one writer and up to 15 readers attached to a distributed cluster volume. Each instance still has its own CPU, memory, page cache, network, connections, database load, and local storage. The reader endpoint balances new connections—not queries.

As of the technology freeze, AWS had announced that the Performance Insights console would redirect to CloudWatch Database Insights after July 31, 2026, while its API would continue. Collectors should use supported APIs and record mode and retention—not depend on screenshots.

The Problem

Aurora’s managed architecture invites several false equivalences:

  • shared cluster storage means readers have no independent performance state;
  • a storage-named wait proves an Aurora storage-service incident;
  • adding a reader distributes existing pooled connections;
  • low cluster-wide replica lag proves every reader is current;
  • the reader endpoint can never resolve to the writer;
  • automatic failover makes client recovery automatic.

None is safe. io/redo_log_flush can reflect excessive commit frequency. A reader can exhaust local temporary storage while the cluster volume is healthy. Long-lived pools can overload one reader. Failover requires clients to reconnect and resolve transaction outcomes.

The core question is: does the incident belong to one DB instance, the writer’s transaction path, the shared cluster volume, reader workload and visibility, endpoint behavior, or a topology transition?

Aurora’s Evidence Boundaries

flowchart TD
    A[application workloads] --> B[writer endpoint — current writer]
    A --> C[reader endpoint — connection balancing]
    A --> D[custom or instance endpoint]
    B --> E[writer — CPU memory cache local storage]
    C --> F[reader one — CPU memory cache local storage]
    C --> G[reader two — CPU memory cache local storage]
    D --> F
    E --> H[shared Aurora cluster volume]
    F --> H
    G --> H
    I[Database Insights — load waits SQL] --> E
    I --> F
    I --> G
    J[CloudWatch — instance and cluster metrics] --> E
    J --> F
    J --> G
    J --> H
    K[RDS events and topology] --> B
    K --> C

Tag every artifact with cluster and instance IDs, role, Availability Zone, endpoint class, engine version, time, and scope. Otherwise, an LLM can correlate the right signals to the wrong node.

Symptoms

SymptomLeading boundaryWhat it does not prove
Writer commit latency and redo-flush load riseWriter transaction pathAurora storage is impaired
One reader’s CPU and DB load riseReader workload or connection skewThe cluster needs another reader
AuroraReplicaLag rises on one readerThat reader is behindEvery reader or global replication is behind
FreeLocalStorage fallsInstance-local temporary or operational filesCluster-volume capacity is low
Read errors appear during failoverEndpoint and connection recoveryData in the cluster volume was lost
Many unrelated SQL digests wait togetherShared resource or contentionEvery query independently regressed

First Five Checks

1. Freeze topology, role, version, and events

Capture cluster members, instance classes, Availability Zones, promotion tiers, writer flag, parameter groups, endpoints, maintenance state, and RDS events. Resolve which instance sampled connections reached.

On Aurora MySQL, innodb_read_only is OFF on the writer and ON on a reader:

SELECT AURORA_VERSION() AS aurora_version,
       @@hostname AS instance_host,
       @@innodb_read_only AS is_reader;

Do not infer role from an instance name. Failover changes the writer while instance endpoints remain tied to instances. Reject deltas across restarts and preserve events before proposing changes.

2. Decompose database load by instance, wait, and SQL

For the writer and affected readers, collect average active sessions, CPU capacity, load by wait, top SQL, and digest references. Cluster aggregation can combine a healthy writer with a saturated reader.

Attach the engine version to waits. Version 3 reports io/redo_log_flush for persistence to Aurora storage; version 2 uses io/aurora_redo_log_flush. Many small commits can raise this wait. io/aurora_respond_to_client points toward result delivery, network capacity, or a slow client. Lock waits still require blocker evidence.

Database load is concurrency, not application latency. Compare it with throughput, commit rate, SQL deltas, instance metrics, and Enhanced Monitoring. Use the vCPU line as a capacity reference only when runnable CPU load dominates.

3. Separate the cluster volume from local storage

The cluster volume contains persistent data and binary logs. VolumeReadIOPs and VolumeWriteIOPs are cluster-level counts over five-minute intervals: useful for trends, but unable to identify the responsible instance or SQL.

Correlate writer CommitLatency, throughput, redo waits, DB load, and SQL commit patterns. For reads, combine volume operations with per-instance cache, load, and digest deltas. Shared storage does not mean shared cache.

Version 3 uses local storage for sorts, index builds, and temporary-table overflow. Readers place TempTable overflow in memory-mapped local files and can return table-full errors. Track FreeLocalStorage, temporary counters, plans, and memory per instance. AuroraVolumeBytesLeftTotal measures cluster-volume capacity, not local space.

4. Diagnose each reader and the connection distribution

Collect AuroraReplicaLag for every reader and pair it with load, resources, connections, query mix, and transaction age. AuroraBinlogReplicaLag describes a separate binary-log topology.

Because the reader endpoint distributes connections, long-lived pools can remain uneven after adding a reader. Record pool lifetime, reconnects, DNS behavior, endpoint, and target. Custom endpoints work only when applications use them.

Long reader queries under default REPEATABLE READ can hold back purge. Correlate RollbackSegmentHistoryListLength with transaction age and query attribution. Reader-specific READ COMMITTED can reduce pressure but permits documented result anomalies; it is a semantic decision.

5. Prove failover readiness before blaming failover

Check whether restart, failover, patch, scaling, or replacement overlaps the symptom. Preserve writer history, event times, dropped connections, ambiguous transactions, endpoint resolution, capacity, and promotion tiers.

Promotion tier 0 is highest and 15 lowest. A cross-AZ reader helps only if it can carry writer load. Use the cluster endpoint for writes; instance endpoints do not follow the writer role.

Validate bounded reconnects, transaction reconciliation, DNS refresh, pool draining, and post-promotion capacity. Forced failover is a controlled production test, not a read-only diagnostic.

Decision Tree

flowchart TD
    A[Aurora MySQL performance incident] --> B{Topology event overlaps symptom}
    B -->|yes| C[failover path — correlate events endpoints and retries]
    B -->|no| D{One instance affected}
    D -->|yes| E{Local storage or memory pressure}
    E -->|yes| F[instance-local path — identify query and resource]
    E -->|no| G{Reader lag or connection skew}
    G -->|yes| H[reader path — inspect pools queries and purge]
    G -->|no| I[instance workload path — inspect waits and SQL]
    D -->|no| J{Redo-flush load and commit latency agree}
    J -->|yes| K[writer transaction path — inspect commit frequency]
    J -->|no| L{Cluster I/O and cache evidence agree}
    L -->|yes| M[shared storage path — identify page demand]
    L -->|no| N[revisit application network and shared dependencies]

In Practice

AWS documents that the Aurora cluster volume writes synchronously across six storage nodes, while readers use asynchronous replication. Separate durable storage from instance visibility and compute.

The wait-event reference defines version-specific redo, client, row-lock, and metadata-lock waits. Waits become actionable through load and workload context, not labels alone.

AWS states that the reader endpoint balances connections rather than queries. The documented pattern explains why adding a replica can leave an existing pool hot on one reader until connections recycle.

The high-availability documentation describes replica promotion, endpoint continuity, promotion tiers, and the brief application-visible failure interval. Aurora automates choosing a new writer; applications still own reconnection and transaction outcome handling.

LLM Correlation Contract

Provide topology and time-series evidence grouped by entity. Every observation must name an instance or cluster scope. A useful hypothesis states:

Hypothesis: writer commit amplification
Supporting: redo-flush load, commit rate, digest and release references
Contradicting: no platform event; reader load remains normal
Missing: transaction batch size by caller
Next diagnostic: attribute commit count to leading write digests
Change authority: none

The LLM may correlate signals and request an approved collector. It must not fail over, reboot, change isolation or tiers, resize, or reroute traffic.

Remediation Options

Proven mechanismCandidate actionValidation and risk
Excessive commit frequencyBatch related writes within correctness limitsCommit rate, redo wait, lock duration, and retry cost
Hot reader from connection skewRecycle pools gradually or route with a custom endpointDistribution by instance; reconnection surge risk
Reader local-temp pressureRewrite or index the query, bound temporary use, or resizeFreeLocalStorage, query result, memory, and cost
Reader-driven purge lagEnd or reshape long reports; consider reader READ COMMITTED only when semantics permitHistory-list trend and result-consistency tests
Writer CPU or SQL regressionApply the digest-and-plan workflow from Article 6 or resize for containmentDB load, SQL latency, cost, and unchanged correctness
Weak failover pathAdd a cross-AZ reader, correct promotion tiers, and fix endpoint or retry behaviorControlled failover recovery objective and reconciliation

Do not treat a replica, resize, reboot, or failover as a substitute for proving the constrained boundary.

Rollback Plan

  1. Restore previous pool lifetime, endpoint, and routing configuration if errors, imbalance, or connection storms increase.
  2. Reverse promotion-tier or parameter-group changes to the recorded values; account for pending-reboot status.
  3. Remove a new reader only after draining connections and proving availability capacity remains sufficient.
  4. Revert reader isolation changes immediately if consistency tests or application contracts fail.
  5. Restore the previous SQL or deployment artifact if digest, latency, error, or correctness gates regress.
  6. For a failover drill, stop the test workload, reconcile ambiguous transactions, verify the intended writer and readers, and restore topology only through an approved runbook. A completed failover cannot be undone like a configuration edit.

Where It Breaks

Failure modeFalse conclusionControl
Per-instance load aggregated at cluster scopeThe writer is saturatedPreserve role and instance dimensions
Five-minute volume counts treated as instantaneousA short storage event is precisely timedRetain waits, SQL, and finer instance telemetry
Redo-flush wait treated as a service faultAurora storage is unhealthyCorrelate commit frequency, latency, SQL, and events
Shared volume treated as shared cacheAll readers have the same read costMeasure cache and DB load per instance
Reader endpoint treated as query balancingA new replica absorbs existing workObserve connection targets and recycle policy
Maximum lag used without per-reader dataEvery reader is staleInspect AuroraReplicaLag on each reader
Local temporary space confused with cluster storageStorage autoscaling will prevent table-full errorsMonitor FreeLocalStorage and temp behavior
Failover tier treated as capacity assuranceThe preferred reader is readyTest instance size, configuration, workload, and AZ placement
Restart crosses the evidence windowCounter drops look like recoveryPreserve uptime, events, and reset provenance
Forced failover treated as harmless validationThe test creates user impactUse approval, retry tests, reconciliation, and abort criteria

What the LLM Cannot Do

Explicit operational boundaries govern the LLM:

  • No Raw Customer Data: The LLM must not receive raw rows, bind values, or unnormalized statement text from the slow query log, events_statements_history, or SHOW PROCESSLIST. Send digest text, digest hashes, timings, and row-examined counts — never literals.
  • No Autonomous Production Execution: The LLM cannot run ALTER TABLE, SET GLOBAL, KILL, index DDL, replication commands, or instance modifications. It proposes; a human authorizes, applies, validates, and holds the rollback condition.
  • No Causal Conclusion Without Contradicting Evidence: Every hypothesis must cite both the telemetry that supports it and the telemetry that would falsify it. A ranked hypothesis with no disconfirming test attached is an assertion, not a diagnosis.
  • No Silent Gap-Filling: Where the evidence pack lacks a required signal, the LLM must name it explicitly under missing evidence rather than inferring a plausible value. “Not collected” and “collected and normal” are different findings and must never be merged.
  • No Change Without a Human Gate: Production changes require human authorization, a stated validation signal, and a defined rollback condition agreed before the change is applied.

What to Do Next

  • Problem: Aurora MySQL incidents cross writer, reader, endpoint, local-storage, cluster-volume, and control-plane boundaries that a MySQL-only evidence pack cannot distinguish.
  • Solution: Freeze topology, analyze load per instance, interpret version-specific waits, separate cluster I/O from local temporary storage, inspect every reader, and treat failover as an application-visible transition.
  • Proof: A second DBA should reproduce the affected entity and failure boundary from immutable evidence, explain contradictions, and predict the metric that changes after remediation.
  • Action: Extend the evidence pack with cluster and instance scope, role history, endpoint targets, promotion tiers, per-reader lag and load, local-storage metrics, RDS events, and failover recovery observations.

The next MySQL article defines the security boundary: diagnostic roles, exported logs, SQL redaction, audit evidence, prompt-injection resistance, and production guardrails.

Sources