When MySQL latency rises on EC2, high CPU is a symptom location—not a root cause—and the first investigation must prove whether work is running, waiting, or stalled below the database.

Technology and product capabilities in this series are evaluated as of June 30, 2026.

Situation

The first engine-specific investigation begins with a deliberately incomplete incident statement: application latency increased, MySQL remains reachable, and EC2 metrics moved in the same window.

That does not distinguish productive CPU work from row locks, EBS latency, memory reclaim, a competing host process, or an EC2 platform limit.

On EC2, the DBA owns the guest and storage layout as well as MySQL. Managed-service assumptions do not apply.

This article classifies the constrained layer and stops. The next article investigates statements and plans; the Aurora article uses Aurora-native evidence rather than treating Aurora as MySQL installed on EC2.

The Problem

Single-signal diagnosis creates predictable mistakes:

  • scaling for high CPUUtilization when sessions are blocked;
  • tuning MySQL while Linux reports I/O or memory stalls;
  • blaming EBS queue length without latency, I/O size, or limit evidence;
  • treating Threads_connected as active work when most connections are sleeping;
  • treating lifetime Performance Schema totals as incident deltas;
  • interpreting absent host memory metrics as healthy memory.

An LLM repeats these mistakes if screenshots and process lists lack timestamps, counter semantics, reset markers, and a comparable baseline.

The core question is: which evidence proves that the incident belongs primarily to compute, memory, storage, connection concurrency, locks, MySQL internal waits, or a layer outside this database host?

Symptoms

Start when user impact and a supporting signal change in the same bounded window.

SymptomWhat it establishesWhat it does not establish
Application latency or errors riseUser impact existsMySQL caused it
Throughput falls while demand remainsWork is not completingWhether it is blocked, stalled, or rejected
EC2 CPU or EBS latency changesResource behavior changedWhich workload caused it
Connections or wait deltas riseSessions or waiting increasedThat either is causal
Lock relationships appearTransactions block one anotherWhich operation created the blocker

First Five Checks

Use a fixed classification order so the investigation does not jump from an application symptom directly to a parameter recommendation.

1. Establish time, restarts, and workload

Align application, CloudWatch, OS, and MySQL timestamps to UTC. Record instance and volume IDs, device mapping, instance type, MySQL and configuration versions, and Uptime. Collect demand, completed work, and connection attempts with latency. Reject counter deltas across a restart or instance replacement.

2. Correlate EC2 service metrics with Linux pressure

AWS documents five-minute periods for basic monitoring and one-minute periods for detailed monitoring. Short events can disappear inside basic averages. CPUUtilization includes physical CPU time used for guest and EC2 code; guest tools can differ.

Collect at least:

  • CPUUtilization, status checks, network bytes, and packets;
  • CPUCreditBalance and surplus-credit metrics when the instance family is burstable;
  • InstanceEBSIOPSExceededCheck and InstanceEBSThroughputExceededCheck on supported Nitro instances;
  • per-volume operations, throughput, queue length, and latency;
  • provisioned volume and instance IOPS and throughput.

EC2 DiskReadOps and DiskWriteOps describe instance-store volumes, not EBS. Use EBS volume metrics and supported Nitro EBS checks.

Use the CloudWatch agent or another collector for guest memory, swap, filesystem, process, and scheduler evidence.

The Linux kernel documentation exposes CPU, memory, and I/O pressure through /proc/pressure/. some measures time with at least some tasks stalled; full measures time with all non-idle tasks stalled for memory or I/O. Capture PSI with run queue, CPU breakdown, memory, swap, major faults, OOM events, filesystem space, device latency, and per-process CPU and I/O.

Compare pressure with the same workload regime; there is no portable safe threshold.

3. Separate connected sessions from running work

MySQL 8.4 defines Threads_connected as currently open connections and Threads_running as threads that are not sleeping. Collect both as gauges, then calculate rates for cumulative variables such as Connections, Aborted_connects, Questions, and statement counters.

SELECT VARIABLE_NAME, VARIABLE_VALUE
FROM performance_schema.global_status
WHERE VARIABLE_NAME IN (
  'Uptime',
  'Connections',
  'Aborted_connects',
  'Threads_connected',
  'Threads_running',
  'Questions',
  'Innodb_buffer_pool_reads',
  'Innodb_buffer_pool_read_requests',
  'Innodb_buffer_pool_wait_free',
  'Innodb_data_pending_reads',
  'Innodb_data_pending_writes',
  'Innodb_data_pending_fsyncs'
);

Label every value as a gauge or counter. A connection storm appears in the rate of Connections. Innodb_buffer_pool_wait_free is cumulative; pending reads and writes are current backlog.

Use performance_schema.processlist for bounded current-state samples. It reads active Performance Schema thread data and can expose other users’ threads with the appropriate privilege. Treat SQL text as restricted evidence; send only digests or redacted text to the LLM.

4. Calculate wait deltas, not lifetime rankings

Performance Schema provides wait summaries by event, thread, account, host, and instance. The global table exposes event count and total, minimum, average, and maximum timer values:

SELECT EVENT_NAME, COUNT_STAR, SUM_TIMER_WAIT,
       AVG_TIMER_WAIT, MAX_TIMER_WAIT
FROM performance_schema.events_waits_summary_global_by_event_name
WHERE COUNT_STAR > 0
ORDER BY SUM_TIMER_WAIT DESC
LIMIT 50;

Summary tables aggregate over time and can be reset. Snapshot baseline, incident, and recovery; subtract by EVENT_NAME; reject deltas across resets; and record instrumentation coverage. Rank added interval wait time, not lifetime totals.

Correlate file, socket, and thread summaries with Linux and EBS evidence. A wait/io/file name alone does not prove storage saturation.

5. Prove locks and classify across layers

MySQL 8.4 exposes active InnoDB transactions through INFORMATION_SCHEMA.INNODB_TRX and lock relationships through Performance Schema data_locks and data_lock_waits. The sys.innodb_lock_waits view makes the dependency easier to inspect:

SELECT wait_age_secs,
       locked_table_schema,
       locked_table_name,
       waiting_pid,
       blocking_pid,
       waiting_query,
       blocking_query
FROM sys.innodb_lock_waits
ORDER BY wait_age_secs DESC
LIMIT 20;

blocking_query can be NULL after the blocker becomes idle. Preserve transaction and process IDs, transaction start time, and prior statement history.

Classify the interval as compute, transaction concurrency, storage, memory, connection management, MySQL internal waiting, or outside the observed host. Give the LLM observations, contradictions, missing evidence, and one next read-only diagnostic—not permission to resize, kill sessions, or change parameters.

Decision Tree

Route the evidence by agreement across layers. A branch names the leading classification, not a proven root cause:

flowchart TD
    A[MySQL latency incident] --> B{Linux pressure changed}
    B -->|yes| C{CPU pressure and mysqld CPU agree}
    C -->|yes| D[compute candidate — inspect statement digests]
    C -->|no| E{IO pressure and EBS latency agree}
    E -->|yes| F[storage candidate — map file waits to volumes]
    E -->|no| G[memory candidate — inspect reclaim swap and allocations]
    B -->|no| H{active lock chain present}
    H -->|yes| I[concurrency candidate — reconstruct blocker chain]
    H -->|no| J{MySQL wait deltas increased}
    J -->|yes| K[engine candidate — inspect dominant interval waits]
    J -->|no| L{connections rose without running work}
    L -->|yes| M[connection candidate — inspect client pools]
    L -->|no| N[outside host candidate — inspect application and network]

In Practice

The documented behavior of the three telemetry layers shows why correlation is necessary.

Amazon EBS documentation defines volume queue length as pending I/O requests and latency as the end-to-end time from sending an operation to receiving its acknowledgement. AWS explicitly says optimal queue length varies by workload. A fixed queue threshold without I/O size, latency, IOPS, and throughput context is therefore not a portable diagnosis.

MySQL’s Performance Schema summary-table documentation describes aggregated wait, stage, statement, transaction, file, socket, memory, error, and status summaries. It also documents reset behavior when summary tables are truncated. The derived operational rule is to capture deltas plus reset provenance rather than present absolute totals to the LLM.

MySQL’s sys.metrics view combines global status, InnoDB metrics, and instrumented memory data, while sys.host_summary groups statement latency, file-I/O latency, connections, scans, and memory by client host. These views can reduce collector complexity, but they inherit Performance Schema coverage and cumulative semantics. A convenient view is not an evidence-completeness guarantee.

The documented pattern is triangulation: the platform reports service and limit behavior, Linux reports resource stalls and competing processes, and MySQL reports sessions, transactions, and instrumented waits. A root-cause hypothesis becomes credible when those layers agree in time and scope—and weaker when one layer contradicts it.

Remediation Options

Triage should select the next bounded action, not authorize a change. Any production mutation follows the proposal, approval, validation, and rollback contract from the preceding article.

ClassificationSafe immediate responseChange that still requires proof and approval
Compute candidateCapture interval statement digests and per-process CPUSQL change, instance resize, or concurrency adjustment
Transaction concurrencyPreserve the blocker chain and identify the owning operationCancel a statement or connection, change transaction scope, or add an index
Storage pathMap MySQL file events to devices and volumes; identify competing I/OProvision more IOPS or throughput, relocate files, or change flush behavior
Host memoryPreserve PSI, reclaim, swap, OOM, allocation, and process evidenceResize memory, alter buffer-pool size, or change per-session limits
Connection managementAttribute creation rate and idle sessions to client hosts and poolsChange pool limits, timeouts, proxy configuration, or admission control
Outside the database hostInspect pool wait time, application queues, DNS, network, and dependenciesMake no database change until new evidence moves the boundary

Emergency containment should use a pre-approved runbook. The LLM may explain why a runbook appears applicable, but it must not choose a target, kill a transaction, resize an instance, or change a parameter directly.

Rollback Plan

The diagnostic workflow is read-only, but it still needs a recovery boundary:

  1. Stop or reduce a collector if its query time, connection use, result volume, or Performance Schema overhead crosses its declared budget.
  2. Retain the last complete evidence-pack version; do not overwrite it with partial recovery samples.
  3. If an approved containment action changes traffic, pool limits, instance capacity, or job scheduling, record the exact prior state and a deterministic restoration step before execution.
  4. Treat session or statement cancellation as non-reversible control input. MySQL may roll back transactional work, but the application still needs a documented retry or reconciliation path.
  5. Roll back on predeclared user-impact, replication, durability, or availability criteria—not because the LLM says the result “looks worse.”

Where It Breaks

Failure modeMisclassificationControl
Five-minute EC2 averages hide a short event“No host anomaly”Enable suitable resolution and retain guest samples
EC2 instance-store metrics treated as EBS metrics“Disk was quiet”Use EBS volume metrics and supported Nitro EBS checks
Wait totals ranked without deltasOld activity becomes current causeSnapshot by window and account for resets and restarts
Performance Schema coverage is incompleteMissing waits look like absence of waitingRecord instruments, consumers, sizing, and lost-event counters
Process list sampled onceTransient concurrency disappearsCollect bounded repeated samples and summaries
Blocking query is nullIdle transaction appears harmlessJoin lock, transaction, thread, and statement-history evidence
High buffer-pool hit ratio treated as proofStorage or flush pressure is dismissedCorrelate pending I/O, file waits, dirty pages, PSI, and EBS latency
Host CPU attributed to MySQL automaticallyBackup or agent load becomes a SQL investigationInclude per-process CPU, I/O, and scheduler evidence
Read-only collector is unboundedDiagnosis worsens the incidentLimit rows, frequency, query time, concurrency, and text collection

What the LLM Cannot Do

Explicit operational boundaries govern the LLM:

  • No Raw Customer Data: The LLM must not receive raw rows, bind values, or unnormalized statement text from the slow query log, events_statements_history, or SHOW PROCESSLIST. Send digest text, digest hashes, timings, and row-examined counts — never literals.
  • No Autonomous Production Execution: The LLM cannot run ALTER TABLE, SET GLOBAL, KILL, index DDL, replication commands, or instance modifications. It proposes; a human authorizes, applies, validates, and holds the rollback condition.
  • No Causal Conclusion Without Contradicting Evidence: Every hypothesis must cite both the telemetry that supports it and the telemetry that would falsify it. A ranked hypothesis with no disconfirming test attached is an assertion, not a diagnosis.
  • No Silent Gap-Filling: Where the evidence pack lacks a required signal, the LLM must name it explicitly under missing evidence rather than inferring a plausible value. “Not collected” and “collected and normal” are different findings and must never be merged.
  • No Change Without a Human Gate: Production changes require human authorization, a stated validation signal, and a defined rollback condition agreed before the change is applied.

What to Do Next

  • Problem: MySQL latency on EC2 spans application, cloud, guest operating system, EBS, connection, transaction, and engine-wait boundaries that no single dashboard can classify.
  • Solution: Align one incident window, collect platform-limit and EBS evidence, measure Linux pressure and competing processes, convert MySQL counters to deltas, and correlate connections, waits, locks, and pending I/O before asking the LLM for hypotheses.
  • Proof: Replay the same evidence pack through the classifier. It should select the same primary layer, expose contradicting and missing evidence, and recommend the same next read-only diagnostic without proposing a production change.
  • Action: Build the EC2 MySQL triage collector with explicit gauge and counter metadata, restart detection, one-minute or finer guest evidence, EBS volume mapping, Performance Schema coverage checks, repeated process samples, and lock-chain capture.

The next article takes the investigation from a classified MySQL workload problem to the responsible statement digest, execution plan, and regression mechanism.

Sources