From Metrics to Root Cause: Why Database Troubleshooting Needs Evidence Correlation
Content reflects the state as of October 2025. AI tooling and model capabilities in this area change frequently.
A database dashboard can show exactly when latency rose, CPU climbed, and lock waits appeared—and still leave the on-call engineer unable to explain which event caused the others.
Technology and product capabilities in this series are evaluated as of June 30, 2026.
Situation
A production request crosses an application service, a connection pool, a database engine, an operating system, storage, and a network before it completes. Each layer emits useful evidence, but most monitoring products present that evidence in separate views. The application dashboard shows P95 latency. The cloud console shows CPU and storage latency. The database exposes waits, sessions, locks, statements, and plans. Deployment and configuration systems record changes somewhere else.
Database-native tools already demonstrate why no single metric is enough. Amazon RDS defines database load as active sessions that are either running on CPU or waiting, then lets operators decompose that load by dimensions such as waits and SQL. MySQL Performance Schema recommends iterative measurement and filtering to improve the signal-to-noise ratio. PostgreSQL’s monitoring guidance explicitly places its cumulative statistics views alongside operating-system tools such as ps, top, iostat, and vmstat.
The evidence exists. The operational gap is assembling it into one defensible explanation while the incident is still happening.
The Problem
Dashboards optimize for visibility, not causal proof. A CPU chart cannot tell whether CPU rose because a deployment changed a query plan, a batch workload arrived, or lock contention finally cleared and released queued work. A slow-query list cannot tell whether a statement became intrinsically slower or merely waited behind storage, locks, or a saturated connection pool.
Adding an LLM directly on top of every raw metric and log does not solve this. It creates a different failure mode: too much context, incompatible timestamps, repeated events, sensitive SQL text, and no explicit boundary between observed facts and generated interpretation. A fluent narrative can then look like a root cause even when the required evidence was never collected.
The core question is therefore: how do we turn fragmented telemetry into a ranked, testable explanation without treating the LLM as the monitoring system or the production operator?
The Evidence-Correlation Control Plane
The architecture separates four responsibilities that are often collapsed into one “AI operations” box:
flowchart TD
A[application symptoms] --> D[deterministic collectors]
B[database waits and workload] --> D
C[host cloud and change events] --> D
D --> E[time-bounded incident evidence pack]
E --> F[rule and statistical anomaly detection]
F --> G[LLM evidence correlation]
G --> H[observations hypotheses missing evidence and actions]
H --> I[DBA verification]
I --> J[approved remediation]
J --> K[post-change validation]
Collectors establish facts. Python, SQL, database APIs, and cloud APIs gather named measurements at known timestamps. They also record provenance: source system, query or API version, collection interval, units, and whether sampling or aggregation occurred.
Anomaly logic establishes what changed. Rules and statistical methods compare the incident with an appropriate baseline. They reduce millions of samples to a tractable set of deltas: which waits appeared, which query digests changed, which resources departed from normal range, and which deployments preceded the transition.
The LLM establishes testable hypotheses. It correlates the reduced evidence across layers, identifies contradictions, and asks for missing evidence. It does not convert correlation into proof. Its output should always separate:
- Observations directly supported by supplied evidence.
- Hypotheses that could explain those observations.
- Missing evidence needed to confirm or reject each hypothesis.
- Actions for the next diagnostic or remediation step.
The DBA establishes authorization. A human verifies the evidence chain, decides whether the root cause is proven, evaluates blast radius, and authorizes any production change. Deterministic automation may execute an approved change, but the language model should not hold production write credentials.
The incident evidence pack is the contract between these stages. It should be sanitized, time-bounded, and organized around four windows:
| Window | Purpose |
|---|---|
| Baseline | Establish the normal workload and resource envelope |
| Pre-incident | Capture the transition and preceding changes |
| Incident | Identify concurrent symptoms, waits, workload, and events |
| Recovery | Test whether suspected causes disappeared with the symptoms |
This temporal shape matters. One snapshot can show a blocked session. The four-window comparison can show that the blocker began after a deployment, lock load rose before API latency, storage stayed inside its baseline, and recovery followed a transaction rollback. That sequence does not automatically prove causality, but it makes the next verification step precise.
In Practice
The documented behavior of existing database systems supports this layered method.
AWS Performance Insights models DB load as average active sessions and decomposes it by wait events and top SQL. That gives an investigator two connected facts: where sessions spend time and which workload contributes to that load. It still does not contain the complete application, deployment, operating-system, or business timeline.
MySQL’s Performance Schema methodology starts with instrumentation, analyzes collected events, filters areas that have been ruled out, and repeats until the relevant signal becomes clearer. The important pattern is iterative diagnosis: evidence eliminates hypotheses and determines what to collect next.
PostgreSQL exposes current activity, wait events, I/O, WAL, replication, checkpointer, and database statistics through separate views while advising operators not to neglect host tools. The documented pattern is cross-layer by design. A wait event describes what a backend is waiting for; it does not, by itself, prove why the underlying resource or conflicting transaction became constrained.
An LLM adds value after these deterministic systems have done their jobs. It can notice that an application deployment, a query-digest shift, a lock-wait increase, and stable storage latency form a stronger transaction-design hypothesis than an infrastructure hypothesis. It can also state that the conclusion remains unproven until the blocking transaction and deployment diff are inspected.
That last sentence is the quality bar: the model must be as explicit about missing evidence as it is about its preferred explanation.
Where It Breaks
| Failure mode | Why the conclusion becomes unsafe | Required control |
|---|---|---|
| Misaligned clocks or time zones | Events appear to occur in the wrong order | Normalize timestamps and retain source clock metadata |
| Missing baseline | Normal batch activity looks anomalous | Compare equivalent workload and business windows |
| Top-N truncation | The actual cause falls outside the retained evidence | Record truncation and allow targeted recollection |
| Sensitive SQL or log payloads | Diagnostic context leaks customer or credential data | Digest, redact, minimize, encrypt, and restrict retention |
| Version-blind recommendations | Valid advice for one engine release is unsafe for another | Include engine, version, deployment model, and feature state |
| Confidence without contradiction testing | A plausible correlation is presented as proof | Require evidence for, evidence against, and a falsifying test |
| Direct production access | A reasoning error becomes an operational incident | Keep model tools read-only and route changes through approval |
What the LLM Cannot Do
Explicit operational boundaries govern the LLM:
- No Raw Customer Data: The LLM must not receive raw result sets, bind or literal parameter values, unrestricted query text, credentials, connection strings, or any column content that can carry PII. It receives shapes, counts, timings, and normalized identifiers only.
- No Live Database Access: The LLM has no network path to any database, host, or cloud control plane. It reasons exclusively over a detached, sanitized incident evidence pack captured by a deterministic collector.
- No Causal Conclusion Without Contradicting Evidence: Every hypothesis must cite both the telemetry that supports it and the telemetry that would falsify it. A ranked hypothesis with no disconfirming test attached is an assertion, not a diagnosis.
- No Silent Gap-Filling: Where the evidence pack lacks a required signal, the LLM must name it explicitly under missing evidence rather than inferring a plausible value. “Not collected” and “collected and normal” are different findings and must never be merged.
- No Change Without a Human Gate: Production changes require human authorization, a stated validation signal, and a defined rollback condition agreed before the change is applied.
What to Do Next
- Problem: Database telemetry is fragmented by layer, so dashboards reveal simultaneous symptoms without establishing their order, relationship, or cause.
- Solution: Build a time-bounded incident evidence pack from deterministic collectors, reduce it with explicit anomaly logic, and ask the LLM for observations, hypotheses, missing evidence, and next actions as separate outputs.
- Proof: Test the design against a resolved incident. The system should reconstruct the transition from baseline through recovery, identify the evidence that actually proved the root cause, and mark unsupported alternatives as rejected or unresolved.
- Action: Choose one recurring database incident and define its evidence contract before writing an LLM prompt: timestamps, application symptoms, host and cloud metrics, database waits, workload deltas, logs, change events, redaction rules, and the human approval boundary.
The next article builds that contract: the sanitized database incident evidence pack an LLM can reason over without receiving an uncontrolled production dump.