Building the Database Incident Evidence Pack for an LLM
Content reflects the state as of October 2025. AI tooling and model capabilities in this area change frequently.
If the input to an LLM is an unlabeled archive of metrics, logs, SQL text, and cloud events, its most confident conclusion may be determined by whichever file is longest—not by the evidence that best explains the incident.
Technology and product capabilities in this series are evaluated as of June 30, 2026.
Situation
The first article in this series separated deterministic collection, anomaly detection, LLM correlation, and DBA authorization. That architecture depends on a clean contract between collection and reasoning: the database incident evidence pack.
An incident evidence pack is not a monitoring export. It is a versioned set of artifacts describing one incident, one set of systems, and explicit time windows. It records how every artifact was collected, transformed, truncated, and redacted. The same pack should support three different consumers:
- deterministic anomaly jobs that compare baseline and incident windows;
- an LLM that needs compact, attributable evidence;
- a DBA who must reproduce or reject the model’s reasoning.
Object storage such as Amazon S3 is a useful landing zone because evidence arrives in different formats and at different rates. But putting files in a bucket does not make them coherent, safe, or suitable for a model.
The Problem
Raw telemetry carries ambiguity that dashboards often hide. A timestamp may describe when an event occurred, when a collector observed it, or the end of an aggregation interval. A missing metric point may mean zero activity, a collection failure, or retention expiration. A counter may have reset after a restart. A “top SQL” export may omit everything beyond its limit. A log line may contain a literal value, token, email address, or customer identifier.
These details affect causal reasoning. If a deployment event uses local time while database waits use UTC, the evidence can reverse cause and effect. If a collector silently converts missing samples to zero, apparent recovery may only be a telemetry gap. If top-N truncation is not declared, the model can incorrectly conclude that an absent query did not contribute to load.
The core question is: what must the evidence pack preserve so an LLM can reason efficiently while a human can still audit every conclusion?
The Evidence Pack Contract
Build the pack as a deterministic pipeline, with a policy boundary before any artifact becomes model-readable:
flowchart TD
A[incident trigger and time windows] --> B[versioned collectors]
B --> C[raw restricted artifacts]
C --> D[normalize timestamps units and identities]
D --> E[aggregate and extract top changes]
E --> F[redact and classify]
F --> G[manifest and model-ready evidence]
G --> H[anomaly analysis]
H --> I[LLM correlation]
I --> J[DBA verification]
The pack needs four layers.
1. A versioned manifest
The manifest identifies the incident and makes collection limitations explicit. Treat a completed manifest as write-once: corrections should create a new version rather than silently changing the evidence record. A compact example:
{
"schema_version": "1.0",
"incident_id": "inc-example",
"generated_at": "2026-06-30T15:30:00Z",
"subject": {
"engine": "postgresql",
"engine_version": "example",
"deployment_model": "ec2",
"environment": "production"
},
"windows": {
"baseline": ["2026-06-30T13:00:00Z", "2026-06-30T13:45:00Z"],
"pre_incident": ["2026-06-30T13:45:00Z", "2026-06-30T14:00:00Z"],
"incident": ["2026-06-30T14:00:00Z", "2026-06-30T14:20:00Z"],
"recovery": ["2026-06-30T14:20:00Z", "2026-06-30T14:45:00Z"]
},
"artifacts": [{
"path": "normalized/database/waits.jsonl",
"collector": "postgres-waits-v3",
"sample_period_seconds": 10,
"completeness": 0.98,
"truncated": false,
"redaction_policy": "diagnostic-v2",
"checksum_sha256": "example"
}]
}
The manifest should also record collector version, requested and returned time range, source system, units, aggregation function, pagination status, row or byte limits, failed collection attempts, and the identity of the policy that approved the artifact for model use. A checksum proves which bytes were analyzed; it does not prove that the collector gathered the right evidence.
2. Normalized evidence
Normalize representation without erasing source semantics. Store timestamps in UTC, but retain both event_time and observed_time where they differ. The OpenTelemetry log data model makes this distinction explicitly. Identify the observed resource separately from the collector so a Kubernetes exporter, cloud API, and SQL collector can refer to the same database instance consistently.
For every metric, preserve:
- unit and statistic, such as bytes per second, count, average, maximum, or percentile;
- sample period and interval boundaries;
- gauge, cumulative counter, or delta semantics;
- missing points and collection errors without converting them to zero;
- reset, restart, failover, and topology-change markers;
- original source dimensions before canonical naming.
Normalization should make two sources comparable, not pretend they measured the same thing.
3. Model-ready summaries and excerpts
Do not make the model discover the incident structure by reading every raw artifact. Produce small, attributable files:
normalized/timeline.jsonl
normalized/application/symptoms.parquet
normalized/host/resource-deltas.parquet
normalized/database/waits.jsonl
normalized/database/locks.jsonl
normalized/workload/query-digests.jsonl
normalized/cloud/service-events.jsonl
normalized/changes/deployments.jsonl
analysis/anomalies.json
analysis/evidence-index.json
Each derived observation should point back to its source artifact and time range. Query digests should replace literals. Log excerpts should include a bounded number of surrounding lines and declare the selection rule. Top-N files should say both limit: 20 and whether additional candidates existed.
The LLM receives the manifest, anomaly summary, evidence index, and selected normalized artifacts. It receives a restricted raw file only when it identifies a specific evidence gap and an authorized retrieval step approves that file.
4. Restricted raw artifacts
Raw SQL text, full diagnostic logs, traces, plans containing literals, and configuration exports belong under a separate prefix and access policy. They are retained for human verification or targeted reprocessing, not automatically copied into prompts.
For S3, enable versioning where overwrites must remain recoverable and store a supported object checksum. AWS documents that versioned objects receive version IDs and that S3 can validate checksums during upload and retain checksum values as metadata. Use encryption, explicit retention, and narrowly scoped read roles. S3 Object Lock can provide write-once-read-many protection when policy or regulation requires it, but it should follow a retention decision rather than become an unexamined default.
In Practice
The documented behavior of AWS telemetry APIs shows why provenance belongs in the pack.
CloudWatch GetMetricData requires start and end times, may round start times to period boundaries, returns an exclusive end boundary, and paginates when response limits are reached. A collector that stores only returned values loses the requested window, rounding behavior, and evidence that another page existed.
Performance Insights GetResourceMetrics similarly returns aligned start and end times, supports explicit periods and aggregate functions, groups results by requested dimensions, paginates, and truncates large response elements such as SQL text. These are not implementation trivia. They determine what “complete evidence for this window” actually means.
The documented pattern is to retain collection semantics alongside values. The LLM should be able to say, “Query text was truncated by the source API” or “the final page was not collected,” instead of treating partial evidence as a complete workload inventory.
Where It Breaks
| Design mistake | Investigation failure | Control |
|---|---|---|
| One bucket prefix for raw and model-ready data | Prompt construction can ingest sensitive artifacts accidentally | Separate prefixes, roles, and classification states |
| Derived summaries without provenance | DBA cannot reproduce the model’s observation | Link every derived item to artifact, query, collector, and time range |
| Silent time alignment | Events appear ordered even when source boundaries differ | Retain requested, aligned, event, and observed timestamps |
| Missing values converted to zero | Collection failure looks like recovery | Preserve nulls, gaps, and collector errors |
| Unreported top-N limits | Absence is interpreted as evidence against a hypothesis | Record limit, population size when known, and truncation |
| Unlimited retention | Diagnostic evidence becomes a permanent sensitive-data lake | Apply classification-specific lifecycle and legal-hold policy |
| Raw files sent by default | Cost and disclosure risk grow while signal quality falls | Start with index and summaries; retrieve narrowly on demand |
What the LLM Cannot Do
Explicit operational boundaries govern the LLM:
- No Raw Customer Data: The LLM must not receive raw result sets, bind or literal parameter values, unrestricted query text, credentials, connection strings, or any column content that can carry PII. It receives shapes, counts, timings, and normalized identifiers only.
- No Live Database Access: The LLM has no network path to any database, host, or cloud control plane. It reasons exclusively over a detached, sanitized incident evidence pack captured by a deterministic collector.
- No Causal Conclusion Without Contradicting Evidence: Every hypothesis must cite both the telemetry that supports it and the telemetry that would falsify it. A ranked hypothesis with no disconfirming test attached is an assertion, not a diagnosis.
- No Silent Gap-Filling: Where the evidence pack lacks a required signal, the LLM must name it explicitly under missing evidence rather than inferring a plausible value. “Not collected” and “collected and normal” are different findings and must never be merged.
- No Change Without a Human Gate: Production changes require human authorization, a stated validation signal, and a defined rollback condition agreed before the change is applied.
What to Do Next
- Problem: Raw monitoring exports conceal collection semantics, mix sensitive and safe evidence, and force an LLM to infer structure that the collectors already know.
- Solution: Create a versioned evidence contract with a write-once manifest, normalized artifacts, model-ready summaries, restricted raw evidence, explicit provenance, and declared truncation.
- Proof: Rebuild a resolved incident from the pack alone. A DBA should be able to trace every generated observation to a source artifact, reproduce its time window, and identify every gap without reopening the original dashboards.
- Action: Implement
manifest.jsonfirst. Require every collector to declare its version, requested and returned window, units, aggregation, completeness, truncation, redaction policy, artifact path, and checksum before adding any LLM integration.
The next article moves one stage downstream: finding statistically and operationally meaningful anomalies before the evidence pack reaches the LLM.