A model should not decide that database latency is abnormal while it is also trying to explain why: detection must be reproducible before interpretation begins.

Technology and product capabilities in this series are evaluated as of June 30, 2026.

Situation

The incident evidence pack gives an investigation a bounded timeline, normalized telemetry, provenance, and declared gaps. The next stage must reduce that evidence without smuggling a conclusion into it.

Database behavior is rarely stationary. Traffic follows hourly and weekly cycles. Month-end processing creates legitimate load. A deployment changes the workload mix. A failover resets counters and moves activity to another host. A connection count of 700 may be normal for one service and a crisis for another.

An anomaly detector therefore has a narrower job than an incident analyst: identify a measurable departure from a comparable baseline, record how it was found, and say whether it passed an operational relevance gate. It should not name a root cause.

The Problem

Simple thresholds confuse unusual with harmful. CPU above 80 percent may be expected during a batch window. A fivefold increase in lock waits may still be immaterial if it lasts one sample and affects no requests. Conversely, a smaller latency shift can matter when it persists across every writer and breaches an application objective.

Unprepared telemetry creates additional traps:

  • cumulative counters look like ever-growing incidents unless converted to rates;
  • a restart looks like a negative workload spike unless the reset is recognized;
  • missing samples look like recovery when nulls are replaced with zero;
  • an incident-contaminated baseline makes degraded behavior appear normal;
  • a fleet-wide average can hide one saturated writer or shard;
  • many simultaneous tests create a stream of statistically unusual but operationally irrelevant points.

The core question is: how should deterministic code convert a time-bounded evidence pack into a small set of anomalies that an LLM can correlate without pretending that correlation is proof?

Build a Deterministic Anomaly Layer

Make anomaly detection an explicit transformation between normalization and LLM analysis:

flowchart TD
    A[normalized evidence pack] --> B[quality and completeness gate]
    B --> C[select comparable baseline]
    C --> D[classify gauge counter event or distribution]
    D --> E[apply deterministic detector]
    E --> F[persistence magnitude and impact gate]
    F --> G[versioned anomaly record]
    G --> H[LLM correlation]

Select a comparable baseline

“The previous hour” is not automatically normal. Baseline selection should be policy, not convenience. Match the incident to the same workload regime where possible:

  • hour of day and day of week;
  • request rate or transaction-volume band;
  • batch, backup, maintenance, and reporting calendar;
  • engine version, instance class, topology, and configuration generation;
  • application release and major feature state.

Exclude known incidents, load tests, failovers, deployments, and incomplete collection periods. Keep those exclusions in the anomaly record. A detector that cannot find enough comparable history should return insufficient_baseline; it should not widen its search silently until it produces an answer.

This matters even with managed anomaly services. Amazon CloudWatch anomaly detection models expected values from historical data and accounts for trends plus hourly, daily, and weekly patterns. AWS also lets operators exclude specified periods from training. The documented behavior reinforces the design principle: seasonal modeling does not remove the need to govern which history represents normal operation.

Apply the detector that matches the signal

No single statistical method fits every diagnostic source.

SignalFirst useful methodWhy
Capacity invariantStatic or derived limitConnections, storage runway, and queue capacity have operational boundaries
GaugeBaseline delta or robust deviationCPU, memory pressure, and active sessions can move above or below normal
CounterReset-aware rate, then deviationRaw cumulative values are not comparable across windows
Latency distributionPercentile envelopeTail regressions can disappear inside an average
State or eventRule and frequency changeRestart, deadlock, failover, and error-class appearance are discrete evidence
TransitionChange-point detectorThe time behavior changed may be more useful than the largest value

For noisy metrics, median and median absolute deviation are a practical robust starting point. MAD measures dispersion around the median and is less affected by an extreme observation than standard deviation, as the SciPy reference documents. That does not make a fixed MAD multiplier universally correct. Calibrate sensitivity per metric and preserve the chosen method, scale, sample count, and threshold.

Counters require semantic preprocessing. Prometheus documents that rate() calculates per-second counter growth and adjusts for breaks in monotonicity such as target restarts. It also advises applying rate() before aggregation so resets remain detectable. The collector should similarly emit reset and restart evidence before any detector compares rates.

Gate statistical anomalies by operational meaning

A point outside an expected band is a candidate, not a root-cause finding. Require deterministic gates such as:

  • Magnitude: Was the absolute or relative change large enough to investigate?
  • Persistence: Did it survive enough consecutive samples or time windows?
  • Scope: Did it affect one session, one instance, all writers, or the service?
  • User impact: Did latency, errors, throughput, or an objective change in the same interval?
  • Capacity proximity: Did the signal approach a known resource or safety limit?

Do not combine these gates into an unexplained “AI confidence” score. Emit their individual results. A lock-wait spike can be statistically extreme but fail persistence; a storage-latency increase can be moderate yet pass persistence, scope, and impact.

Emit an auditable anomaly record

The output is structured evidence, not prose:

{
  "schema_version": "1.0",
  "anomaly_id": "anomaly-example",
  "metric": "database.commit_latency",
  "entity": "writer-example",
  "method": "robust_mad",
  "baseline": {
    "selection_policy": "same_workload_regime",
    "sample_count": 0,
    "median": null,
    "mad": null,
    "excluded_windows": []
  },
  "observed": {
    "window": ["example-start", "example-end"],
    "value": null,
    "duration_seconds": null
  },
  "gates": {
    "magnitude": "example",
    "persistence": "example",
    "scope": "example",
    "user_impact": "example"
  },
  "related_evidence": [],
  "caveats": [],
  "provenance": {
    "artifact": "normalized/example.parquet",
    "collector": "example-version"
  }
}

Version the detector configuration with the record. Preserve candidates that failed operational gates separately from promoted anomalies so a DBA can audit suppression. Related evidence may identify co-occurrence—CPU rose in the same interval as query latency—but must not label one signal as the cause of another.

In Practice

The documented behavior of monitoring systems shows why preprocessing and context cannot be delegated to a language model.

CloudWatch builds a band for a specific metric and statistic; a model for AVG is distinct from one for another statistic. Its training can include up to two weeks of data, adapts as values evolve, and supports exclusions for unusual periods. A band crossing is therefore attributable to a particular model configuration, not a timeless definition of abnormality.

Prometheus distinguishes gauges from counters in its function semantics. rate() is for counters, while predict_linear() is documented for gauges. Applying the wrong transformation changes the question being asked before any LLM sees the result.

The NIST Engineering Statistics Handbook frames a control-limit crossing as a reason to investigate an assignable cause and notes that limit placement changes the risk of investigating normal variation. The documented pattern is detection followed by investigation—not detection presented as causal proof.

Where It Breaks

Failure modeMisleading resultControl
Baseline includes the incidentPersistent degradation becomes “normal”Curate exclusions and version baseline membership
Baseline ignores workload regimeLegitimate cycles become anomaliesMatch time, volume, topology, and calendar state
Cold start or sparse dataUnstable bands imply false precisionReturn insufficient evidence and use explicit rules
Counter treated as gaugeAccumulation or restart becomes a spikeDerive reset-aware rates before detection
Fleet average onlyOne hot writer, node, or shard disappearsDetect at resource level before aggregation
Missing values imputed as zeroTelemetry loss looks like recoveryPreserve nulls and fail the completeness gate
Many ungoverned testsCandidate volume overwhelms investigationApply persistence, impact, and capacity gates
Co-occurrence labeled as causeSymptoms become circular proofEmit related evidence without causal direction
Detector adapts during degradationDrift absorbs the failureFreeze or quarantine learning during incidents

What the LLM Cannot Do

Explicit operational boundaries govern the LLM:

  • No Raw Customer Data: The LLM must not receive raw result sets, bind or literal parameter values, unrestricted query text, credentials, connection strings, or any column content that can carry PII. It receives shapes, counts, timings, and normalized identifiers only.
  • No Live Database Access: The LLM has no network path to any database, host, or cloud control plane. It reasons exclusively over a detached, sanitized incident evidence pack captured by a deterministic collector.
  • No Causal Conclusion Without Contradicting Evidence: Every hypothesis must cite both the telemetry that supports it and the telemetry that would falsify it. A ranked hypothesis with no disconfirming test attached is an assertion, not a diagnosis.
  • No Silent Gap-Filling: Where the evidence pack lacks a required signal, the LLM must name it explicitly under missing evidence rather than inferring a plausible value. “Not collected” and “collected and normal” are different findings and must never be merged.
  • No Change Without a Human Gate: Production changes require human authorization, a stated validation signal, and a defined rollback condition agreed before the change is applied.

What to Do Next

  • Problem: Raw metrics contain seasonality, resets, gaps, outliers, and aggregation effects that can mislead both statistical detectors and LLMs.
  • Solution: Select comparable baselines, classify signal semantics, run versioned deterministic detectors, and promote candidates only after explicit persistence, magnitude, scope, and impact gates.
  • Proof: Re-run the same detector version against the same evidence pack. It should produce the same anomaly records, exclusions, gate decisions, and source references without an LLM call.
  • Action: Implement one end-to-end detector for a user-impact metric and one database-native metric. Include counter-reset handling, an insufficient_baseline result, failed-gate retention, and provenance before expanding coverage.

The next article defines the safety boundary after correlation: what an LLM may observe, request, recommend, or never execute when database performance remediation reaches production.

Sources