A full WiredTiger cache is not a root cause, a lagging secondary is not an explanation, and a busy primary does not prove that MongoDB needs more hardware. Triage starts by preserving which member did what, when, and under which role.

Technology and product capabilities in this series are evaluated as of June 30, 2026.

Situation

A self-managed MongoDB incident crosses the application, driver, replica set, operating system, storage, and network. One API latency spike can coincide with a primary workload burst, cache eviction on a secondary, flow-control waits, and an election. Each observation is real; only some belong to the causal chain.

MongoDB exposes strong evidence through serverStatus, replSetGetStatus, diagnostic logs, Full Time Diagnostic Data Capture (FTDC), and host telemetry. The challenge is not finding another graph. It is comparing synchronized evidence without losing node identity, process lifetime, or replica role.

The LLM belongs after deterministic collection and anomaly reduction. It correlates bounded evidence, ranks explanations, identifies contradictions, and requests the next safe diagnostic. It does not poll production, invent thresholds, or change the replica set.

The Problem

MongoDB metrics are easy to misread in isolation. WiredTiger is designed to use its internal cache; occupancy alone does not prove pressure. opcounters and many other fields are cumulative since process start, so raw totals cannot be compared across restarts. Replica-set averages hide a single constrained member. Replication lag can exist without flow control engaging, and an old optime may simply reflect an idle primary.

Collection can distort the timeline too. A primary at 10:01 may be a secondary at 10:03. Host CPU collected at one cadence and database counters at another produce false correlations. A single rs.status() snapshot misses transition and recovery.

How do we classify a MongoDB slowdown as workload demand, cache and storage pressure, connection stress, replication trouble, topology change, or an external host failure before investigating individual query shapes?

Build a Role-Aware MongoDB Evidence Pack

flowchart TD
    A[application latency throughput and errors] --> E[time-bounded incident evidence pack]
    B[host CPU memory disk and network per member] --> E
    C[server status and replica status per member] --> E
    D[diagnostic logs FTDC and change events] --> E
    E --> F[validate timestamps roles uptime and counter resets]
    F --> G[derive rates deltas and anomalies]
    G --> H[LLM observations hypotheses contradictions and gaps]
    H --> I[DBA verification]
    I --> J[remediation validation and rollback]

Collect baseline, pre-incident, incident, and recovery windows in UTC. Every sample needs replica-set name, member, role, MongoDB version, process start or uptime, collection timestamp, interval, source, and collector version. Store gauges as observed values and cumulative counters as restart-bounded deltas.

Keep per-member evidence intact. A cluster summary may be derived later, but it cannot replace primary and secondary timelines. Record election, restart, deployment, index-build, backup, maintenance, and infrastructure events beside performance data.

The analysis contract has four outputs: observations tied to evidence IDs; ranked hypotheses with supporting and contradicting facts; missing evidence; and the next bounded action. Confidence is an investigation priority, not a probability of truth.

Symptoms

SymptomPlausible classesEvidence that separates them
Read latency risescache misses, disk, query mix, secondary stalenesspage-read deltas, storage latency, operation mix, read preference, member optimes
Write latency risesprimary saturation, journal or checkpoint work, flow control, networkhost saturation, WiredTiger I/O, checkpoint time, flow-control wait delta, commit progress
One secondary lagsdisk, CPU, network, slow oplog apply, competing readsper-member resources, pingMs, sync source, REPL log events, applied optime
Connections climbtraffic growth, pool leak, churn, slow operationscurrent, active, available, totalCreated delta, request rate, latency
CPU climbsquery work, compression, eviction, replication, background workoperation rates, query-executor deltas, eviction evidence, role, concurrent events
Latency jumps during electiontopology transition, driver discovery, retry behaviorterm, member states, election logs, application errors and retry timing

First Five Checks

1. Validate time, role, and process continuity

Confirm clock alignment, sample gaps, member identity, uptime, replica state, term, and restart events before calculating rates. Reject a delta if the later counter is smaller, uptime moved backward, or the process identity changed. Annotate every primary transition.

This prevents two common errors: interpreting a reset counter as reduced demand and attributing a primary symptom to a host that had already changed roles.

2. Classify host saturation per member

Compare CPU utilization with run queue and throttling; memory availability with swap and cgroup limits; disk latency with throughput, IOPS, and queue depth; and network throughput with loss and retransmits. Use hostInfo.system.memLimitMB to verify the memory limit MongoDB sees, especially in containers.

Host evidence defines constraints, not causality. High disk latency on only the lagging secondary is stronger than a cluster-wide storage average. High CPU with no run queue may still have headroom. Page faults can be limited and sporadic without indicating a fault.

3. Test cache pressure as a chain, not a percentage

From wiredTiger.cache, retain configured bytes, current bytes, dirty bytes, page-read and page-write counters, and eviction-related counters available in the deployed version. Derive interval rates, then align them with application latency, disk reads, disk latency, CPU, and operation mix.

MongoDB uses both the WiredTiger internal cache and the operating system’s filesystem cache. Collection data in the internal cache has a different, uncompressed representation, while filesystem-cached data retains the on-disk format. Increasing the internal cache can therefore reduce memory available to the filesystem and other consumers.

A persuasive cache-pressure hypothesis needs concurrent eviction or read activity, host impact, and workload evidence. Occupancy near its configured maximum is not enough.

4. Separate connection count from connection stress

Track connections.current, active, available, rejected, and the delta of totalCreated. Compare them with request volume and operation latency. A stable pool can keep many connections open harmlessly; a steep creation rate suggests churn even when the current count stays flat. Rejections or exhausted availability require file-limit and process-limit evidence as well as MongoDB metrics.

Do not immediately raise a connection limit. Slow database work can retain connections and make the pool a downstream victim; excessive client concurrency can also amplify queueing and cache churn.

5. Reconstruct replication progress and topology

Capture replSetGetStatus from the appropriate members, including state, health, term, sync source, heartbeat timing, pingMs, applied and durable optimes, and majority commit progress. MongoDB 8.0 also reports written optimes. Compare progress over the same interval instead of treating one timestamp as a rate.

On the primary, derive deltas for flowControl.timeAcquiringMicros, engagement count, and engagement time. Flow control delays writes as majority-commit lag approaches its configured target, but a secondary may lag when there is insufficient load to engage it. Record the oplog window: a disconnected secondary that falls beyond it cannot catch up incrementally and needs initial sync.

Decision Tree

flowchart TD
    A[MongoDB latency or throughput incident] --> B{role change restart or sample break}
    B -->|yes| C[segment timeline and analyze topology]
    B -->|no| D{one member or all members constrained}
    D -->|one| E[compare its host storage network and role]
    D -->|all| F[compare workload and shared infrastructure]
    E --> G{replication progress diverging}
    F --> H{operation mix or concurrency changed}
    G -->|yes| I[test apply disk network and sync-source hypotheses]
    G -->|no| J[test cache workload and host hypotheses]
    H -->|yes| K[identify workload class then inspect query shapes]
    H -->|no| L[test storage checkpoint connection and external causes]
    I --> M[collect missing evidence and verify]
    J --> M
    K --> M
    L --> M

In Practice

MongoDB’s serverStatus documentation defines opcounters as totals since the process started and notes that received operations increment them even when unsuccessful. It also distinguishes operation counts from document-level work. The documented behavior makes interval deltas, restart boundaries, and workload context mandatory.

The WiredTiger documentation states that MongoDB uses an internal WiredTiger cache and the filesystem cache, and warns that container memory limits require appropriate sizing. The serverStatus reference recommends avoiding an internal-cache increase above the default without a specific use case. The derived practice is to prove pressure across cache, OS, disk, and workload before changing size.

MongoDB’s replSetGetStatus exposes state, term, sync source, heartbeat information, and written, applied, durable, and majority-commit progress. Its replication-lag guidance identifies network, secondary resource exhaustion, long-running operations, and inadequate write capacity as possible causes. Lag is therefore a classification input, not a diagnosis.

FTDC is enabled by default and periodically captures serverStatus, replSetGetStatus, oplog collection statistics, and host data. MongoDB documents that it excludes query predicates, query results, user collection samples, credentials, and certificates, but includes hostnames, operating-system details, and startup settings that may still be confidential. It is valuable historical evidence, not an automatically safe LLM payload.

The documented pattern is a two-level investigation: use synchronized member and host evidence to locate the constrained layer, then use query-shape evidence only when workload behavior remains a supported hypothesis.

Remediation Options

Proven causeCandidate responseValidation
Working set causes sustained cache and disk pressurereduce scanned data, improve access path, add memory after sizing analysislatency, page-read rate, disk latency, CPU, query proof
Connection churn or excess concurrencycorrect pool lifecycle, cap client concurrency, stagger reconnectscreation rate, active work, rejects, throughput
Secondary storage or CPU constraintremove competing load, repair host path, resize the memberapply progress, resource headroom, recovery rate
Network degradation between memberscorrect route, loss, bandwidth, or sync-source conditionpacket evidence, heartbeat latency, optime convergence
Oplog window below recovery requirementresize oplog with disk and retention reviewwindow exceeds tested outage and maintenance need
Flow control reflects majority-commit lagrepair the sustaining secondary or capacity bottleneckflow-control wait rate and commit lag fall
Election exposes driver behaviorcorrect topology discovery, timeouts, and retry policycontrolled failover meets error and recovery objectives

Rollback Plan

Define rollback before modifying cache size, oplog size, read routing, pool concurrency, sync behavior, or infrastructure. Record old configuration, target members, expected effect, stop condition, and restoration command. Change one causal lever at a time.

For a cache change, watch process memory, filesystem-cache headroom, swap, eviction, disk latency, and application latency; restore the prior value if host pressure or latency worsens. For connection controls, roll back if throughput falls or rejection increases without relieving the database. For secondary or topology work, preserve voting and failure tolerance, verify the member reaches SECONDARY, and confirm optime convergence before returning traffic.

An oplog enlargement consumes disk and is not a cure for slow application. A sync-source override is temporary and can move load to another member. Neither should survive validation merely because replication lag briefly fell.

Where It Breaks

Failure modeWhy reasoning failsBetter evidence boundary
Cluster averagesHide the constrained member and its rolePer-member aligned timelines
Cache occupancy thresholdConfuses intended use with pressureEviction, I/O, latency, and workload chain
Raw cumulative totalsMix process age with activityRestart-bounded interval deltas
One replica-status snapshotMisses direction, election, and recoveryRepeated status plus event timeline
Lag means networkDisk, CPU, apply work, reads, and sync source can contributeMember host, REPL logs, progress, and network
Flow control is not engagedDoes not prove replication healthDirect optime and commit-progress evidence
More cache is always betterRemoves filesystem and process headroomMemory-limit-aware sizing and rollback
LLM confidence proves causeCorrelation cannot establish mechanismContradiction, targeted test, DBA validation

What the LLM Cannot Do

Explicit operational boundaries govern the LLM:

  • No Raw Customer Data: The LLM must not receive documents, profiler entries containing literal filter values, or currentOp output with embedded query predicates. Send normalized query shapes, planSummary, docsExamined/nReturned ratios, and timings.
  • No Autonomous Production Execution: The LLM cannot create or drop indexes, run moveChunk, alter the balancer, step down primaries, or change cluster configuration. It proposes; a human authorizes, applies, validates, and holds the rollback condition.
  • No Causal Conclusion Without Contradicting Evidence: Every hypothesis must cite both the telemetry that supports it and the telemetry that would falsify it. A ranked hypothesis with no disconfirming test attached is an assertion, not a diagnosis.
  • No Silent Gap-Filling: Where the evidence pack lacks a required signal, the LLM must name it explicitly under missing evidence rather than inferring a plausible value. “Not collected” and “collected and normal” are different findings and must never be merged.
  • No Change Without a Human Gate: Production changes require human authorization, a stated validation signal, and a defined rollback condition agreed before the change is applied.

What to Do Next

  • Problem: MongoDB symptoms cross members and layers, while raw totals, role changes, and cache occupancy create convincing false narratives.
  • Solution: Build a synchronized, role-aware incident evidence pack with restart-bounded deltas across application, host, WiredTiger, connections, replication, topology, and events.
  • Proof: Require every hypothesis to identify supporting and contradicting evidence, survive a targeted diagnostic, and predict a measurable recovery signal.
  • Action: Instrument per-member collection and timeline segmentation first. If the workload remains implicated, continue to Part 2 and prove which query shape, access path, or aggregation stage consumed the resources.

Sources