A balanced MongoDB collection can still have a hot shard. The balancer distributes ranges under placement constraints; it does not equalize traffic, scatter-gather work, moving write frontiers, or replica health.

Technology and product capabilities in this series are evaluated as of June 30, 2026.

Situation

MongoDB Parts 1 and 2 classified cluster pressure and traced expensive query shapes. Now suppose one shard remains at 90% CPU while its peers sit near 30%. Latency rises, replication lag appears inside that shard, and the balancer reports the collection as compliant.

That is not contradictory evidence. A sharded cluster has three independent distributions:

  1. Routing: how many shards each operation targets.
  2. Ownership: where owned documents, bytes, and ranges reside.
  3. Service capacity: how much useful work each shard replica set can perform.

MongoDB’s balancer runs on the config-server primary, respects zones, and moves ranges when placement policy requires it. Its compliance verdict addresses ownership under those rules. It does not prove that traffic or capacity is balanced.

An LLM can correlate router, shard, migration, cleanup, and replica evidence. It cannot infer a shard-key defect from CPU alone or safely turn a suspected hotspot into an immediate resharding command.

The Problem

The same symptom—one shard is slow—has very different causes. A high-frequency shard-key value can concentrate targeted traffic. A monotonic key can send current inserts to the range containing MaxKey. Queries missing the shard key can scatter to every shard. Zone placement can intentionally concentrate data. Range migration can tax the donor and recipient, while delayed orphan cleanup creates later I/O pressure. Or the ownership may be sound and one shard’s primary, storage, or secondary replication path may simply be unhealthy.

These mechanisms can overlap. Migration begins after imbalance, so migration counters correlated with latency do not prove migration caused the original event. Replica lag may be a consequence of hot writes, not the cause of application latency.

How do we prove which distribution failed before changing placement or the shard key?

Build a Three-Axis Sharding Evidence Pack

flowchart TD
    A[incident interval and topology snapshot] --> B[router targeting and query shapes]
    A --> C[owned data ranges zones and orphans]
    A --> D[per-shard hosts replica roles and capacity]
    C --> E[migration and range-deletion timeline]
    B --> F[deterministic interval aggregates]
    D --> F
    E --> F
    F --> G[LLM observations hypotheses contradictions and gaps]
    G --> H[DBA verification]
    H --> I[bounded remediation]
    I --> J[traffic ownership and replica validation]

Capture a time-bounded topology manifest: versions, mongos identities, config-server primary, shards, replica-set roles, namespaces, shard keys, zones, and balancer state. Every metric needs UTC timestamps, source, units, counter semantics, and process start time.

For routers, aggregate operation counts by query shape and number of shards targeted. For ownership, capture per-shard owned documents and bytes, orphaned documents and bytes, zone constraints, and collection compliance. For each shard replica set, retain CPU, memory, disk latency and throughput, network, connections, operations, WiredTiger cache pressure, tickets or queues where applicable, replication lag, oplog window, elections, and member state.

Migration and deletion counters are cumulative and reset with process restart. Convert them to restart-bounded interval deltas; never compare raw totals across differently aged processes.

Symptoms

Evidence patternPlausible mechanismEvidence still required
One shard owns similar bytes but serves far more operationshot key, query-frequency skew, or moving write frontiershard-key value distribution and targeted shape counts
All shards receive most readsscatter-gather routingmongos targeting counters and representative explain()
One shard owns materially more dataplacement imbalance or zone constraintowned bytes, ranges, zones, and balancer status
Donor and recipient latency rise during migrationclone, catch-up, critical-section, or network workmigration phase, interval deltas, host and network metrics
Orphaned bytes or deletion queue growsrange cleanup delayed or resource constrainedcursor activity, deletion-task trend, disk and cache pressure
Only one shard primary is saturatedreplica-set member or storage problemmember-level metrics, elections, lag, and host comparison
Balancer says compliant but latency remains skewedtraffic or capacity imbalancerouter fan-out and per-shard service evidence

First Five Checks

1. Freeze the topology before interpreting load

Record which mongos observed each request, which node was config primary, and which members were shard primaries during baseline, transition, incident, and recovery. Elections and role changes otherwise make host graphs look like abrupt capacity loss.

Record balancer enablement and window, but do not disable it reflexively. MongoDB warns that leaving balancing disabled can degrade performance as ownership diverges. A scheduled window uses the config-server primary’s local time, so verify timezone before concluding that a migration ran “outside” the window.

2. Measure routing fan-out

A query containing the shard key or a usable compound-key prefix can target one shard or a subset. Without it, mongos commonly broadcasts and merges results. Cost depends on returned work, network, and the slowest target.

Use shardingStatistics.numHostsTargeted from serverStatus on each mongos to derive interval rates for oneShard, manyShards, and allShards operations. Break the change down by command and query shape. Then use representative explain() evidence to inspect the shards, mergeType, and splitPipeline fields where applicable.

A rising all-shard ratio aligned with latency supports scatter-gather amplification. It does not prove the shard key should change: a missing application predicate, changed aggregation pipeline, or routing-unfriendly index/query design may be the narrower defect.

3. Separate owned data from hot traffic

Run $shardedDataDistribution through mongos with an approved diagnostic role. It reports numOwnedDocuments, ownedSizeBytes, numOrphanedDocs, and orphanedSizeBytes per shard and namespace. Pair that snapshot with sh.balancerCollectionStatus(), which can identify draining-shard, zone, or imbalance violations.

These are placement facts, not traffic facts. Equal bytes can hide a high-frequency tenant or current-time range. Low shard-key cardinality limits distribution; frequent values create indivisible hot ranges; monotonic keys focus new inserts at one edge. Hashed sharding may distribute those writes but makes range predicates less targetable.

Use analyzeShardKey only as a bounded investigation with representative sampling. Its key characteristics and read-write distribution help test candidate keys, but sampling has cost and historical migrations can distort its monotonicity inference. The LLM should report the sample interval and coverage, not convert the output into certainty.

4. Correlate migration with cleanup

During range migration, the donor continues serving writes while the recipient creates missing indexes, clones documents, and catches up. Metadata then commits, and donor cleanup happens asynchronously. MongoDB can begin another migration without waiting for deletion of the previous range.

Derive interval deltas for donor migrations started, committed, and aborted; donor migration time; recipient documents and bytes cloned; and documents deleted by the range deleter. Track shardingStatistics.rangeDeleterTasks for queued or active work. Use $shardedDataDistribution to verify whether orphan counts fall.

This sequence matters. A recipient CPU spike during cloning, a short metadata-transition pause, and donor I/O after commit have different remedies. Long-running reads or cursors can delay deletion. cleanupOrphaned is deprecated in MongoDB 8.0 and only waits for cleanup; it is not a manual deletion remedy.

5. Inspect the replica set inside every shard

Each shard is a replica set, not one capacity number. Compare its primary, secondaries, and equivalent peer-shard members. Check write rate, disk latency, WiredTiger cache, replication lag, oplog window, elections, network, and read-preference routing.

If one primary is slow while ownership and targeted traffic are normal, investigate member or infrastructure health. If secondaries lag only on the hot shard, determine whether replication pressure follows concentrated writes or whether a slow secondary is constraining durability and failover readiness. Do not solve a primary hotspot by shifting reads to secondaries without proving staleness tolerance and query capacity.

Decision Tree

flowchart TD
    A[one shard is hot] --> B{router fan-out changed}
    B -->|yes| C[identify broadcast shapes and missing targeting predicates]
    B -->|no| D{traffic per owned byte is skewed}
    D -->|yes| E[test hot values monotonicity and zone constraints]
    D -->|no| F{ownership is noncompliant}
    F -->|yes| G[inspect balancer state windows and migration blockers]
    F -->|no| H{migration or cleanup overlaps}
    H -->|yes| I[separate donor clone commit and deletion phases]
    H -->|no| J[inspect shard replica members storage and elections]
    C --> K[bounded remediation and validation]
    E --> K
    G --> K
    I --> K
    J --> K

The model’s output must separate observations, hypotheses, missing evidence, and actions. “Shard 2 handled 71% of targeted writes” is an observation. “A high-frequency tenant key caused the hotspot” is a hypothesis until value-frequency and routing evidence support it.

In Practice

The documented MongoDB behavior produces a useful diagnostic pattern. Suppose owned bytes are within policy, collection status is compliant, all-shard reads remain flat, and one shard receives most targeted inserts. Migration activity begins only after the traffic divergence. That evidence contradicts “the balancer caused the incident” and supports a traffic hotspot—possibly a monotonic frontier or high-frequency key value.

Now change the evidence: targeted traffic is stable, donor and recipient clone counters rise with network and disk activity, and range-deletion tasks accumulate after commits. Migration is then a plausible latency amplifier, while the original reason for balancing still needs explanation.

The LLM can rank these hypotheses and identify contradictions across hundreds of interval summaries. A DBA must verify the actual shard key, zone policy, representative query plans, migration phase, and replica health before authorizing a change.

Remediation Options

Prefer the narrowest reversible action:

  • Fix a missing shard-key predicate, aggregation routing defect, or unintended multi-shard write, then canary the application change.
  • Move discretionary maintenance away from the incident interval or adjust the balancing window after confirming the config-primary timezone. Preserve a maximum pause and automatic re-enable procedure.
  • Relieve a degraded shard member or infrastructure bottleneck using the platform’s tested failover, replacement, or capacity procedure.
  • Correct zone ranges or membership only after proving the current constraint causes concentration and verifying data-residency requirements.
  • Split a divisible oversized range where appropriate. Do not clear a jumbo flag blindly; an indivisible high-frequency value remains indivisible.
  • Refine the shard key when adding suffix fields improves cardinality or range divisibility. Refinement keeps the existing key as a prefix and does not immediately redistribute existing ranges.
  • Reshard only when the current key cannot support required routing and distribution. MongoDB 8.0 can force redistribution with the same key, but resharding is write- and storage-intensive, permits one collection operation at a time, and includes a write-blocking phase.

Rollback Plan

Define rollback per change. Application routing changes need feature flags and old-query compatibility. Balancer-window changes need the prior schedule, a timer, and ownership-drift thresholds. Zone changes need an exported mapping. Member replacement needs election and replication headroom.

Shard-key refinement is not a simple metadata toggle, and completed resharding has no one-command return to the former design. Rehearse resharding on representative scale, validate storage and oplog headroom, monitor its phase with $currentOp, define pre-commit abort criteria, and preserve a tested recovery procedure. Do not describe an untested re-reshard as rollback.

After any change, validate all three axes: per-shape shard targeting, owned and orphaned data, and per-member capacity and replication. Also verify P95 and P99 latency, throughput, errors, elections, and the absence of a shifted hotspot.

Where It Breaks

Failure modeWhy the analysis failsGuardrail
Single mongos sampledrouting behavior differs across routers or deploymentsaggregate every relevant router with process identity
Raw cumulative counters comparedprocess age looks like incident workuse restart-bounded interval deltas
Balancer compliance treated as healthpolicy compliance ignores traffic and capacityevaluate all three axes
Correlation treated as causationbalancing often follows the original skewpreserve baseline, transition, incident, and recovery order
Orphans treated as duplicate application datamigration leftovers have ownership semanticsuse supported distribution and cleanup evidence
Resharding recommended from averageshot values and query patterns disappear in meanstest frequency, monotonicity, targeting, and adverse values
LLM receives unrestricted command accessa hypothesis can become an irreversible topology changeread-only evidence role and human-approved runbook

What the LLM Cannot Do

Explicit operational boundaries govern the LLM:

  • No Raw Customer Data: The LLM must not receive documents, profiler entries containing literal filter values, or currentOp output with embedded query predicates. Send normalized query shapes, planSummary, docsExamined/nReturned ratios, and timings.
  • No Autonomous Production Execution: The LLM cannot create or drop indexes, run moveChunk, alter the balancer, step down primaries, or change cluster configuration. It proposes; a human authorizes, applies, validates, and holds the rollback condition.
  • No Causal Conclusion Without Contradicting Evidence: Every hypothesis must cite both the telemetry that supports it and the telemetry that would falsify it. A ranked hypothesis with no disconfirming test attached is an assertion, not a diagnosis.
  • No Silent Gap-Filling: Where the evidence pack lacks a required signal, the LLM must name it explicitly under missing evidence rather than inferring a plausible value. “Not collected” and “collected and normal” are different findings and must never be merged.
  • No Change Without a Human Gate: Production changes require human authorization, a stated validation signal, and a defined rollback condition agreed before the change is applied.

What to Do Next

  • Problem: A hot shard can originate in routing, ownership, migration, cleanup, or replica capacity.
  • Solution: Build synchronized router, placement, migration, and member evidence before asking the LLM to correlate causes.
  • Proof: Confirm the hypothesis against targeting ratios, owned and orphaned data, restart-aware migration deltas, and replica health; then demonstrate recovery without moving the hotspot.
  • Action: Implement read-only collectors and reversible runbooks now. Part 4 will secure profiler data, diagnostic logs, access boundaries, audit evidence, and production guardrails.

Sources