A single $O(N)$ command in Valkey can stall thousands of concurrent client requests, yet identifying the offending key or query pattern must never rely on invasive commands like KEYS * or unmetered memory scans. Triage requires extracting differential command statistics, analyzing sampled slowlogs, and inspecting data structure encodings safely.

Technology and product capabilities in this series are evaluated as of June 30, 2026.

Situation

When an application experiences tail latency spikes on an in-memory datastore like Valkey (versions 7.2 and 8.0), host-level metrics frequently show a single CPU core pinned at 100% while overall system memory remains stable. Because Valkey executes data-manipulation commands on a single main event loop, any command that takes 5 milliseconds to execute introduces an immediate 5-millisecond queueing delay for every subsequent operation in the pipeline.

Valkey provides built-in telemetry to diagnose execution bottlenecks: INFO commandstats, SLOWLOG, LATENCY DOCTOR, LATENCY HISTORY, MEMORY USAGE, CLIENT LIST, and internal encoding statistics. However, naive diagnostic queries—such as executing KEYS *, unthrottled --bigkeys scans, or running MONITOR on high-throughput nodes—amplify the incident and can cause complete service outages.

The diagnostic objective is to collect deterministic execution metrics, calculate rate-of-change deltas across command families, and use an LLM to formulate and rank testable hypotheses regarding algorithmic complexity, hot-key contention, and collection anti-patterns without exposing sensitive payload data or running unsafe operations in production.

The Problem

Valkey’s speed relies on sub-microsecond in-memory operations. When performance degrades due to workload patterns, the root cause usually falls into one of five distinct categories:

  1. Algorithmic Complexity Traps: Running $O(N)$ or $O(M \log N)$ commands where $N$ is the total keyspace or collection size (such as KEYS, unbounded HGETALL, SMEMBERS, SUNION, or ZRANGEBYSCORE on large sorted sets).
  2. Big Keys and Memory Serialization: Keys holding hundreds of megabytes of nested data structures require proportional CPU time to serialize across the network socket and allocate memory, blocking the main thread during both retrieval and deletion.
  3. Hot-Key Access Skew: A single popular cache key accessed by thousands of concurrent application threads saturates single-core throughput and fills network socket buffers, regardless of cluster size or shard count.
  4. Blocking Lua Scripts and Functions: Atomic execution of Lua scripts (EVAL/EVALSHA) or Valkey Functions (FCALL) ensures consistency, but unoptimized loops or heavy string manipulations inside a script stall the entire server until the script completes or reaches lua-time-limit.
  5. Pipelining and Buffer Bloat: While pipelining amortizes network round-trip times, unconstrained client pipelines (issuing tens of thousands of commands in a single batch) monopolize the event loop and inflate client output buffers, exhausting server memory.

How do we systematically isolate the exact command, key pattern, and data structure responsible for event-loop saturation while protecting production stability and data privacy?

Build a Workload-Aware Command Evidence Pack

flowchart TD
    A[application p99 latency spikes and queueing alerts] --> E[time-bounded Valkey command evidence pack]
    B[INFO commandstats deltas and latency percentiles] --> E
    C[SLOWLOG samples and latency history events] --> E
    D[sanitized MEMORY USAGE samples and client tracking metrics] --> E
    E --> F[calculate command CPU share and per-call microsecond rates]
    F --> G[correlate slowlog timestamps with client buffers and blocked states]
    G --> H[LLM observations hypotheses contradictions and gaps]
    H --> I[SRE and DBA verification]
    I --> J[remediation validation and rollback]

To isolate expensive commands safely, construct an evidence pack covering baseline, onset, and degradation windows. The collection process must extract differential metrics without executing blocking keyspace scans:

  • Command Statistics (INFO commandstats): For each command family, record cumulative calls, usec, usec_per_call, rejected_calls, and failed_calls. Convert these cumulative totals into interval deltas ($\Delta \text{usec} / \Delta t$) to determine the exact percentage of main-thread CPU consumed by each command.
  • Slow Query Log (SLOWLOG GET 128): Extract timestamp, execution duration in microseconds, command name, client IP/port, and key name (with payload arguments redacted to protect privacy).
  • Latency Diagnostics (LATENCY LATEST, LATENCY HISTORY command): Track historical execution latency spikes across engine events.
  • Client and Buffer State (INFO clients, CLIENT LIST): Capture connected_clients, blocked_clients, client_recent_max_output_buffer, client_recent_max_input_buffer, and identify clients with large output buffers. Pin the collector to these field names for the Valkey versions you support; the older client_longest_output_list and client_biggest_input_buf names belong to earlier Redis-era INFO output and will not be present.
  • Data Structure Telemetry: Sampled memory usage via MEMORY USAGE <key> SAMPLES 5 for suspect keys, and OBJECT ENCODING <key> to determine if collections have expanded beyond optimized compact encodings (e.g., listpack to hashtable/skiplist).

The LLM analyzes this structured payload to generate four distinct outputs:

  1. Observations: Quantitative facts tied to specific metric deltas (e.g., “HGETALL accounted for 74.2% of total main-thread CPU time over a 60-second window, averaging 4,200 µs per invocation across 18,000 calls”).
  2. Hypotheses: Ranked architectural explanations with supporting and contradicting metrics (e.g., “Hypothesis 1: Large hash traversal on user session objects; Hypothesis 2: Client output buffer starvation caused by unbounded batch sizes”).
  3. Missing Evidence: Additional safe metrics required to confirm the hypothesis (e.g., “Need OBJECT ENCODING and HLEN for top slowlog hash keys to confirm if hash exceeds listpack boundaries”).
  4. Actions: Non-blocking validation commands and application-level query refactoring steps.

Symptoms

SymptomPlausible failure classesEvidence that separates them
Main-thread CPU pinned at 100%$O(N)$ keyspace command, complex Lua script, large collection traversalINFO commandstats $\Delta \text{usec}$, SLOWLOG duration, used_cpu_user_main_thread
Sudden p99 latency spike to >10msBig key serialization, blocking function, active defragmentation surgeSLOWLOG entries, LATENCY LATEST, latest_fork_usec, MEMORY USAGE
Client timeout during write burstBlocking Lua script, synchronous eviction of big keys, unbounded pipelineblocked_clients, lazyfree-lazy-eviction, client_recent_max_output_buffer
High network egress with low ops/secUnbounded read commands (HGETALL, SMEMBERS, LRANGE 0 -1)total_net_output_bytes delta, INFO commandstats cmdstat_hgetall
blocked_clients metric climbingBlocking list/stream reads (BLPOP, BRPOP, XREAD BLOCK)INFO clients blocked_clients, application consumer connection counts
Memory ballooning without key count increaseNested data structures expanding from listpack to hashtable/skiplisttracking_total_keys stable while used_memory_dataset climbs, OBJECT ENCODING

First Five Checks

flowchart TD
    A[Valkey command latency or CPU alert] --> B[1. Compute commandstats deltas and CPU time distribution]
    B --> C[2. Inspect SLOWLOG for long-running command signatures]
    C --> D[3. Audit big keys and structural encodings via non-blocking sampling]
    D --> E[4. Identify hot keys and client access skew via LFU metrics]
    E --> F[5. Evaluate Lua script execution times and client output buffers]
    F --> G[Synthesize observations hypotheses and remediation steps]

1. Compute commandstats deltas and CPU time distribution

Never evaluate raw cumulative numbers from INFO commandstats. Calculate the rate of change over a discrete sample interval ($T_1$ to $T_2$):

$$\Delta \text{CPU_Share}(cmd) = \frac{\Delta \text{usec}(cmd)}{\Delta t_{\text{elapsed_microseconds}}} \times 100%$$

If cmdstat_keys, cmdstat_hgetall, cmdstat_smembers, or cmdstat_eval accounts for more than 20% of total elapsed time, the workload is suffering from algorithmic inefficiency. Compare usec_per_call: sub-microsecond commands (GET, SET) typically average 0.5–2.0 µs, whereas problematic collection queries average 500–10,000 µs per call.

2. Inspect SLOWLOG for long-running command signatures

Execute SLOWLOG GET 32 to capture the most recent commands that exceeded slowlog-log-slower-than (default 10,000 µs; recommended 1,000 µs for low-latency triage).

Analyze the slowlog payload:

  • Command Name and Duration: Identify whether latency is driven by retrieval (HGETALL), modification (SADD), or deletion (DEL on large collections).
  • Execution Timestamps: Correlate slowlog timestamps with application latency alarms and CPU spikes.
  • Client IP and Port: Determine if slow commands originate from a specific batch service, background worker, or misconfigured microservice.

3. Audit big keys and structural encodings via non-blocking sampling

Do not execute valkey-cli --bigkeys on a heavily loaded production node without rate limiting, as it runs continuous SCAN iterations that increase CPU overhead. Instead, perform targeted sampling on suspected keys identified in the slowlog:

  • Memory Footprint: Execute MEMORY USAGE <key> SAMPLES 5 to estimate total memory consumption including allocator overhead.
  • Internal Encoding: Run OBJECT ENCODING <key>. Valkey stores small hashes, sets, and zsets using memory-efficient, compact listpack structures. When element counts or value lengths exceed configured thresholds (hash-max-listpack-entries, hash-max-listpack-value), the engine converts the structure to a standard hashtable or skiplist, increasing memory overhead by 3x–5x and changing traversal CPU characteristics.

4. Identify hot keys and client access skew via LFU metrics

When a single key experiences massive concurrency, it saturates the execution thread. If Valkey is configured with an LFU eviction policy (such as allkeys-lfu or volatile-lfu), retrieve the access frequency counter using OBJECT FREQ <key>.

To scan for hot keys safely:

  • Use valkey-cli --hotkeys -i 0.1 during non-peak windows with an explicit sleep interval (-i 0.1 introduces a 100 ms pause every 100 SCAN iterations) to avoid CPU contention.
  • Monitor CLIENT LIST to detect whether hundreds of connections are querying the exact same database index or executing identical query strings.

5. Evaluate Lua script execution times and client output buffers

Lua scripts and Valkey Functions execute with strict ACID atomicity, locking the main thread for their entire duration:

  • Lua Execution Stalls: Check cmdstat_eval and cmdstat_fcall in INFO commandstats. If script duration exceeds lua-time-limit (default 5,000 ms), the engine logs warnings and allows read-only SCRIPT KILL operations.
  • Client Output Buffers: Inspect INFO clients for client_recent_max_output_buffer. Large pipelines (e.g., 50,000 GET commands in a single pipeline) or wide queries (LRANGE 0 -1) force Valkey to buffer megabytes of reply data in memory per client. If client consumption stalls, the engine spends excessive time managing buffer memory allocations. Because this field is a recent maximum across clients rather than a per-connection gauge, follow it with CLIENT LIST and sort by omem to find the offending connection.

Decision Tree

flowchart TD
    A[Valkey tail latency spike or event loop stall] --> B{Commandstats shows high usec per call}
    B -->|Yes| C{Command type identified in SLOWLOG}
    C -->|KEYS or Full Keyspace Scan| D[Replace with SCAN and cursor pagination]
    C -->|HGETALL SMEMBERS LRANGE| E[Check collection size and switch to HSCAN or HMGET]
    C -->|EVAL FCALL Lua Script| F[Decompose script and remove blocking loops]
    C -->|DEL on Large Collection| G[Enable lazyfree or use UNLINK instead of DEL]
    B -->|No| H{High ops per sec on single key}
    H -->|Yes| I[Implement client-side caching or key sharding]
    H -->|No| J{Blocked clients climbing}
    J -->|Yes| K[Inspect BLPOP BRPOP XREAD queue consumer saturation]
    J -->|No| L[Check client pipeline sizes and network output buffer limits]

In Practice

The official Valkey command reference documents the time complexity of every engine operation. Commands like KEYS are $O(N)$ where $N$ is the total number of keys in the database, making them prohibited in production. Collection operations such as HGETALL and SMEMBERS are $O(N)$ where $N$ is the number of elements in the specific collection; for hashes containing 100,000 fields, executing HGETALL forces the engine to serialize 100,000 key-value pairs synchronously.

The Valkey memory optimization guide specifies that hashes, lists, and sets below configured thresholds (such as hash-max-listpack-entries 512 and hash-max-listpack-value 64) are encoded as a single contiguous memory block (listpack). This representation provides extreme memory compactness and high CPU cache locality. Once a collection exceeds these limits, it converts to a hashtable or skiplist, increasing memory pointer overhead and altering allocation behavior.

According to the Valkey latency troubleshooting guide, deleting a large collection containing millions of elements using standard DEL is an $O(N)$ operation that deallocates memory synchronously on the main thread. The documented best practice is using UNLINK (non-blocking delete) or setting lazyfree-lazy-user-del yes to offload memory reclamation to background bio threads.

Documented client architecture patterns establish that while pipelining eliminates network round-trip delays, issuing unbounded pipelines (e.g., >10,000 operations per pipeline) forces the server to buffer all response objects simultaneously. This amplifies memory consumption in client-output-buffer-limit and creates head-of-line blocking for other connected clients.

Remediation Options

Proven root causeCandidate remediationValidation criteria
Unbounded hash read (HGETALL)Refactor application to use HMGET for required fields or HSCAN for paginationcmdstat_hgetall rate drops to 0; p99 latency normalizes
Synchronous collection deletion (DEL)Replace application DEL calls with UNLINK; set lazyfree-lazy-user-del yesSLOWLOG entries for deletions disappear; latency spikes eliminate
Keyspace scanning (KEYS *)Replace with cursor-based SCAN with bounded COUNT parameters (e.g., COUNT 100)cmdstat_keys drops to 0; event-loop stalls resolve
Hot-key access saturationEnable Client-Side Caching (CLIENT TRACKING) or shard keys (key_{1..N})CPU distribution balances across cores/shards; OBJECT FREQ skews flatten
Long-running Lua scriptRefactor complex business logic to client application or split script into smaller batchescmdstat_eval average usec_per_call drops below 500 µs
Unbounded client pipeliningEnforce maximum client pipeline batch size (e.g., 500 commands per batch)client_recent_max_output_buffer stabilizes; client buffer memory drops
Blocking list queue thundering herdStagger worker retry intervals and add exponential backoff on empty BLPOPblocked_clients stabilizes; worker connection churn drops

Rollback Plan

All production workload adjustments and configuration changes must maintain a clear, low-risk rollback strategy:

  1. Configuration Parameter Rollback: Any runtime setting modified via CONFIG SET (such as lazyfree-lazy-user-del, hash-max-listpack-entries, or slowlog-log-slower-than) can be immediately restored to its baseline value using a reciprocal CONFIG SET. Always record initial parameters using CONFIG GET * before modifications.
  2. Client Application Feature Flags: When rolling out query refactorings (e.g., migrating from HGETALL to HMGET/HSCAN, or implementing client-side caching), gate changes behind dynamic configuration flags. If application-side batching introduces unforeseen race conditions, toggle the feature flag to revert to baseline behavior instantly.
  3. Key Sharding / Salting Reversal: If partitioning hot keys across multiple sub-keys (key_0, key_1), ensure the client read layer supports dual-reading from both the unpartitioned and partitioned key paths until data migration completes.
  4. Client-Side Caching Evacuation: If enabling CLIENT TRACKING causes invalidation message storming over application connections, disable tracking globally at the client library level without modifying database server configurations.

Where It Breaks

Failure modeWhy naive reasoning failsBetter diagnostic boundary
”Slow database means slow server”A single client issuing KEYS * stalls an otherwise idle, multi-gigabyte instanceAudit INFO commandstats and SLOWLOG to isolate specific commands
”Low CPU means all queries are fast”Bursts of $O(N)$ commands lasting 5ms may barely move 1-minute CPU averagesInspect p99/p999 latency percentiles and LATENCY LATEST history
”Big keys must be deleted immediately”Executing synchronous DEL on a 500MB key will freeze the server for secondsUse UNLINK or asynchronous background deallocation
”Pipelining is always better”Massive pipelines create head-of-line blocking and trigger memory buffer limitsBenchmark pipeline batch sizes and monitor client output buffers
”Lua scripts run faster than commands”Lua executes synchronously; complex loops inside scripts block all clientsProfile Lua execution duration via cmdstat_eval and slowlog
”Hot keys can be fixed by vertical scaling”Valkey command execution is single-threaded per shard; more CPU cores will not helpImplement client-side caching or key partitioning

What the LLM Cannot Do

To guarantee production security, data privacy, and operational stability, strict guardrails define LLM operations:

  • No Raw Payload Inspection: The LLM must never receive unredacted key names, user IDs, authentication tokens, or payload data from SLOWLOG or DUMP outputs. All diagnostic evidence packs must mask payload parameters before analysis.
  • No Direct Query Execution: The LLM does not connect directly to the database to execute diagnostic queries, run SCAN loops, or modify configurations.
  • No Autonomous Key Deletion: The LLM cannot execute DEL, UNLINK, or FLUSHDB. Remediation actions must be authorized and applied by human engineers through validated deployment pipelines.
  • No Guesses on Incomplete Commandstats: If INFO commandstats lacks baseline rate-of-change deltas, the LLM must explicitly flag missing evidence rather than guessing workload distribution from cumulative lifetime counters.

What to Do Next

  • Problem: Expensive commands, big collections, and hot keys stall Valkey’s single-threaded event loop, introducing severe tail latency across upstream services.
  • Solution: Construct a workload-aware command evidence pack capturing differential INFO commandstats, sampled SLOWLOG entries, and structural encoding diagnostics.
  • Proof: Correlate specific command families with main-thread CPU time share, verify collection encodings via OBJECT ENCODING, and validate execution durations against slowlog timestamps.
  • Action: Instrument differential commandstat collection across your Valkey nodes. If host-level triage and command profiling confirm engine health, proceed to Part 3 to diagnose multi-node clustering, shard balancing, and ElastiCache managed failover behavior.

Sources