High memory occupancy in Valkey is by design, elevated CPU can reflect healthy I/O threading rather than a runaway command, and tail latency spikes frequently originate outside the core engine loop. Triage begins by isolating kernel virtual memory, EC2 hypervisor constraints, and allocator fragmentation from single-threaded engine stalls.

Technology and product capabilities in this series are evaluated as of June 30, 2026.

Situation

Deploying self-managed Valkey (versions 7.2 and 8.0) on Amazon EC2 powers latency-critical caching, session stores, rate limiters, and real-time state machines. Operating at sub-millisecond p99 latency targets leaves zero margin for diagnostic ambiguity. When an upstream API alerts on cache timeouts, the environment surfaces dozens of concurrent signals: elevated instance CPU, climbing RSS, sudden key evictions, replication buffer growth, and microbursts on the Elastic Network Adapter (ENA).

Valkey exposes granular internal telemetry through INFO commands (such as memory, cpu, stats, clients, replication, and commandstats), SLOWLOG, and LATENCY DOCTOR. Linux provides /proc/vmstat, sysfs, and perf, while EC2 surfaces CloudWatch metrics, ENA driver counters, and EBS volume telemetry.

The operational bottleneck is deterministic correlation. An LLM cannot poll production or execute unchecked configuration commands. Its role is strictly to correlate synchronized, sanitized evidence packs, eliminate contradictory hypotheses, surface missing diagnostic dimensions, and produce ranked, auditable remediation options for human authorization.

The Problem

Valkey’s in-memory, single-threaded execution core (augmented by multi-threaded network I/O) creates distinctive failure modes that mislead generic database triage:

  1. Memory Occupancy vs. Memory Pressure: used_memory reaching maxmemory is expected when using an eviction policy like allkeys-lru. True pressure arises from allocator fragmentation (mem_fragmentation_ratio), jemalloc metadata bloat, unevictable keys under volatile-* policies, or Linux swap invocation.
  2. Main Thread vs. I/O Thread Saturation: High instance vCPU utilization can reflect efficient network offloading across I/O threads or execution starvation on the single main event loop caused by $O(N)$ commands, synchronous large-key evictions, or blocking functions.
  3. Burstable and Hypervisor Throttling: On burstable EC2 instances (such as T4g or T3), CPU credit depletion surfaces as host CPU steal and sudden event-loop freezes, mimicking an internal database lock.
  4. Copy-on-Write Memory Bloat: Background persistence operations (BGSAVE or BGREWRITEAOF) fork child processes. If Linux Transparent Huge Pages (THP) are enabled, kernel 2 MB page-fault allocations during write traffic rapidly duplicate memory, exhausting RAM and triggering out-of-memory (OOM) termination.
  5. Network Packet and Buffer Saturation: Client timeouts often stem from exhausting EC2 ENA packet-per-second (PPS) allowances, TCP backlog drops (somaxconn), or replication client output buffer overflows rather than internal command latency.

How do we classify a Valkey incident into host virtual memory, kernel configuration, EC2 instance constraints, network buffer exhaustion, or engine event-loop stalls before running invasive diagnostics?

Build an Engine-Aware Valkey Evidence Pack

flowchart TD
    A[application p95 p99 latency and timeouts] --> E[time-bounded Valkey incident evidence pack]
    B[host CPU memory swap and ENA counters] --> E
    C[Valkey INFO commandstats memory and clients] --> E
    D[Linux kernel sysctl THP and EBS storage metrics] --> E
    E --> F[validate timestamps uptime role and counter resets]
    F --> G[derive interval rates deltas and ratios]
    G --> H[LLM observations hypotheses contradictions and gaps]
    H --> I[SRE and DBA verification]
    I --> J[remediation validation and rollback]

To diagnose Valkey accurately, build a time-bounded incident evidence pack capturing baseline, onset, peak, and recovery windows in synchronized UTC. Cumulative counters from INFO stats, INFO commandstats, and network interfaces must be converted to interval rates bounded by process uptime.

The telemetry payload captures:

  • Engine State (INFO): Process uptime_in_seconds, instantaneous_ops_per_sec, connected_clients, blocked_clients, rejected_connections.
  • Memory Subsystem (INFO memory): used_memory, used_memory_rss, maxmemory, maxmemory_policy, mem_fragmentation_ratio, mem_allocator.
  • Eviction and Expiration Counters (INFO stats): evicted_keys, expired_keys. These live in the stats section, not memory — a collector that scrapes only INFO memory will silently return nothing for them. They are cumulative since start, so record deltas over the collection interval rather than absolute values.
  • Compute and Execution (INFO cpu): used_cpu_sys, used_cpu_user, used_cpu_sys_main_thread, used_cpu_user_main_thread.
  • Persistence and Replication (INFO persistence, INFO replication): rdb_bgsave_in_progress, aof_rewrite_in_progress, role, master_repl_offset, repl_backlog_size.
  • Host and Kernel State: Instance type, total RAM, swap usage, vm.overcommit_memory, vm.swappiness, THP state (/sys/kernel/mm/transparent_hugepage/enabled), CPU steal percentage, and ENA driver counters (ethtool -S <interface>).

The LLM processes this sanitized bundle to generate four distinct, auditable artifacts:

  1. Observations: Direct facts tied to specific metric deltas (e.g., “evicted_keys rate surged to 35,000/s while used_memory remained at 99.8% of maxmemory”).
  2. Hypotheses: Ranked potential root causes with explicit supporting and contradicting metrics.
  3. Missing Evidence: Unobserved metrics required to confirm the primary hypothesis (e.g., “Need SLOWLOG GET to verify whether evictions are stalling the main thread synchronously”).
  4. Actions: Safe, read-only verification commands and reversible configuration adjustments.

Symptoms

SymptomPlausible failure classesEvidence that separates them
p99 latency spikesO(N) command, synchronous eviction, THP memory allocation, EBS fsync stallused_cpu_user_main_thread, evicted_keys delta, latest_fork_usec, aof_delayed_fsync, THP state
Rapid memory climb to OOMMemory fragmentation, unevictable keys, replication buffer overflow, fork COWmem_fragmentation_ratio, used_memory vs used_memory_rss, client-output-buffer-limit, Linux swap
Sudden eviction spikeWorking set exceeds maxmemory, TTL expiration lag, data structure expansionevicted_keys delta, expired_keys delta, maxmemory_policy, write ingestion rate
Client connection timeoutsExhausted maxclients, TCP backlog drops, ENA PPS allowance exhaustionrejected_connections, somaxconn, netstat listen drops, ethtool allowance counters
High CPU with low ops/secSpin-wait contention, regular expression matching, heavy Lua script executioninstantaneous_ops_per_sec, used_cpu_sys_main_thread, SLOWLOG, INFO commandstats
Replication lag / disconnectBacklog buffer wrap, network bandwidth throttling, large snapshot transmissionmaster_repl_offset vs slave_repl_offset, repl_backlog_size, replication client buffer drops

First Five Checks

flowchart TD
    A[Valkey latency or saturation alert] --> B[1. Check process uptime counter resets and host CPU steal]
    B --> C[2. Inspect memory breakdown RSS fragmentation and eviction rates]
    C --> D[3. Compare main thread vs system CPU and verify THP status]
    D --> E[4. Check connection backlog ENA allowances and client buffers]
    E --> F[5. Evaluate persistence fork duration and replication offsets]
    F --> G[Generate ranked hypotheses with missing evidence]

1. Validate process uptime, counter continuity, and host CPU steal

Check uptime_in_seconds to confirm the process has not crashed or restarted under Linux OOM-killer supervision (dmesg -T | grep -E -i "valkey|oom-killer").

On the host, inspect CPU steal (%steal via vmstat or top) and, on burstable instances (T3/T4g), the CloudWatch credit metrics. These are two independent signals and conflating them produces a confident but wrong diagnosis:

SignalWhat it actually meansWhat it does not prove
%stealThe hypervisor scheduled another tenant while this vCPU had runnable workCredit exhaustion — it is a virtualization scheduling signal, present on non-burstable types too
CPUCreditBalanceAccrued earned credits remainingThrottling by itself; a low balance matters only in Standard mode
CPUSurplusCreditBalanceSurplus credits consumed in Unlimited modeThrottling — in Unlimited mode the instance keeps bursting and accrues charges instead of being capped

Read the credit mode first, because it determines what a depleted balance does. In Standard mode, exhausted credits restrict the instance toward its baseline performance — that is genuine throttling. In Unlimited mode, the instance continues to burst and bills for surplus credits, so a rising CPUSurplusCreditBalance is a cost signal, not evidence of a capped CPU.

Do not apply a universal 5% steal threshold as proof of credit throttling. Evaluate credit mode, CPUCreditBalance, surplus accrual, CPUUtilization against the instance’s documented baseline, and OS scheduling metrics separately, then state which of them actually moved. Where the host is genuinely constrained, internal Valkey latency metrics reflect external virtualization limits rather than engine behavior — but that conclusion needs the specific signal named, not “steal was high.”

2. Characterize memory breakdown: RSS, fragmentation, and eviction dynamics

Evaluate the relationship between used_memory, used_memory_rss, and maxmemory:

  • Fragmentation Ratio ($used_memory_rss / used_memory$): A ratio above 1.5 indicates significant allocator fragmentation, where jemalloc holds unpurged arenas. A ratio below 1.0 indicates that the operating system has paged Valkey memory to swap disk, which destroys in-memory latency.
  • Eviction vs. Expiration Rates: If used_memory reaches maxmemory and evicted_keys climbs, Valkey is pruning keys to accommodate incoming writes. If maxmemory_policy is noeviction, subsequent write commands fail immediately with OOM command not allowed errors.
  • Unevictable Key Traps: Under volatile-lru, volatile-lfu, or volatile-ttl, keys without an explicit TTL cannot be evicted. If unexpired or TTL-less keys fill maxmemory, writes fail despite memory exhaustion.

3. Deconstruct CPU saturation and event-loop blocking

Valkey handles network parsing across multiple I/O threads when configured, but core command execution runs on a single main event loop.

  • Main Thread CPU (used_cpu_user_main_thread): If main thread user CPU approaches 100% of a single core while overall instance CPU remains low (e.g., 25% on a 4-vCPU instance), the single-threaded engine is CPU-bound. Inspect INFO commandstats for $O(N)$ operations (KEYS, HGETALL, SMEMBERS).
  • Synchronous Eviction Blocking: Freeing large strings, sets, or hashes during eviction happens synchronously on the main thread by default. If lazyfree-lazy-eviction is no, evicting large keys blocks the event loop.
  • Transparent Huge Pages (THP): Check /sys/kernel/mm/transparent_hugepage/enabled. If set to [always], write traffic during background persistence (BGSAVE) forces 2 MB page allocations, stalling the event loop and inflating memory via Copy-on-Write.

4. Trace latency amplification through network, buffers, and client queues

Sub-millisecond engine performance does not prevent upstream timeouts if transport queues are saturated:

  • Connection Backlog Drops: Compare connected_clients with maxclients. Check the Linux socket listen backlog (/proc/sys/net/core/somaxconn) against Valkey’s tcp-backlog. If client connect bursts exceed this queue, TCP SYN packets are dropped, adding connection timeouts.
  • ENA Network Throttling: On EC2, check ethtool -S <interface> | grep -E "allowance_exceeded|drop". Nitro instances enforce strict Packet Per Second (PPS) allowances; exceeding them causes packet drops at the hypervisor level.
  • Client Output Buffer Overflow: Check INFO clients for client_recent_max_output_buffer and client_recent_max_input_buffer. When clients issue large queries faster than socket buffers drain, Valkey allocates expanding output buffers, consuming memory until reaching client-output-buffer-limit. Both fields report a recent maximum across all clients, so a high value proves a large buffer occurred but does not identify the connection holding it — use CLIENT LIST and read its omem and tot-mem columns to attribute it.

5. Audit persistence, replication lag, and fork-induced stalls

Inspect background persistence activity and replica synchronization status:

  • Fork Latency (latest_fork_usec): During BGSAVE or BGREWRITEAOF, the engine calls fork(). Duplicating a 32 GB instance page table can stall the main thread for 50–200 ms.
  • AOF Fsync Stalls (aof_delayed_fsync): Under appendfsync everysec, if an EBS volume hits IOPS limits during background flushing, the main thread delays subsequent writes by up to 2 seconds.
  • Replication Backlog Wrap: Compare master_repl_offset with replica offsets. If replication lag exceeds repl-backlog-size, the primary cannot perform a partial resync (PSYNC) and forces a full snapshot resync.

Decision Tree

flowchart TD
    A[Valkey latency or saturation incident] --> B{Process restart or CPU steal detected}
    B -->|Yes| C[Investigate OOM crash or EC2 burstable credit starvation]
    B -->|No| D{used_memory near maxmemory}
    D -->|Yes| E{Evictions climbing rapidly}
    E -->|Yes| F[Check lazyfree settings and working set growth]
    E -->|No| G{mem_fragmentation_ratio > 1.5}
    G -->|Yes| H[Check allocator arenas and active defragmentation]
    G -->|No| I[Check volatile policy unevictable keys]
    D -->|No| J{Main thread CPU at 100%}
    J -->|Yes| K[Inspect SLOWLOG and O-N commandstats]
    J -->|No| L{ENA allowance exceeded or backlog drops}
    L -->|Yes| M[Resize EC2 network tier or tune somaxconn]
    L -->|No| N[Inspect AOF fsync stalls and fork latency]

In Practice

The official Valkey documentation establishes that memory management relies on jemalloc chunk allocation. When keys are allocated and released in rapid, variable-sized bursts, jemalloc can retain virtual memory pages, resulting in a high mem_fragmentation_ratio. The documented behavior mandates enabling activedefrag yes or tuning active-defrag-threshold-lower rather than restarting the node or adding physical RAM.

Valkey’s latency troubleshooting guide details the single-threaded execution model: any command with $O(N)$ complexity blocking the main thread prevents all other client commands from executing. Furthermore, freeing keys containing large collections synchronously during eviction blocks the event loop; enabling lazyfree-lazy-eviction, lazyfree-lazy-expire, and lazyfree-lazy-server-del offloads memory reclamation to background bio threads.

The Linux Kernel Virtual Memory documentation documents that vm.overcommit_memory = 1 is mandatory for in-memory datastores relying on fork() for snapshotting. Without this setting, background save operations fail if the kernel estimates that allocated address space exceeds physical RAM plus swap.

AWS EC2 Nitro documentation emphasizes that network performance on EC2 is bounded by per-instance bandwidth and packet-per-second (PPS) allowances. ENA metrics surfaced via ethtool (pps_allowance_exceeded, bw_in_allowance_exceeded) indicate when network traffic is throttled at the hypervisor, demonstrating why high p99 latency can occur while instance CPU and internal database execution latency appear normal.

Remediation Options

Proven root causeCandidate remediationValidation criteria
Main thread blocked by synchronous evictionsSet lazyfree-lazy-eviction yes and lazyfree-lazy-expire yes via CONFIG SETlatency_percentiles_usec drops; background freeing visible in INFO memory
Allocator fragmentation causing memory ballooningEnable activedefrag yes; configure active-defrag-threshold-lower 10mem_fragmentation_ratio drops toward 1.1–1.2 without spike in CPU latency
O(N) command execution stalling event loopReplace KEYS with SCAN; replace large HGETALL with HSCAN or HMGET in appSLOWLOG clears; used_cpu_user_main_thread normalizes
Transparent Huge Pages causing COW latencyDisable THP: echo never > /sys/kernel/mm/transparent_hugepage/enabledlatest_fork_usec and p99 latency during BGSAVE drop by >80%
Burstable CPU credit exhaustion (T3/T4g)Upgrade instance to memory-optimized Nitro type (e.g., r7g.large / r7g.xlarge)%steal drops to 0; CPU throttling alerts clear
ENA PPS allowance exhaustionUpgrade instance size for higher network baseline or implement client-side pipeliningpps_allowance_exceeded counter delta returns to 0
AOF disk fsync blocking writesSet no-appendfsync-on-rewrite yes or migrate EBS to gp3 with dedicated IOPSaof_delayed_fsync counter stops incrementing during AOF rewrites

Rollback Plan

Operational adjustments on a live Valkey EC2 instance require an explicit reversal path:

  1. Dynamic Configuration Changes: Any parameter modified via CONFIG SET (such as lazyfree-lazy-eviction, maxmemory, or activedefrag) can be reverted instantaneously with a reciprocal CONFIG SET. Always capture prior settings via CONFIG GET <parameter> before applying changes.
  2. Active Defragmentation Rollback: If enabling activedefrag introduces unwanted CPU overhead on the main thread during peak traffic, disable it immediately with CONFIG SET activedefrag no.
  3. Kernel Parameter Adjustments: If modifying sysctl parameters (such as net.core.somaxconn or vm.overcommit_memory), record existing values from sysctl -a before applying sysctl -w. Revert using the saved baseline.
  4. Instance Resizing Rollback: If vertical instance scaling requires stopping the instance, ensure a standby replica has taken over the master role or maintain an EBS snapshot before initiating the resize. Revert using standard AWS CLI commands if needed.

Where It Breaks

Failure modeWhy naive reasoning failsBetter diagnostic boundary
High CPU means “needs bigger instance”Single main thread can be pegged at 100% of 1 core while 15 cores sit idleMeasure used_cpu_user_main_thread and profile INFO commandstats
High memory means “memory leak”jemalloc allocator fragmentation or LRU cache design mimics memory leaksCorrelate used_memory, used_memory_rss, and mem_fragmentation_ratio
Latency spikes mean “slow network”Kernel THP 2 MB page allocations during snapshotting cause silent engine freezesCheck /sys/kernel/mm/transparent_hugepage/enabled and latest_fork_usec
Zero evictions means “healthy memory”noeviction policy causes hard application write errors instead of evictionsVerify maxmemory_policy and check for OOM command errors in application logs
Low ops/sec means “idle database”A single multi-key blocking command or Lua script can stall the engine completelyCheck blocked_clients, SLOWLOG, and main thread CPU utilization
CloudWatch CPU looks normalCloudWatch averages CPU across all cores, hiding a 100% pinned main execution coreInspect per-vCPU utilization and internal engine thread metrics

What the LLM Cannot Do

Explicit operational boundaries govern the LLM:

  • No Unsanitized Payload Ingestion: The LLM must not receive raw MONITOR output or unredacted DUMP payloads, as these contain raw key names, user sessions, and sensitive payload values.
  • No Real-Time Monitoring or Polling: The LLM does not continuously poll database ports or manage live telemetry streams. It operates exclusively on static, structured evidence packs.
  • No Autonomous Production Execution: The LLM cannot execute CONFIG SET, issue FLUSHALL, trigger failovers, or terminate client connections. Production modifications require human verification and authorization.
  • No Intuitive Guesswork on Missing Data: If evidence lacks SLOWLOG samples, ethtool network counters, or jemalloc memory breakdowns, the LLM must explicitly list these as missing evidence rather than guessing a root cause.

What to Do Next

  • Problem: Valkey performance incidents on EC2 present overlapping symptoms across single-threaded execution, memory fragmentation, kernel page allocation, and hypervisor limits that mislead generic database triage.
  • Solution: Assemble a role-aware, time-bounded evidence pack capturing synchronized Valkey INFO deltas, Linux virtual memory counters, and EC2 ENA driver metrics.
  • Proof: Formulate hypotheses that explicitly reconcile main thread CPU against I/O threads, mem_fragmentation_ratio against eviction dynamics, and network allowance limits against application p99 latency.
  • Action: Deploy deterministic telemetry collectors on your Valkey EC2 fleet. If command execution latency is confirmed as the primary bottleneck, proceed to Part 2 to identify expensive commands, data structure anti-patterns, and blocking functions.

Sources