In Amazon ElastiCache for Valkey, high host CPU does not prove engine saturation, sub-second failover does not guarantee zero data loss, and cluster rebalancing can introduce cross-slot command errors under live traffic. Triage requires reconciling CloudWatch host metrics with internal cluster topology and managed service events.

Technology and product capabilities in this series are evaluated as of June 30, 2026.

Situation

Amazon ElastiCache for Valkey provides a fully managed, in-memory caching and data structure service with automated Multi-AZ failover, online resharding, and automated parameter management. Running Valkey (versions 7.2 and 8.0) in ElastiCache removes the operational burden of OS provisioning, kernel tuning, and manual replication setup.

However, operating behind a managed cloud boundary alters the diagnostic workflow fundamentally:

  • No Host Shell Access: SREs and DBAs cannot inspect /proc/vmstat, run ethtool for network driver drops, or attach debuggers.
  • Restricted Command Interface: Administrative commands such as CONFIG SET, SHUTDOWN, SLAVEOF, and BGREWRITEAOF are blocked by AWS IAM and engine ACLs. Configuration changes must flow through ElastiCache Parameter Groups.
  • Dual Telemetry Streams: Performance data is split between CloudWatch metrics (EngineCPUUtilization, DatabaseMemoryUsagePercentage, NetworkBandwidthInAllowanceExceeded, ReplicationLag), ElastiCache Service Events, and client-accessible engine commands (INFO, CLUSTER SLOTS, SLOWLOG).

When an upstream microservice experiences intermittent connection timeouts or cache miss storms during an ElastiCache incident, the diagnostic challenge is correlating CloudWatch infrastructure metrics, Valkey engine counters, and AWS control-plane maintenance events without making unsupported assumptions about inaccessible host layers.

The Problem

Managed ElastiCache clusters introduce complex multi-node dynamics across shards, replicas, and routing topologies:

  1. Host CPU vs. Engine CPU Misinterpretation: CPUUtilization averages compute across all instance vCPUs (including background threads and network I/O). If the single-threaded Valkey main loop is 100% saturated by an $O(N)$ query, CPUUtilization on a 4-vCPU node may show only 25%, while EngineCPUUtilization is pegged at 100%.
  2. Cluster Mode Topology and Slot Skew: In Cluster Mode Enabled (CME) architectures, 16,384 hash slots are partitioned across shards. Workload imbalance, hot hash tags (e.g., {tenant_42}:data), or uneven slot allocation concentrates write traffic onto a single shard, causing hot-node degradation while other shards remain idle.
  3. Replication Lag and Multi-AZ Failover Data Loss: Valkey replication to read replicas is asynchronous. When a primary fails, ElastiCache promotes a replica automatically. If ReplicationLag was elevated due to heavy write throughput or large key transfers prior to failover, writes acknowledged by the old primary but unapplied on the replica are permanently lost.
  4. Online Resharding and Slot Migration Errors: Adding or removing shards in a live cluster migrates hash slots online. Multi-key operations spanning slots in migration return TRYAGAIN or CROSSSLOT errors unless client drivers handle slot redirection (MOVED and ASK) correctly.
  5. Connection Storms on Primary Replacement: During planned maintenance or unplanned failover, all connected application clients disconnect and reconnect simultaneously to the promoted primary, causing connection queue drops and thread pool starvation in upstream microservices.

How do we systematically isolate whether an ElastiCache performance degradation stems from engine CPU saturation, shard key skew, asynchronous replication delay, managed maintenance failover, or client driver misconfiguration?

Build a Managed ElastiCache Evidence Pack

flowchart TD
    A[application p99 latency timeouts and MOVED errors] --> E[time-bounded ElastiCache incident evidence pack]
    B[CloudWatch EngineCPU FreeableMemory and ENA allowances] --> E
    C[Valkey INFO SLOWLOG and CLUSTER SLOTS per node] --> E
    D[ElastiCache service events and parameter group state] --> E
    E --> F[validate node roles cluster topology and counter deltas]
    F --> G[reconcile CloudWatch metrics with engine commandstats]
    G --> H[LLM observations hypotheses contradictions and gaps]
    H --> I[SRE and Cloud DBA verification]
    I --> J[remediation validation and rollback]

To diagnose ElastiCache incidents, assemble an evidence pack synchronizing CloudWatch metrics, ElastiCache Service Events, and Valkey INFO outputs across every primary and replica node in the cluster:

  • CloudWatch Infrastructure Metrics (per node): EngineCPUUtilization, CPUUtilization, FreeableMemory, BytesUsedForCache, DatabaseMemoryUsagePercentage, NetworkBytesIn, NetworkBytesOut, NetworkBandwidthInAllowanceExceeded, NetworkPacketsPerSecondAllowanceExceeded, ReplicationLag, CurrConnections, Evictions, CacheHits, CacheMisses.
  • ElastiCache Service Events: Capture event logs filtered by replication group and cache cluster ID (e.g., “Failover to replica completed”, “Added shard”, “Applying parameter group”, “Automated backup started”).
  • Valkey Engine State (INFO, CLUSTER NODES, CLUSTER SLOTS): Extract node IDs, roles (master/slave), assigned slot ranges, connected client counts, instantaneous_ops_per_sec, and differential INFO commandstats.
  • Parameter Group Configuration: Inspect active parameter values, specifically reserved-memory-percent (default 25%), maxmemory-policy, timeout, tcp-keepalive, and cluster-enabled.

The LLM correlates these synchronized dimensions to produce four distinct diagnostic artifacts:

  1. Observations: Quantitative facts tied to specific node IDs and timestamps (e.g., “Shard 0002 primary cache.r7g.large reached 99.4% EngineCPUUtilization while Shards 0001 and 0003 remained below 15%”).
  2. Hypotheses: Ranked explanations identifying the failure layer (e.g., “Hypothesis 1: Hash tag slot skew concentrating multi-key operations on Shard 0002; Hypothesis 2: Asynchronous replication buffer exhaustion during snapshotting”).
  3. Missing Evidence: Inaccessible or unobserved telemetry (e.g., “Need client-side cluster routing logs to verify if application driver is retrying on MOVED redirections”).
  4. Actions: AWS CLI or management console operations (e.g., scaling cluster shards, modifying parameter groups, or refactoring client hash tags).

Symptoms

SymptomPlausible failure classesEvidence that separates them
Latency spike on one shard onlyKey distribution skew, hash tag hotspot, heavy single-key writesShard-specific EngineCPUUtilization, BytesUsedForCache, CLUSTER SLOTS
CROSSSLOT or TRYAGAIN errorsMulti-key command during resharding, missing hash tag {...}Application error logs, ElastiCache resharding status, INFO commandstats
Read replica lag climbingHeavy write throughput, replica CPU saturation, network allowance limitsReplicationLag, replica EngineCPUUtilization, NetworkBandwidthInAllowanceExceeded
Sudden failover and connection dropsUnhealthy primary health check, managed maintenance event, OOM crashElastiCache Service Events, EngineCPUUtilization spike prior to event, CloudWatch
High client timeouts with low CPUNetwork PPS allowance exceeded, connection pool exhaustionNetworkPacketsPerSecondAllowanceExceeded, CurrConnections delta, client socket metrics
Memory evictions while memory looks freereserved-memory-percent limit reached, unevictable keys under volatile policyDatabaseMemoryUsagePercentage, FreeableMemory, Evictions, maxmemory-policy

First Five Checks

flowchart TD
    A[ElastiCache latency or failover alert] --> B[1. Check ElastiCache Service Events and node role stability]
    B --> C[2. Reconcile EngineCPUUtilization against host CPUUtilization across shards]
    C --> D[3. Inspect slot distribution and memory skew across cluster nodes]
    D --> E[4. Evaluate ReplicationLag and network allowance throttling]
    E --> F[5. Verify reserved-memory-percent and parameter group configuration]
    F --> G[Synthesize observations hypotheses and Cloud DBA actions]

1. Check ElastiCache Service Events and node role stability

Before analyzing engine metrics, inspect ElastiCache Service Events for the cluster over the preceding 2 hours:

aws elasticache describe-events \
  --source-type cache-cluster \
  --start-time $(date -u -v-2H +"%Y-%m-%dT%H:%M:%SZ")

Verify whether AWS initiated an automated node replacement, Multi-AZ failover, parameter group update, or automated snapshot. If a failover occurred, map which node was promoted and check whether client timeouts align precisely with the promotion window.

2. Reconcile EngineCPUUtilization against host CPUUtilization across shards

Compare EngineCPUUtilization and CPUUtilization across all shards:

  • If EngineCPUUtilization is near 100% while CPUUtilization is below 30%, a single-threaded Valkey workload bottleneck exists (e.g., $O(N)$ command or blocking Lua script).
  • If both EngineCPUUtilization and CPUUtilization are elevated, the node is experiencing heavy multi-threaded network I/O parsing, TLS encryption overhead, or high connection churn.
  • Compare metrics across shards: if only one shard is elevated, the issue is workload or slot skew, not cluster-wide traffic growth.

3. Inspect slot distribution and memory skew across cluster nodes

Run CLUSTER SLOTS or CLUSTER NODES to map slot assignments:

  • Verify that each shard owns an approximately equal slice of the 16,384 total slots (~5,461 slots per shard in a 3-shard cluster).
  • Compare BytesUsedForCache and DatabaseMemoryUsagePercentage across shards in CloudWatch. If one shard stores 80% of total cluster memory, inspect application key patterns for broad hash tags (e.g., {app}:user:1 forces all user keys into the same hash slot).

4. Evaluate ReplicationLag and network allowance throttling

Inspect CloudWatch ReplicationLag for all read replicas:

  • A rising ReplicationLag (measured in seconds) indicates that write volume on the primary exceeds the replica’s ingestion capacity, or network bandwidth between AZs is constrained.
  • Check NetworkBandwidthInAllowanceExceeded and NetworkPacketsPerSecondAllowanceExceeded. Nitro-based ElastiCache nodes enforce per-instance network bandwidth and PPS caps. If write traffic or snapshot synchronization exceeds allowances, packets are throttled at the hypervisor.

5. Verify reserved-memory-percent and parameter group configuration

ElastiCache manages memory allocation via the reserved-memory-percent parameter in custom parameter groups (default 25% for Valkey):

  • If reserved-memory-percent is set to 0, background snapshots (BGSAVE) and replica synchronization can consume all free RAM during write bursts, triggering Linux OOM termination and unexpected node failover.
  • As DatabaseMemoryUsagePercentage approaches 100%, Valkey begins evicting keys according to maxmemory-policy even when CloudWatch FreeableMemory appears adequate.

That threshold is easy to get wrong, and getting it wrong delays the alert until after eviction has already started. DatabaseMemoryUsagePercentage measures used_memory against the engine’s configured maxmemory, and reserved-memory-percent has already been subtracted when ElastiCache derives that maxmemory. The reserved fraction is therefore baked into the denominator — subtracting it a second time to build a 100 - reserved-memory-percent alarm double-counts it and sets the threshold far below the real eviction boundary.

Compare against 100% and reserve alerting headroom explicitly:

MetricDenominatorUse it for
DatabaseMemoryUsagePercentageConfigured maxmemory (reserved memory already excluded)Eviction proximity — alert in the high 80s/90s, not at 100 - reserved
DatabaseMemoryUsageCountedForEvictPercentageMemory actually counted toward evictionThe closer read on eviction pressure where the two diverge
FreeableMemoryHost RAMOS and fork headroom, not engine eviction

Where AWS documentation describes this behavior for “Valkey and Redis OSS” together, that reflects a shared engine lineage in the documentation — it is not a statement that this track covers both products. This series targets Valkey.

Decision Tree

flowchart TD
    A[ElastiCache latency or failure incident] --> B{Service Events show recent failover or maintenance}
    B -->|Yes| C[Correlate client reconnect storm and DNS TTL propagation]
    B -->|No| D{EngineCPUUtilization near 100% on one shard}
    D -->|Yes| E{Is memory or write traffic skewed to this shard}
    E -->|Yes| F[Refactor hash tags and rebalance slot distribution]
    E -->|No| G[Profile expensive commands via SLOWLOG on that shard]
    D -->|No| H{ReplicationLag climbing on replicas}
    H -->|Yes| I[Inspect replica EngineCPU and NetworkBandwidth allowances]
    H -->|No| J{Network allowances exceeded on primary}
    J -->|Yes| K[Scale up node instance family for higher network baseline]
    J -->|No| L[Verify client connection pooling and timeout settings]

In Practice

The AWS ElastiCache for Valkey User Guide establishes that ElastiCache provides native support for open-source Valkey with automated cluster management. AWS documentation highlights that EngineCPUUtilization is the primary health metric for the Valkey core process; relying on standard CPUUtilization will obscure single-threaded event loop saturation on multi-core instances.

AWS ElastiCache Best Practices documentation dictates maintaining reserved-memory-percent at a minimum of 25% (or allocating dedicated reserved-memory). Without reserved memory, failover synchronization and automated backups can trigger memory exhaustion during write-heavy workloads.

According to the AWS ElastiCache Resharding and Scaling documentation, online horizontal scaling adds or removes shards without cluster downtime. However, multi-key operations (such as MGET, MSET, or transactions) executed against keys residing in migrating slots will fail with CROSSSLOT or TRYAGAIN errors unless the keys share an identical hash tag {...} that pins them to a single slot.

Documented Multi-AZ architecture patterns confirm that Valkey replication across Availability Zones is asynchronous. During an unexpected primary outage, the promoted replica may experience an RPO (Recovery Point Objective) greater than zero if ReplicationLag was non-zero at the time of failure. SREs must monitor ReplicationLag continuously as an availability and durability metric.

Remediation Options

Proven root causeCandidate remediationValidation criteria
Shard slot / key skew from broad hash tagsRemove broad hash tags {...}; distribute keys across independent slotsBytesUsedForCache and EngineCPUUtilization equalize across shards
Engine CPU saturation on primary nodeScale horizontally (add shards via online resharding) or vertically (upgrade node size)EngineCPUUtilization drops below 70% across all primary nodes
Replica falling behind on write throughputScale read replica node size vertically or add read replicas to distribute read loadReplicationLag returns to 0 seconds
ENA network allowance throttlingUpgrade instance type (e.g., from cache.m7g.large to cache.m7g.xlarge / 2xlarge)NetworkBandwidthOutAllowanceExceeded delta drops to 0
Premature evictions under write burstsIncrease reserved-memory-percent to 25–30% in ElastiCache Parameter GroupNode failovers during snapshots stop; eviction rates normalize
Client disconnection storm during failoverConfigure exponential backoff with jitter in client connection poolApplication p99 latency stabilizes during failover simulations

Rollback Plan

When modifying managed ElastiCache clusters, follow defined rollback procedures:

  1. Parameter Group Reversion: If modifying parameters (such as maxmemory-policy, timeout, or reserved-memory-percent) causes unexpected eviction or connection behavior, modify the parameter group to restore previous values and apply changes immediately (ApplyMethod: immediate).

  2. Resharding Rollback: If adding shards introduces application routing errors due to client driver limitations, reduce the shard count with aws elasticache modify-replication-group-shard-configuration, specifying the target --node-group-count and, where required, --node-groups-to-remove.

    Do not reach for decrease-replica-count here: it removes replica nodes within existing shards, not the shards themselves, so it will leave the new shards in place while silently stripping their redundancy.

    Treat this as a rollback in name only. Removing shards starts another online resharding operation that migrates slots off the departing nodes — it is not an instantaneous revert to the prior topology. Before declaring rollback complete, verify that all 16,384 slots are covered, that clients have refreshed their slot maps, that the surviving nodes have memory headroom for the returning keyspace, and that migration has actually finished rather than stalled mid-flight.

  3. Vertical Scaling Rollback: If vertical node resizing causes performance regressions, initiate a reverse modification to the prior node type. ElastiCache executes rolling replacements without dropping data.

  4. Client-Side Routing Flags: Ensure application client driver updates (e.g., switching from single-node to cluster-mode routing) are controlled by application feature flags to revert to previous connection endpoints if connection storms occur.

Where It Breaks

Failure modeWhy naive reasoning failsBetter diagnostic boundary
”Cluster CPU is 20%, we have headroom”One primary shard is at 100% EngineCPUUtilization while others idleMonitor per-node EngineCPUUtilization and slot traffic
”Failover was seamless with zero loss”Asynchronous replication means non-zero ReplicationLag loses data on failoverVerify ReplicationLag prior to failover and audit write offsets
”More replicas will speed up writes”In Valkey, replicas only serve reads and add replication network overheadScale shards horizontally to scale write capacity
”Adding shards fixes all hot keys”A single hot key resides on exactly one shard; more shards will not split the keyImplement client-side caching or key salting
”CloudWatch shows high FreeableMemory”OS cache memory is counted as freeable, but Valkey may be evicting at maxmemoryTrack DatabaseMemoryUsagePercentage and Evictions
”Multi-AZ prevents all downtime”DNS CNAME propagation and client reconnection take 5–15 seconds to stabilizeMeasure client-perceived error rates during failover tests

What the LLM Cannot Do

To prevent outages and protect cloud infrastructure, strict boundaries govern LLM diagnostic assistance:

  • No Direct AWS API Execution: The LLM cannot execute aws elasticache modify-cache-cluster, initiate failovers, or alter parameter groups autonomously.
  • No Unsanitized Metric Ingestion: The LLM must not receive unredacted key names, connection string credentials, or internal VPC IP addresses.
  • No Inferences on Missing Shard Telemetry: If telemetry only provides cluster-wide aggregates without per-node breakdown, the LLM must explicitly flag missing per-shard metrics rather than inferring uniform load distribution.
  • No Real-Time Traffic Routing: The LLM cannot dynamically redirect client traffic between primary and reader endpoints.

What to Do Next

  • Problem: Managed ElastiCache for Valkey abstracts host infrastructure while introducing distributed cluster failure modes—including shard skew, replication lag, and failover data loss.
  • Solution: Build a synchronized evidence pack correlating CloudWatch EngineCPUUtilization, network allowance counters, ElastiCache Service Events, and Valkey CLUSTER SLOTS topology.
  • Proof: Verify load balance across all shards, confirm reserved-memory-percent headroom, and reconcile ReplicationLag against client write throughput.
  • Action: Instrument per-node CloudWatch alarms for your ElastiCache clusters. Then proceed to Part 4 to secure LLM diagnostics with key privacy redaction, audit evidence, and safe operational guardrails.

Sources