Topic
Architecture reviews, scalability, failure modes, guardrails, distributed systems, reliability boundaries, and production tradeoffs.
Good entry points for this topic before browsing the full archive.
Most system designs fail for reasons visible at review time: overloaded dependencies, ambiguous ownership, unsafe retries, unbounded queues, and missing rollback paths — a checklist senior engineers use to surface those risks early.
Capacity planning fails when teams size for the average request and ignore fanout, hot keys, and bursty traffic — a framework for sizing from QPS, read/write ratios, and peak multipliers before the first incident teaches the lesson.
Healthy systems preserve their ability to recover by refusing work before a failure becomes contagious — how to design backpressure at the queue boundary, connection pool, and API layer so overload stops propagating upstream.
L2 and L3 posts with architecture, reliability, and tradeoff detail.
Cloudflare's November 2023 outage is a case study in correlated failure. Redundancy protects against independent failures. It does nothing when every node runs the same defective code.
Why paging an engineer solely because CPU hit 85% is an anti-pattern, and how to build correlated alerts that require real operational evidence.
Review questions a staff engineer asks to surface cascade failures, missing fallbacks, state boundaries, and load assumptions that design docs bury.
Pre-positioned capacity, elastic response, bounded queues, and overload shedding — controls for peak traffic without permanent fleet waste.
Ownership boundaries for OLTP, search, cache, queue, and warehouse in a commerce data plane — so no datastore becomes source of truth during an incident.
Managed database selection across operational burden, feature fit, cost trajectory, and exit risk — with failure modes the easy adoption story hides.
Architectural strategies to eliminate waste in Dev, Test, and Staging database environments.
Cloudflare's November 2023 outage is a case study in correlated failure. Redundancy protects against independent failures. It does nothing when every node runs the same defective code.
Why paging an engineer solely because CPU hit 85% is an anti-pattern, and how to build correlated alerts that require real operational evidence.
Review questions a staff engineer asks to surface cascade failures, missing fallbacks, state boundaries, and load assumptions that design docs bury.
Pre-positioned capacity, elastic response, bounded queues, and overload shedding — controls for peak traffic without permanent fleet waste.
Ownership boundaries for OLTP, search, cache, queue, and warehouse in a commerce data plane — so no datastore becomes source of truth during an incident.
Architectural strategies to eliminate waste in Dev, Test, and Staging database environments.
Cloudflare's November 2023 outage is a case study in correlated failure. Redundancy protects against independent failures. It does nothing when every node runs the same defective code.
Why paging an engineer solely because CPU hit 85% is an anti-pattern, and how to build correlated alerts that require real operational evidence.
Review questions a staff engineer asks to surface cascade failures, missing fallbacks, state boundaries, and load assumptions that design docs bury.
Pre-positioned capacity, elastic response, bounded queues, and overload shedding — controls for peak traffic without permanent fleet waste.
Ownership boundaries for OLTP, search, cache, queue, and warehouse in a commerce data plane — so no datastore becomes source of truth during an incident.
Managed database selection across operational burden, feature fit, cost trajectory, and exit risk — with failure modes the easy adoption story hides.
Splitting a service without relocating the database boundary creates distributed coordination overhead worse than the monolith the split was meant to fix.
The four failure boundaries in event-driven systems: schema evolution contracts, ordering guarantees, consumer replay safety, and dead-letter queue handling.
Database migration cutover using dual writes, CDC, backfill, and freeze phases — with rollback boundaries for when 'almost synchronized' is not an operational state.
Cloud cost triage across compute, storage, data transfer, logs, and managed services — a repeatable workflow for finding runaway spend before the bill arrives.
Designing a failover game day that validates DNS cutover, replication lag thresholds, and traffic routing before a real region failure forces the test.
Producer spikes, consumer lag, poison messages, and retry storms each need a different intervention — the diagnosis order matters as much as the fix.
Cache hit-rate collapse leads to stampede, TTL misconfiguration, and unprotected database load — a workflow for diagnosing each failure in sequence.
API gateway incidents are misdiagnosed when teams treat them as proxy failures instead of control-plane failures with downstream saturation blast radius.
Cart writability, inventory oversell, order durability, and analytics isolation are the real failure boundaries in commerce data architecture.
PII boundary enforcement breaks when consent, encryption, and regional residency are conventions scattered across services, queues, and warehouses.
Order count discrepancies between OLTP and the warehouse often trace to CDC pipeline schema drift redefining what counts as a committed order.
Propagating a catalog update from database commit through Elasticsearch, CDN edge cache, and application cache without stranding stale reads downstream.
Reservation, release, and reconciliation for inventory systems where carts, payments, and retries generate conflicting stock counts across writes.
Hot key contention, connection pool exhaustion, and cache miss bursts each hit local thresholds before aggregate dashboards show anything alarming.
Elasticsearch is a read index, not a record system — routing writes through it creates catalog drift that surfaces only after orders are placed.
Order state machines prevent checkout duplication by constraining which database transitions are legal — so a paid order cannot be paid twice.
Session cache versus durable cart: the recovery semantics that determine data survival across session loss, browser closure, and checkout failure.
Catalog, cart, orders, inventory, and payments as five distinct consistency problems — why a shared transaction boundary causes e-commerce system failures.
OCI disaster recovery gaps that emerge when teams rely on regional failover alone, and how Data Guard and GoldenGate address the database replication tier.
Isolating the OCI Autonomous Transaction Processing write path from catalog and analytics load using GoldenGate replication and Object Storage offloading.
Exadata Cloud Service exposes RDMA interconnects and Smart Scan offload tiers that matter when Oracle workload latency cannot be fixed with software alone.
Oracle Autonomous Database automates patching and scaling, but cannot substitute for query intent, schema decisions, and access patterns the team must own.
How OCI load balancing, OKE, Autonomous Database, cache, and queue layers interact — and why cross-service ambiguity assumptions cause the first failure.
Control plane coupling, Spanner split boundaries, and untested Pub/Sub failover are why GCP multi-region architectures break before the region goes dark.
Slot contention and multi-second scan latency are the failure modes when BigQuery gets used as the transactional backend of a user-facing service.
Pub/Sub ordering keys control which events serialize together, determining whether failures stall the whole stream or only the affected partition.
Cloud Spanner vs Cloud SQL turns on failure domain tolerance — whether your SLA survives a regional primary outage, not on scale or throughput alone.
Azure multi-region design tradeoffs: Front Door routing, Cosmos DB consistency, and SQL failover group lag — and which failures each bet absorbs.
Azure database recovery beyond 'we have backups': failover group cutover, geo-replication lag, and backup restore testing as the real reliability floor.
The wrong Azure database choice announces itself when one tenant or region becomes hot enough to make every clean abstraction expensive — how to decide between Azure SQL and Cosmos DB based on access patterns, consistency needs, and operational cost.
Azure applications typically fail first at the edges: Front Door configuration, App Service connection pools, SQL failover groups, Redis cache invalidation, and Service Bus backlog — a reference architecture that makes these failure boundaries explicit.
AWS multi-region failover fails most often in traffic steering, write promotion, and schema drift — how Route 53, Global Accelerator, Aurora global databases, and DynamoDB global tables behave under a real regional failure.
Database bills grow when ownership, workload shape, and control loops drift apart — a structured triage approach for RDS, Aurora, DynamoDB, ElastiCache, and OpenSearch spend before it becomes an emergency.
Most AWS data leaks happen when identity, network, encryption, and audit boundaries are designed as separate controls by separate teams — a multi-account architecture that treats VPCs, KMS, IAM, and CloudTrail as a unified boundary.
Checkout fails when payment, inventory, order history, and notification are treated as one synchronous request — how to model checkout as one committed decision followed by recoverable asynchronous consequences using SQS, Lambda, Aurora, and DynamoDB.
S3 event processing is durable and cheap but the event stream and the bucket tell different stories — how to design S3-driven pipelines around ordering guarantees, duplicate delivery, and eventual consistency without data loss.
The real difference between Aurora and RDS shows up during storage stall, replica lag, and failover at 03:00 — how the two products behave differently under failure and what those differences mean for operational choice and cost.
Most system designs fail for reasons visible at review time: overloaded dependencies, ambiguous ownership, unsafe retries, unbounded queues, and missing rollback paths — a checklist senior engineers use to surface those risks early.
Multi-region is usually a failure-containment project, not a scalability project — and deploying across regions exposes every weak assumption in your data model, write ownership strategy, and cross-region blast-radius planning.
Healthy systems preserve their ability to recover by refusing work before a failure becomes contagious — how to design backpressure at the queue boundary, connection pool, and API layer so overload stops propagating upstream.
Capacity planning fails when teams size for the average request and ignore fanout, hot keys, and bursty traffic — a framework for sizing from QPS, read/write ratios, and peak multipliers before the first incident teaches the lesson.
Queues and streams solve different problems: commands vs events, at-most-once delivery vs replay, immediate consumption vs historical processing — and teams that choose without understanding the difference reverse the decision under load.
The first system design question is not 'what are the services' — it is 'what breaks, how fast does it spread, and what evidence tells us the damage is contained.' A framework for failure-mode-first design.