Applications should not care which database instance is the primary writer, nor should they crash when the database topology shifts. Yet, most systems rely on DNS propagation or fragile client-side driver logic to handle High Availability (HA) failovers, resulting in guaranteed downtime and catastrophic connection storms.

When your database serves the world’s largest repository of source code, 30 seconds of DNS propagation delay is an unacceptable eternity. This is why the industry relies on intelligent proxy layers and topology orchestrators.

Situation

As a platform scales, the database topology inherently becomes more complex. You move from a single primary to a primary with read replicas, then to a massive sharded fleet.

During a hardware failure or a routine maintenance window, a database failover occurs. A replica is promoted to become the new primary. However, the application servers are still holding onto thousands of established TCP connections pointing to the dead primary. When those connections break, the application pods desperately attempt to reconnect, overwhelming the newly promoted primary in a “connection storm.”

DNS-based failover (updating a Route53 CNAME) is the traditional industry standard, but it relies on the application language, the OS resolver, and the local network caching honoring Time-To-Live (TTL) values correctly. In reality, DNS caches are sticky. The application throws Connection Refused or Read-only transaction errors until humans intervene or automated backoff scripts finally succeed.

The Problem

The database layer and the application layer have misaligned expectations. The database expects to be able to shift roles (writer to reader, active to passive) instantly for HA purposes. The application expects a persistent, unbreakable TCP connection for its connection pool.

When these layers are tightly coupled, any database topological change cascades as a failure into the application. If 50 application pods simultaneously detect a disconnect and aggressively attempt to recreate 100 connections each, the resulting 5,000 connection requests will instantly exhaust the database’s max_connections limit, or max out the CPU handling the TLS handshakes, killing the database before it can serve a single query.

The core question is: how do you decouple the application’s connection lifecycle from the physical database topology so that failovers become completely invisible to the end user?

The ProxySQL Decoupling Architecture

flowchart TD
    A[Application Fleet] --> B[ProxySQL Tier]
    B --> C[Database Primary — Writer]
    B --> D[Database Replica 1 — Reader]
    B --> E[Database Replica 2 — Reader]
    
    F[Orchestrator Topology Manager] --> C
    F --> D
    F --> E
    F -->|Admin API Hooks| B

In Practice: GitHub’s Orchestrator, Paired with a Proxy Layer

The documented pattern for extreme HA survival relies on deploying an intelligent database proxy between the application fleet and the database layer, governed by an autonomous topology manager.

GitHub’s engineering team authored and open-sourced the definitive solution to this problem: Orchestrator. GitHub runs a massive fleet of MySQL servers. When a primary database hardware node died, they could not afford the downtime of manual intervention or DNS TTL propagation.

They built Orchestrator to act as the “brain” of the MySQL topology. Orchestrator continuously discovers, maps, and monitors the replication state of every MySQL node. When a primary fails, Orchestrator detects the failure, calculates the best replica to promote (based on replication lag and configuration) faster than DNS-based approaches can react — I have not independently verified a specific sub-15-second figure against current GitHub documentation, so treat that number as illustrative rather than a cited benchmark.

However, promoting a new database doesn’t magically update the applications. To solve the traffic routing, GitHub’s own documented architecture pairs Orchestrator with GLB (GitHub’s load balancer) and HAProxy, not ProxySQL specifically. ProxySQL is the wider industry’s equivalent pattern: many MySQL operators pair Orchestrator’s topology decisions with a SQL-aware proxy tier so application connections survive a failover. The mechanics below describe that general ProxySQL pattern, not a GitHub-specific implementation detail.

ProxySQL acts as a connection multiplexer and query router. In this architecture, the application never connects directly to the database. It connects to ProxySQL. ProxySQL maintains two separate pools of connections: one pool facing the application (frontend) and one pool facing the databases (backend).

When Orchestrator detects a failure and promotes a new primary, it immediately fires a web hook to the ProxySQL Admin API. Orchestrator tells ProxySQL, “Node A is dead; Node B is the new writer.” ProxySQL updates its internal hostgroups in milliseconds.

Surviving the Connection Storm

Because the application is connected to ProxySQL, its frontend TCP connections are never dropped. During the 10-second window while Orchestrator promotes the new database, ProxySQL simply holds the incoming write queries in a queue.

Once Orchestrator informs ProxySQL of the new writer, ProxySQL flushes the queued queries to the new primary. The application experiences a slight latency bump (e.g., a query taking 5 seconds instead of 5 milliseconds) rather than a hard Connection Refused exception. The failover is entirely transparent to the application code.

Furthermore, ProxySQL mathematically solves the connection storm problem through multiplexing. Even if 50 application pods open 5,000 frontend connections to ProxySQL, ProxySQL might only open 200 persistent backend connections to the MySQL database. It rapidly routes queries from the 5,000 idle frontend connections through the 200 active backend connections.

When the new primary is promoted, it only has to absorb 200 connection handshakes from ProxySQL, protecting the database engine from connection thrashing and CPU saturation. To prevent cascading failures, Orchestrator implements “anti-flapping” policies, ensuring that if the new primary is unstable, it won’t repeatedly trigger automated failovers that destabilize the entire fleet.

Where It Breaks

TopologyFailure ModeMitigation
Direct DNS FailoverApplication caches old IP address; writes fail against demoted primary.Implement ProxySQL or smart drivers; stop relying on DNS TTLs for database HA.
ProxySQL TierProxySQL becomes the single point of failure (SPOF) and crashes under load.Deploy ProxySQL as a highly available cluster or run it as a sidecar alongside application pods.
Connection StormsApplication reconnect logic lacks jitter, DDOSing the database during recovery.Use connection multiplexing via proxy; enforce exponential backoff with jitter in application code.
Prepared StatementsProxy multiplexing breaks session state or server-side prepared statements.Configure ProxySQL to appropriately map session variables or use client-side prepared statements.

What to Do Next

  • Problem: Direct application-to-database connections guarantee application errors during database HA failovers and risk connection storms that can crash newly promoted primary databases.
  • Solution: Introduce an intelligent proxy tier (ProxySQL) paired with a topology manager (Orchestrator) to decouple application connection pools from physical database instances.
  • Proof: GitHub Engineering explicitly documented the creation and open-sourcing of Orchestrator to achieve fast, low-downtime failovers across one of the world’s largest MySQL deployments; pairing that topology manager with a SQL-aware proxy (GitHub uses GLB/HAProxy, many other operators use ProxySQL) is the documented industry pattern for making the failover transparent to applications.
  • Action: Audit your application’s connection pool settings. Deploy ProxySQL in a staging environment and perform a forced failover (e.g., kill -9 the primary database process) to verify that your application only experiences latency spikes, not hard errors or dropped connections.