cal · writing

Designing for high availability

Feb 15, 2026 · 2m read

The patterns I used to build the HA Infrastructure project — VIP failover, Postgres replication, and shared storage across APIs, databases, and Airflow.

Postgres HA with Patroni

Patroni handles automatic leader election and failover for Postgres clusters. Each node runs a Patroni agent; leader election is mediated through a distributed consensus store (etcd or Consul). On primary failure, the agent with the most up-to-date WAL position is promoted automatically.

Key configuration choices:

  • synchronous_mode: true — replica must acknowledge writes before the primary returns success. Eliminates data loss on failover at the cost of write latency.
  • max_lag_on_failover — the agent won't promote a replica that's too far behind the primary.
  • pg_hba.conf managed by Patroni so all nodes stay consistent.

Virtual IPs with Keepalived

Keepalived manages a floating Virtual IP (VIP) shared across the Postgres nodes. Applications connect to the VIP; on failover, Keepalived moves the VIP to the new primary within seconds. VRRP (Virtual Router Redundancy Protocol) handles the election.

API → VIP:5432 → Patroni primary (Keepalived moves VIP on failover)

The VIP approach means no application-side connection-string change on failover — the connection pool reconnects to the same address.

Shared storage with GlusterFS

GlusterFS provides distributed, replicated storage for Airflow DAGs and logs. A replica 2 volume mirrors data across two nodes; writes are synchronous. This means Airflow workers on any node see the same DAG files without an NFS single point of failure.

For smaller setups NFS is simpler, but GlusterFS eliminates the NFS server as an SPOF.

The full stack

Load balancer (Nginx + Route53/Azure DNS health checks)
    ↓
Application nodes (Gunicorn / FastAPI)
    ↓
Patroni primary (VIP via Keepalived)
    ↓
GlusterFS replicated volume (DAGs + logs)

Airflow's scheduler and workers run on the application nodes and share DAG storage over GlusterFS. Database connections go through the VIP. If either Postgres node dies, Patroni promotes the replica and Keepalived moves the VIP — no application restart required.

← all writing