Patroni failover: 10 scenarios to test before production

31 August 202611 min read

PostgreSQLPatroniHigh availability

Patroni does not prevent split-brain. The distributed configuration store does, and Patroni is the thing that respects it. Conflating those two is how clusters end up with two primaries taking writes.

Below are the failure scenarios worth testing before a Patroni cluster carries production traffic, and the configuration mistakes that turn each one into an incident.

What Patroni actually guarantees

Patroni is a control loop. Each node reports to a DCS — etcd, Consul or ZooKeeper — and the node holding the leader key is primary. The guarantee comes from the leader key having a TTL that must be renewed. A node that cannot reach the DCS cannot renew, and a node that cannot renew must demote itself.

That last clause is the whole safety model. It only holds if demotion is reliable, which is where testing goes.

Scenarios to test before production

Scenario	What you break	Expected behaviour	What it catches
Clean switchover	patronictl switchover	Role swap, no data loss, brief pause	Application reconnect handling
Primary process killed	kill -9 on postgres	Local restart or failover within TTL	Whether failover is faster than the app's timeout
Primary host lost	Power off the VM	Replica promoted after TTL expiry	Real recovery time, not the theoretical one
DCS unreachable from primary	Firewall the primary off etcd	Primary demotes itself, cluster read-only	The most important test on this list
Whole DCS down	Stop every etcd node	All nodes demote, no primary	That losing etcd fails safe rather than open
DCS quorum lost	Stop 2 of 3 etcd nodes	Same as above	Whether etcd is genuinely 3+ nodes
Network partition	Split primary from replicas and DCS	Isolated node demotes, majority side promotes	Split-brain resistance
Replica far behind	Pause replay, then fail the primary	Lagging node not promoted, or promoted only within maximum_lag_on_failover	Silent data loss on failover
Disk full on primary	Fill the data volume	PostgreSQL stops, Patroni fails over	A failure mode monitoring often misses
Return of the old primary	Bring the failed node back	Rejoins as replica, pg_rewind if needed	Whether the old primary can rejoin at all

The row that matters most is "DCS unreachable from primary". If the primary keeps accepting writes while it cannot see etcd, the cluster has no split-brain protection at all, whatever the rest of the configuration says.

Configuration mistakes that cause split-brain

etcd on the database nodes

A three-node cluster where etcd runs on the same three machines as PostgreSQL looks tidy and removes the safety margin: the same event takes out both a database node and a DCS member. Lose two machines and you lose etcd quorum at the same moment you need it. Run the DCS separately, or accept that this cluster tolerates exactly one failure.

Two etcd nodes

Two nodes have a quorum of two. Losing either one loses quorum entirely, so a two-node DCS is less available than a single node. Three is the minimum that means anything.

No watchdog

Demotion assumes the Patroni process is healthy enough to demote. If the host is swapping, or Patroni is stopped while PostgreSQL keeps running, that assumption breaks. A hardware or softdog watchdog reboots the node instead of trusting it to behave.

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576   # 1MB, refuse to promote a laggard
    synchronous_mode: true             # no silent data loss on failover

watchdog:
  mode: required        # refuse to start as primary without a watchdog
  device: /dev/watchdog
  safety_margin: 5

ttl, loop_wait and retry_timeout that do not add up

These three are related and are often tuned individually. The rule Patroni documents is that ttl must be greater than loop_wait plus twice retry_timeout. Set ttl too low and the cluster fails over on a network hiccup; set it too high and a genuinely dead primary keeps its lock for a minute while the application times out.

Asynchronous replication with an unread failover policy

With the defaults, a failover can promote a replica that is behind, and the writes that had not reached it are gone. Sometimes that is the accepted trade. It should be an explicit decision recorded somewhere, not a discovery made during the post-incident review. synchronous_mode makes the trade the other way: no data loss, at the cost of write availability when the synchronous standby is unreachable.

The load balancer that does not ask

Patroni exposes an HTTP endpoint that answers whether a node is primary. If HAProxy or the application routes on a static IP or on plain TCP checks instead, the cluster can fail over correctly and traffic will keep going to the old node.

# HAProxy backend that routes on role, not on IP
backend postgres_primary
    option httpchk GET /primary
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server pg1 10.0.0.11:5432 check port 8008
    server pg2 10.0.0.12:5432 check port 8008
    server pg3 10.0.0.13:5432 check port 8008

on-marked-down shutdown-sessions matters: without it, connections already open to the demoted node stay open and keep trying to write.

Backups are not replication

A Patroni cluster survives a node failure. It does not survive a DROP TABLE, and it replicates corruption faithfully to every replica. pgBackRest alongside the cluster, with restores actually tested, covers the failure modes that HA does not. An untested backup is a hypothesis.

A pre-production checklist

•DCS on separate hosts, at least three of them.
•Watchdog enabled, mode set to required.
•ttl > loop_wait + 2 x retry_timeout, and the numbers written down with the reasoning.
•maximum_lag_on_failover set deliberately, or synchronous_mode on.
•Routing based on the Patroni REST endpoint, not a static IP.
•Every scenario in the table above executed at least once, with the observed recovery time recorded.
•pgBackRest configured, and a restore performed into a scratch instance.
•Monitoring alerts on: no primary, replication lag, DCS quorum, and time since last successful backup.

Where this fits

We build and run PostgreSQL HA on Patroni, including the part most people skip — sitting down and actually breaking the cluster before it holds real data. That is a fixed-scope engagement, billed at €65/hour, and it ends with a written record of how the cluster behaved in each scenario rather than an assurance that it should be fine.

Running PostgreSQL HA?

We design, build and test Patroni clusters, including the failure testing most setups skip. €65/hour, fixed-scope, with a written record of how the cluster actually behaved.

Talk to a DBA Database services