prty Contact

Architecture · data plane · quorum and failover

Quorum and failover.

The members of a cluster vote on who is the primary. A primary that cannot get a majority to confirm it stops itself. Only then can another member be promoted. The rule behind all of it: never two writable primaries, and no acknowledged write lost.

The concept

Each member runs a small Rust program, the sidecar, beside PostgreSQL. The sidecars run Raft (the consensus algorithm behind etcd and many others) and replicate one tiny record per cluster: the generation (a counter that goes up by one at every promotion), who the primary is, which replicas must hold every acknowledged commit, and whether a planned switchover or maintenance is in progress. A member can only be primary if the committed record says so, at the current generation.

Three ideas make this safe:

  1. A majority elects and a majority acknowledges. With three voters, two are needed to change the primary. Synchronous replication means a commit is acknowledged to the client only when a replica also has it, so the member that wins the vote has every acknowledged write.
  2. A primary needs a lease. Every second the primary asks the quorum “am I still the primary at this generation?”. If it gets no answer for 10 s it stops PostgreSQL. A hardware watchdog (a kernel timer, armed only on the primary) resets the whole machine at 15 s if the sidecar itself hangs.
  3. Promotion waits for the lease to be dead. The member that coordinates promotes a replica only 20 s after the primary’s last confirmation, longer than the primary can possibly keep writing. The argument relies only on clocks running at about the same rate, not on synchronised time.
A three-member cluster: voting, replication and the primary's lease Three zones. The primary and the replica each hold data; the arbiter only votes. All three sidecars talk Raft. The primary streams to the replica synchronously and renews a lease every second through the quorum. Clients use a DNS name or a list of members. The management server is outside the vote and only observes and repoints the name. zone a zone b zone c Primary PostgreSQL (writes) sidecar + Raft vote lease: renewed every 1 s no renewal 10 s: stop no pet 15 s: host reset Replica PostgreSQL (hot standby) sidecar + Raft vote has every acknowledged commit (sync repl.) Arbiter sidecar only Raft vote, no data never promoted sync WAL Raft Client host=name,primary,replica target_session_attrs=read-write Management server (outside the vote) reads the record, repoints the DNS name after a promotion, asks for planned switchovers, shows the status page
Losing any one box leaves two voters, a majority. Losing two leaves one, which can neither elect nor confirm: that is an outage by design (below).

When the primary is lost

Timeline of an unplanned failover At 0 seconds the primary is lost. By 10 seconds it would have stopped itself. By 15 seconds its machine would have been reset by the watchdog. At 20 seconds the coordinator may promote. About 22 seconds later PostgreSQL accepts writes. Around 30 seconds clients using only the DNS name are connected again. 0 sprimary lost 10 sstops itself 15 slatest: host reset 20 spromote allowed ≈ 22 s: new primary writable (lab) ≈ 30 sname-only clients
The 20 s wait is the price of certainty that the old primary has stopped. It is the largest single part of the failover and it is tied to the watchdog; shortening it needs new measurements on AWS first.
  1. The old primary stops confirming. Its own sidecar stops PostgreSQL; if that sidecar hung, the watchdog resets the machine.
  2. The coordinator (whichever member leads Raft, usually the arbiter) waits until 20 s have passed since the last confirmation, and checks the preconditions: the chosen replica is in the synchronous set with the highest received log position, the cluster is not degraded, nobody has set maintenance, and no automatic failover happened in the last 10 minutes.
  3. It commits Promote, generation + 1. The replica promotes itself, takes over synchronous replication, and the management server repoints the cluster’s DNS name.
  4. When the old primary comes back it sees the higher generation, rewinds the part of its log that was never acknowledged (pg_rewind), and rejoins as a replica. A machine that restarts never starts as primary unless the quorum confirms it within the last 10 s.

Planned switchover

For maintenance the primary stops cleanly, the replica waits until it has the final checkpoint, and only then is it promoted: no acknowledged write can be lost because nothing is ambiguous. A planned switchover does not wait for any lease.

Failure modes that matter

What failsWhat happensAcknowledged writes
The primary’s VM, disk or zoneFailover as above, about 30 s.None lost.
The primary is cut off from the others but still aliveIt cannot renew its lease, stops itself at 10 s, and the others promote after 20 s. A client connected to it gets errors, never an acknowledged write that will vanish.None lost.
The replica is lostIn a cluster with two replicas, nothing: either replica’s acknowledgement is enough, so writes continue with no loss of guarantee. With one replica, writes would block, because the replica no longer confirms them. By default (relaxed) the cluster notices after 10 s, records that it is degraded and carries on with only the primary. A cluster can instead be set to strict: writes then wait for the replica.While degraded, a second failure can lose recent writes. A degraded cluster does not fail over by itself; a person decides, after the loss is measured.
The arbiter is lostNothing visible: the primary and replica are still a majority. The margin is gone until it is replaced.None lost.
Two members at onceNo majority. The cluster cannot elect or confirm, so it stops accepting writes and stays stopped until a member returns. In one soak run a spot interruption of a test instance while another was already down stopped writes for 21 minutes.None lost; availability lost.
The management serverThe cluster promotes by itself. The DNS name keeps pointing at the old primary until the server returns; clients with the member list (the default) are unaffected.None lost.
A flapping or slow machineThe leader lease keeps a restarting follower from disturbing a healthy leader, and at most one automatic failover per 10 minutes is allowed.None lost.
The whole region, or the cloud’s control APIOut of scope. Every member is in the same region.—

The honest limit. Three members survive any one failure and no more. Two failures at once is an outage until a member is back. We are working on making that case rarer and shorter (reliability, roadmap); we cannot make it disappear with three voters.

Problems this design solves

  • Split brain. A primary stops itself before anyone else is promoted, the watchdog backs up a hung sidecar, and a rebooting machine must be confirmed before it may be primary.
  • A failover that needs a single server. The members decide and act, in about 30 s, and the management server can be down.
  • A wrong guess about a dead primary. Promotion is a committed decision with preconditions, not a side effect of who answered first. A partitioned minority cannot commit anything.
  • Losing data when a replica was not caught up. Only a replica in the synchronous set, with the highest log position, can be promoted.
  • Clients that cache DNS. The connection string lists every member and asks for a writable one, so a stale name pointing at a standby is skipped instead of written to.
  • Flapping. A minimum interval, a lease, and a maintenance mode that pauses automatic failover (but never the self-fencing).

Objectives

ObjectiveTargetStatus
Primary lost → writes accepted again≤ 60 sMeasured 29.5 s and 30.1 s on AWS (the writer’s longest gap); 21–29 s in the local lab. Design: 25–35 s.
Acknowledged commits lost in an automatic failover0Measured none in the AWS chaos suite (r1–r9) or the soak: 119,654 writes, none missing. Exception stated above: a degraded cluster.
Planned switchover: writes pausedsecondsMeasured up to 10.2 s on AWS for clients using only the DNS name, mostly propagation; 1.3 s to a writable new primary and 2.3 s longest client pause in the local lab. No loss in 8 runs.
Primary stops itself when it loses the majority≤ 10 s + 3 sDesign Startup refuses timing settings that break the ordering of the timers.
Name-only clients reconnected after a failover+ 8–10 sMeasured on AWS: DNS change plus TTL. The member list avoids it.
Recovery from losing two membersno targetEstimate It is the time to bring one member back: minutes if the machine restarts, longer if a disk or machine must be replaced. To figure out: time the manual replacement of a member and decide whether the system may do it unattended.
Replace a lost member by handminutesEstimate New VM plus a copy of the data. To figure out: measure by database size.