Architecture · data plane · minor upgrades

Minor upgrades.

Moving from PostgreSQL 17.x to 17.y by adding new machines from a newer image beside the old ones, not by changing the old ones. Nothing is removed until everything that replaces it has proven itself. The only moment a client notices is the switchover. You start it, when you want it.

Status. Built and tested on AWS for quorum clusters (a primary and a replica with an arbiter, or a primary and two replicas). Clusters of more than three data members and other clouds have no minor upgrade. Machines are never patched in place: a database machine has no route to the internet to pull an image, and patching a running machine is the riskier way. See the roadmap.

The concept

A minor version changes only the PostgreSQL program, not the on-disk format, and a server of one minor can stream its log to a standby of another minor of the same major. Machine images are never patched: a new minor is a new image, with the operating system, PostgreSQL and our agent that were tested together. An upgrade adds machines from that image, lets them catch up as followers that do not vote, hands them the old machines’ votes one at a time, switches the primary over, and only then retires the old machines. Everything before the first retirement can be undone.

It is a durable workflow, like every operation (control plane). The new machines’ data does not come from copying the database over the network: on AWS each new machine’s volume is a clone of the old machine’s volume in the same zone, usable at once while AWS fills it in the background. If a clone cannot be made, the new machine copies the database with pg_basebackup instead.

The cluster always keeps its three voters, one in each zone, and never has fewer copies of the data than it began with.

The procedure

Before it starts, the control plane refuses the upgrade, and lists every reason, unless: the target image is a newer minor of the same major and someone has marked it safe to upgrade onto after reading its release notes; every member answers and runs an agent version the image is compatible with; there is a leader, nothing is degraded, and the replicas are close behind; and no other operation is running on the cluster, or any other upgrade.

Safety copy. A snapshot of the primary’s data volume and of one replica’s (or a recent completed one, or none, if you say so and acknowledge it). It is the only copy that the new PostgreSQL has never touched.
New machines join. One per data machine, in the same zone as the machine it will replace, each on a clone of that machine’s volume. They follow the primary and do not vote or count for commits.
Verify. They stream without a break, on the new version, close behind the primary, for a verification period (10 minutes by default). Any doubt undoes the upgrade.
Votes move. One vote at a time, a new machine takes an old replica’s vote in a single membership change; while it does, commits wait for two machines, so none is acknowledged by a machine that could not be promoted.
Switchover. Once the safety snapshots are complete and no transaction older than 5 minutes is open (unless you allow it), the planned switchover from quorum and failover moves the primary to a new machine. No acknowledged write is lost.
Last vote, optional hold. The new machine in the old primary’s zone takes the last vote. You can hold the old machines as followers for a while, so a regression is answered by aborting: the primary goes back to an old machine.
Old machines leave, one at a time, once the clone made from each one’s volume is fully filled in. They are stopped and kept for a retention period (7 days), a near-current copy for disaster recovery, then deleted if the cluster has been healthy.

There are no automatic or forced upgrades, including security releases. An upgrade changes the operating system as well as PostgreSQL, because the new image is built with the current one; the answer to the request says which. Up to the first retirement you can abort and it undoes itself; after it, a failure parks the operation for a person to resume.

Failure modes that matter

What fails	What happens
The image is not allowed, or the cluster is not whole	Refused before anything is made, with every reason. Nothing has changed.
A new machine never starts PostgreSQL on its copy, stops answering, or falls behind	The upgrade is undone: the new machines are removed and the cluster is as it began. Tested on AWS by terminating a new machine while it joined: undone, no instance left over, no acknowledged write lost.
A vote or the switchover is refused	Asked again for a minute, in case the state passes; then undone. A failure after the switchover puts the primary back on an old machine and the votes back, then removes the new machines.
The primary dies in the middle	The cluster fails over by itself, as always (quorum and failover); the upgrade notices, and goes on from where the cluster is. The new machines only add copies, so a replica that can be promoted remains.
A retirement fails	The point of no return has passed: the operation parks as needs attention with the cluster healthy and every member in place. Extra old machines cost money, nothing else; a person resumes it.
The management server dies mid-upgrade	The workflow resumes when it returns, reading the cluster again at every step. Quorum clusters keep serving meanwhile.
The new minor version has a defect	During a hold: abort, and the old machines lead again. After the old machines are retired: the safety snapshot and the stopped old machines are the way back, and both lose the writes since they were made. Without backups or point-in-time recovery (not built) this stays an incident, not a button.
The operating system’s C library changes under a database that uses a locale other than C	Text indexes can be corrupt. Our own clusters are created with the C locale, which has no such dependency; the upgrade does not check for other locales yet (roadmap).

Problems this design solves

Patching a database means an outage. Only the switchover is visible: a pause of seconds, not the length of the upgrade.
A change nobody can undo. The old machines are untouched until the new ones have proven themselves, so almost every failure ends with the cluster exactly as it was.
Untested combinations. The new machine runs the image that was tested as a whole, and the sidecar’s version is checked against what the image allows.
A copy of the database over the network. New machines start from clones of the old volumes, so the old machines are not read for hours.
Surprise upgrades. You start it, and the pre-checks tell you what to fix first.

Objectives

Objective	Target	Status
Acknowledged commits lost	0	Measured none lost under a continuous writer in the AWS runs of the upgrade (5,842 acknowledged writes in one run), in the undo after a terminated machine (4,637), and in a switchover afterwards. A small cluster on small machines.
Write pause seen by clients	≤ 30 s	Measured 10.2 s for the upgrade’s switchover and 10.1 s for a plain switchover on the same cluster, writing through the cluster’s DNS name. To figure out: members-list clients, larger databases.
Duration of the whole upgrade	hours for big volumes	Measured 207 s for two 1 GiB volumes with a 2-minute verification period. Estimate for large volumes: AWS fills a clone at about 35–45 MiB/s of written data (measured: 10 GiB written in 5 minutes, 50 GiB in 19 minutes); an old machine is not retired before its clone is filled in. To figure out: a large database end to end.
Fewer copies of the data than at the start	never	Measured the cluster has more members during the upgrade, never fewer; one brief degraded episode is the switchover’s own, as for any planned switchover.
Forced upgrades	never	Design
Disaster recovery from the safety snapshot	minutes	Measured a database started from the snapshot of the old primary’s volume answered a query 31 s after the restore began, with the writes made since lost. Adding a member on a restored volume to a live cluster is not built.