Architecture · control plane

The control plane.

The part of prty that creates, changes and deletes clusters, and the Autopilot that decides when a change is due. It is one management server per deployment. It is deliberately not on the path of a running database: while it is down you cannot make changes, but the clusters keep serving and keep failing over.

The concept

A deployment is one private network in one cloud region. It holds one management server (a VM) and any number of database clusters (the data plane). The control plane is what runs on the management server:

The API and the operations engine. Every change to a cluster is an operation: a list of steps, recorded step by step in a small embedded database (SQLite) on the server’s disk.
The Autopilot (below), a separate process that only ever calls that API.

The management server also hosts the metrics, logs and status page that the monitoring block uses. The control plane makes no decisions of its own. It does what it is asked, by a person or by the Autopilot, and records who asked and why. It is also the only component with permission to change cloud resources: create and stop machines, attach disks, change the private DNS name.

The operator’s tooling runs once to create a deployment. After that, the management server talks to an agent on each database machine, over the private network.

How it is rolled out

Machine images are baked ahead of time with Packer, one for the management server and one for database machines. The control plane, Autopilot, status page and agents are Rust binaries copied into the image. Nothing is compiled or fetched from the internet on a database machine at boot.
Infrastructure as code (Terraform) creates the deployment: the private network with one private subnet per zone, firewalls (one for the management server, one for database machines, and a “fenced” one that admits nothing but the management server), a private DNS zone, access roles, and the management server from its image.
First boot of the management server writes generated secrets (the token its agents accept, the Autopilot’s own token) and the settings that tell it which images, zone and subnets to use, then starts its services. Secrets are generated per deployment.
The control plane resumes its unfinished work. On start it re-runs any operation that was running: finished steps are skipped and the step that was in flight starts again from its beginning, which is why every step is written to be safe to repeat.

Replacing it with a new version

A new control plane version is a new management-server image. The server is treated as replaceable: its only state is the SQLite files on its disk and the metrics store. A control plane release does not touch the database machines; their sidecar and agent versions change only with the machine image of new machines (minor upgrades).

To figure out later: the exact procedure and timing of replacing a live management server (waiting for running operations to finish, moving or restoring the SQLite files, and what to do if a release changes the steps of an operation that is half-done) are not written down as a tested runbook yet. The page will be updated when they are.

Operations: how a change is made safely

Creating a cluster, growing a disk, a minor upgrade, replacing a member: each is a durable operation with the same rules.

Every step is written down before and after it runs, and is written to be repeated, so a crash or a restart resumes the operation instead of losing it.
Transient errors are retried with backoff, and each attempt is recorded.
Before its point of no return, a failing operation is undone in reverse order and ends rolled back. After it, it goes forward or parks as needs attention. A failed undo parks as compensation failed. It is never left looking successful when half-done.
A person can abort a running operation (until its point of no return) or resume a parked one. The Autopilot cannot.
One operation per cluster at a time. A second request is refused while one is running or parked.
Every operation records who asked, a person or the Autopilot with its decision, and why.

The Autopilot: deciding when

The control plane knows how to do things. The Autopilot decides when. It reads signals from monitoring, checks them against policies you set, and calls the same API a person would, after writing down why. It is a separate process with its own small database for its decision records. If it is down, nothing breaks; things just stop being automatic.

A policy is a typed document: what to watch (selector), when to look (trigger), the limits (guardrails) and how much freedom it has (level). The one policy built today is Grow: when a database’s disk use passes a threshold, grow the persistent disks online, in bounded steps, up to a ceiling you set, then wait a cooldown. There is no downtime. Replicas are grown before the primary, so a failure part-way never leaves a replica with less room than its primary.

The Autopilot does not fail clusters over. Quorum clusters decide that for themselves, without the management server, and the control plane would refuse the request (quorum and failover).

If the Autopilot crashes or cannot see its inputs, it does nothing. The control plane re-checks every request and can refuse it.

Levels

The level is a dial per policy: manual (the Autopilot does nothing), suggest (it records what it would do), approve (it proposes, a person approves) and auto (it acts within the guardrails). Whatever the level, every evaluation that passes the gate is recorded. A shadow flag evaluates and records without ever calling the API, to run a new policy for weeks before it acts.

A policy

Policies are YAML, applied with prtyctl policy apply and validated against a typed definition when applied. Anything you leave out takes the definition’s default. There is no fleet-wide policy: a selector is required, and a policy without one is rejected. A policy the provider cannot carry out is rejected too, when you apply it.

kind: Grow
name: web-prod-grow
selector: { database: web-prod }   # which database it applies to
level: auto                        # act, and record why
trigger: { disk_used_pct: 70 }     # the gate: look when use passes 70%
guardrails:
  target_used_pct: 60              # aim to land back near 60%
  max_step_gib: 10                 # never grow by more than 10 GiB at once
  max_gib: 1000                    # never grow past this ceiling
  cooldown: 1d                     # then wait before growing again

With a 100 GiB disk at 72 GiB used, this asks for a grow to 110 GiB. Use afterwards is about 65%, still above the 60% target because the step is capped; the next cycle after the cooldown grows again if use keeps climbing.

A decision on record

For each evaluation that passes the gate, the Autopilot stores a structured reasoning record: the inputs it saw, each check in order with its result, and the outcome (proceed, skip or defer). The record is written before the call, so a crash cannot leave an action without its explanation. The operation it starts carries a reference to the decision, and repeating a call for the same decision returns the operation already started instead of starting another. The layout below is simplified for the page; the real record is structured JSON.

policy: web-prod-grow           # and the definition version it ran against
level: auto
outcome: proceed
summary: web-prod disk 72% used (72 of 100 GiB); growing to 110 GiB
checks:
  - trigger:      pass          # disk_used_pct 72 >= 70
  - capability:   pass          # the provider can grow disks online
  - enabled:      pass
  - cooldown:     pass          # no earlier grow of this database
  - within_max:   pass          # 110 <= 1000
action: grow web-prod to 110 GiB   # carries this decision's id

Failure modes that matter

What fails	What happens	What you can do
The management server is down	No new clusters, upgrades, disk growth or automation. Clusters keep serving and fail over by themselves (measured: a primary lost with the control plane stopped was replaced in 22 s). One thing waits for it: moving the cluster’s DNS name after a failover.	Clients that list every member in their connection string (the default string does) find the new primary without DNS. Replace the server from its image.
Its disk is lost	The record of which clusters exist, their credentials and the history of operations goes with it. The clusters themselves keep running.	This is the case we are least ready for: the state files are not backed up off the machine yet. Treat it as an open item.
It restarts in the middle of an operation	The operation resumes. Finished steps are not repeated; the interrupted one runs again.	Nothing, unless a step cannot be repeated safely, in which case it parks for a person.
An operation step fails	Retried, then undone before the point of no return; forward or parked after it (see above).	A person resumes or aborts a parked operation. The Autopilot cannot.
The Autopilot is down, or sees stale or missing metrics	No automatic disk growth; it never acts on data it cannot see. A full disk is then yours to prevent by hand, and nothing outside the server alerts you yet.	Grow a disk by hand with the same API. Run a policy at suggest first to learn how it would behave.
A disk-growth loop	Growth is stepped, capped at the policy’s maximum, and followed by a cooldown.	A disk that grows faster than that is a person’s decision.
The cloud’s control API is down	The server cannot create, stop or move anything. Running clusters are unaffected.	Wait. Nothing we run can replace machines while the provider cannot.
A token leaks	The token is the only thing that authorises commands to a deployment’s database machines, and it only works from the private network.	Rotate by redeploying. There is no tested rotation procedure yet.

Problems this design solves

One thing to operate per network, instead of one per customer or one per database. A release is one image.
No SSH and no key distribution. Commands go through a small agent on each machine with a per-deployment token. Remote commands run as local processes, so there are no hung sessions.
Database machines hold no cloud permissions. A compromised database machine cannot stop another cluster’s machines or rewrite another cluster’s DNS name; only the management server can.
No internet route from a database machine, so no NAT gateway to run. Images are baked.
Long operations survive crashes and never end half-done without saying so.
Mechanism and decision are separate. The control plane never starts anything on its own; the Autopilot or a person does, through the same API, so every action has an author and a reason on record.
Automation you can trust slowly. Suggest and shadow modes let a policy run for weeks without acting, and the API refuses the Autopilot what only a human may do.

Objectives

Objective	Target	Status
Impact of a management-server outage on a running cluster’s availability	none	Measured failover with the control plane stopped: 22 s, no acknowledged write lost (AWS chaos suite).
Availability of the management server itself	no target	Estimate One VM, in one zone, so at best the VM’s own 99.5%–99.9% from the cloud’s commitment. We set no objective until we have operated it.
Time to replace a lost management server	≈ 30 min	Estimate Boot from the image and start services. To figure out: time a real replacement, and whether the state files can be restored.
Time from nothing to a ready deployment	≈ 10 min	Estimate To figure out: time infrastructure creation plus first boot.
Operations lost to a restart	0	Design Resumed from the recorded step.
Every Autopilot action has a reasoning record	100%	Design The record is written before the call.
Autopilot calls the API would refuse a person	0	Design Same endpoints, same checks; “force” is not available to it.
Disk-use signal to a grow decision	≤ 2 min	Estimate 30 s cycle plus 30–75 s metric lag. To figure out: measure from the crossing of the threshold to the call.
A disk grow, start to finish	minutes	Estimate Cloud volume modification plus filesystem growth, online. To figure out: time on AWS at several sizes.

Known gap, stated plainly: the management server is a single instance and nothing outside it alerts a person when it, or the Autopilot, stops. Control-plane high availability and an outside alert are on the list; neither is built.