Architecture · monitoring

Monitoring.

Monitoring is the link between the other two blocks: the data plane feeds it, the control plane reads it. It runs on the management server today, but it is a block of its own because nothing that keeps your data safe depends on it.

The concept

We do not build our own metrics system. Three small pieces:

A monitor agent on every database machine (a small Rust program in its own container). Every 15 seconds by default it collects host and container figures (CPU, memory, disk) and PostgreSQL statistics, and ships the PostgreSQL and system logs. It pushes; nothing connects in to it.
A metrics store and a log store on the management server: VictoriaMetrics and VictoriaLogs, off the shelf. Metrics are kept for 30 days.
A status page on the management server: every cluster and its members, which ones need attention, and links into the metrics and log browsers. It reads the control plane, the metrics and the logs. It is an operator’s tool today.

Two other kinds of reader use what monitoring holds. The Autopilot polls the metrics store every 30 seconds for the signals its policies need (today, disk use). And the control plane itself watches each cluster in a way that does not use the metrics store: it reads the cluster’s own record from the members every 5 seconds, which is how it notices and records a promotion.

Solid: pushed by the data plane. Dashed: read by the control plane, the Autopilot and operators. Nothing flows back into the clusters from here.

Where it sits: it is fed by the data plane and read by the control plane, and it is deliberately on neither’s critical path. A cluster does not slow down, stop or fail over because a metric is missing; and the control plane’s safe operations do not wait for a metric.

Failure modes that matter

What fails	What happens
The metrics or log store is down, or the management server is	The clusters are not affected. Agents cannot deliver, so there is a gap in the history. The Autopilot cannot see its inputs and does nothing. Estimate We have not yet checked whether agents hold their samples and send them later; plan for a gap.
The monitor agent crashes or misbehaves	It runs in its own container with its own CPU and memory limits, so it cannot take memory from PostgreSQL. It is restarted by the machine, and metrics from that member stop until it is back.
The metrics are late	Expect 30–75 s of lag between an event and the store. That is fine for disk use and is why no safety decision uses metrics: promotion, fencing and failover work from the members’ own state.
The metric store fills its disk	The management server’s disk is shared with the control plane’s state. Retention is 30 days. Estimate To figure out: how much a cluster writes per day, and an alert before the disk fills.
A cluster is degraded and nobody is told	A known gap. The status page shows what needs attention, but nothing pages a person, and nothing outside the management server alerts when it is down. Alerting is on the roadmap and not built.

Problems this design solves

Reinventing metrics. We run VictoriaMetrics and VictoriaLogs, not our own store, and keep the agent small.
Monitoring that can hurt the database. The agent is a separate container with its own limits and only makes outbound connections.
Safety that depends on a dashboard. It does not: monitoring is observed, never consulted, by the cluster’s own decisions.
A signal that is too slow for the job. Slow, aggregated metrics are used for slow things (a disk filling over hours); the cluster’s own state is used for fast ones.
One more thing to operate. The stores run on the management server that is already there.

Objectives

Objective	Target	Status
Impact of monitoring being down on a cluster’s availability or safety	none	Design No cluster decision reads a metric. To figure out: stop the stores during an AWS chaos run and confirm.
Sampling interval	15 s	Design The agent’s default.
Delay from an event to the metrics store	≤ 75 s	Design The Autopilot is built to tolerate 30–75 s. To figure out: measure it end to end on AWS.
Metrics retention	30 days	Design The store’s setting. Log retention to be confirmed.
Agent’s share of a database machine	≤ 256 MiB	Estimate A memory ceiling for its container; the real use, and its CPU, are not yet measured. Measuring and capping our agents’ footprint is a roadmap item.
Time to tell a person that a cluster is degraded	none yet	Estimate Not built. To figure out: an alert from outside the management server, and a target for how soon.