Skip to content

Recover from a PostgreSQL failover

Use this runbook when the control-plane database has failed and you must fail over or restore it. The control-plane database is PostgreSQL. It holds every mutable record: workspaces, leases, mount tickets, snapshots, audit events, and idempotency keys. The goal is to bring a new primary online, point awc-control at it, and confirm that no acknowledged snapshot is lost.

PostgreSQL alone determines the acknowledged workspace head. The system MUST never reconstruct a head by guessing from object-store contents. Object-store snapshots are immutable candidates. PostgreSQL picks which one is current.

AWC does not ship a failover manager or a backup tool. PostgreSQL MUST be deployed with automated failover and point-in-time recovery (PITR). You provision that layer. AWC only consumes the primary through DATABASE_URL.

Detect

Watch for any of these signals:

  • awc-control fails to start, or exits after it logs a PostgreSQL connection error. It reads DATABASE_URL at startup and refuses to run without it.
  • Lease renewals fail. awcd renews each leased mount session against the control plane in the background. A dead primary makes those renewals fail. Each renewal pass adds its failed sessions to awc_lease_renewal_failures_total on the awcd /metrics endpoint (needs health_http_listen). Failures also surface as RPC errors on the next lease-protected operation, and as fenced sessions on the next reconcile.
  • awc_api_errors_total rises on the worker. The worker emits this counter (label component=worker), so a broad rise means background jobs are failing against the dead primary.
  • Lifecycle RPCs fail across the board. awcctl workspace fork, awcctl workspace mount, and awcctl workspace checkpoint return errors. A sustained run of the awcctl retryable set (408, 425, 429, and 5xx other than 501 and 505) against many workspaces is a plane-wide signal.
  • No new publication, lease acquisition, or authoritative lifecycle mutation succeeds. This is the database row of the failure matrix.
  • PostgreSQL itself reports the primary as down or read-only, a replica is being promoted, or you are replaying a write-ahead log (WAL) archive.

Health HTTP will not detect this outage

The control_plane and object_store health components report Unknown with the detail connectivity probe not wired. They do not flip to Down, so /healthz stays 200 and /readyz does not change. Detect this outage from the awc-control startup error and the awc_api_errors_total rise on the worker, not from readiness probes.

Warm reads and local writes on a mounted workspace do not touch the control plane. The outage is invisible to a running workspace until a lease needs renewal or a checkpoint must publish.

Procedure

  1. Confirm PostgreSQL is the failed dependency. Correlate the signals across more than one host so you do not chase a single host's network. Check the Postgres primary directly: it is down, read-only, or unreachable. Decide between replica promotion and backup restore.
  2. Promote the healthy replica, or restore from a point-in-time backup. Run this in your Postgres layer, with streaming replication plus a WAL archive or a managed equivalent. Do not run AWC against a primary that is still promoting or still replaying.
  3. Keep mutations disabled until the new primary is authoritative. Stop your scheduler so it does not pile new forks, mounts, and checkpoints onto a primary that is not yet consistent.
  4. Point awc-control at the new primary. Set DATABASE_URL to the new primary's connection string, then restart awc-control. It connects through a fixed 16-connection pool and refuses to start when DATABASE_URL is unset.
  5. Confirm migrations applied. On startup awc-control runs every pending migration before it accepts traffic. awc-worker runs the same migrations on its own startup, so restarting either binary against the new primary applies pending migrations. The migrations are forward-only and additive, so an older binary reading a newer schema still works during a rolling deploy. Check the sqlx migrations ledger for failures. Never revert a migration; a faulty migration is a forward corrective migration.
  6. Verify snapshots and workspaces are intact. Confirm every workspaces.head_snapshot_id resolves to a snapshots row. Confirm that workspace version and lease constraints held through the failover. A checkpoint publishes only through a locked compare-and-set on the workspace row, so a half-applied publication cannot advance the head.
  7. Run metadata and object reconciliation. Restart each awcd host so its restart reconcile classifies sessions against the restored control plane. Then compare the five records the model requires: published snapshot records, workspace versions, audit events, object-store snapshot manifests, and host-reported mount sessions.
  8. Compare audit sequence and idempotency records, and fence stale sessions. Confirm audit_events are contiguous and idempotency_keys match the publications you expect. A replay with the same key MUST return the committed result; a different result signals divergence. Fence any session created after the restore point unless you can prove it published. awcd preserves a session whose lease is missing as ControlPlaneUnknown rather than deleting it.
  9. Resume mutations in controlled stages. Re-enable the scheduler one tenant or host at a time. Do not infer missing publications from orphan snapshot objects. PostgreSQL alone sets the acknowledged head, and unreferenced immutable objects are left for garbage collection (GC).

Idempotent retries need no new key

Lifecycle and publication retries that carry a stable idempotency key return the committed result. Let them run after the failover. Do not reissue them with a new key, or you may create a duplicate.

Verify

Confirm recovery when all of these hold:

  • awc-control logs show both listeners up: REST on AWC_CONTROL_LISTEN and gRPC on AWC_CONTROL_GRPC_LISTEN. The startup migration reported no error.
  • The sqlx migrations ledger shows the full set applied, with no new failures, and it matches your other environments.
  • Lease-protected RPCs succeed. A fork, a mount, and a checkpoint publish all return without error, which means renewals reach the primary again.
  • awc_api_errors_total returns to baseline.
  • A new mount ticket issues. A fork and a checkpoint publish both succeed.
  • Each host reconcile report shows sessions renewed as ActiveLeaseValid or correctly fenced as StaleGeneration. The preserved count (ControlPlaneUnknown plus PreservedDelta) is what you expect.
  • Database and object-store inventories agree. Every snapshots.manifest_digest is present in the object store. No head_snapshot_id is dangling. An idempotency replay returns the committed result.

Impact

State the data risk plainly:

  • A clean replica failover loses no committed data. Streaming replication carried every committed mutation to the replica before promotion.
  • A point-in-time restore loses every database mutation after the restore point. That includes workspace head advances, leases, mount tickets, and audit events. Those rows are gone. AWC cannot rebuild them.
  • The system never reconstructs a head from the object store. Orphan snapshot objects after a restore are immutable candidates, not inferred publications. GC handles them.
  • Host writes are untouched by a database failover. Warm reads and local writes continue without the control plane. If a host is also lost, only durable artifacts survive: the latest acknowledged checkpoint and imported snapshots. See Recover from host loss.
  • Stale sessions are fenced on purpose. A session that predates the restore point and lost its lease is fenced and cannot publish. awcd preserves its uncheckpointed upper as ControlPlaneUnknown for offline export, then GC collects it. This is the safe behavior.

Current scope

The build consumes PostgreSQL but does not manage it. Some operator surfaces lag the spec. Be aware of these gaps:

  • AWC ships no failover manager or PITR tooling. Provision automated failover and point-in-time recovery in your Postgres deployment, for example with Patroni or a managed Postgres with a WAL archive. AWC consumes the primary through DATABASE_URL only. For the backup procedure, see Back up and restore PostgreSQL.
  • No database-specific metric exists. There is no database, migration, or reconcile gauge, and the lease-renewal failure counter is defined but not yet emitted. Detect a database failure indirectly: awc-control fails to start, or awc_api_errors_total climbs on the worker. The control_plane and object_store health components report Unknown (connectivity probe not wired), so /healthz and /readyz do not reflect a database outage.
  • Migrations run automatically on startup. Both awc-control and awc-worker run pending migrations on startup. There is no separate migrate subcommand and no dry run. The set is forward-only and additive, enforced in CI. For the upgrade and rollback contract, see Upgrade a deployment.
  • Cross-check reconciliation is manual. awcd reconciles host sessions against the control plane on restart, but no single command diffs the database against the object store. Run the audit, idempotency, and manifest comparisons from SQL and an object-store listing.
  • The connection pool is fixed at 16. awc-control opens at most 16 PostgreSQL connections. Confirm the new primary's connection budget can absorb awc-control, awc-worker, and your job concurrency together.

When the database outage is part of a wider control-plane failure, also follow Recover from a control-plane outage. When the failover exposes a corrupt or missing object, follow Handle object-store corruption. When the object store itself is the failed dependency, follow Recover from an object-store outage.