Skip to content

Recover from a control-plane outage

Use this runbook when hosts cannot reach the control plane. The control plane is the awc-control API plus its PostgreSQL database. The goal is to keep hosts serving mounted workspaces while you restore the plane, then renew leases and confirm no workspace was fenced by mistake.

The control plane and the data plane are separate. Existing warm reads and local writes on a host do not require the control plane (spec §17.5). What fails is coordination: lease renewal, checkpoint publication, new mount tickets, and workspace fork or delete. See Architecture for the split.

Detect

Watch for any of these signals:

  • The awc_lease_renewal_failures_total counter rises and keeps climbing. Scrape it from GET /metrics on each host's health HTTP port.
  • awc_api_errors_total rises across many workspaces at once.
  • Lifecycle RPCs fail. awcctl workspace fork, awcctl workspace mount, and awcctl workspace checkpoint return errors, and a workspace cannot be forked or deleted (spec §17.5). A sustained run of retryable statuses (408, 425, 429, and 5xx other than 501 and 505) against many workspaces signals a plane-wide outage, not one bad call.
  • A new mount ticket does not issue. The mount path cannot authorize without the control plane (spec §17.5).
  • awc-control or PostgreSQL is down on the control-plane host. Confirm both, because the API cannot serve without the database (spec §17.6).

Health HTTP does not detect this outage today

The control_plane health component reports Unknown with the detail connectivity probe not wired. It does not flip to Down, so /healthz stays 200 and /readyz does not change. Do not rely on readiness probes to detect a control-plane outage. Use the lease-renewal counter and the lifecycle RPC failures instead.

A host often looks healthy while the plane is down. awcctl status (the host-local GetHostStatus RPC) reports the host's own mount state, not the control plane. A healthy host status plus failing lifecycle RPCs is the signature of this outage.

Procedure

  1. Confirm the control plane is the failed dependency. Check awc-control and PostgreSQL together. Correlate the failure across more than one host, so you do not chase a single host's network.
  2. Stop scheduling new lifecycle operations. New forks, mounts, deletes, and checkpoint publications MUST fail or queue with explicit errors, not retry onto a dead plane (spec §25.4). Pause your scheduler so it does not pile up retries.
  3. Let hosts keep serving mounted workspaces. Existing warm reads and local writes continue without the control plane (spec §17.5). Do not restart or drain hosts only because the plane is down.
  4. Watch the lease grace windows. awcd MAY keep a writable mount through the disconnected grace, default leases.disconnected_grace_seconds: 60. After grace it MUST deny checkpoint publication and SHOULD remount the workspace read-only or pause the sandbox per policy (spec §17.5, §22.9).
  5. Hold checkpoint publication. Do not force a publish while the plane is down. A checkpoint cannot be published without the authoritative lease (spec §17.5). Uncheckpointed writes stay local and ephemeral.
  6. Restore or fail over the control plane. Bring back PostgreSQL first, then the API. If PostgreSQL failed, follow Recover from a PostgreSQL failover before you trust the API.
  7. Confirm hosts re-register and renew leases. The lease-renewal task refreshes on its interval (default leases.renew_interval_seconds: 30) and moves each session out of the ControlPlaneUnknown class (spec §11.12, §22.9).
  8. Drain the idempotent retry backlog. Lifecycle and publication retries with a stable idempotency key return the committed result (spec §17.2). Let them run; do not reissue with a new key.
  9. Confirm no workspace was fenced by mistake. Compare the acknowledged workspace head in PostgreSQL against each host session. A session fenced for an expired or stale lease is correct fencing, not a false fence (spec §17.5).

Verify

Confirm recovery when all of these hold:

  • awc_lease_renewal_failures_total stops climbing. Its rate returns to zero over a few renewal intervals.
  • awc_active_writer_leases reflects the expected writer count.
  • A new mount ticket issues, and a new fork succeeds.
  • A checkpoint publishes without a lease or publication error.
  • The reconcile report holds no ControlPlaneUnknown sessions. Each session is either ActiveLeaseValid (renewed) or correctly fenced as StaleGeneration.
  • The acknowledged workspace head version in PostgreSQL matches the host sessions. No head diverged.

Impact

State the data risk plainly:

  • Reads and local writes on mounted workspaces are safe during the outage. They do not touch the control plane (spec §17.5). The outage itself loses no data.
  • Uncheckpointed writes are ephemeral. Writes that have not been published are not durable (spec §17.1, §6.8). An outage blocks publication. If a host is then lost, those writes are lost. See Recover from host loss.
  • Publication is blocked, not lost. A checkpoint held by the outage can be retried with the same idempotency key once the plane returns (spec §17.9). Do not reissue it with a new key.
  • Fencing is deliberate. A host whose lease expired or whose generation is stale is fenced and cannot publish (spec §17.2, §17.5). This prevents stale hosts from diverging the head; it is the safe behavior.
  • The system never guesses a head from the object store. PostgreSQL alone determines the acknowledged workspace head (spec §17.6). Unreferenced objects are left for garbage collection.

Current scope

The build implements the reconcile and lease-renewal paths, but some operator surfaces lag the spec. Be aware of these gaps:

  • The control_plane health component is not wired. It reports Unknown with the detail connectivity probe not wired, so /healthz and /readyz cannot confirm or deny an outage. Detect the outage from awc_lease_renewal_failures_total and lifecycle RPC failures instead. The object_store component is Unknown for the same reason.
  • Lease provisioning from the mount path is partial. The renewal task and lease cache run, but a session whose record carries no lease id is never renewed. Reconciliation preserves it as ControlPlaneUnknown until a lease is provisioned, which is safe but means the renewal step may be a no-op for those sessions.
  • Health HTTP defaults to off. Set health_http_listen to expose /metrics, /healthz, and /readyz. Without it, the lease counter is not reachable over HTTP.
  • Per-host head comparison is manual. There is no single command that diffs the acknowledged head against every host session. Run the comparison from the database and from awcctl status on each host.