Skip to content

Recover from host loss

Use this runbook when a host disappears while it holds active writers. The goal is to fence the lost host, recover each workspace from its last acknowledged checkpoint, and return capacity to the pool.

Host loss is permanent loss of that host's local state. The local cache, the active mount state, the uncheckpointed upper deltas, and any preserved local recovery deltas on that host are gone. What survives is the immutable truth: imported snapshots, acknowledged checkpoints, workspace metadata, audit events, and retained patch exports. See The durability model.

Leases and fencing prevent split-brain. Every writer lease carries a monotonic generation. A lost host keeps an old generation. Even if its process resumes, it cannot publish. See Fencing and split-brain.

Detect

Watch for one or more of these signals:

  • The host does not answer awcctl status. The GetHostStatus host RPC fails, times out, or refuses the connection. The host's SSH session and health probe also fail.
  • Renewals for the host's writer leases stop. The control plane still shows live leases with host_id equal to the lost host, but their expires_at passes with no renewal.
  • awc_active_writer_leases keeps counting the lost host's leases as live. They stay live until a new acquire fences them (see Current scope).
  • Lifecycle RPCs that target the host fail. A new awcctl workspace mount on that host returns an error, because the host cannot serve it.

A short partition is not host loss

Confirm the host is gone, not briefly unreachable. A host that recovers on its own needs Handle a stale writer, not this runbook. Wait long enough to rule out a transient network failure.

Procedure

  1. Mark the lost host unschedulable. Stop your scheduler from sending new mounts to it. This prevents new work from targeting a host that cannot serve it.

  2. Confirm the host is lost. Probe awcctl status, the health endpoint, and SSH. Correlate the failure with the lost host's stalled leases. If the host might return, follow Handle a stale writer instead.

  3. List the affected workspaces. Query the control-plane database for live leases whose host_id equals the lost host. Record each workspace_id, the acknowledged head, and the lease generation.

  4. Wait out the lease TTL (time to live). Renewals from the lost host have stopped. After expiry_seconds (default 90), each lease is past-due and a new acquire can fence it. Do not delete lease rows by hand.

  5. Confirm the acknowledged checkpoint survived. The workspace head in PostgreSQL points at a published snapshot. Verify that snapshot's objects are readable in the object store. Only an acknowledged checkpoint is guaranteed to survive host loss.

  6. Reassign each workspace to a healthy host. Mount the same workspace there, or fork a new workspace from the acknowledged head.

    awcctl workspace mount ws_01J... --target /workspace
awcctl workspace fork --workspace ws_01J... --from head

    The new lease acquire fences the lost host's past-due lease to EXPIRED and issues the next generation. This is the fence.

  7. Treat uncheckpointed writes on the lost host as lost. They were never remotely durable. Do not try to make them the head. If the host's disk returns later, you MAY export its upper as an offline patch bundle that never advances the head automatically.

  8. Retire the lost host's identity. Once you are sure it will not return, retire it so its mTLS (mutual TLS) identity is no longer honored. A resurrected host then cannot reauthenticate as a trusted writer.

Verify

Confirm recovery when all of these hold:

  • Each affected workspace has one active writer lease on a healthy host. That lease holds a generation higher than the lost host's.
  • awc_active_writer_leases shows the expected writer count. No live lease remains on the lost host.
  • The workspace head and version in PostgreSQL match the last acknowledged checkpoint. They are unchanged unless a new checkpoint was published on the replacement host.
  • A publish attempt with the lost host's old generation fails with StaleGeneration. No double-write was possible, because only one generation can publish.
  • The lost host is UNSCHEDULABLE or RETIRED in the host table.

Impact

Uncheckpointed writes on the lost host are lost

Writes that were not checkpointed before the host disappeared are gone. They lived only in the host-local upper directory. They were never remotely durable, and they cannot become the workspace head. The authoritative head is the last acknowledged checkpoint.

Acknowledged checkpoints and metadata survive

Imported snapshots, acknowledged checkpoints, workspace metadata, audit events, and retained patch exports live in the object store and PostgreSQL. Host loss does not touch them. The workspace recovers from its last published head.

No split-brain

The generation fence means the lost host could never have published alongside the new writer. Even if the old process resumes, its stale generation is rejected at publication. There is no window in which two writers advance the same head.

Current scope

The build implements the lease-acquire fence and the publication predicate, but some operator surfaces lag. Be aware of these gaps:

  • Leases are fenced lazily, not by a background sweeper. A lease moves to EXPIRED only when the next writer lease for that workspace is acquired. Waiting out the TTL makes the lease past-due. Step 6 of the procedure performs the fence. There is no timer that auto-expires a lease on its own.
  • There is no awcctl host drain command. Marking a host UNSCHEDULABLE or RETIRED is a control-plane database or API operation.
  • Finding workspaces by host is manual. There is no single command that lists the active leases for a host. Query the workspace_leases table for rows with the lost host_id and a live state.
  • Lease provisioning from the CLI mount path is partial. A session created by awcctl workspace mount may not yet carry a lease_id. The fence path applies only to sessions the mount path issued a lease for. See Handle a stale writer.
  • The control_plane health component is not wired. It reports Unknown, so readiness probes cannot confirm or deny host loss. Detect the loss from the unreachable host and its stalled leases instead. See Recover from a control-plane outage.

See also