Drain and restart a host¶

Take a host out of scheduling, let in-flight work settle, restart awcd, and return the host to service. This is the drain and restart half of the daemon upgrade runbook (spec §25.8). For the full upgrade, with binary and format compatibility checks and rollback, see Upgrade a deployment.

A clean drain and restart is not a crash. On a clean drain, awcd receives SIGTERM, finishes or cancels in-flight installs, persists mount-session metadata, and leaves mounted views in a defined state (spec §15.13). On restart it reconciles existing mounts rather than assuming they disappeared (spec §15.13, §17.3). A crash skips the drain. Restart recovery still runs, but writes that never reached the upper directory are gone. See Impact below.

Detect¶

Run this procedure when any of these is true:

You planned a restart. You are patching the kernel, changing awcd configuration, swapping the binary, or doing host maintenance.
You are rolling a deployment. Follow Upgrade a deployment first; this page is the per-host drain and restart step inside it.
The unit restarted on its own and you want to confirm whether the host drained cleanly or crashed.

Read the current state before you drain:

awcctl status

The HostStatus response carries the fields you need:

Field	Meaning	Drain target
`state`	`HostState`: `STARTING`, `HEALTHY`, `DEGRADED`, `STOPPING`, `STOPPED`	Starts at `HEALTHY` (or `DEGRADED`); ends at `HEALTHY`.
`base_mount_state`	Shared base filesystem state	`MOUNTED` before and after.
`active_mount_count`	Live mount sessions on the host	Falls to the post-drain count, then recovers.

Watch these metrics from GET /metrics on the health HTTP port while you drain:

awc_workspace_active_mounts falls as sessions unmount.
awc_active_writer_leases falls as writers release or finish.
awc_workspace_checkpoint_total rises if you flush uppers before drain.

Health HTTP is opt-in

awcd does not serve /healthz, /readyz, or /metrics unless you set health_http_listen (for example 127.0.0.1:9100). Without it, use awcctl status and journalctl -u awcd -f instead. Configure it before you need it (spec §19.6).

Distinguish a clean drain from a crash by the events that precede the restart:

Clean drain. The log shows the awcd shutdown sequence. HostState moves to STOPPING, then STOPPED. The unit goes inactive on its own.
Crash. The log stops mid-event with no shutdown sequence. systemd restarts the unit under Restart=on-failure (see systemd units). The next awcd started event follows no clean STOPPED.

Procedure¶

Mark the host unschedulable for new mounts. Stop routing new mount tickets to the host at your scheduler or load balancer. The build has no in-daemon unschedulable toggle; you MUST stop admission at the scheduler. Confirm new mounts have stopped by watching awc_workspace_active_mounts hold steady.
Wait for in-flight checkpoints to finish. A publication transaction that is open when you drain is recoverable (spec §17.9), but a clean finish is preferred. Check each writer:
```
awcctl workspace status ws_01J...
```
Let CHECKPOINTING workspaces return to MOUNTED. Do not cancel a checkpoint in progress unless you accept its recovery class.
Preserve mount-session metadata. Do not delete the workspace upper or work directories, the local cache index, or the durable session store. On restart awcd recovers sessions from this durable state (spec §11.12, §17.3). The daemon MUST NOT delete an upper directory merely because the control plane is temporarily unreachable (spec §17.3).
Drain awcd with a bounded stop. Send SIGTERM through systemd so the graceful shutdown path runs (spec §15.13):
```
sudo systemctl stop awcd
# or, with a bounded wait and a status report:
packaging/drain.sh awcd
```
On SIGTERM, awcd stops accepting new mounts and checkpoints, finishes or cancels in-flight object installs safely, persists local mount-session metadata, attempts to release leases, and avoids deleting uppers with uncheckpointed data (spec §15.13). The unit's TimeoutStopSec bounds the drain. If the unit is still active after the timeout, investigate before you force a kill. Do not SIGKILL a host that holds uncheckpointed uppers.

Restart awcd:

sudo systemctl start awcd
# or, if you only stopped it:
sudo systemctl restart awcd

Confirm restart reconciliation ran. On startup awcd acquires the singleton host lock, scans the durable session store, classifies each session against the authoritative lease, and dispatches recovery (spec §17.3, §11.12). Read the reconcile summary in the health detail, or in the log:
```
reconciled N session(s): renewed=A fenced=B deleted=C preserved=D
```
For a clean drain against a reachable control plane, expect preserved=0. A non-zero preserved count means the control plane was unreachable for those sessions; their uppers are kept, not deleted (spec §17.3). See Recover from a control-plane outage if preserved is large.
Renew valid leases and run read and write probes. The lease-renewal task refreshes on its interval (default leases.renew_interval_seconds: 30). Leases whose fencing generation is still current are renewed; stale generations are fenced (spec §17.3). Mount one workspace read-only and one writable, and confirm both serve:
```
awcctl workspace mount ws_01J... --target /workspace --read-only
awcctl workspace mount ws_01J... --target /workspace
```
Then unmount the probe sessions.
Return the host to scheduling. Re-enable the host in your scheduler or load balancer. Confirm new mount tickets route to it and that awcctl status reports HEALTHY with base_mount_state MOUNTED.

Verify¶

Confirm recovery when all of these hold:

awcctl status reports state HEALTHY and base_mount_state MOUNTED.
GET /healthz returns 200 and GET /readyz returns 200 (when health HTTP is enabled). /readyz reflects readiness to accept a new writable mount: base FUSE, cache, leases, kernel compat, no disk pressure, and control-plane and object-store connectivity (spec §19.6).
The reconcile summary reads renewed=... for the writer set you drained, with no unexpected fenced or preserved counts.
awc_active_writer_leases matches the writer count you expect on the host.
A read and a write probe both succeed on a freshly mounted workspace.

Impact¶

State the data risk plainly:

A clean drain preserves uncheckpointed uppers. The shutdown path persists mount-session metadata and does not delete uppers with uncheckpointed data (spec §15.13, §17.3). On restart the daemon reconciles those sessions against the authoritative lease.
A checkpoint interrupted by drain is recoverable. Local checkpoint state records phases and idempotency keys (spec §17.9). Retry publication with the same idempotency key. Draining cleanly is still preferred over interrupting a checkpoint.
A crash loses writes that never reached the upper. Restart recovery reconstructs state from durable records (spec §11.12). It does not recover writes that the agent process or the kernel had not yet flushed to the upper directory.
Host loss destroys uncheckpointed uppers. If the host is lost instead of cleanly restarted, only the latest acknowledged checkpoint survives (spec §17.4). See Recover from host loss.
Fencing is deliberate. A session whose lease expired or whose generation is stale is fenced and cannot publish (spec §17.2, §17.3). This preserves the acknowledged workspace head.

Current scope¶

The build implements graceful shutdown and restart reconciliation, but some operator surfaces lag the spec. Be aware of these gaps:

No in-daemon unschedulable toggle. Mark the host unschedulable at your scheduler or load balancer. /readyz reports readiness, but it does not stop a scheduler that ignores it. You MUST stop routing new mounts yourself.
Health HTTP defaults to off. Set health_http_listen to expose /metrics, /healthz, and /readyz. Without it, rely on awcctl status and journald.
Control-plane and object-store health components are not wired. Both report Unknown with the detail connectivity probe not wired. As a result /readyz can read 200 while the control plane is down. After a restart, do not trust /readyz alone to confirm the control plane is up. Check the lease counters and a probe publish instead. See Recover from a control-plane outage.
Lease provisioning from the mount path is partial. The renewal task and lease cache run, but a session whose record carries no lease id is never renewed. Reconciliation preserves it as ControlPlaneUnknown, which is safe but means step 7 may be a no-op for sessions created through awcctl until a lease is provisioned.
Restart reconciliation runs on the startup path. A live fence without a restart needs the lease-renewal task, which only renews sessions that carry a lease id.