Skip to content

Handle host disk pressure

Use this runbook when the host cache or the workspace filesystem runs low on free space or free inodes. The goal is to relieve pressure before it blocks new mounts or risks uncheckpointed writes.

awcd drives one host through six pressure states, in this order (spec §17.8):

NORMAL → CACHE_EVICTION → PREFETCH_DISABLED → NEW_MOUNTS_DISABLED
       → UPPER_WRITES_AT_RISK → EMERGENCY

The state is a single integer gauge, awc_host_disk_pressure_state (spec §19, §17.8):

Value State What the host does
0 NORMAL Nothing.
1 CACHE_EVICTION Evicts disposable cache objects.
2 PREFETCH_DISABLED Stops speculative prefetch. Demand reads continue.
3 NEW_MOUNTS_DISABLED Refuses new writable mounts.
4 UPPER_WRITES_AT_RISK Existing upper writes may fail with ENOSPC.
5 EMERGENCY Only warm reads continue.

Containment is strictly ordered. The cheap, safe lever fires first: the host MUST evict disposable cache before it touches workspace uppers (spec §17.8). The state machine computes and reports all six states; not every action is wired in the current build. See Current scope for what fires today.

Detect

Watch for any of these signals:

  • The awc_host_disk_pressure_state gauge rises above 0. Scrape it from GET /metrics on the health HTTP port.
  • GET /readyz returns 503 while GET /healthz returns 200. This means the host is warm_read_ready: existing reads continue, but new writable mounts are not admitted.
  • The /healthz body shows the disk_pressure component as Degraded or Down. Its detail line reads state=<STATE>; uncheckpointed_upper_bytes=<N>.
  • A mount request fails with ResourceExhausted and the text disk pressure blocks new mounts.
  • An upper write returns ENOSPC (state UPPER_WRITES_AT_RISK or worse).
  • Free inodes drop to the inode threshold. The default is 1024 free inodes. At or below it, the host escalates to at least NEW_MOUNTS_DISABLED, even if free bytes are plentiful. The escalation shows in awc_host_disk_pressure_state; check the raw inode count with df -i on cache_root.
  • The event log (stderr) prints disk pressure elevated at UPPER_WRITES_AT_RISK or EMERGENCY.

Health HTTP is opt-in

awcd does not serve /healthz, /readyz, or /metrics unless you set health_http_listen (for example 127.0.0.1:9100). Without it, you have no HTTP signal for pressure. Configure it before you need it (spec §19.6).

The byte thresholds derive from one operator reserve, cache.min_free_bytes (spec §22.9). With the default 50 GiB reserve, the ladder is:

State Triggers when free bytes fall to
CACHE_EVICTION 150 GiB (3 × reserve)
PREFETCH_DISABLED 100 GiB (2 × reserve)
NEW_MOUNTS_DISABLED 50 GiB (== reserve)
UPPER_WRITES_AT_RISK ~33 GiB (2/3 × reserve)
EMERGENCY ~16 GiB (1/3 × reserve)

Pressure stall information (PSI) can only escalate the state, never relax it. High I/O stall pushes the host up the ladder even when free bytes remain.

Procedure

  1. Read the current state. Fetch awc_host_disk_pressure_state from /metrics, or read the disk_pressure component in /healthz. Decide from the table above.
  2. Separate cache bytes from upper bytes. Cache objects live under cache_root (default /var/lib/awc/cache). Uncheckpointed uppers live in per-workspace overlay upper directories. The cache is disposable; the uppers are not.
  3. Do not rely on the cache reclaiming itself. The segmented-LRU policy is specified to evict on every admission to stay under cache.max_bytes and to preserve cache.min_free_bytes (spec §10.9), but in the current build eviction is not wired into the live cache path. The cache does not self-reclaim under pressure, and lowering cache.max_bytes will not free space until it is wired. Use the levers that work now: free upper bytes (step 5) or add capacity (step 7).
  4. Stop scheduling new writable mounts to the host. At NEW_MOUNTS_DISABLED the daemon already refuses them with a visible disk-full error. Drain your scheduler regardless, so it does not retry onto a host that cannot admit.
  5. Checkpoint or unmount healthy workspaces. Run awcctl workspace checkpoint to flush upper deltas to the object store. Then run awcctl workspace unmount with a disposition. This frees upper bytes on the host. See Checkpoint a workspace and Unmount a workspace.
  6. Do not delete active uppers. Never remove a workspace upper directory by hand to relieve pressure. The daemon MUST NOT silently delete uncheckpointed data (spec §17.8, §18.5). An upper may be removed only after a successful unmount disposition and the configured recovery grace.
  7. Add capacity or drain the host. Grow the filesystem that holds cache_root, or move workloads off the host. See Drain and restart a host.
  8. Verify the reserve and return to normal. Confirm the steps in Verify below.

Prefetch is specified to disable itself

You do not disable prefetch by hand. At PREFETCH_DISABLED the state machine flags prefetch off, and interactive demand reads keep priority over prefetch by design (spec §10.10). In the current build the prefetch path does not yet consult the pressure monitor, so prefetch is not actually stopped at this state. See Current scope below.

Verify

Confirm recovery when all of these hold:

  • awc_host_disk_pressure_state returns to 0.
  • GET /readyz returns 200.
  • The /healthz body shows disk_pressure as Ok.
  • A new writable mount is admitted without a disk-full refusal.
  • Free bytes are above the cache.min_free_bytes reserve, and free inodes are above the inode threshold.

Impact

State the data risk plainly:

  • Cache eviction is safe. Cached objects are disposable copies of immutable content-addressed objects (spec §8.5, §17.1). Eviction loses no committed data. A cold miss refetches the object from the object store.
  • Uncheckpointed uppers are ephemeral. Writes that have not been checkpointed are not durable (spec §6.8, §17.1). At UPPER_WRITES_AT_RISK or worse, existing upper writes may fail with ENOSPC.
  • Host loss destroys uncheckpointed uppers. If the host is lost while pressure is high, only durable artifacts survive: the latest acknowledged checkpoint and imported snapshots (spec §17.4). If pressure threatens the host, checkpoint first. See Recover from host loss.
  • The daemon never frees an upper to relieve pressure. It reports the disk-full state and refuses new mounts instead (spec §17.8).

Current scope

The build computes and reports the full pressure state machine and wires the new-mount admission gate, but two containment actions and several operator surfaces lag the spec. Be aware of these gaps:

  • Health HTTP defaults to off. Set health_http_listen to expose /metrics, /healthz, and /readyz. Without it, only the disk pressure blocks new mounts mount error and the stderr event surface the condition.
  • The pressure gauges are populated every 5 s. With health_http_listen set, /metrics reports awc_host_disk_pressure_state, awc_host_free_bytes, awc_host_free_inodes, and awc_host_upper_usage_bytes. awc_host_cache_bytes follows every 30 s and needs the fuse feature. Without the listener, fall back to df and df -i on cache_root.
  • Per-workspace upper usage is not yet wired. The monitor can account per-workspace upper bytes (spec §17.8), but the mount path does not feed it yet. The uncheckpointed_upper_bytes detail reads 0 until it is connected.
  • Cache eviction is not wired into the live path. The segmented-LRU EvictionPolicy is specified (spec §10.9) and unit-tested, but its enforce step has no production caller; the cache store defers admission and eviction to task T024. The CACHE_EVICTION state is reported, but no cache objects are reclaimed automatically. There is also no pressure-triggered eviction loop and no awcctl command to force cache eviction. To reclaim space now, free upper bytes by checkpointing and unmounting workspaces, or add capacity. Durable object-store garbage collection is a separate job; see Run garbage collection.
  • Prefetch is not gated by pressure. The PREFETCH_DISABLED state is reported, but the prefetch path does not yet consult the pressure monitor, so speculative prefetch continues at this state. Spec §10.10 is the target.
  • Some health components report unknown. Control-plane and object-store connectivity probes are not yet wired, so they cannot confirm or deny an outage alongside pressure.