Skip to content

Break it on purpose

You build confidence in a cache by breaking it. In this tutorial you inject four failures into a running AWC (Agent Workspace Cache) deployment and watch what survives. You crash the host daemon mid-work, prove the durability contract, run the repo outage harness, and provoke the lease fence.

AWC keeps mutable coordination separate from immutable truth (spec §17.1). The five user-level primitives are unchanged (spec §5.1):

snapshot → fork → mount → checkpoint → diff/export

This tutorial adds failure on top of them. You do not change the surface.

Before you start

You need a host where the three AWC services run as systemd units:

  • awc-control, the control plane that issues mount tickets and publishes heads.
  • awcd, the host daemon that serves the FUSE (Filesystem in Userspace) mount and the local cache.
  • awc-worker, the background validator.

This is the Proxmox pilot or a VM provisioned by deploy on Proxmox or deploy a VM with systemd. You also need root privileges, because the failure commands stop services and drop the page cache.

Local-dev smoke is not enough

The repo outage harness reads /etc/awc/awcd.toml and drives the awc-control and awcd systemd units. The local-dev smoke runs the binaries as foreground processes and has no such units. You can do the first experiments on local-dev, but the outage harness needs a systemd deployment.

Set up your shell the way Your first workspace shows. Export AWC_CONTROL_ENDPOINT, AWC_HOST_SOCKET, AWC_TOKEN, AWC_TENANT, and AWC_PROJECT. Confirm the daemon answers:

awcctl status

1. Set up a workspace with an uncommitted change

Pick a READY snapshot and fork a private workspace from it. Forking is metadata only; it MUST NOT copy content objects (spec §5.3, §6.10):

SNAP=$(awcctl --output json snapshot list \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["snapshots"][0]["id"])')

awcctl --output json workspace fork --snapshot "$SNAP" --ttl 4h --name break-ws \
  > fork.json
WS=$(python3 -c 'import json;print(json.load(open("fork.json"))["id"])')
echo "$WS"

Mount the workspace. The control plane issues a signed ticket; awcd redeems it, takes an exclusive writer lease, and mounts an OverlayFS view at your target (spec §5.4, §13.8, §13.9):

MNT=/var/lib/awc/binds/break
mkdir -p "$MNT"
awcctl --output json workspace mount "$WS" --target "$MNT" > mount.json
SESS=$(python3 -c 'import json;print(json.load(open("mount.json"))["mount_session_id"])')
echo "$SESS"

Write a file you will keep, and a file you will not commit:

printf 'checkpoint me\n' > "$MNT/kept.txt"
printf 'forget me\n'   > "$MNT/lost.txt"
ls "$MNT"

Neither file is durable yet. Ordinary local writes are not remotely durable; only an acknowledged checkpoint is guaranteed to survive host loss (spec §6.8).

2. Crash the host daemon, then recover

A crash of awcd is not catastrophic. The OverlayFS upper directory lives on local disk, separate from the daemon process. Kill the daemon hard, the way a power loss or an out-of-memory killer would:

sudo systemctl kill   --signal=KILL awcd
sudo systemctl is-active awcd          # prints: failed

While awcd is down, the kernel mount may return errors for new filesystem operations. That is expected. The on-disk upper bytes are intact. Bring the daemon back:

sudo systemctl start awcd
until [ "$(sudo systemctl is-active awcd)" = active ]; do sleep 1; done
awcctl status

On restart, awcd acquires a singleton host lock, reconciles local cache metadata with the files on disk, discovers the existing mounts, and renews or fences lease state (spec §17.3). The cache reconciliation step walks cache metadata rows, drops broken or incomplete cache entries, and removes orphan temp files left by a crash. It does not touch workspace upper directories, and awcd MUST NOT delete an upper directory while the control plane is unreachable (spec §10.5, §17.3). Your two files are still in the mount:

ls "$MNT"          # kept.txt and lost.txt are still here

What this proves

The local upper layer survives a daemon crash, and awcd reconciles the cache on restart without operator help (spec §11.12, §17.3).

What it does not prove

This proves nothing about remote durability. Both files live only on this host. The spec also requires awcd to classify recovered sessions on startup (spec §11.12). The cache reconciliation is wired; the broader session classification is spec-required and tracked in the recovery runbooks.

3. Prove the durability contract

Now separate the two files. Checkpoint kept.txt into an immutable snapshot. Pass the mount session id, not the workspace id (spec §5.5, §13.11):

awcctl --output json workspace checkpoint "$SESS" \
  --mode cooperative --message "keep me" > ckpt.json
HEAD=$(python3 -c 'import json;print(json.load(open("ckpt.json"))["snapshot_id"])')
echo "$HEAD"

The new head is durable only after the control plane commits publication in one atomic transaction (spec §11.6 Phase E, §13.12). Now lost.txt is still uncommitted. Discard the local upper on unmount:

awcctl workspace unmount "$SESS" --disposition discard-local

discard-local deletes uncheckpointed upper state. Destructive discard MUST be explicit when uncheckpointed changes exist (spec §11.11). Re-mount the workspace and look inside:

awcctl --output json workspace mount "$WS" --target "$MNT" > mount2.json
SESS=$(python3 -c 'import json;print(json.load(open("mount2.json"))["mount_session_id"])')
ls "$MNT"

kept.txt is present, because the checkpoint published it. lost.txt is gone, because no checkpoint held it.

What this proves

An acknowledged checkpoint survives; an uncommitted write does not. That is the whole durability model in two files (spec §6.8). A fsync on a writable mount applies to the local upper filesystem only and does not imply remote durability (spec §14.8).

Host loss is the destructive form of the same rule. If this host is destroyed, you lose the local cache, the active mount state, and every uncheckpointed upper delta. You do not lose imported snapshots, acknowledged checkpoints, workspace metadata, or audit events (spec §17.4). For the recovery procedure, see recover from host loss. For the reasoning behind explicit durability, read the durability model.

4. Run the outage harness

The repo ships a failure-injection harness at awc_outage_test.sh. It stops the control plane only and keeps awcd, PostgreSQL, and MinIO (an S3-compatible content-addressed store, or CAS) running. The property under test is: once a mount is established, ordinary reads and writes MUST continue during a temporary control-plane outage, while new mounts and publication correctly fail (spec §17.5).

The harness reads three lease timers from /etc/awc/awcd.toml. The defaults are (spec configuration block):

renew_interval_seconds: 30     # how often a healthy writer renews
expiry_seconds: 90             # lease lifetime with no renewal
disconnected_grace_seconds: 60 # how long awcd keeps serving a detached writer

Run it as root from the repo root:

sudo bash awc_outage_test.sh

The script forks a workspace, mounts it, starts a rotating reader, an append writer, and an fsync writer, then runs this sequence:

  1. Stop awc-control only. A monitor process, started before the stop, records that awcd, PostgreSQL, and MinIO stay up.
  2. Sustain the workload past disconnected_grace_seconds and toward expiry_seconds, dropping the host page cache so reads are not served from RAM.
  3. Probe a new mount and a checkpoint during the outage. Both need the control plane, so both are expected to fail or stall.
  4. Restart awc-control, re-sync the dev token, and run a post-recovery checkpoint.

Read the verdicts the script prints at the end. Against the defaults above, you see results of this shape:

[reads sustained during outage]         PASS
[writes sustained WITHIN grace (60s)]   PASS
[writes survive PAST grace boundary]    FENCED
[cold read during outage succeeded]     PASS
[new mount needs control plane]         EXPECTED (failed or stalled-timeout)
[checkpoint needs control plane]        EXPECTED (failed or stalled-timeout)
[post-recovery checkpoint succeeded]    PASS

Reads are not fenced, so a read-only remount keeps serving them. Writes inside the grace window keep going, because awcd MAY continue a writable mount during a bounded lease grace period (spec §17.5). Writes past grace are fenced, because after grace the daemon MUST deny checkpoint publication and SHOULD remount the workspace read-only or pause it (spec §17.5). The harness labels the fenced result as expected lease-safety design, not a regression.

What this proves

The data plane stays up while the control plane is down. Lease grace gives a writer a bounded window; after that window, publication is fenced.

What it does not prove

awcd stayed up the whole time. This is a control-plane outage, not host loss, so it says nothing about what survives a destroyed host. Section 3 is the host-loss answer.

For the operator recovery steps, see recover from a control-plane outage and drain and restart a host.

5. Watch the lease fence a stale writer

Split-brain prevention does not rely on lease expiry alone. Every writer lease carries a fencing generation, a number that increases each time a new lease is issued (spec §9.10). A host with an older generation may keep its local bytes, but it MUST NOT publish a checkpoint (spec §9.10, §14.4).

The publication path enforces this. The checkpoint publication step runs in one PostgreSQL transaction. Within that transaction it fences the active writer lease on generation, state, expiry, and mount session (spec §11.6 Phase E). If the generation is stale, the transaction rolls back and the workspace head is unchanged.

You already saw the fence act in the outage harness. When the control plane was down, the lease could not be renewed. Once the outage passed disconnected_grace_seconds, writes were fenced. That is the same rule: a writer whose lease is no longer authoritative cannot advance the head.

A true split-brain needs two hosts

To reproduce two writers racing for one workspace, you need the lease issued on one host and a second writer attempt elsewhere. A single-host tutorial cannot do that safely. The generation fence in the publication path is the guarantee that the second writer cannot win (spec §14.4). For the full procedure, read fencing and split-brain and handle a stale writer.

A stale host is not silently erased. Its local upper bytes may survive, and an operator MAY export them as an offline patch bundle. That export MUST never advance the workspace head automatically and MUST record that the bytes came from unacknowledged local state (spec §11.12, §14.4, §17.11).

6. Recover with the runbooks

Each failure class has a runbook. Reach for the one that matches the symptom:

Every failure path preserves the last acknowledged workspace head (spec §17.1). If a step in this tutorial left a workspace behind, unmount and remove it:

awcctl workspace unmount "$SESS" --disposition discard-local

What you learned

You broke the system four ways and watched it hold:

  • A daemon crash leaves the upper layer intact, and awcd reconciles on restart.
  • An acknowledged checkpoint survives; an uncommitted write does not (spec §6.8).
  • A control-plane outage keeps the data plane alive through lease grace, then fences publication (spec §17.5).
  • A stale fencing generation cannot publish, so a split writer cannot corrupt the head (spec §14.4).

The durability contract is explicit, not automatic. Checkpoint the work you want to keep.

Where to go next: