Skip to content

Handle a stale writer

Fence a workspace whose writer lease expired or was superseded by a higher generation. The goal is one outcome: only the authoritative generation can publish, and the old writer cannot advance the workspace head (spec §14.4, §25.2).

A stale writer is a mount session whose lease no longer authorizes publication. Its local bytes may still exist on the host. Those bytes are not remotely durable and cannot become the canonical head through normal publication (spec §14.4, §6.8).

Detect

Watch for one or more of these signals:

  • The control plane reports the lease as EXPIRED or REVOKED. Only an ACTIVE or RELEASING lease may publish (spec §12.4).
  • A higher generation was issued elsewhere for the same workspace. Every writer lease carries a monotonically increasing fencing generation (spec §9.10).
  • A checkpoint publish is rejected. Publication succeeds only when lease.generation == supplied_generation, the lease is active, the mount session matches, and the expected workspace version and head match (spec §14.3). A mismatch fails without changing the head.
  • The metric awc_lease_renewal_failures_total rises on the awcd /metrics endpoint (needs health_http_listen). awc_active_writer_leases is declared but not emitted in this build.
  • The awcd health detail reads reconciled N session(s): renewed=... fenced=... deleted=... preserved=.... A non-zero fenced count means the daemon has classified a session as StaleGeneration.

Procedure

  1. Confirm the workspace and its head. Run status against the workspace id.

    awcctl workspace status ws_01J...

    Record the head, the version, and the mount session id of the suspect writer.

  2. Identify the authoritative generation. The authoritative writer is the live lease with the highest generation for the workspace. A host with an older generation MUST NOT publish a checkpoint (spec §9.10, §14.4).

  3. Confirm publication is already blocked for the old generation. The awcd reconciler classifies a live mount whose lease does not authorize as StaleGeneration, sets may_publish to false, and emits a Fence action (spec §11.12). This is the fence. The old writer cannot advance the head even if its process resumes.

  4. Run the fence pass. In the current build, the reconciler executes on the daemon restart path. Restart the daemon to reconcile all sessions against the authoritative leases.

    sudo systemctl restart awcd

    The fence executor force-closes the stale session record (MountSessionState::Closed). It never publishes.

  5. Pause the fenced writer if its process is still running. Stop the agent, or freeze the execution cgroup that the mount ticket sealed (spec §13.8). The daemon does not remount the kernel overlay read-only itself; you MUST stop new writes from the stale writer by hand.

  6. Preserve the stale upper for the recovery grace period. Do not delete the session directory. The daemon refuses to delete a session whose upper still holds uncommitted data (spec §17.3). The PreservedDelta class promotes to SafeToDelete only after the configured grace window (spec §11.11, §11.12).

  7. Choose a disposition with the workspace owner. Pick one.

    The uncheckpointed upper is not the workspace head. Once the owner confirms, let the grace window elapse and let GC reclaim the session.

    Recover the upper as an offline patch bundle (spec §17.11). This path MUST NOT advance the original head automatically. It records that the bytes came from unacknowledged local state and needs explicit authorization. Do not execute any workspace content from the bundle.

  8. Never merge the stale delta into the authoritative head automatically. A stale host MAY keep local bytes, but they can become canonical only through an explicit recovery path (spec §14.4, §17.11).

  9. Release host resources after disposition. Once the owner discards or exports the delta, the session moves to SafeToDelete and the daemon reclaims its upper directory.

Verify

Confirm the workspace now has exactly one active writer:

  • awcctl workspace status ws_01J... shows the head and version that the authoritative generation published.
  • Exactly one active writer lease exists, at the authoritative generation. awc_active_writer_leases is not emitted in this build, so confirm with the awcctl workspace status output.
  • The awcd health detail shows fenced=N for this pass and no new stale-publication rejections.
  • A retry of the stale writer's checkpoint fails with the §14.3 publication predicate. It does not change the head.

Impact

Uncheckpointed writes are not durable

The fenced writer's local upper is not remotely durable (spec §6.8). It will not become the workspace head through publication. It survives only in the host-local upper directory for the grace window. If the owner discards it, or the grace window elapses, those writes are lost. The authoritative head is whatever the higher-generation writer published.

Current build: leases are not yet issued for CLI mounts

The control-plane lease path is still being wired. Sessions created by awcctl workspace mount do not yet carry a lease_id, so the renewal task does not renew them and the reconciler preserves them rather than fencing. A live fence against a confirmed stale generation requires a session that the mount path issued a lease for. Until then, treat any CLI-mounted writer as preserved-delta, not fenced.

See also