Skip to content

Recover from an interrupted checkpoint

Resume a checkpoint that crashed during publish. The goal is one outcome: the workspace head advances exactly once, or not at all.

Publication is one atomic database transaction. A crashed publish either committed the new head or left the prior head unchanged. You never repair a partial head by hand. Recovery is idempotent retry: re-run the checkpoint with the same idempotency key, and the control plane returns the committed result for that key instead of duplicating work.

Detect

Watch for one or more of these signals:

  • A checkpoint call returned an error, was cancelled, or its event stream dropped before the Result event.
  • awcctl workspace status ws_01J... shows the head and version unchanged from before the attempt. The head references the complete prior snapshot or the complete new snapshot, never both.
  • The daemon restart reconciliation line reads reconciled N session(s): renewed=... fenced=... deleted=... preserved=.... An interrupted checkpoint counts toward N but not toward renewed, fenced, deleted, or preserved. The session is class CheckpointInterrupted or CleanupIncomplete.
  • The metric awc_workspace_checkpoint_total shows no outcome="published" sample for the attempt (the interrupted stream, if any, counted outcome="failed"). Requires health_http_listen on the awcd host.
  • The file checkpoint.json exists in the session state directory with a phase other than COMPLETE.

Procedure

  1. Capture the workspace id, the mount session id, and the current head and version.

    awcctl workspace status ws_01J...

    Record head, version, and the session id of the attempt.

  2. Read the persisted recovery record. The record is checkpoint.json in the session state directory, <workspace_root>/<workspace-id>/<generation>/ (default workspace_root is /var/lib/awc/workspaces).

    sudo cat /var/lib/awc/workspaces/ws_01J.../3/checkpoint.json

    Read phase and idempotency_key. The phase tells you how far the attempt reached. Each phase maps to one recovery case:

    phase Reached Recovery action
    STARTED Identity recorded only Re-run construction, upload, then publish
    BUILT Candidate manifest persisted Re-run the closure upload, then publish
    UPLOADED Closure durable Re-issue publication only
    PUBLISHED Head already advanced Run post-publish cleanup only
    COMPLETE Fully done Nothing to recover

  3. Confirm the writer lease still authorizes publication. Only an ACTIVE lease at the current generation may publish. If the lease is EXPIRED, REVOKED, or fenced by a higher generation, stop here. Follow Handle a stale writer. A stale fencing token MUST NOT publish.

  4. Restart the daemon to run restart reconciliation. The pass finishes post-publish cleanup for a PUBLISHED attempt and preserves the session for every earlier phase.

    sudo systemctl restart awcd

    The daemon never deletes an upper directory while the checkpoint state is uncertain.

  5. Re-run the checkpoint with the same idempotency key. Pass the same session id and mode as the original attempt.

    awcctl workspace checkpoint sess_01J... \
  --mode cooperative \
  --idempotency-key cp-run991-3

    The re-run is safe for every phase. Immutable uploads are no-ops on objects already present, and the build is deterministic, so the re-run reproduces the same candidate.

  6. Capture the result snapshot id and version.

    snapshot      snap_01J...
digest        sha256:...
version       4

    If the original publish committed and only its response was lost, this re-run returns that same committed snapshot. The version does not advance a second time.

Verify

Confirm the head advanced exactly once:

  • awcctl workspace status ws_01J... shows head equal to the result snapshot and version exactly one higher than before the attempt.
  • The checkpoint.json record now reads phase: COMPLETE. The attempt retired.
  • awc_workspace_checkpoint_total incremented once with outcome="published", not twice. Requires health_http_listen on the awcd host; otherwise confirm via the head, version, and phase checks above.
  • Mount the workspace and read back the checkpointed state. The merged view equals original lower + accumulated upper.

Impact

Uncheckpointed writes are not durable

If the crash happened before publication and you do not re-run the checkpoint, the upper delta stays local to the host. It is not remotely durable. A later host loss destroys it. The acknowledged head is whatever the last successful publish committed.

Current build: the daemon does not auto-resume a crashed publish

Restart reconciliation classifies an interrupted checkpoint (CheckpointInterrupted, CleanupIncomplete) and finishes post-publish cleanup. It does not re-issue publication. You MUST re-run awcctl workspace checkpoint by hand. The library functions that decide and drive the resume are implemented and unit-tested, but the live daemon executor does not yet dispatch them.

Current build: CLI mounts have no writer lease

A checkpoint requires a writer lease. The call fails with session has no writer lease when the session has no lease_id. Sessions created by awcctl workspace mount do not yet carry one, so the recovery re-run needs a session that the mount path issued a lease for.

The checkpoint counter needs the health listener

awc_workspace_checkpoint_total counts every terminal checkpoint event with an outcome label (published, failed, cancelled). It renders only when awcd runs with health_http_listen set. Without the listener, detect and verify the recovery with the head and version values and the checkpoint.json phase instead.

Publication is atomic

You cannot advance the head twice. The control plane completes every mutable update in one PostgreSQL transaction and deduplicates a repeated publish on the idempotency key. A retry either commits the new snapshot now or returns the one already committed.

See also