Skip to content

Upgrade a deployment

This page is the operational upgrade and compatibility runbook for AWC. It states the compatibility guarantees the implementation must preserve across a rolling upgrade, the daemon upgrade procedure operators follow, the artifacts every release ships, and how each guarantee is verified in CI. It is written for an operator who is not on the core implementation team.

The testable guarantees below are turned into machine-checked assertions run in CI.

Scope

An "upgrade" is any change to the awcd, awc-worker, awc-control, or awcctl binaries deployed to a running fleet without erasing state — a rolling deploy where, for a time, mixed binary versions and mixed on-disk/object formats coexist. This document covers four compatibility axes:

  1. Manifest formats — immutable, versioned on-disk and object content formats.
  2. Control-plane database schema — PostgreSQL migrations.
  3. API and RPC protocols — external /v1 HTTP and internal gRPC, including host and control registration.
  4. Mount tickets — short-lived signed capabilities carried between control plane and host.

Compatibility guarantees

These guarantees are normative. Each is verified in CI (see "Verification" column); a release that breaks one must not ship.

# Guarantee What it means operationally Verification
G1 Manifest formats are versioned and immutable. Every serialized manifest carries a format version; a given (version, payload) always decodes to the same logical object. A format, once published, is never silently reinterpreted. Decoded against golden manifest vectors in CI.
G2 New daemons MUST read every non-retired prior format. After an upgrade, awcd, awc-worker, and the import and checkpoint paths still decode snapshots and trees written by every older, non-retired version. No on-disk rewrite is required to read them. Decodes vectors for every supported read version in CI.
G3 Writers create only the configured current format. The version written by import and checkpoint is a single configurable current write version (default = the newest supported version), never an arbitrary or stale one. Rolling forward is therefore monotone and intentional. Asserts a single configured current write version in CI.
G4 Control-plane schema migrations MUST be backward-compatible during rolling deploys. During the window where old and new binaries share one database, both must function. Migrations are additive only (no destructive DDL), so an old binary reading a newer schema continues to work. Scans the embedded migration set for destructive DDL and pins monotonic versions in CI.
G5 Mount tickets include protocol version and expiry. A ticket names the protocol version it was minted under and an absolute expiry, so a host can reject a ticket minted for an incompatible version or one that is stale after a long partition. Ticket versioning and expiry exercised in CI.
G6 A daemon upgrade MUST not require rewriting existing snapshots. Upgrading awcd never rewrites or re-hashes existing snapshots or CAS objects. Snapshots are immutable; the upgrade only changes code and (additively) schema. Asserts no snapshot is rewritten on upgrade in CI.

API and RPC protocol compatibility

  • External API is versioned by path (/v1). Additive fields are allowed within a version; removing a field or changing its semantics requires a new API version. Clients MUST tolerate unknown additive fields.
  • Internal gRPC (host ↔ control plane over mTLS) reserves removed protobuf field numbers so an old reader ignores fields a newer writer adds.
  • Protocol-range negotiation: during host registration, the host and the control plane exchange the ranges of protocol versions each supports and select the newest mutually-supported version. A pair with empty overlap fails registration with an explicit error rather than proceeding with mismatched assumptions.

The negotiation is pure logic: a set intersection over [min, max] ranges. The helper is unit-tested in isolation so the contract is fixed before the live registration RPC is wired.

Daemon upgrade runbook

To upgrade awcd on a host without losing in-flight work:

  1. Mark the host unschedulable for new mounts — stop admitting new privileged mount operations. Existing mounts keep serving reads.
  2. Wait for in-flight checkpoints to finish. Do not proceed while a publication transaction is open; an interrupted checkpoint is recoverable, but draining it cleanly is preferred.
  3. Preserve mount-session metadata. Do not delete upper or work directories or the local cache index on upgrade. They are reconciled on restart. Never delete an upper merely because the control plane is unreachable.
  4. Verify binary and format compatibility. Confirm the new binary supports every non-retired manifest version present in the cache and object store.
  5. Restart awcd (for example, systemctl restart awcd).
  6. Reconcile shared FUSE and overlay mounts. On restart awcd re-walks mount-session metadata and re-establishes overlays that were live before the restart, recovering all host-local crash classes.
  7. Renew valid leases. Leases whose fencing generation is still current are renewed; stale generations are fenced off and routed to offline export so the acknowledged head is preserved.
  8. Run read and write probes (health and readiness, plus a sample mount).
  9. Return the host to scheduling.
  10. Roll back if compatibility or latency checks fail.

Rollback

Rollback is the reverse procedure against the previous binary and the previous configuration, not a schema rollback: because every migration is additive (G4), the newer schema remains readable by the older binary, so rolling back the code is safe without touching the database. Operators:

  1. Mark the host unschedulable (step 1 above).
  2. Reinstall the previous binary (the versioned layout under /opt/awc keeps prior versions; rollback.sh repoints /opt/awc/current).
  3. Restart awcd and re-run the probes (steps 5–8).
  4. Return the host to scheduling.

Never roll back by reverting a database migration. If a migration is genuinely faulty, that is a forward fix (a corrective additive migration), not a revert.

Release artifacts

Every release MUST produce the artifacts below. package.sh puts the ones marked tarball inside share/awc/ of the release tarball, so SHA256SUMS covers them (see Build and package):

  • signed binaries or images (sign.sh, package.sh);
  • software bill of materials (tarball, sbom.spdx.json via syft);
  • database migration bundle (the additive, forward-only migration set) (tarball, migrations/);
  • manifest-format compatibility report (G1–G3, verified against golden vectors in CI, not a standalone file);
  • supported kernel matrix (tarball, docs/kernel-compatibility.md);
  • upgrade and rollback notes (this document; tarball copy at docs/upgrade.md);
  • security scan results;
  • performance regression report;
  • golden manifest vectors (tarball, golden-vectors/golden_vectors.json).

Current status (honest)

Axis Status
Manifest format versioning (G1–G3) Implemented: every manifest carries a version. The decoder accepts every supported read version (minus retired versions) and rejects all others. Writers stamp the current write version (1 today). Verified in CI.
Database migrations (G4) Implemented: embedded migrations are additive and enforced forward-only in CI.
Mount-ticket versioning (G5) Implemented: mount tickets carry a protocol version and an absolute expiry, and are Ed25519-signed. The daemon rejects an incompatible protocol version. Verified in CI.
Snapshot immutability on upgrade (G6) Guaranteed by construction: snapshots and CAS objects are immutable and are never rewritten by an upgrade. Verified in CI.
Protocol-range negotiation Implemented: host registration exchanges and negotiates supported protocol-version ranges, rejecting disjoint ranges. Verified in CI.
Live host drain / reconcile (runbook) awcd reconciles host-local state and renews leases on restart. The operator-facing drain toggle and rollback tooling ship as systemd units and packaging scripts.
awc-control Live REST and mutually authenticated internal gRPC service.
Worker CAS transport Production releases include the S3 transport; an in-memory transport remains available for tests and explicit development.
awcd base mount Production release builds use the real FUSE base mount; featureless CI builds retain a no-op lifecycle test double.

See also