Skip to content

Handle object-store corruption

Use this runbook when a durable object is missing or when its bytes no longer match its content digest. The goal is to restore integrity without serving wrong bytes.

Every AWC object is content-addressed. An object is accepted only when its computed digest equals its claimed digest. Content addressing makes corruption detectable, but objects are immutable. You restore them from a trusted copy. You do not repair them in place.

Two failures are distinct and actionable:

  • Missing — no object exists under the digest. It may be recoverable from a trusted replica.
  • Corrupt — an object exists, but its bytes do not re-hash to the digest. The key is immutable, so wrong bytes under a correct address cannot be repaired by re-writing.

Detect

Watch for any of these signals:

  • A read or mount returns an integrity error. The API returns ObjectCorrupt or ObjectMissing as HTTP 422. awcctl exits with code 8 (INTEGRITY) for ObjectCorrupt, and code 7 (REMOTE_IO) for ObjectMissing.
  • The local cache rejects a digest mismatch. On download the cache recomputes the digest. A mismatch raises DigestMismatch and the cache row moves to the CORRUPT state.
  • The durable store returns a distinct integrity error. A verified read returns DigestMismatch for wrong bytes, or ObjectNotFound for a missing object. The fault classifier maps a mismatch to Corruption.
  • A background scrub marks the snapshot DEGRADED. The scrub job re-verifies every object in a snapshot's closure.
  • awcctl snapshot list --state DEGRADED returns snapshots the scrub has flagged.

Missing is not always corruption

A transient object-store outage can also surface ObjectNotFound. Confirm the outage is over before you treat a missing object as durable loss. See Recover from an object-store outage.

Procedure

  1. Distinguish local-cache corruption from durable-object corruption. A local mismatch quarantines a disposable cache copy. A durable mismatch affects the immutable object under its content address.
  2. Clear a local-cache mismatch first. The cache row is already CORRUPT. Evict it and refetch from the object store. The fetch path re-verifies the digest on download. If the refetched bytes verify, the corruption was local only. If they mismatch again, the durable object is corrupt. Continue.
  3. Identify the affected object and the snapshots that reference it. Read the snapshot.scrub audit event for the snapshot. Its detail lists each incident: digest, kind, severity (structural or leaf), problem (missing or corrupt), recovery, and actual_digest for corruption.

  4. List every degraded snapshot. Run the command below to find all snapshots the scrub has flagged.

    awcctl snapshot list --state DEGRADED

  5. Pin the closure against garbage collection (GC). The scrub writes the closure digests into the gc_integrity_holds table. Do not run a GC sweep that deletes held objects. See Run garbage collection.

  6. Restore the object from a trusted copy. Pick a source.

    Restore verified bytes from a configured replica or an object-store version. The scrub writes the object only if the key is absent, then re-verifies it. A missing object may clear this way. A corrupt object will not, because the key already holds bytes.

    Re-import the original source to produce a fresh, verified object graph. See Import from Git and Import from a directory.

  7. Restore only with verified bytes. A corrupt object is immutable under its address. The store has no overwrite. You cannot repair wrong bytes in place. You must restore the correct bytes from a trusted source, or into a clean store.

  8. Keep the snapshot unmounted while degraded. A DEGRADED snapshot is readable where objects survive, but it is not mountable as a writable workspace. Do not fork or mount it until it returns to READY.
  9. Record the incident and rotate credentials if you must. The scrub emits a snapshot.scrub audit event. If you suspect overwrite or unauthorized access, rotate the object-store credentials. See Rotate credentials.

Verify

Confirm recovery when all of these hold:

  • The snapshot state returns to READY. Confirm with awcctl snapshot inspect <id>.
  • A re-run of the scrub returns status: clean with zero incidents.
  • The affected snapshot mounts. awcctl workspace mount succeeds. Only a READY snapshot is mountable.
  • The gc_integrity_holds rows for the closure are released.
  • The restored object's digest matches its content address.

Impact

State the data risk plainly:

  • Objects are immutable. Content addressing makes corruption detectable, but you restore objects rather than repair them.
  • Corruption is unrecoverable without a trusted copy. The store has no overwrite, so wrong bytes under a correct address cannot be fixed in place. You must restore from a replica, a version, or a re-import.
  • A missing object may be recoverable. A trusted replica can restore a missing object. Without one, the object is lost.
  • Structural loss is worse than leaf loss. A missing or corrupt structural object (snapshot manifest, tree, or file manifest) makes the closure untraversable. A leaf blob loss degrades file content, but the snapshot is still walkable.
  • A degraded snapshot is not mountable. It stays readable where objects survive. It cannot be mounted or forked until it returns to READY.
  • Workspace writes are not affected. Uncheckpointed writes live in the host upper directory and are independent of the durable store. See Checkpoint a workspace.

Current scope

The build implements the integrity primitives and the scrub engine, but some operator surfaces lag the spec. Be aware of these gaps:

  • No scrub is scheduled yet. The scrub handler is registered in the worker, but no scheduler enqueues scrub jobs. There is no awcctl snapshot scrub command. To run a scrub, enqueue a scrub job directly. The control plane enqueues only a closure presence check at publication. That check does not re-verify the bytes of published objects over time.
  • No recovery replica is configured. The worker wires the trusted recovery store as None. The automated recovery path therefore returns NoSource for a missing object and Unrecoverable for a corrupt one. Restoring from a replica or re-importing the source is an operator action until the recovery store is wired.
  • GC holds are empty until a scrub runs. The gc_integrity_holds table exists and the GC sweep honors it, but nothing populates it until a scrub job runs.
  • The cache-corruption metric is not emitted. awc_host_cache_corruption_total is defined but is not recorded at runtime. Until it is wired, detect local-cache corruption from the CORRUPT cache row state and the DigestMismatch error.

See also