Recover from an object-store outage¶

Use this runbook when hosts cannot reach the object store. The goal is to keep mounted workspaces on the host cache while the store is down, then restore the store and confirm that fetches resume and nothing was marked corrupt by mistake.

The object store holds the immutable objects behind every snapshot. The read hierarchy does not pass through the control plane. It is:

Linux page cache
  → awcd in-memory manifest cache
  → host-local NVMe CAS cache
  → regional/network object storage

Only the last tier is affected by this outage. Objects already resident on the host keep serving. Objects not yet cached do not. See Architecture for the read path.

During the outage, awcd MUST keep cached reads serving, and cold misses MUST return a clear transient I/O error after bounded retries. It MUST NOT fabricate empty or stale content. Checkpoint publication MUST fail without advancing the workspace head.

Detect¶

Watch for any of these signals:

A cold read fails with a transient I/O error. The FUSE layer maps a fetch failure to EIO. The agent sees the read fail; the mount does not return wrong bytes.
awc_singleflight_leader_fetches_total climbs and keeps climbing on the host health HTTP port (GET /metrics). Every cache miss that becomes a remote fetch is a leader fetch. A sustained rise with failing reads means the store is down, not one bad object.
A checkpoint fails at the upload phase. The pipeline uploads every immutable object before it advances the head. An upload that cannot complete cancels the checkpoint and leaves the head unchanged.
The object-store provider endpoint fails its own health check, or the provider reports an availability, throttling, or authentication fault.

awcd classifies every object-store fault into one of six categories:

Class	Retryable	Meaning
`Availability`	Yes	General failure. Connection reset, DNS, or 5xx.
`Throttling`	Yes	Provider rate-limited the request.
`Timeout`	Yes	The operation timed out.
`Authentication`	No	Bad or denied credentials.
`MissingObject`	No	No object exists under the digest.
`Corruption`	No	Bytes do not re-hash to the digest.

Availability, Throttling, and Timeout drive the retry path. Authentication, MissingObject, and Corruption do not.

Health HTTP will not detect this outage today

The object_store health component reports Unknown with the detail connectivity probe not wired. It does not flip to Down, so /healthz stays 200 and /readyz does not change. Do not rely on readiness probes. Use the leader-fetch counter and the cold-read EIO failures instead.

Missing is not corruption during an outage

A down store surfaces ObjectNotFound, which classifies as MissingObject, not Corruption. Confirm the outage is over before you treat a missing object as durable loss. See Handle object-store corruption.

Procedure¶

Confirm the object store is the failed dependency. Correlate the EIO reads and the rising leader-fetch counter across more than one workspace, and check the provider endpoint. Do not chase one bad object or one host's network.
Disable prefetch and nonessential imports. Prefetch issues remote fetches for metadata and high-value files on mount. Stop it, and pause any import or fork that must pull from the store. Demand reads take priority over prefetch.
Keep warm reads serving. Cached objects and resident manifest metadata MUST continue to resolve. Do not restart or drain hosts only because the store is down. A host that holds the working set stays useful.
Let cold misses fail clearly. A cold miss retries with bounded exponential backoff, then returns a transient EIO. The terminal failure is negatively cached for a short bounded interval so repeated misses do not hammer the store. Do not script an unbounded retry loop around the FUSE request.
Hold checkpoint publication. Do not force a publish while the store is down. The pipeline uploads objects before it advances the head, so a failed upload cancels the checkpoint and leaves the head unchanged. Uncheckpointed writes stay local and ephemeral.
Watch local upper growth. Local upper writes MAY continue during the outage. They land on the host upper directory and do not touch the store. A long outage lets the upper grow, which costs delta efficiency and disk space. Track awc_host_upper_usage_bytes. If the upper fills the disk, follow Handle host disk pressure.
Restore the object store. Fix the provider: credentials, network, capacity, or the backend itself. For an authentication fault, rotate the object-store credentials before you declare recovery (see Rotate credentials).
Let the caches clear and refill. The negative cache expires on its short TTL, and the retry path fetches on demand. If a provider circuit breaker has tripped, reset it gradually by letting half-open probes through rather than flooding the store. Watch awc_singleflight_leader_fetches_total fall back to its idle rate.
Retry idempotent uploads and publications. Re-issue a held checkpoint with the same idempotency key. It returns the committed result, or it publishes now that the store is back. Do not reissue with a new key.

Verify¶

Confirm recovery when all of these hold:

A fresh cold read succeeds. Pick a path the host has not cached, mount its snapshot, and read it. The read returns bytes, not EIO.
awc_singleflight_leader_fetches_total settles. Its rate returns to the idle pattern over a few sample intervals, and the negative cache stops serving remembered errors.
A checkpoint publishes without a cancellation. The upload phase completes and the workspace head advances exactly once.
The acknowledged workspace head in PostgreSQL matches the host sessions. No head advanced during the outage.
No snapshot was marked corrupt by mistake. The outage surfaces MissingObject, not Corruption. Run the command below and confirm it returns no new entries.
```
awcctl snapshot list --state DEGRADED
```
A sampled object re-verifies. Read a few objects from a recovered snapshot and confirm their digests match their content addresses. If any object still mismatches, that is real corruption, not an outage artifact. See Handle object-store corruption.

Impact¶

State the data risk plainly:

Cached reads and local writes survive. Objects resident on the host, the manifest cache, and the upper directory are independent of the store. The outage itself loses no data on a host that holds the working set.
Cold reads fail. A read for an object that is not cached returns EIO after bounded retries. The agent sees a transient I/O error.
Checkpoint publication is blocked, not lost. A checkpoint held by the outage can be retried with the same idempotency key once the store returns. The head does not advance until publication succeeds.
Uncheckpointed writes are ephemeral. Writes that have not been published are not durable. If the host is lost during the outage, those writes are lost. See Recover from host loss.
The system never fabricates content. A cold read fails rather than return empty or stale bytes.
Prefetch warming pauses. Metadata and content prefetch on mount stall until the store returns. Mounts of cold snapshots are slower to warm.

Current scope¶

The build implements the fetch, retry, and negative-cache path, but some operator surfaces lag the spec. Be aware of these gaps:

The provider circuit breaker is not wired into the running daemon. The fetch coordinator is constructed without the remote controls, so no per-provider breaker or concurrency bound is attached. In a live outage the active mechanisms are singleflight, bounded retry, and the negative cache. "Reset the breaker gradually" (step 8) is, in the current build, letting the negative-cache TTL expire and letting on-demand retries through.
Retry and negative-cache numbers are fixed at build time. The coordinator defaults to 3 attempts, a backoff of 10 ms to 1000 ms, and a negative-cache TTL of 5000 ms. There is no runtime knob to widen them for a long outage.
The object_store health component is not wired. It reports Unknown, so /healthz and /readyz cannot confirm or deny the outage. Detect it from awc_singleflight_leader_fetches_total and cold-read EIO failures instead.
Per-tenant and global remote-fetch bounds are not enforced yet. The concurrency limiter ships with the coordinator but is not attached. A storm of cold misses during the outage is bounded only by singleflight and the negative cache, not by an admission limit.