Skip to content

Run garbage collection

Use this runbook to reclaim durable content-addressed objects that no retained snapshot, workspace, or checkpoint can reach. The reclaim target is the object store, not the host cache.

Garbage collection (GC) runs as a mark-and-sweep epoch over one tenant (spec §18.3, §18.4). It deletes only objects that are both unreachable and older than the safety grace. Reachable content survives. Shared content survives deletion of one fork, because objects are deduplicated inside a tenant (spec §18.2, §16.12). See content addressing for the object model.

GC is separate from host cache eviction (spec §18.5). Cache loss affects speed only (spec §6.7). To reclaim space on a full host, run Handle host disk pressure instead.

Detect

No gauge reports unreachable bytes yet (see Current scope). Detect the condition from what has changed and measure it with a dry run:

  • A tenant deleted workspaces or snapshot aliases. Logical deletion removes the reference; it does not free durable bytes at once (spec §16.12, §18.2).
  • Tenant-unique durable bytes, or the storage bill, do not fall after those deletions. Tenant-unique bytes are the basis GC reclaims (spec §18.7).
  • You want to confirm that no orphaned objects remain from a failed import or an interrupted checkpoint. Such objects are safe orphans until GC runs (spec §11.6, §17.6).

The authoritative measurement is a dry-run GC epoch. It plans candidates and records their bytes without deleting anything (spec §18.3 step 7). Run it first, then read the counts back from the database.

Procedure

GC is implemented in the awc-gc crate and runs as an awc-worker job of kind gc_sweep (spec §8.2). The worker claims the job, drives one epoch, and records the result. A worker process must be running to claim the job.

The worker reads four knobs from the environment:

Variable Default Meaning
AWC_GC_GRACE_SECS 86400 Safety grace. Objects newer than now - grace are never swept (spec §18.4).
AWC_GC_BATCH_SIZE 1000 Max candidates one epoch deletes. Bounds each run as a canary.
AWC_GC_CANARY_SIZE 10 Size of the leading batch inside an epoch.
AWC_GC_DRY_RUN false Plan only. Delete nothing (spec §18.3 step 7).

The worker reads these variables once at startup. Restart the worker when you change one, or a run you expect to be a dry run can delete for real.

  1. Start a dry run to measure. Set AWC_GC_DRY_RUN=1 on the worker, then enqueue a gc_sweep job for the tenant. The epoch snapshots the retained roots and the grace cutoff, completes the mark phase, ingests the inventory, and writes the plan (spec §25.7 steps 1 to 4). Enqueue against the job queue:
INSERT INTO jobs
  (kind, tenant_id, state, payload, max_attempts,
   scheduled_at, available_at, created_at, updated_at)
VALUES
  ('gc_sweep', 'tnt_demo', 'PENDING',
   '{"tenant_id":"tnt_demo"}'::jsonb, 3,
   now(), now(), now(), now());

The handler reads the tenant from payload.tenant_id. A missing or blank tenant fails the job permanently; any epoch error is retried (spec §26.11).

  1. Wait for the epoch to reach COMPLETED, then compare the marked set against the inventory. The gc_runs row records every count (spec §25.7 step 3):
SELECT epoch, state, dry_run, cutoff, roots_count, marked_count,
       inventory_count, candidate_count, deleted_count, reclaimed_bytes
FROM gc_runs
WHERE tenant_id = 'tnt_demo'
ORDER BY epoch DESC
LIMIT 5;

marked_count is the set reached from retained roots. inventory_count is what the object store holds for the tenant. The gap is the unreachable set.

  1. Inspect the deletion candidates and their bytes (spec §25.7 step 4). Each candidate is one row in gc_deletion_plans:
SELECT kind, state, count(*) AS n, sum(size_bytes) AS bytes
FROM gc_deletion_plans
WHERE epoch = <EPOCH>
GROUP BY kind, state;

In a dry run every candidate stays in state CANDIDATE and deleted_count is 0. Confirm the bytes are worth reclaiming before you delete.

  1. Enforce the safety grace and the holds (spec §25.7 step 5). Confirm cutoff equals now - AWC_GC_GRACE_SECS. An object uploaded after the roots were snapshotted is protected by timestamp, even if it is not yet reachable (spec §18.4). Integrity holds fence off any digest pinned by a degraded snapshot's recovery, so recovery content is never swept (spec §17.10 step 6, §25.3). Objects of unknown age are never candidates.

  2. Disable dry run and run one bounded epoch. Set AWC_GC_DRY_RUN=0 and enqueue a second gc_sweep job. The sweep deletes a leading canary batch of AWC_GC_CANARY_SIZE objects, then continues in batches up to AWC_GC_BATCH_SIZE total (spec §25.7 step 6). Treat the first real epoch as the canary. Keep AWC_GC_BATCH_SIZE modest until you trust the run.

  3. Verify retained snapshots still mount (spec §25.7 step 7). Fork and mount a known snapshot, then read files from it:

awcctl workspace fork --snapshot <SNAPSHOT_ID> --ttl 1h
awcctl workspace mount <WORKSPACE_ID> --target /workspace
ls /workspace

A missing-object error here means GC deleted reachable content. Stop and open an incident (spec §17.10).

  1. Expand the batches by running further epochs (spec §25.7 step 8). Each epoch deletes at most AWC_GC_BATCH_SIZE candidates and leaves the rest pending for the next epoch. Repeat until a dry run reports candidate_count = 0.

  2. Record reclaimed bytes and keep the manifests (spec §25.7 steps 9 and 10). Each COMPLETED epoch records reclaimed_bytes on its gc_runs row, and a gc.sweep audit event is appended (spec §16.12). The gc_deletion_plans rows persist as the auditable deletion manifest; the physically removed rows move to state DELETED.

A failed epoch deletes nothing

If the mark phase hits a missing object or a kind mismatch, the epoch is recorded FAILED and nothing is deleted (spec §26.11). Do not treat a FAILED row as a partial reclaim. Read the error column and resolve the corruption first. See Handle object corruption.

Verify

Confirm a safe reclaim when all of these hold:

  • The epoch row is COMPLETED, not FAILED.
  • deleted_count and reclaimed_bytes match what you expected, or are 0 if nothing was unreachable.
  • A known retained snapshot still forks, mounts, and reads.
  • A follow-up dry run reports candidate_count = 0, or only objects younger than the grace remain.
  • The gc.sweep audit event is present for the epoch.

Impact

State the data risk plainly:

  • GC deletes only unreachable, grace-expired durable objects (spec §18.3, §18.4). It does not touch the host cache, workspace uppers, or any live root.
  • Shared content survives. Deleting one snapshot alias does not delete an object that another snapshot or workspace still reaches (spec §18.2, §16.12). GC reclaims tenant-unique durable bytes, not summed logical snapshot bytes (spec §18.7).
  • A reachability bug could delete shared reachable content (spec §26.11). The mitigations are immutable roots, the epoch cutoff, the safety grace, the dry run, the canary batch, integrity holds, and the audit trail. The mark phase aborts on the first inconsistency, so a buggy run fails closed.
  • A FAILED epoch is safe. It records the reason and deletes nothing.
  • Cache eviction is separate and safe. Losing every host-local cache affects performance only (spec §6.7, §18.5). It does not affect recoverability.

Current scope

The engine is complete and tested, but the operator surfaces lag the spec:

  • No awcctl gc command. The CLI exposes only status, snapshot, and workspace. Run GC by enqueuing a gc_sweep job into the PostgreSQL job queue, with an awc-worker process running to claim it.
  • No GC auto-scheduler. The control plane enqueues snapshot_validate on import, but it never enqueues gc_sweep. You must trigger each epoch by hand.
  • The storage gauges are not yet emitted. awc_tenant_unreachable_grace_bytes, awc_tenant_unique_durable_bytes, and awc_snapshot_logical_bytes are specified (spec §19.2) but are not wired in awc-telemetry. Until they ship, measure unreachable bytes with a dry-run epoch, not with a gauge.
  • GC knobs are environment-wide, not per tenant. The worker reads the AWC_GC_* variables once at startup. A per-project retention policy (spec §18.6) is prospective and does not yet drive the sweep.
  • The mark phase roots at READY snapshots and integrity holds. The epoch marks reachability from the manifest digests of every READY snapshot, plus the digests pinned by gc_integrity_holds for degraded-snapshot recovery (spec §18.1, §17.10). It does not mark from an in-flight import session directly. Objects an import has uploaded but not yet sealed into a READY snapshot are protected by the safety grace timestamp, not by reachability. An import that runs longer than AWC_GC_GRACE_SECS (default 24 hours) without producing a READY snapshot can have its not-yet-reachable objects become candidates. Finish long imports first, or raise the grace for the run.