Retention and garbage collection reference

Durable garbage collection (GC) reclaims content-addressed objects that no retained root can reach. It runs as one mark-and-sweep epoch over a single tenant. For the operational runbook, see Run garbage collection.

GC targets the durable object store only. It does not evict from the host cache.

The epoch

One epoch is the unit of work. The handler gc_sweep drives it end to end. The table maps each phase to its action.

Step	Phase	What happens
1	Roots	Read the manifest digests of every READY snapshot for the tenant.
2	Mark	Traverse the object graph from the roots. Collect every reachable digest.
3	Inventory	Enumerate every object the store holds for the tenant.
4	Plan	Diff the inventory against the marked set under the grace cutoff.
4b	Holds	Remove candidates pinned by an integrity hold.
5	Record	Write one CANDIDATE row per candidate digest.
6	Sweep	Delete candidates in bounded batches with a leading canary.
7	Mark deleted	Flip physically removed rows to DELETED.
8	Complete	Record the counts and the summary on the epoch row.
9	Audit	Append one gc.sweep audit event.

A failed epoch deletes nothing. See Failure behavior.

Retained roots

The mark phase roots at two sources only:

Root source	How it protects content
Manifest digest of every READY snapshot	Traversal reaches its trees, files, file-chunk blobs, and symlink blobs.
Digests pinned in gc_integrity_holds	Removed from the candidate set before the sweep. Protects the closure of a snapshot degraded by a missing or corrupt object.

The SQL for the READY roots is one query:

SELECT manifest_digest FROM snapshots
WHERE tenant_id = $1 AND state = 'READY';

More roots are live than the epoch marks: a workspace base or head, a retained checkpoint alias, an active import session, an active export job, a legal hold, a GC grace record, and an administrative pin. The current build does not mark from those directly. Objects they reference stay safe through the grace cutoff when their age beats the cutoff.

Narrow root gap

An import that runs longer than AWC_GC_GRACE_SECS without producing a READY snapshot has its uploaded objects protected by timestamp alone. Finish long imports before you sweep, or raise the grace for the run.

Candidate eligibility

An object becomes a deletion candidate when both conditions hold.

Condition	Rule	Edge case
Unreachable	The digest is absent from the marked set.	None.
Grace-expired	created_at is set and is strictly before the cutoff (now - safety_grace).	An object created exactly at the cutoff is protected.
Unknown age	created_at is None.	Never a candidate. The sweep is conservative about anything it cannot age.

Safety grace

The grace protects an object uploaded by a concurrent import or checkpoint whose root snapshot has not yet reached READY and been observed by this epoch.

Item	Value
Cutoff	now - AWC_GC_GRACE_SECS
Default	86400 seconds (24 hours)
Stored on	gc_runs.cutoff
Comparison	Strictly before. Equal is protected.

The epoch records the cutoff it used. An object uploaded after the roots were snapshotted is protected by timestamp even when it is not yet reachable.

Integrity holds

An integrity hold pins one digest against GC for one tenant. It fences the sweep around a degraded snapshot's recovery content.

Property	Value
Storage	PostgreSQL table gc_integrity_holds. Survives worker restarts.
Producer	The scrub path. It pins the closure of a degraded snapshot, then releases it after restore.
Read by sweep	Held digests are removed from the candidate set before the sweep.
Failure mode	If holds cannot be read, the epoch is recorded FAILED. Nothing is deleted.
Tenant scope	One tenant. No cross-tenant leak.

Configuration

The worker reads these variables once at startup. Restart the worker when you change one.

Variable	Type	Default	Meaning
AWC_GC_GRACE_SECS	integer	86400	Safety grace in seconds. Objects newer than now - grace are never swept.
AWC_GC_BATCH_SIZE	integer	1000	Max candidates one epoch deletes across all batches. Bounds each run as a canary.
AWC_GC_CANARY_SIZE	integer	10	Size of the leading batch inside an epoch.
AWC_GC_DRY_RUN	string	unset	Plan only, delete nothing. Truthy values are 1, true, and TRUE.

The knobs are environment-wide, not per tenant. A per-project retention policy is prospective and does not drive the sweep in this build.

Sweep batching

The sweep deletes candidates in batches.

Batch	Cap
First batch (canary)	min(AWC_GC_CANARY_SIZE, remaining)
Later batches	min(AWC_GC_BATCH_SIZE, remaining)
Total per epoch	min(AWC_GC_BATCH_SIZE, candidate_count)

Candidates not deleted this epoch are pending. They are left for a subsequent epoch. The run reports the pending count in the job result.

Dry run

Set AWC_GC_DRY_RUN=1 to plan candidates and record them without deleting anything. In a dry run every candidate stays in state CANDIDATE, deleted_count is 0, and every candidate is reported pending.

Database tables

Three tables record the durable-GC accounting.

gc_runs

One row per epoch (migration 20260807000003_gc_runs.sql).

Column	Type	Notes
epoch	bigserial PRIMARY KEY	Monotonic epoch id.
tenant_id	TEXT NOT NULL	References tenants(id), cascades on delete.
cutoff	timestamptz NOT NULL	Objects newer than this are grace-protected.
state	TEXT NOT NULL	RUNNING, COMPLETED, or FAILED.
dry_run	boolean NOT NULL DEFAULT false	Whether this epoch deleted anything.
roots_count	bigint NOT NULL DEFAULT 0	Retained roots snapshotted.
marked_count	bigint NOT NULL DEFAULT 0	Digests reached by the mark phase.
inventory_count	bigint NOT NULL DEFAULT 0	Objects enumerated from the store.
candidate_count	bigint NOT NULL DEFAULT 0	Unreachable and grace-expired.
deleted_count	bigint NOT NULL DEFAULT 0	Physically removed this epoch.
reclaimed_bytes	bigint NOT NULL DEFAULT 0	Bytes reclaimed this epoch.
summary	jsonb NOT NULL DEFAULT '{}'	Per-kind counts and run config. See Run summary.
error	TEXT	Set when state = FAILED.
started_at	timestamptz NOT NULL DEFAULT now()	Epoch start.
completed_at	timestamptz	Epoch completion.

Index: gc_runs_tenant_started_idx on (tenant_id, started_at DESC).

State transitions:

From	To	Trigger
(new row)	RUNNING	The epoch opens.
RUNNING	COMPLETED	The counts are written.
RUNNING	FAILED	The error column is written.

gc_deletion_plans

One row per candidate digest in an epoch (migration 20260807000003_gc_runs.sql).

Column	Type	Notes
epoch	bigint NOT NULL	References gc_runs(epoch), cascades on delete.
tenant_id	TEXT NOT NULL	References tenants(id), cascades on delete.
digest	TEXT NOT NULL	Canonical sha256:<hex>.
kind	TEXT NOT NULL	BLOB, FILE, TREE, or SNAPSHOT.
size_bytes	bigint NOT NULL	Object size.
state	TEXT NOT NULL	CANDIDATE, DELETED, or SKIPPED.
created_at	timestamptz NOT NULL DEFAULT now()	Plan row creation.

Primary key: (epoch, digest). Index: gc_deletion_plans_tenant_state_idx on (tenant_id, state).

State transitions in the current write path:

From	To	Trigger
(new row)	CANDIDATE	One row is inserted per candidate.
CANDIDATE	DELETED	After the physical delete.

The schema permits SKIPPED. The current write path does not set it.

gc_integrity_holds

One row per pinned digest per tenant (migration 20260807000004_gc_integrity_holds.sql).

Column	Type	Notes
tenant_id	TEXT NOT NULL	References tenants(id), cascades on delete.
digest	TEXT NOT NULL	Canonical sha256:<hex>.
created_at	timestamptz NOT NULL DEFAULT now()	Hold creation.

Primary key: (tenant_id, digest). Index: gc_integrity_holds_tenant_idx on (tenant_id).

Audit events

One audit event is appended per epoch.

Field	Value
Action	gc.sweep
Resource kind	gc_run
Resource id	The epoch id, as a string.
Principal	GC_PRINCIPAL

The detail payload records the run outcome:

Key	Meaning
cutoff	The grace cutoff timestamp used.
dry_run	Whether this was a dry run.
candidate_count	Candidates after holds were applied.
deleted_count	Candidates physically deleted.
reclaimed_bytes	Bytes reclaimed.
held_blocked	Candidates removed because a hold pinned them.

Run summary

The gc_runs.summary JSON column stores per-kind candidate counts and the run config.

Key	Meaning
candidate_bytes	Sum of candidate size_bytes.
by_kind.BLOB	Candidate count of kind BLOB.
by_kind.FILE	Candidate count of kind FILE.
by_kind.TREE	Candidate count of kind TREE.
by_kind.SNAPSHOT	Candidate count of kind SNAPSHOT.
safety_grace_secs	The grace used, in seconds.
batch_size	The run cap used.
canary_batch_size	The canary size used.

The job result carries the same counts plus pending_count.

Failure behavior

The mark phase aborts on the first traversal inconsistency. A failed epoch is recorded FAILED with its reason in gc_runs.error, and nothing is deleted.

Variant	Meaning
MissingObject	A referenced object is absent mid-traversal.
KindMismatch	A referenced object's stored kind does not match its referencer.
MalformedObject	An object failed to parse as its declared kind.
OversizedObject	A single object exceeded the per-object byte budget.
OversizedTree	A tree exceeded the maximum entry count.
ExcessiveDepth	Tree nesting exceeded the depth budget.
Store	The object store failed a fetch or a delete.
Inventory	The object store failed to enumerate its inventory.
InvalidRoot	A retained-root manifest digest was not well-formed.
Db	A control-plane (PostgreSQL) failure.

The gc_sweep handler maps errors to job outcomes. A missing or blank tenant_id in the payload fails the job permanently. Any epoch error retries, because the run is idempotent. A genuine reachability inconsistency records a FAILED epoch each attempt and dead-letters once max_attempts is exhausted.

Retention policy

A project retention policy defines these prospective fields:

default workspace TTL
checkpoint retention duration
failed import retention
preserved-local grace
export retention
minimum GC grace
legal-hold behavior

Retention changes apply prospectively unless an administrator explicitly requests re-evaluation of existing resources. The current build does not drive the sweep from a retention policy.

Local cache GC

Host-local cache eviction is independent from durable GC. Cache entries may be removed at any time if they are not being installed and local readers tolerate normal Unix unlink semantics. Local workspace uppers are deleted only after a successful unmount disposition, lease release or fencing, no active bind mounts, the configured recovery grace, and confirmation that no checkpoint operation still references the directory.

Cross-links

• Run garbage collection — the operational runbook.
• Environment variables — the AWC_GC_* knobs.
• Database schema — the full schema.
• Error codes — error surfaces.

Current scope

The GC engine is complete and unit-tested. These items are specified but not present in the current build, or lag the spec:

• No awcctl gc command. Run GC by enqueuing a gc_sweep job into the PostgreSQL job queue with an awc-worker process running to claim it.
• No GC auto-scheduler. The control plane enqueues snapshot_validate on import, but it never enqueues gc_sweep. You must trigger each epoch by hand.
• The storage gauges are not emitted. awc_tenant_unreachable_grace_bytes, awc_tenant_unique_durable_bytes, and awc_snapshot_logical_bytes are specified but are not emitted. Measure unreachable bytes with a dry-run epoch, not with a gauge.
• GC knobs are environment-wide, not per tenant. The AWC_GC_* variables apply to every tenant the worker serves. A per-project retention policy does not drive the sweep.
• The mark roots are narrower than the full set. The epoch marks from READY snapshot manifest digests and gc_integrity_holds only. It does not mark from a workspace base or head, a checkpoint alias, an import session, an export job, a legal hold, or an administrative pin. Content those roots reference is protected by the safety grace timestamp when its age beats the cutoff.
• The gc_deletion_plans.state value SKIPPED is unused. The schema permits it. The write path sets CANDIDATE then DELETED.