awc-worker reference

awc-worker is a background process. It drains the PostgreSQL job queue and dispatches each claimed job by kind to a handler (spec §8.2). The worker only claims and runs jobs. It does not enqueue them. The control plane enqueues jobs.

The process runs a bounded pool of workers. Each worker loops: claim a ready job, run the handler, record the outcome, then claim the next job.

Startup

The startup sequence is fixed:

1. Read DATABASE_URL and AWC_CAS_URL. Both are required.
2. Connect the database pool.
3. Apply pending migrations (Repositories::migrate).
4. Select the object store from the AWC_CAS_URL scheme.
5. Register the job handlers.
6. Start the supervisor pool.

If any step fails, the process prints awc-worker: <error> to standard error and exits 1. Startup error messages carry no secrets (spec §16.5). Database errors surface by type and message, never by the connection string.

Migrate on startup

The worker runs schema migrations before it accepts jobs. The call is Repositories::migrate, which runs the sqlx MIGRATOR against the pool. Point two workers at one database and both migrate; sqlx tracks the applied set in the migration bookkeeping table.

Object store selection

The AWC_CAS_URL scheme selects the content-addressed object store. The same store backs snapshot validation, the integrity scrub, and durable garbage collection.

Scheme	Store	Requirement
memory://	In-process store	Local and integration use.
s3://	S3-compatible transport	Requires the s3 cargo feature. Configure it with the AWC_S3_* variables (spec §16.3).
any other	none	Startup error.

Exit codes

Code	Meaning
0	The supervisor ran and shut down cleanly.
1	A startup or runtime error occurred.

Environment variables

Variable	Default	Purpose
DATABASE_URL	required	PostgreSQL connection string for the job queue.
AWC_CAS_URL	required	Object-store URL. Selects the transport by scheme.
AWC_WORKER_ID	awc-worker	Per-process id. Each concurrent worker appends -<n> to its leases.
AWC_WORKER_CONCURRENCY	4	Bounded worker count. The pool spawns this many tasks.
AWC_WORKER_LEASE_SECS	300	Visibility-lease length, in seconds.
AWC_WORKER_POLL_INTERVAL_MS	1000	Idle poll interval, in milliseconds. The worker sleeps this long when no job is ready.
AWC_RETENTION_GRACE_SECS	86400	How long an EXPIRING workspace stays reversible before it may be reaped (spec §12.5). Used by workspace_expire.
AWC_GC_GRACE_SECS	86400	Durable-GC safety grace, in seconds. Objects newer than now - grace are never swept (spec §18.4). Used by gc_sweep.
AWC_GC_BATCH_SIZE	1000	Maximum candidates deleted per GC epoch. Used by gc_sweep.
AWC_GC_CANARY_SIZE	10	Size of the leading canary batch per GC epoch. Used by gc_sweep.
AWC_GC_DRY_RUN	false	Plan only; delete nothing (spec §18.3). Accepted true values: 1, true, TRUE.

Integer variables that fail to parse cause a startup error. The error names the setting and the parse failure.

Job kinds

The worker registers one handler per kind. An unknown kind is a permanent failure. The supervisor dead-letters it; it does not retry.

Kind	Handler	Payload key	Work
snapshot_validate	ValidationHandler	snapshot_id	Verifies the content-addressed closure of a VALIDATING snapshot. Moves it to READY, FAILED, or skips it if already terminal.
snapshot_scrub	ScrubHandler	snapshot_id	Re-verifies every object in a READY snapshot's closure. Marks decayed snapshots DEGRADED and pins their closure against GC. Emits the snapshot.scrub audit event.
workspace_expire	WorkspaceExpireHandler	workspace_id	Advances one workspace through the TTL state machine (spec §12.5).
snapshot_retain	SnapshotRetainHandler	snapshot_id	Evaluates retention for one snapshot.
gc_sweep	GcHandler	tenant_id	Runs one durable-GC epoch for a tenant (spec §18.3).

A malformed payload is a permanent failure for every kind. The job is dead-lettered; it is not retried.

Job lifecycle

A job moves through these states: PENDING, RUNNING, COMPLETED, FAILED, CANCELLED.

Claim and lease

Each worker calls claim with its per-worker id and the lease length. A claim sets the job to RUNNING and records locked_by and locked_until. The claim commits before the handler runs, so handler I/O stays outside the database transaction (spec §8.2).

Outcome

A handler returns one of three outcomes. The supervisor records it.

Outcome	Record action	Final state
Complete(result)	complete	COMPLETED
Retry(error)	fail with backoff	PENDING, or FAILED once attempts reaches max_attempts
Fail(error)	dead_letter	FAILED

A job whose subject failed is still a completed job. For example, a snapshot that does not close produces a COMPLETED validation job with the verdict in result.

Retry and dead-letter

Retry is for transient store errors. The supervisor reschedules the job with a bounded exponential backoff and full jitter (spec §17.7, §8.2). The retry path dead-letters the job once attempts reaches max_attempts.

max_attempts is a per-job column. The producer sets it at enqueue time; the worker reads it from the row. The worker does not set max_attempts.

Fail is permanent. It dead-letters immediately. The worker never retries a Fail.

Backoff

The deterministic base is 5 * 2^attempts seconds, capped at 300 seconds.

Attempts (after this claim)	Base delay
0	5 s
1	10 s
2	20 s
3	40 s
>= 6	300 s (cap)

Full jitter adds a uniform [0, base] offset. A retried job's total delay lies in [base, 2 * base]. Each worker seeds one xorshift32 PRNG from its worker id, so retried jobs across the pool do not fire on the same tick.

Cancellation and lease expiry

After the handler runs, the supervisor re-reads the job. If the job is no longer RUNNING under this worker, the supervisor discards its outcome. This covers two cases:

• The job was cancelled while the handler ran.
• The lease expired and another worker reclaimed the job.

Validation is idempotent, so a concurrent run is harmless (spec §8.2).

Metrics

The worker registers three instruments when the supervisor runs with observability attached (spec §19.2). Label values are closed-set.

Metric	Type	Labels	When recorded
awc_background_jobs_total	counter	kind, outcome	Once per processed job.
awc_background_job_lag_seconds	gauge	kind	Once per processed job. The ready-to-processed lag, in seconds.
awc_api_errors_total	counter	component=worker	On a transient store error during the claim or record step.

The outcome label is a closed set.

outcome value	Job state that produced it
completed	COMPLETED
failed	FAILED
cancelled	CANCELLED
retried	PENDING (rescheduled)
other	any unrecognised state

The metrics registry caps distinct label-sets per instrument at 128. Recordings beyond the cap roll into the synthetic cardinality_overflow label value.

Structured events

The worker emits one structured event per processed job to standard error (spec §19.5). The event id is job_<job-id>. It carries these fields: kind, outcome, state, attempts. Events carry no secrets (spec §16.5): only the kind and an outcome class, never a payload or error text.

Outcome	Log level	Message
completed	info	job completed
retried	warn	job retried after transient failure
failed	error	job dead-lettered
cancelled	error	job cancelled

Signals

SIGINT and SIGTERM trigger a graceful shutdown. A shared stop signal is sent to every worker. Each worker stops accepting new claims and exits after the job it is mid-flight on. The supervisor returns once all workers have joined.

Current scope

The MVP spec (§8.2) lists more job types than the current build implements. The worker ships handlers for these kinds only: snapshot_validate, snapshot_scrub, workspace_expire, snapshot_retain, and gc_sweep.

Spec §8.2 job types with no shipped handler:

• Git import
• Snapshot logical-size calculation
• Structural diff generation for large diffs
• CAS garbage-collection planning (the shipped gc_sweep runs the epoch; the separate planning job is not implemented)
• Object-store inventory ingestion
• Billing aggregation
• Host-health reconciliation
• Audit export

The worker does not enqueue jobs and does not expose a control API. Job scheduling is the responsibility of the control plane.

No trusted recovery store is wired today. The scrub handler detects decay and degrades without attempting replica recovery (the recovery source is None).