Observability reference¶

The operator surfaces of AWC are the awcd health HTTP endpoints, the /metrics scrape, structured logs, traces, and the audit-event log.

Health HTTP endpoints¶

awcd serves three HTTP/1.1 routes on a separate loopback listener. The host API stays on its Unix-domain gRPC socket; only health and metrics use this listener.

Route	Method	Returns	HTTP `200`	HTTP `503`
`/healthz`	`GET`	Liveness plus the full report body	The process is up and the base filesystem is serving.	A foundational component is hard-down.
`/readyz`	`GET`	Readiness for new writable mounts plus the full report body	The host can accept a new writable mount.	The host cannot accept a new writable mount. Warm reads may still be served.
`/metrics`	`GET`	Prometheus text exposition	Always, when the listener is up.	Never.

Any other method returns 405. Any other path returns 404. Both /healthz and /readyz return the same JSON body, a HealthReport. Only the status code differs.

The listener is opt-in. When health_http_listen is unset, the HTTP server does not start and GetHostStatus is the only host-status surface (over gRPC).

Knob	Where	Value
Config key	`health_http_listen`	A loopback `host:port`, for example `127.0.0.1:9100`.
Env var	`AWC_HEALTH_HTTP_LISTEN`	Same format.
Type	`Option<SocketAddr>`	The listener address is optional.
Default	unset	The listener is not started.

The value must parse as a host:port or awcd returns a config error. It should be a loopback address so the endpoints are not exposed broadly. See configuration and environment variables.

Health report body¶

/healthz and /readyz return a JSON HealthReport.

Field	Type	Meaning
`live`	bool	Liveness. The process is up and its event loop makes progress.
`ready_for_new_mounts`	bool	The host can accept a new writable mount.
`warm_read_ready`	bool	The host can keep serving existing warm reads.
`components`	array	One object per component. See components.

Each components entry has three fields:

Field	Type	Meaning
`name`	string	A canonical component name. See components.
`state`	string	One of `ok`, `degraded`, `down`, `unknown`. See component states.
`detail`	string or null	A closed-set, non-sensitive verdict, for example `active_mounts=2`.

Health components¶

awcd derives the report from seven components.

Component	`name`	State in the current build
Base filesystem	`base_fuse`	`ok` when mounted, `degraded` while mounting or unmounting, `down` otherwise
Cache	`cache`	`ok`
Kernel compatibility	`kernel_compat`	`ok`, `degraded`, or `unknown`
Mounts and leases	`mounts_leases`	`ok`
Control plane	`control_plane`	`unknown`
Object store	`object_store`	`unknown`
Disk pressure	`disk_pressure`	`ok`, `degraded`, `down`, or `unknown`

Component states¶

Each component state has one of four values. They serialize in snake_case.

State	Meaning
`ok`	Healthy or nominal.
`degraded`	Serving but impaired. For example, elevated disk pressure.
`down`	Required for readiness and currently failing.
`unknown`	Not yet probed. The transport for this build is not wired.

Readiness model¶

A host may stay ready for existing warm reads while it is unready for new writable mounts. The report preserves that distinction.

Flag	Derived rule
`live`	`base_fuse` is not `down`.
`warm_read_ready`	`base_fuse` and `cache` are `ok` or `degraded`.
`ready_for_new_mounts`	`warm_read_ready` and `kernel_compat` is `ok` or `degraded`, and `mounts_leases`, `disk_pressure`, `control_plane`, and `object_store` are not `down`.

/healthz returns 200 when live is true. /readyz returns 200 only when ready_for_new_mounts is true. For kernel_compat, unknown fails readiness closed: an unverified kernel is not admitted. The mounts_leases, disk_pressure, control_plane, and object_store gates fail readiness only when down, not when unknown.

Because the readiness rule for control_plane and object_store is not_down, their unknown state does not by itself fail /readyz. In the shipped binary kernel_compat is wired to the live probe, so on a schedulable, non-pressured host /readyz returns 200 even while those two components report unknown. A real down connectivity verdict cannot be asserted until the probes are wired. Warm reads still return 200 on the liveness probe.

Disk pressure states¶

The disk_pressure component maps a pressure state to a component state. States escalate monotonically by severity rank.

Rank	State string	Component state	Effect
`0`	`NORMAL`	`ok`	No containment.
`1`	`CACHE_EVICTION`	`degraded`	Disposable cache is evicted.
`2`	`PREFETCH_DISABLED`	`degraded`	Speculative prefetch yields.
`3`	`NEW_MOUNTS_DISABLED`	`down`	New writable mounts are rejected.
`4`	`UPPER_WRITES_AT_RISK`	`down`	Existing uppers may hit `ENOSPC`.
`5`	`EMERGENCY`	`down`	Only warm reads continue.

The detail string for disk_pressure is state=<STATE>; uncheckpointed_upper_bytes=<n>. The monitor never silently deletes uncheckpointed upper data to relieve pressure. The rank is also exported as the awc_host_disk_pressure_state metric. See metrics.

Metrics¶

GET /metrics returns a Prometheus-style text exposition. The registry caps distinct label-sets per instrument at 128. Recordings beyond the cap roll into the synthetic value cardinality_overflow. Do not put a path, digest, workspace ID, or tenant ID in a label.

awcd serves /metrics on its health HTTP listener (see above). awc-control serves /metrics on its REST listener, alongside /v1/health; that route is unauthenticated, so bind the listener to a private interface.

The full metric catalog, the label sets, the emitted-versus-declared split, and the histogram buckets are on the metrics reference page.

Structured logs¶

Logs are structured JSON lines that carry a request or operation ID.

Field	Type	Meaning
`op_id`	string	The request or operation id.
`level`	string	One of `INFO`, `WARN`, `ERROR`.
`message`	string	A short event summary.
`fields`	object	Closed-set, non-sensitive facts. For example, a stage name or a count.
`ts`	string	The event timestamp, RFC 3339, UTC.

A deployment wires an event sink for the events. Two sinks ship: one writes a JSON line per event to standard error; one collects lines in memory for in-process tests. Logs should record state transitions, not every successful file read.

Under systemd, all three units send standard output and standard error to journald (StandardOutput=journal, StandardError=journal). Tail with journalctl -u awcd -f. See systemd units.

The following must never appear in a log line:

file contents;
access tokens;
object-store credentials;
raw secret-bearing paths.

Redact a path before you log it. See tracing and redaction.

Tracing¶

A trace should connect the workspace operation to the object-store request and the publication transaction.

workspace operation
  -> mount ticket
  -> host mount preparation
  -> FUSE lookup/read
  -> cache lookup
  -> singleflight wait
  -> object-store request
  -> checkpoint quiesce
  -> object upload
  -> publication transaction

Trace attributes may include tenant, project, host, snapshot digest, workspace ID, cache outcome, object size, and error class. File paths must be redacted or hashed by default because they may contain sensitive information.

The redaction behavior is as follows.

Operation	Input	Output
Path redaction	A path and a `keep_depth`	`awc:<head>:<hash>`. The first `keep_depth` components are kept; the rest collapses to a short SHA-256 hash.
Secret redaction	Any secret-bearing string	The fixed placeholder `awc:redacted:secret`.
ID hashing	Any text	`sha256:` plus the first eight bytes of the SHA-256 digest, in lower-case hex.

An OpenTelemetry-compatible OTLP exporter is not layered on. The metrics registry render path is the integration point for one. See Current scope.

Audit events¶

The control plane records immutable audit events for privileged mutations. Each event is one row in the append-only audit_events table. The action and resource_type columns use a fixed set of values so the log stays greppable by category.

`audit_events` table¶

The table schema is as follows.

Column	Type	Meaning
`id`	`bigserial`	Row primary key.
`tenant_id`	`TEXT`	The tenant the event belongs to.
`principal_id`	`TEXT`, nullable	The actor. A system principal for background passes.
`action`	`TEXT`	The action string. See actions.
`resource_type`	`TEXT`	The resource family. See resource types.
`resource_id`	`TEXT`	The resource identifier.
`request_id`	`TEXT`, nullable	The inbound request id, when one is in scope.
`detail`	`jsonb`	Actor, resource, request, and state-transition facts.
`created_at`	`timestamptz`	The event time.

The detail column records only identifiers, digests, counts, and enum verdicts. It must not contain object contents, byte payloads, manifest bodies, file names from user data, credentials, tokens, or keys.

Audit actions¶

Every action follows the <resource>.<verb> shape. The table below lists each action constant and its string value.

Constant	`action` value
`CHECKPOINT_PUBLISH`	`checkpoint.publish`
`WORKSPACE_EXPIRE`	`workspace.expire`
`WORKSPACE_DELETE`	`workspace.delete`
`SNAPSHOT_MARK_DELETING`	`snapshot.mark_deleting`
`SNAPSHOT_CREATE`	`snapshot.create`
`SNAPSHOT_TRANSITION`	`snapshot.transition`
`SNAPSHOT_SCRUB`	`snapshot.scrub`
`GC_SWEEP`	`gc.sweep`
`HOST_REGISTER`	`host.register`
`HOST_REVOKE`	`host.revoke`
`HOST_ROTATE`	`host.rotate`
`HOST_COMPATIBILITY`	`host.compatibility`
`IMPORT_SESSION_CREATE`	`import_session.create`
`SIGNING_KEY_ROTATE`	`signing_key.rotate`
`SIGNING_KEY_RETIRE`	`signing_key.retire`
`SIGNING_KEY_REVOKE`	`signing_key.revoke`
`OBJECT_CREDENTIAL_ISSUE`	`object_credential.issue`

Audit resource types¶

The resource_type column takes these values.

Constant	`resource_type` value	Status
`WORKSPACE`	`workspace`	Emitted.
`SNAPSHOT`	`snapshot`	Emitted.
`IMPORT_SESSION`	`import_session`	Emitted.
`HOST`	`host`	Emitted.
`GC_RUN`	`gc_run`	Emitted.
`SIGNING_KEY`	`signing_key`	Emitted.
`OBJECT_CREDENTIAL`	`object_credential`	Emitted.
`LEASE`	`lease`	Reserved. No action is emitted.
`MOUNT_SESSION`	`mount_session`	Reserved. No action is emitted.
`EXPORT`	`export`	Reserved. No action is emitted.

System principals¶

Background passes record a system principal in principal_id.

Constant	`principal_id` value	Pass
`RETENTION`	`system:retention`	The retention and TTL pass.
`GC`	`system:gc`	The durable garbage collector.
`VALIDATION`	`system:validation`	The closure-validation job.
`SCRUB`	`system:scrub`	The durable-object integrity scrub.

Current scope¶

What the current build does not yet do:

Connectivity probes are not wired. The control_plane and object_store components report unknown. Their readiness gate is not_down, so unknown does not by itself fail /readyz; on a schedulable, non-pressured host /readyz still returns 200. A real down verdict for connectivity cannot be asserted until the probes are wired. The cache reports ok with the detail corruption probe not wired.
Tracing is not exported. The redaction helpers are available, and the span chain is the target shape, but no OTLP exporter is wired. The metrics render path is the integration point.
Several audit categories have no emit site. The reserved LEASE, MOUNT_SESSION, and EXPORT resource types have no actions. Events are required for writer lease acquisition, revocation, and expiration; mount ticket issuance; mount and unmount; export; retention-policy changes; administrative recovery; and authorization failures for privileged operations. None of these are emitted yet. Workspace creation and fork also have no dedicated action.
Two histograms are recorded. awc_snapshot_validation_duration_seconds is recorded by awc-worker (no HTTP listener, so not scrapeable). awc_fuse_request_duration_seconds is recorded by awcd per served FUSE request (fuse feature). See metrics.
Many metrics are declared but not emitted. See the declared-not-emitted table on the metrics reference page.