Metrics reference¶

AWC exposes a bounded Prometheus-style metrics registry. Instrument names are defined by the telemetry module. This page reflects what the current build records.

Two registries ship: one in the awcd host daemon, one in the awc-worker job runner. Only awcd exposes a /metrics endpoint (see Scrape endpoint); awc-worker records into its registry but serves no HTTP, so its metrics are not scrapeable in this build.

Cardinality model¶

The registry caps distinct label-sets per instrument at 128. A recording that would add a new label-set beyond the cap rolls into the synthetic value cardinality_overflow instead of being dropped. Use closed-set label values only. Do not put a path, digest, workspace ID, or tenant ID in a label. The primary backend forbids these.

Emitted metrics¶

Every row below has a real recording site in non-test code. A declared name alone is not enough to qualify.

Name	Type	Labels	Binary	Meaning
`awc_workspace_mounts_total`	counter	none	`awcd`	Adds one for each accepted workspace mount.
`awc_workspace_unmounts_total`	counter	none	`awcd`	Adds one for each workspace unmount.
`awc_workspace_active_mounts`	gauge	none	`awcd`	Current count of active mount sessions. Set to `0` at start.
`awc_workspace_checkpoint_total`	counter	`outcome`	`awcd`	Adds one at each checkpoint stream terminal event.
`awc_host_disk_pressure_state`	gauge	none	`awcd`	Host disk-pressure severity rank, `0..=5`. Sampled every 5 s. See pressure ranks.
`awc_host_free_bytes`	gauge	none	`awcd`	Free bytes on the filesystem holding the cache. Sampled every 5 s.
`awc_host_free_inodes`	gauge	none	`awcd`	Free inodes on that filesystem. Sampled every 5 s.
`awc_host_upper_usage_bytes`	gauge	none	`awcd`	Bytes held in uncheckpointed upper directories. Sampled every 5 s.
`awc_host_cache_bytes`	gauge	none	`awcd` (`fuse` feature)	Total verified bytes in the cache index. Sampled every 30 s.
`awc_cache_hits_total`	counter	none	`awcd` (`fuse` feature)	Reads served from a verified cached object. Folded every 5 s.
`awc_cache_misses_total`	counter	none	`awcd` (`fuse` feature)	Read passes that went to the singleflight path. Folded every 5 s.
`awc_singleflight_leader_fetches_total`	counter	none	`awcd` (`fuse` feature)	Cumulative singleflight leader fetches. Folded from the fetch coordinator every 5 s.
`awc_singleflight_follower_requests_total`	counter	none	`awcd` (`fuse` feature)	Cumulative singleflight follower requests. Folded every 5 s.
`awc_fuse_request_duration_seconds`	histogram	`op`	`awcd` (`fuse` feature)	Wall-clock duration of one served FUSE request, recorded at completion.
`awc_fuse_inflight_requests`	gauge	none	`awcd` (`fuse` feature)	FUSE requests currently being served. Sampled every 5 s; set to `0` at start.
`awc_lease_renewal_failures_total`	counter	none	`awcd`	Lease-bearing sessions whose renewal RPC failed. Counted per renewal pass.
`awc_background_jobs_total`	counter	`kind`, `outcome`	`awc-worker`	Adds one for each processed background job.
`awc_background_job_lag_seconds`	gauge	`kind`	`awc-worker`	Ready-to-processed lag for the most recent job of each kind, in seconds.
`awc_api_errors_total`	counter	`component=worker`	`awc-worker`	Adds one for each transient worker store error.
`awc_snapshot_validation_duration_seconds`	histogram	`outcome`	`awc-worker`	Wall-clock duration of one snapshot validation.
`awc_workspace_forks_total`	counter	none	`awc-control`	Adds the batch size for each committed workspace fork. Idempotent replays do not count.
`awc_active_writer_leases`	gauge	none	`awc-control`	Live writer leases (issued, active, or releasing, and unexpired). Sampled every 15 s; set to `0` at start.

Label value sets:

awc_workspace_checkpoint_total outcome: one of published, failed, cancelled.
awc_background_jobs_total kind: the job's kind string from the job row.
awc_background_jobs_total outcome: one of completed, failed, cancelled, retried, other.
awc_api_errors_total component: the fixed value worker.
awc_snapshot_validation_duration_seconds outcome: one of validated, invalid, skipped, error.
awc_fuse_request_duration_seconds op: one of lookup, getattr, read, readdir, readlink, statfs.

The singleflight and cache counters are gated behind the fuse build feature. The sampler reads cumulative counters from the fetch coordinator and folds their deltas into the registry. The FUSE instruments share that gate: awc_fuse_request_duration_seconds is recorded directly in the FUSE mount thread as each request completes, and awc_fuse_inflight_requests is folded by the 5 s sampler.

A cache miss counts one per get_or_fetch pass that reaches the singleflight path, not per RPC: a pass whose leader died and retried counts one miss, and if the retry finds the object verified it also counts one hit. Hits and misses therefore do not sum to a request count.

The free-space gauges come from the newest successful probe sample. A probe that keeps failing leaves the last value in place. awc_lease_renewal_failures_total counts sessions, not passes: one pass over two unreachable sessions adds two.

Singleflight counters are a cross-host signal

On one loopback host, the local cache fast-path serves a repeat read before a second request joins the in-flight fetch. Expect awc_singleflight_follower_requests_total to stay at 0 in single-host traffic. The counters measure cross-host coalescing, where two hosts race for the same missing object.

Pressure ranks¶

awc_host_disk_pressure_state records the pressure severity as an integer. The states escalate monotonically.

Value	State
`0`	`NORMAL`
`1`	`CACHE_EVICTION`
`2`	`PREFETCH_DISABLED`
`3`	`NEW_MOUNTS_DISABLED`
`4`	`UPPER_WRITES_AT_RISK`
`5`	`EMERGENCY`

Declared but not emitted¶

Every row below is declared in the telemetry module but has no recording site in shipped code. Each stays unemitted for a concrete reason:

Name	Type	Intended meaning	Why it is not emitted
`awc_workspace_checkpoint_uploaded_bytes`	counter	Unique bytes a checkpoint uploads.	The checkpoint result reports zeros for the upload/reuse split today; emitting them would lie. Lands with a `CheckpointStats` split.
`awc_workspace_checkpoint_reused_bytes`	counter	Bytes a checkpoint reuses from prior objects.	Same as above: the split does not exist in the stats yet.
`awc_host_cache_evictions_total`	counter	Cache evictions.	No shipped binary invokes the eviction policy.
`awc_host_cache_corruption_total`	counter	Detected cache corruption events.	No shipped binary runs cache reconcile (the corruption detector) on a schedule.

Notes on this table:

The declared histogram bucket upper bounds are 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0 seconds.
The metric catalog lists more names than the code declares (for example awc_read_logical_bytes_total, awc_workspace_mount_duration_seconds, awc_snapshot_logical_bytes). Those have no declared constant and no recording site. They are omitted here.

Scrape endpoint¶

GET /metrics on the awcd health HTTP listener returns the Prometheus text exposition. The same listener serves three routes:

Route	Returns
`GET /healthz`	Liveness. `200` when the process is up.
`GET /readyz`	Readiness for new writable mounts. Preserves the warm-read flag.
`GET /metrics`	Prometheus text exposition.

The listener is off by default. The host API stays on its Unix-domain socket; only health and metrics use the HTTP listener.

Knob	Where	Value
Config file key	`health_http_listen`	A loopback `host:port`, for example `127.0.0.1:9100`.
Environment variable	`AWC_HEALTH_HTTP_LISTEN`	Same format as the file key.
Type	`Option<SocketAddr>`	Optional.
Default	`None`	The listener is not started.

awcd starts the listener only when health_http_listen is set. The value must parse as a host:port, or awcd returns a config error.

awc-control serves GET /metrics on its REST listener (AWC_CONTROL_LISTEN), alongside /v1/health. The route is unauthenticated, like the liveness probe: bind the listener to a private interface. It exposes the two awc-control rows above. A failed lease sample keeps the last gauge value and logs.

Current scope¶

What the current build does not yet do:

The FUSE instruments cover the six read-path operations only (lookup, getattr, read, readdir, readlink, statfs). Other kernel callbacks, including the mutating operations that reply EROFS, are not timed.
Cache eviction and corruption have no recording sites. awc_host_cache_evictions_total and awc_host_cache_corruption_total stay unset until a binary schedules eviction and reconcile.
Only a Prometheus text exposition is implemented. An OTLP exporter is not layered on. The render path is the integration point for one.
awc-worker records its metrics (awc_background_jobs_total, awc_background_job_lag_seconds, awc_api_errors_total, awc_snapshot_validation_duration_seconds) into an in-process registry but starts no HTTP listener. These counters have no /metrics scrape target in this build; wire them to an exporter before relying on them.