Measuring AWC

AWC is a cache. That makes it easy to benchmark badly. Warm the cache, compare against a naive full copy, and report one headline number. That run will flatter AWC, cheat AWC, or both. This page explains why the measurement discipline is the way it is, and where the measured evidence stops.

Why a naive benchmark lies

The AWC win is a fan-out win. One agent reading one base pays a local tax for routing reads through a FUSE content cache. Ten agents reading the same base, or one agent forking a base it already paid for, get the reuse. The economic bet is that physical storage is approximately base plus unique deltas, not workspace count times complete size. Many agents requesting the same object on one cache domain collapse to one remote fetch.

Two bad baselines follow from misunderstanding that shape. The first is a plain cp -a or a full eager download as the comparison. Both pay the whole base for every workspace. Against them, AWC looks like a large win at fan-out even when it does nothing clever. At a fan-out of one, AWC loses to a plain copy, because it pays a FUSE tax a plain copy does not.

The second bad baseline is the opposite error. A local OverlayFS mount gives base amortization away for free. If you compare AWC to OverlayFS on one host, AWC loses, because it pays a FUSE tax OverlayFS does not. The apparent amplification is generic base amortization, not an AWC feature. OverlayFS delivers it with no FUSE cost.

A fair measurement needs both baselines, and it needs to say which regime each applies to. The naive full copy is the cross-host baseline, where OverlayFS structurally cannot follow because the bytes must cross a network. OverlayFS is the honest local baseline, and locally AWC is candid about being slower.

Cache state is the variable you are not measuring

A warm read and a cold read of the same object differ by orders of magnitude. A number that does not say which state produced it says nothing.

AWC stacks three caches: the Linux page cache, the daemon in-memory manifest cache, and the host-local object cache on disk, in front of the object store. A cold result must name which of these were cleared. No single headline result may mix cold and warm states. A run must also report the environment that sets those states: kernel, NVMe model, object-store region, network bandwidth, snapshot file count, and read pattern.

On a live pilot you usually cannot clear the on-disk object cache while the daemon runs. So a trial is often page-cache cold but object-cache warm. That understates the true first-command cost. The coldest trial, not the median, is the honest first-command number.

This is the discipline: fix the variables you are not testing. Same host, same repository, same agent, same model, same prompts, same evaluator. Fix concurrency, and report queue time separately from service time so a saturated scheduler does not masquerade as a slow read. Use unique idempotency keys per trial so a fork does not replay a prior workspace. Drop the page cache before every cold trial, and verify it actually ran. Each control removes one alternative explanation for the number.

Correctness is absolute, performance is conditional, economics is open

The targets fall into three classes, and a fair measurement treats them differently.

Correctness gates are absolute and must never regress. Fork work must be independent of snapshot logical size. A ready snapshot must never reference a missing object. Injected corruption must be detected before bytes are returned. Every failure injection must yield either the old head or the complete new head. A stale lease generation must fail to publish. These are not latency questions. They hold on any hardware. The failure-injection and recovery suites exercise the integrity gates: snapshot closure, digest correctness, checkpoint atomicity, and fencing. The count-based invariants — fork invariance, delta storage, fan-out coalescing, and singleflight suppression — are asserted as counts and ratios, not wall clock.

Performance targets are conditional on a published reference host. Fork latency has a budget. A batch of forks has a budget. Warm mount has a budget, excluding sandbox startup. Cold metadata mount for a large snapshot has a budget, with no full materialization. Many simultaneous reads of one absent object must produce one underlying fetch per host and tenant. These are service targets measured on a named environment. They are not guarantees across arbitrary customer networks.

Economic gates are the open ones. The fan-out test bounds the remote bytes many workspaces pull when they scan one base. The delta-storage test asks that many forks, each changing a little, cost about base plus unique deltas in physical storage, not the total that workspace count times base would imply. The launch gate asks for a clear improvement on at least one real design-partner workload. The last is not measured.

The validation ladder

You cannot interpret an economic number unless the rungs below it hold. The specification orders its own evidence that way, from absolute correctness up to contingent economics. The measurement follows the same ladder.

The contract rung is the correctness set: fork invariance, snapshot closure, digest correctness, checkpoint atomicity, and fencing. If these fail, no latency number matters. The lifecycle rung is the set of state machines: snapshot, workspace, mount session, and lease. Measuring here means showing transitions are clean, and that a ready snapshot, an active lease, and a mounted writer behave as defined.

The crash-matrix rung is the failure matrix. Every failure path must preserve the last acknowledged workspace head. With the control plane down past its grace and expiry windows, durability writes complete with zero errors, and reads and writes continue during the outage. New mounts and checkpoints fail as expected, because both need the control plane. A checkpoint that crosses the expiry window is rejected after recovery. You can work through a long outage, but you cannot publish that work until the lease is re-established.

The sparse-scale rung is the delta-storage test. The workload and economics rung is the fan-out test and the launch gate. The attribution rung is the set of product ratios, backed by the storage accounting. Attribution is what stops a headline from crediting the wrong mechanism.

Attribute the win to the right mechanism

A flat-remote-bytes result has at least three causes on one host: the object cache, the Linux page cache, and singleflight. They are not interchangeable. The measurement gives you the ratios to tell them apart. Storage amplification is logical retained divided by tenant-unique durable bytes. The I/O suppression ratio is one minus actual remote bytes over logical immutable bytes requested. Singleflight efficiency is follower requests served over total cold-miss requesters. Delta efficiency is checkpoint unique uploaded bytes over logical workspace bytes.

Attribution matters because a flat-bytes result on one host can be explained by the caches alone. The deterministic gates prove coalescing directly in the software data path, with no page cache in the loop. Many concurrent readers of one absent object produce exactly one remote fetch, and warm repeated reads make no remote requests. When you report a number, say which mechanism produced it.

Measure the first useful command, not the fork

Fork latency is metadata work. It is the wrong headline. The metric that matters is the time from a fork request to the successful completion of a command that represents useful readiness, such as git status, rg, pytest --collect-only, or cargo metadata.

This matters because the local taxes show up here, not in the fork. A cold build costs more than native, and warm collapses toward native. A read- or transform-heavy cold build costs more than a build-heavy one. The pure-read tax is larger and less stable, because the per-file cold-fetch cost dominates when the native work is small. Editing a base file pays copy-up. metacopy=off is forced, so the first cold edit of a base file is expensive. The fork is instant. The command after it is not.

For how these local taxes stack against the cross-host reuse that pays them back, read economics and cost model. For the read and write path that sets the cold-versus-warm gap, read overlay, cache, and singleflight.

Report what you actually proved

The honest measurement states its scope. Three layers carry different weight.

The regression gates are deterministic and hold on any host. They assert the ratios and counts a release rates on: singleflight suppression, warm cache hit rate, remote requests equal to distinct objects, fan-out coalescing, delta storage as base plus unique deltas, fork invariance, and checkpoint scaling with changes rather than base.

The reference-host wall clock is conditional on a named environment. The runner measures the software path against an in-memory store. It isolates FUSE, NVMe, and the object-store network. It is not the same as the service targets, and it must not be compared to those targets. Absolute latency needs the published reference environment.

The cross-host claim is open. The economic launch gate needs a second host, a real network, the coordinator on the read path, and its counters published. Until a real cross-host run exists, the cross-host edge stays a thesis.

Read this before you quote an AWC number

State the cache. Name the baseline. Pick the rung. One-host numbers are candid, and locally AWC is slower than plain OverlayFS for cold reads and for editing base files. The fan-out and delta-storage properties are proven in the software data path. The cross-host economic launch gate is not yet measured.

For what survives the loss of a host, which any cross-host measurement assumes, read the durability model.