Overlay, cache, and singleflight

AWC serves a mounted workspace from an immutable base plus a private writable delta. The read path behind that view is layered, and the layers share one purpose. They keep the common case local, and they make many readers of one cold object pay for a single remote fetch. This page explains why the layers are arranged as they are, why OverlayFS carries the writes, and what the singleflight coordinator is worth today.

A read walks four layers

A read for an immutable object descends through four caches, cheapest first:

Linux page cache
  → awcd in-memory manifest cache
  → host-local object cache (NVMe)
  → object storage

Each layer answers a different question. The Linux page cache answers "did some process on this host already read these bytes?" The kernel owns it. The manifest cache answers "do I already know the structure for this path?" It holds decoded SNAPSHOT, TREE, and FILE manifests in memory, with a byte budget and a bounded negative cache. A storm of stat and readdir calls therefore never reaches the bytes. The host-local object cache answers "is the verified blob on local disk yet?" Object storage, last, is the durable source of truth.

The control plane is absent from this list on purpose. A read must not depend on PostgreSQL, on a lease check, or on an API call. Once a mount is authorized, reads and writes continue through a control-plane outage. Putting the database on the read path would make every fleet read share one scaling limit.

The layers are disposable by design. Losing every host-local cache affects performance only. It cannot lose an acknowledged snapshot. That license is what lets the cache serve an object without re-verifying it on every read. Verification happens once, at install. The daemon streams the object into a temporary file, computes SHA-256, and rejects any digest or length mismatch. Then it atomically renames the file into place. Every later read trusts that one check.

One shared FUSE base, one writable upper

All workspace lower directories on a host point into one read-only FUSE filesystem mounted at /var/lib/awc/basefs. Snapshots appear under digest-qualified paths:

/var/lib/awc/basefs/tenants/<tenant-id>/snapshots/<root-digest>/

There is one FUSE daemon per host, not one per workspace. A per-workspace FUSE process would multiply daemons and file descriptors with fan-out. That is the cost AWC exists to avoid. The base is read-only. A write attempt returns EROFS. All mutation happens above it.

The writable layer is OverlayFS. A mounted workspace is the kernel's merge of the read-only lower, the snapshot inside basefs, and a private upper directory:

merged = immutable lower snapshot + private local upper

OverlayFS is the native kernel mechanism for stacking a writable tree over a read-only one. The agent therefore sees an ordinary filesystem with local read-your-writes semantics. Writes land on the local upper only. They never synchronously reach object storage. Durability comes later, at checkpoint.

The initial OverlayFS configuration disables redirect_dir, metacopy, index, and xino. The choice trades performance for simplicity. With metacopy=off, a copy-up is a full data copy, not a metadata stub. Every changed file is then fully present in the upper, which keeps checkpoint delta interpretation simple and unambiguous. The price is local. On the pilot, editing one byte of a base file cost 13 to 23 times a native edit when cold. The kernel pulls the whole file up through the FUSE lower. That tax is the cost of a delta a checkpoint can read back without reconstructing kernel metadata.

Why singleflight

The motive is the thundering-herd problem. When many agents on one host all miss the same cold object, each would open its own fetch. The object store would see N times the load for one logical read. The spec states the inverse as the goal. Many readers of one object become a single remote fetch.

The fetch coordinator is the mechanism. It keeps an in-memory table of in-flight fetches keyed by (tenant, digest). The first miss for a key becomes the leader and runs the fetch. Every later miss for the same key, while the leader is still in flight, becomes a follower. The follower awaits the leader's result.

Three design choices matter, and each answers a failure mode.

The leader runs in a detached task, not on the requester's future. A follower that cancels, or a requester that became leader and then aborted, cannot strand the fetch while other followers wait. A drop guard removes the in-flight entry on every exit, success or failure. A dead leader is detected, and a follower takes over.

Followers share one outcome through a single-slot channel. They all observe the same verified bytes, or the same error. A failed fetch retries with bounded, jittered backoff while the failure is transient. A permanent failure is negatively cached for a short, bounded interval, so a broken object does not trigger a fresh fetch on every reader.

The coordinator bounds concurrent fetches, globally and per tenant, and trips a circuit breaker when the object store is unhealthy. The daemon attaches both through the remote-control hooks. A leader must obtain an admission permit before it contacts the store. When the queued-miss ceiling is reached, the permit request is retried with the same bounded, jittered backoff as a failed fetch, and a refusal after those retries is returned to the waiting readers but is never negatively cached: the next read of that digest may spawn a fresh leader at once. This keeps a large cold fan-out from poisoning digests. A backpressure refusal is transient by contract, so caching it would fail reads across the host for the negative-cache interval. The intended effect is that one slow tenant cannot consume the host fetch budget, and a down store is failed fast rather than hammered.

What is proven, and what is not

The coordinator is on the read path. Both the manifest source and the byte source route every read through it. The daemon builds both sources over one shared coordinator. This wiring is committed, not a working-tree change. A sampler folds the leader and follower counters into /metrics every five seconds.

On a single loopback host, however, the follower path is typically idle. The pilot staged 16 concurrent reads of identical content, released together by a pthread_barrier. Across six repeats the counters showed two leader fetches and zero followers. The cause is in the code. The read path checks the verified cache first. On loopback the leader's fetch finishes in roughly five to ten milliseconds. It installs the cache entry before the other readers reach the singleflight check, so they take the cache fast-path instead.

The follower path itself is proven, but by a regression test, not by live traffic. The test holds the leader's fetch open with a gated remote. That widens the window so a follower arrives while the leader is still in flight. Without that artificial gate, loopback is too fast for the window to open.

The value of singleflight is therefore cross-host. Real network round-trip widens the fetch window to tens of milliseconds, and many readers can land in it at once. That regime is not measured. The cross-host emulation with netem failed its methodology, and the pilot has no second host. The byte-amplification property itself holds on loopback, but there it is explained by the object cache plus the page cache, not by the follower path. The cross-host economic edge remains open.