Skip to content

See the economics

What this tutorial shows

AWC is an agent workspace cache. Its economic bet is narrow and specific. One base snapshot feeds many workspaces, and all of them share the same underlying bytes. In this tutorial you fan out 100 workspaces from one snapshot, mount a few, and read the metrics that collapse the remote-fetch bill.

You will see why N forks of one base cost about 1 times the remote bytes, not N times.

Before you start

You need:

  • a running awcd host daemon;
  • the awcctl CLI on your path;
  • a directory that holds a real repository image (a few hundred megabytes makes the effect visible).

If you do not have a daemon yet, follow install AWC and run AWC locally.

Step 1 — Import one base snapshot

A snapshot is an immutable, content-addressed image of a directory (spec §5.2). Pick a directory that represents a real repo, and import it.

awcctl snapshot create --from-dir ./my-repo --name econ-base --output json

The command prints a JSON object. Note three fields: snapshot_id, logical_bytes, and unique_bytes.

  • logical_bytes is the total file size a reader sees.
  • unique_bytes is the dedup-aware footprint. The content store keeps each distinct object once per tenant (spec §9.2).

The snapshot is not readable until it is READY. Poll it.

awcctl snapshot inspect <snapshot_id>

When state reads READY, the base snapshot S0 is in place. Every fork you make next will share its objects.

Step 2 — Fork one seed workspace

A fork is private and metadata-only. Forking does O(1) work regardless of the snapshot size (spec §20.2). Fork one seed workspace from the snapshot.

awcctl workspace fork --snapshot <snapshot_id> --ttl 2h --name econ-seed

Note the workspace id it returns. This seed is the anchor for the batch fork.

Step 3 — Batch-fork 100 workspaces from the seed

Now fan out. Batch-fork 100 workspaces from the seed, all sourced from its base snapshot.

awcctl workspace fork \
  --workspace <seed_id> \
  --from base \
  --count 100 \
  --name econ-agent \
  --ttl 2h

--from base makes every child share the seed's base snapshot S0. In batch mode --name is a prefix; the 100 children get distinct names. The --ttl flag is required. It sets how long a workspace lives before cleanup.

This matches the shape of the spec §20.7 fan-out scenario: many workspaces on one host cache domain, all sharing one base.

Step 4 — Mount several workspaces

Pick three children and mount them at separate targets. The target path must fall under the host allow-list.

awcctl workspace mount <child_id_1> --target /mnt/awc/ws-1
awcctl workspace mount <child_id_2> --target /mnt/awc/ws-2
awcctl workspace mount <child_id_3> --target /mnt/awc/ws-3

Each mount returns a mount_session_id. You use it to unmount later.

Step 5 — Trigger reads that hit the same bytes

Run the same scan in every mount. Read every file once.

find /mnt/awc/ws-1 -type f -print0 | xargs -0 cat > /dev/null
find /mnt/awc/ws-2 -type f -print0 | xargs -0 cat > /dev/null
find /mnt/awc/ws-3 -type f -print0 | xargs -0 cat > /dev/null

The three workspaces are separate forks, but they resolve the same paths to the same object digests. That is the key fact. The next step shows what it does to the remote-fetch count.

Step 6 — Read the metrics that collapse

awcd exposes a Prometheus-style metrics endpoint at /metrics on its health listener (spec §19.6). The default config binds that listener to 127.0.0.1:9100.

curl -s http://127.0.0.1:9100/metrics | grep -E 'awc_singleflight'

Two counters matter for this tutorial.

  • awc_singleflight_leader_fetches_total counts the fetches that ran as the single leader.
  • awc_singleflight_follower_requests_total counts the readers that waited on a leader and shared its result.

After the three scans, the leader count sits close to the number of distinct objects in S0. The second and third scans add no leader fetches. They hit the local cache. Singleflight coalesces concurrent misses for the same object into one leader fetch, and the followers share the result. In a full-tree scan, though, concurrent readers are usually at different files at any instant, so they rarely contend on one digest. On one host the follower count often stays near zero, and the flat bytes come from the caches, not from coalescing. The regression gate below is where coalescing is proven directly.

Why the counts behave this way

The cache key is tenant_id + hash_algorithm + object_digest (spec §10.3). All forks of one base, under one tenant, resolve a shared file to one key. The first miss fetches the object once (spec §10.5). The rest are cache hits.

The mechanism — three layers that flatten the bill

AWC collapses repeated remote fetches through three mechanisms.

  1. Tenant-scoped content addressing (spec §9.2, §10.3). Each object lives once per tenant. Identical bytes share one object, whether they appear in one fork or in a hundred.
  2. The cache hierarchy (spec §10.2). A read walks the Linux page cache, then the awcd in-memory manifest cache, then the host-local object cache, before it touches object storage. Once an object is cached, repeated reads are free of remote cost.
  3. Singleflight (spec §10.4). awcd keeps an in-memory map of active fetches keyed by cache key. When several readers miss the same object at once, one becomes the leader and fetches. The rest become followers and share the leader's result (spec §10.4: "followers MUST share the leader result"). The build routes the read path through the fetch coordinator, and the /metrics endpoint exposes the leader and follower counters (spec §19.2).

The measured numbers

The figures below come from two sources. The regression gates are deterministic tests over the software data path. The pilot run is one end-to-end measurement on a single loopback host. Both are published in the repo; cite them when you reuse a number.

Singleflight and fan-out

Quantity Measured value Source
100 concurrent readers of one object collapse to exactly 1 remote fetch; singleflight efficiency ≥ 0.99 singleflight regression gate (spec §20.6, §19.3)
100 concurrent scans of one base remote bytes ≤ 1.25 times one cold scan fan-out regression gate (spec §20.7)
N=100 fan-out on the pilot, remote bytes vs N=1 ratio 1.00878 (gate ≤ 1.25; PASS, margin 0.241) pilot measurement, experiment A

On the pilot, the wire bytes stayed in a 3.57 MiB band (405.98 to 409.55 MiB, a 0.88 percent spread) while the scan time grew 3.53 times from N=1 to N=100. Local FUSE and CPU work scaled with concurrency. Remote bytes did not.

Storage amplification

Quantity Measured value Source
Physical unique bytes after many forks base plus the sum of unique deltas, not forks times base delta_storage regression gate (spec §20.8, §19.3)
Storage amplification greater than 2.0 times delta_storage regression gate

Economic meaning: forking a 10 TiB snapshot 100 times costs 10 TiB + unique deltas in object storage, not 1000 TiB.

Cache hit rate and cold-versus-warm read

Quantity Measured value Source
Warm repeated reads 0 remote requests, a 100 percent hit rate cache_hit_rate regression gate (spec §20.6)
4 KiB cold read, p50 3.56 ms pilot read-latency measurement
4 KiB warm read, p50 19.7 microseconds pilot read-latency measurement
Cold-versus-warm ratio, p50 about 181 times pilot read-latency measurement

Where these numbers hold, and where they do not

The pilot fan-out figure (1.00878) was recorded on one loopback host with a local object store. On loopback, the host-local cache and the Linux page cache also flatten remote bytes, so the figure is a local result, not a cross-host proof. The regression gates prove the coalescing property directly in the software data path, with no page cache in the loop. A cross-host measurement against a real network — the regime where the spec §20.11 economic launch gate applies — is not in these sources.

Interpret the result

One base plus N forks costs about 1 times the remote bytes, not N times, for three reasons.

  • The base bytes are stored once (content addressing).
  • The base bytes are fetched once per host cache domain (the cache hierarchy).
  • Concurrent misses for the same object coalesce to one fetch (singleflight).

So the remote cost is O(distinct objects), not O(distinct objects times concurrency). This is why the spec says fork operations SHOULD be free or nearly free (spec §19.8). The bill tracks new content and cold reads, not the fork count.

Clean up

Unmount the three sessions. The default disposition discards local changes.

awcctl workspace unmount <mount_session_id_1>
awcctl workspace unmount <mount_session_id_2>
awcctl workspace unmount <mount_session_id_3>

The 100 workspaces expire when their --ttl elapses. To reclaim space sooner, or to learn what garbage collection keeps and drops, read run garbage collection.

What you learned

  • One base snapshot feeds many forks, and each fork is metadata-only.
  • Forks of one base resolve shared files to one cache key, so the bytes are stored and fetched once per host.
  • Singleflight coalesces concurrent misses into one leader fetch, and followers share the result.
  • The spec §20.7 fan-out gate (≤ 1.25 times one cold scan) passed at 1.00878 on the pilot, and the regression gates prove the coalescing property in the software path.

Where to go next