Skip to content

Architecture

AWC splits into three planes. A stateless control plane owns mutable metadata. A privileged data plane on each host serves bytes and builds checkpoints. A durable object store holds immutable content. The split exists because the system handles two different kinds of truth, and they scale differently (spec §7.2, §17.1).

Two stores of truth

AWC keeps mutable state and immutable bytes apart. PostgreSQL is the source of truth for mutable metadata: workspace heads, leases, states, and retention (spec §7.2, §8.7). The object store is the source of truth for immutable content: blobs, trees, files, and snapshots (spec §8.8). The store is content-addressed (a CAS), scoped per tenant (spec §9.2, §16.3).

These two stores do different work. A workspace head must move atomically and linearizably, so it advances in one PostgreSQL transaction (spec §14.3). File bytes are immutable, so they live in the object store, where they deduplicate and fetch by range (spec §8.8). Folding both into one system forces a bad choice. The transactional store would carry every byte, or the byte store would have to do transactions. AWC refuses both.

Host-local state is disposable, not truth (spec §3.2, §6.7). The local cache and the writable upper directory are acceleration and scratch space. Losing the cache costs performance, not correctness. Only an acknowledged checkpoint survives host loss (spec §6.8).

Why the control plane is off the read path

The control plane owns identities, authorization, leases, publication, and retention (spec §7.2, §8.1). It does not touch ordinary file reads. That choice is what lets a mounted workspace keep serving reads during a control-plane outage (spec §17.5).

A warm read stays in the kernel and on the host:

agent process
  → Linux VFS
  → shared read-only FUSE base
  → host-local CAS file
  → page cache / local NVMe

It needs no control-plane request, no database query, no object-store request, and no per-workspace FUSE process (spec §7.3). A cold read adds the parts that cost a round trip:

→ host cache lookup
→ singleflight coordinator
→ object storage
→ streaming digest verification
→ atomic cache install

Concurrent cold reads for the same (tenant, digest) on one host share one fetch (spec §7.4, §10.4). A singleflight coordinator sits over the one shared host cache. A hundred readers that miss the same object produce one remote fetch, not a hundred.

The object store, not the control plane, is the data dependency for reads. The control plane must not proxy file reads or writes (spec §8.1). Hosts fetch objects directly from the tenant-scoped CAS.

Why one daemon owns the host

awcd is one privileged Linux service per compute host (spec §8.3). It holds the shared read-only FUSE base mount, the host-local cache, the singleflight table, the OverlayFS mount manager, and the checkpoint builder. There is one multiplexed base filesystem for the whole host, not one FUSE daemon per workspace (spec §8.4).

The privilege is narrow and forced. The daemon needs capabilities for FUSE, OverlayFS, bind mounts, cgroup freeze, and namespace operations (spec §15.3). Nothing else needs those powers. So the control plane and the worker are kept stateless, unprivileged, and horizontally replaceable instead.

awc-control runs as stateless replicas behind a load balancer, with PostgreSQL as the only coordination dependency (spec §8.1, §21.2). awc-worker drains a PostgreSQL job queue with bounded, idempotent workers (spec §8.2). Neither mounts a filesystem or talks to the kernel. If a replica dies, another takes over with no host-local state to recover.

The spec also asks awcd to split its own privilege internally: a small privileged mount helper, an unprivileged cache and FUSE service, and hashing workers with no mount rights (spec §15.3). The current build runs awcd as one process, as the spec permits for the initial implementation. That is a scope decision for now, not a departure from the model.

How the planes connect

Mount authorization crosses the planes by capability, not by trust. The control plane issues a short-lived, signed mount ticket (spec §9.11, §13.8). awcd verifies the ticket without a synchronous control-plane call and fails closed if verification or host authorization fails (spec §8.3, §13.9). Internal host-to-control traffic uses gRPC over mutual TLS (spec §13.1).

On checkpoint, the data plane does the byte work and the control plane does the metadata work. awcd quiesces the writer, scans the upper delta, hashes and uploads new objects directly to the CAS, then asks the control plane to advance the workspace head in one transaction (spec §11.6, §13.12). Bytes go to the object store. The head moves in PostgreSQL. The two stores of truth stay on their own sides.

Known limits

These are architecture-relevant gaps between the spec and the committed build.

  • Control-plane REST surface. awc-control serves the core lifecycle over REST today: import sessions, snapshots with :diff, workspace create/fork/get, and mount-ticket issuance (T112, done). The internal gRPC LeaseService and CheckpointService are wired. Authentication is narrower than spec §13.2: service-account tokens verify, but OIDC discovery still resolves to an empty verifier, so id tokens do not yet authenticate.
  • Singleflight concurrency bounds. The singleflight coordinator is wired onto the read path, but the remote-read admission limiter and object-store circuit breakers are not attached yet. Coalescing works. A global fetch bound does not (spec §10.10, §10.11).
  • Worker job coverage. awc-worker runs snapshot validation, a decay scrub, workspace expiry, snapshot retention, and durable GC. Several job types in spec §8.2 are not yet separate handlers.
  • Cross-host evidence. The fan-out gate (spec §20.7) is defined for one host cache domain (100 workspaces on one host), and it is measured there: 100 concurrent readers fetch about 1.00878 times the remote bytes of one reader, under the 1.25 gate. On loopback that flat-byte result is the host object cache and the Linux page cache, not singleflight: the follower path stays idle on a single host, and coalescing is proven only by the regression gate. The cross-host economic case for singleflight — the reason the coordinator exists — is not yet measured; it needs a second host.