Skip to content

Design decisions

AWC fixes twelve architecture decisions in spec Appendix F. They are not preferences. The spec labels them "architecture decisions fixed by this specification." This page walks the ones that carry a real trade-off: the choice, the alternative that was rejected, what the choice costs, and the risk that remains (spec §26). For the mechanism behind any one decision, read the deeper page this one links to.

The decisions are not independent. Most of them follow from one root choice: durable data is immutable and content-addressed (spec Appendix F.1). Once the truth is a set of immutable, hash-named objects, the rest of the shape falls out of it. Local writes become a private delta. Fork becomes a metadata pointer. Dedup becomes a key lookup. Publication becomes an atomic pointer swap. The sections below make those consequences explicit.

Immutable, content-addressed durable data

The first decision is that every durable object is immutable and addressed by the SHA-256 digest of its content (spec Appendix F.1, §9.3). The alternative is a mutable or block-addressed filesystem, where bytes live at a location and change in place.

Immutability is what makes the rest of the design cheap. Two snapshots that contain the same file share one object, so deduplication is a key lookup, not a scan (spec §3.1). A cached object is verified once, by recomputing its digest, and then trusted forever, because it cannot change (spec §6.2). A snapshot reaches READY only when every object reachable from its root is present and valid, and that closure never needs rechecking (spec §6.4).

The cost is that there is no in-place edit. Changing one byte of a file writes a new blob and a new chain of manifests up to the root. That is the trade the system makes on purpose. The risk it leaves behind is object count: a tree of millions of small files becomes millions of small objects, which stresses the object store and the garbage collector. Spec §26.5 names this directly, and small-object packing is deferred, not solved (spec §27.2). The MVP keeps the format simple until traces justify packing (spec §26.5).

For how content addressing works in detail, read Content addressing.

Local writes, explicit durability

Ordinary writes are local, and remote durability is an explicit checkpoint (spec Appendix F.4, §6.8). The rejected alternative is synchronous remote durability, where fsync(2) means the byte is safe off-host.

This choice exists to keep the writer cheap. A mounted workspace is an immutable lower plus a private local upper (spec Appendix F.5, §11.3). The agent writes to the upper with normal OverlayFS copy-on-write semantics. No write crosses the network, so no write waits on a quorum. That is what makes the writer disposable: lose the host, and you lose only work that was never checkpointed (spec §6.8).

The cost is a sharp split between what the agent sees and what survives. Inside a mount, fsync and fdatasync touch the local upper only (spec §14.8). They do not imply remote durability. A user who treats fsync as durability will be wrong. Spec §26.9 names this as a first-class risk, and the mitigation is documentation and CLI language, not a different design: the product must state the distinction in the CLI, the APIs, the mount metadata, and the dashboards (spec §26.9).

The consequence is that checkpointing is on the critical path of recoverability, not of correctness. An uncheckpointed workspace can be lost at any time. Only an acknowledged checkpoint is guaranteed to survive host loss (spec §6.8). For the full model, read The durability model.

One writer, single-authority publication

Each workspace has at most one active writer, and publication is single-authority (spec Appendix F.3, F.10). The rejected alternative is concurrent writers, where several processes or hosts publish to one workspace through a distributed-coherence protocol.

Concurrent writers would force the very thing the cache model exists to avoid. Two writers on one workspace need distributed locking, coherence, and conflict resolution. That is a shared mutable filesystem, which the spec refuses as a non-goal (spec §4.2). Picking one writer lets AWC drop all of that machinery.

Publication still has to be safe against a paused or partitioned old host. The spec solves this with a monotonically increasing fencing generation and an optimistic workspace version, not with distributed consensus (spec Appendix F.10, §14.4). A checkpoint commits only inside one PostgreSQL transaction that checks version, head, lease.generation, and mount_session together (spec §14.3). A stale host may keep writing to its local upper, but those bytes can never become the canonical head, because its generation fails the check (spec §14.4).

The residual risk is operational. A partitioned writer keeps mutating local state that will be discarded. An operator may need to recover that work through an explicit path, or accept its loss (spec §14.4, §17.11). For the full fencing model, read Fencing and split-brain.

PostgreSQL as the metadata authority

Mutable metadata lives in PostgreSQL, and the workspace head advances in one transaction (spec §7.2, §8.7, §14.3). The rejected alternative is a purpose-built consensus service, or a per-workspace Raft group.

PostgreSQL is on the publication path and the lease path. It is not on the read path (spec Appendix F.7, §7.2). That separation is what lets the control plane fail without stalling mounted workspaces: once a mount is established, reads and writes continue during a control-plane outage, subject to the local lease (spec §17.5). The database only has to be strongly consistent for the one operation that needs linearizability, which is moving a head.

The cost is a single metadata dependency and a scaling ceiling. Spec §26.10 names the risk that large workspace fan-out overloads PostgreSQL. The mitigation is to keep reads off the database hot path, batch forks, use compact rows, and partition the audit tables (spec §26.10). A specialized metadata service is deferred until measured need (spec §26.10). For how the planes split, read Architecture.

FUSE and OverlayFS, not a custom filesystem

The active Linux view is an immutable lower plus a private upper, built on FUSE and kernel OverlayFS (spec Appendix F.5, §15.6, §15.8). The rejected alternative is a custom filesystem, or a custom kernel module. The spec forbids a custom kernel module outright (spec §4.2).

Using the kernel OverlayFS for the writable composition avoids a second copy of overlay semantics inside FUSE. The immutable lower is served by one shared read-only FUSE mount per host, not one FUSE process per workspace (spec §8.4, Appendix F.7). That keeps the fast path in the kernel and on local NVMe (spec §7.3).

The cost is dependence on kernel behavior that the project does not control. OverlayFS stacked over FUSE is kernel-version-sensitive, and spec §26.1 names it as the first-listed risk. The mitigation is a certified kernel matrix, host admission probes, and a held-in-reserve fallback such as a merged writable FUSE path or eager lower materialization (spec §26.1, §28). The spec does not claim universal Linux compatibility.

The performance risk is real and measured. A cold read of many small files through FUSE is expensive: one pilot measurement reads 300 source files at 75.6x native time, because each file pays a cold-fetch cost. The feared tax on synchronous builds did not appear: a cold build measures 1.99x native and a warm build 1.06x, because compile compute dilutes the read cost. Spec §26.2 keeps the risk open and points at virtio-fs or a specialized kernel path only after traces show FUSE is the dominant cost. For the read and write model, read Overlay, cache, and singleflight.

Tenant-scoped deduplication

Deduplication is scoped to a tenant (spec Appendix F.2, §6.3, §16.3). The rejected alternative is a global content-addressed store that deduplicates across all tenants.

Global dedup would save storage when two tenants hold the same public base. It would also create a cross-tenant side channel. An attacker who controls upload timing can learn whether a digest already exists for another tenant, and so leak information about what that tenant stores (spec §16.3). The MVP refuses that trade. Every object-store request is scoped to a tenant prefix (spec §9.2, §16.3), and a digest collision across tenants does not imply a shared object (spec §9.2).

The cost is duplicate storage for identical content held by different tenants. The spec accepts that cost as the price of isolation. A future deployment may deduplicate across an explicitly configured organization trust domain, but only after a separate security review (spec §16.3). For the trust model, read Security model.

A shared, disposable host cache

The host-local cache is disposable and shared across all mounts on that host (spec Appendix F.8, §6.7). The rejected alternative is a rigid per-workspace cache partition.

A shared cache is what makes cross-workspace dedup pay off locally. Two workspaces from the same tenant that read the same object share one cache file, with no per-workspace copy (spec §4.1). Losing the entire cache costs performance only, never correctness (spec §6.7). Recovery is trivial, because there is nothing to recover.

The cost is contention. One scan-heavy workload can evict another tenant's hot set, which spec §26.7 names as a risk. The mitigation is segmented admission, per-tenant concurrency limits, optional reservations, and scheduling locality, and the spec explicitly avoids rigid per-workspace partitions (spec §26.7). The cache admission policy is the place where that tension is managed. For how singleflight and eviction fit together, read Overlay, cache, and singleflight.

Fork is metadata-only

Fork creates a logical workspace without copying file content (spec Appendix F.6, §6.10). The rejected alternative is a copy, however fast.

This decision is a direct consequence of immutability. Because the base is immutable and content-addressed, a new workspace only needs a metadata row that points at the base snapshot (spec §6.10). Fork cost stays independent of snapshot size. Any work proportional to the number or size of files is classified as import, mount warmup, or checkpoint work, not fork work (spec §6.10).

There is no real residual risk here. The invariant is a guard against drift. If a future change made fork proportional to size, it would break the economic premise of the product (spec §3.1). The invariant exists to make that regression visible.

The five operations are the whole surface

The product surface is exactly snapshot, fork, mount, checkpoint, and diff/export (spec Appendix F.12). Cross-host shared mutable POSIX semantics are out of scope (spec Appendix F.11, §4.2). These two are treated together in Scope discipline, because the refusal is the point. Each neighbor the product refuses (orchestration, sandboxing, a build system, a memory database) would widen the correctness kernel and blur the economic wedge. Read that page for the full argument and for the deferred capabilities in spec §27.

Known limits

Appendix F is normative. These are decisions fixed by the specification, not a description of a finished system. The current build does not yet match the spec everywhere. Treat each decision as a design commitment, and check the deeper page for what is implemented today.

  • Cross-host evidence is local so far. The fan-out acceptance target (spec §20.7) is itself a one-host test: 100 workspaces on one host cache domain. Measured evidence in this repo is single-host, too. The build-tax and read-tax numbers cited above come from one pilot host. The cross-host economic edge that motivates singleflight (coalescing the same read across hosts) needs a second host with a real network path to the object store, which the repo does not yet have.
  • The overlay fallback is a held design, not a built path. Spec §26.1 names a merged writable FUSE path or eager lower materialization as the fallback if OverlayFS-over-FUSE proves unreliable. That fallback is specified, not implemented.
  • Deferred capabilities stay deferred. Small-object packing (§27.2), live rebase (§27.5), virtio-fs (§27.8), and the other items in §27 are candidates that need economic evidence before they enter the surface (spec §26.12).