Skip to content

Fencing and split-brain

A workspace has one writer at a time. That rule is easy to state and hard to enforce when a host can be paused, partitioned, or simply slow. The writer may believe it still owns the workspace long after the control plane has moved on. Fencing is how AWC keeps that confused writer from corrupting the head.

The mechanism is one number: a fencing generation on each writer lease. This page explains why that number exists, what a stale writer sees, and why the obvious alternatives do not work.

The problem a generation solves

A read-write mount holds an exclusive writer lease. The lease lets one host publish checkpoints that advance the workspace head. "Exclusive" must survive failure. Consider what happens when host A freezes for a minute, or loses its network link.

While A is unreachable, the control plane cannot tell whether A is gone or merely paused. A new mount request may arrive for the same workspace. The control plane issues a fresh lease to host B. Now two hosts believe they can write. This is the split-brain window. If both can publish, the head can advance along two divergent histories, and the workspace has no single canonical state.

The lease cannot solve this by time alone. A's lease may still read as unexpired on A's own clock. Even if the control plane marks A's lease as expired, A may resume a second later, still convinced it is the active writer. Something must let B win and make A's attempt fail, no matter when A wakes up.

That thing is the generation.

Why a generation, not just a timer

Every new writer lease for a workspace gets a monotonically increasing generation. B's lease carries a generation strictly higher than A's. Publication is allowed only when the host presents the one generation the control plane granted for that lease. A host cannot invent a token. A writer that presents a stale generation is fenced off, and a writer that presents a higher generation than its lease holds is also rejected.

The decision does not depend on the writer's wall clock. Correctness rests on the generation and one database transaction, not on synchronized time. Wall clocks drive expiry and observability. They do not fence.

This is why lease expiration alone is not enough. A partitioned host holding an older generation is rejected even if it resumes later. The token it holds is now permanently behind.

The lease lifecycle

A lease moves through a small set of states, from ISSUED to ACTIVE and on to RELEASED, with a timeout path to EXPIRED and an admin path to REVOKED.

Only two states may publish: ACTIVE and RELEASING. RELEASING exists so a final checkpoint can publish during a clean unmount. The other states cannot advance the head.

The host drives the lifecycle. It activates its lease when the kernel mount succeeds, renews it in the background, and then releases or is fenced. Renewal is a control-plane RPC. The reply is one of four verdicts: extend the lease, remount read-only, revoke, or report the workspace gone. The last three demote the cached lease to a non-authorizing state. The host then fences on the next reconcile.

What publication checks

A checkpoint ends in one transaction that advances the head. Before it advances anything, that transaction checks that the writer is still the writer. It confirms the lease, its generation, its state, and the head the caller expected. If the check fails, the transaction rolls back and the head is unchanged.

The caller gets a distinct rejection, not a vague failure. The stable error codes include STALE_LEASE_GENERATION and WORKSPACE_VERSION_CONFLICT. A stale writer learns precisely that it no longer holds the workspace.

What a stale writer keeps and what it loses

Fencing stops publication. It does not delete the writer's work in progress. A stale host may keep modifying its local upper directory. Those bytes remain useful to the agent that wrote them. The merged mount is still a correct local view.

What those bytes cannot do is become the canonical head through any normal path. They leave only through an explicit, operator-assisted recovery that never advances the head automatically. The stale generation is fenced at the control plane. The host daemon also fences itself on restart: its reconciler detects a non-authorizing lease and refuses to publish from that generation.

This split is deliberate. The agent's local work is not destroyed from under it. But the single durable history is protected from a confused writer.

The rejected alternative: last writer wins

A simpler design would let any host that can reach the database publish. The last publication wins. The head converges on whoever wrote last. This is the "last writer wins" model.

It fails the workload contract. Two hosts that checkpoint the same workspace would clobber each other. A host partitioned for an hour could resume, publish its stale view, and silently overwrite hours of newer work. There is no durable history, only a race. AWC forbids this. At most one unexpired writer lease generation may publish, and every publication must carry that generation.

A second rejected design fences by time alone. The lease has an expiry. On expiry, the old host is presumed dead. This is the naive timer model. It fails because time is not authoritative across a partition. The frozen host's clock may disagree, or it may wake at the boundary and race. A monotonic token has no such failure mode. Once B's generation is higher, A's token can never catch up, no matter when A resumes.

How this connects to the rest of the model

Fencing is the publication half of a larger promise about durability. The other half is that the head advances atomically, or not at all, in one transaction. See the durability model for that side of the contract.

The lease state machine sits inside the wider workspace and mount-session lifecycle. See lifecycle states. When a host finds itself fenced, the operator recovery path is in handle a stale writer. The broader outage behavior is in recover from a control-plane outage.

Limits and what is not proven here

Be precise about what the evidence supports.

  • The fence is enforced in code and covered by unit tests. The tests reject a stale generation, a higher presented generation, a moved head, a moved version, a non-publishable state, an expired lease, a foreign workspace's lease, and a mismatched mount session.
  • The behavior is exercised beyond the unit tests. Fault-injection tests model a superseded lease and assert that a stale generation is rejected at publish and is not retried. Security and soak tests cover the stale-lease publication fence and credential rotation. The transactional path also has a database-gated integration test.
  • A live two-host split-brain has not been demonstrated in a recorded chaos test. Recorded measurements cover economics, cold-start, and amplification. They do not measure partition recovery or fencing under a real network split. Treat split-brain prevention as specified behavior confirmed against the code, not as a measured result.
  • Fencing protects the head. It does not reclaim the stale host's local disk usage. Those bytes persist on the host until the daemon fences the session, or until an operator exports and removes them.