Limits, quotas, and backpressure¶

This page lists every limit, quota, and backpressure signal that AWC defines. Each entry names the constant, the default, the knob you tune, and whether the current build enforces it.

Two error classes follow from every limit:

Class	Meaning	Error code	Retry
`Capacity`	A shared ceiling is full. Retry after backoff.	`RESOURCE_EXHAUSTED` (HTTP 429)	Transient
`Structural`	The request shape itself is too large. Retry cannot help.	`INVALID_ARGUMENT` (HTTP 422)	Permanent

See error codes for the full code set.

Disk-pressure state ladder¶

awcd drives the host through six monotone pressure states as free bytes, free inodes, and PSI stall fractions cross configured thresholds. Containment fires in order: the cheap, safe lever runs first.

Severity	State	Wire label	Containment action
0	`Normal`	`NORMAL`	None.
1	`CacheEviction`	`CACHE_EVICTION`	Reclaim disposable cache objects.
2	`PrefetchDisabled`	`PREFETCH_DISABLED`	Stop speculative prefetch. Demand reads continue.
3	`NewMountsDisabled`	`NEW_MOUNTS_DISABLED`	Reject new writable mounts.
4	`UpperWritesAtRisk`	`UPPER_WRITES_AT_RISK`	Existing upper writes may fail with `ENOSPC`. Warn agent and platform.
5	`Emergency`	`EMERGENCY`	Only warm reads continue.

Inode scarcity and PSI can only escalate the byte-driven state (take the max). They never relax it. Workspace uppers are never silently deleted; uncheckpointed data is preserved for an operator.

Byte thresholds¶

The five byte thresholds are the free-bytes levels at or below which each state triggers. They must be strictly ordered from mildest to most severe. The daemon derives a correctly spaced ladder from one operator reserve, typically the resolved cache.min_free_bytes.

Threshold	Multiplier of the reserve	At default reserve (50 GiB)
Eviction	3×	150 GiB
Prefetch disabled	2×	100 GiB
New mounts disabled	1× (equals the reserve)	50 GiB
Upper writes at risk	2/3×	~33.3 GiB
Emergency	1/3×	~16.7 GiB
Checkpoint reserve	1× (the always-preserved floor)	50 GiB

Inode and PSI escalators¶

Input	Field	Default	Effect
Free inodes	`new_mounts_disabled_free_inodes`	1024	Free inodes at or below this escalate the state to at least `NewMountsDisabled`.
PSI "some" stall	`prefetch_disabled_psi_some_bp`	5000 bp (50%)	Stall at or above this disables prefetch.
PSI "full" stall	`new_mounts_disabled_psi_full_bp`	8000 bp (80%)	Stall at or above this disables new mounts.

A PSI value of None (the host does not expose /proc/pressure/*) disables that escalator. An inode count of 0 (unknown) is ignored, not treated as scarcity.

Reserve enforcement¶

New writable mounts and checkpoints need the metadata and checkpoint temporary file reserve (checkpoint_reserve_bytes). The reserve admission check tests both the pressure state and the reserve. A refusal returns a visible reserve error that carries the current free bytes, the required reserve, and the pressure state.

Uppers are never deleted to relieve pressure

Cache eviction is the only space-freeing lever. A disk-full condition is surfaced as an error, never papered over by deleting uncheckpointed upper data. See handle disk pressure.

Status: the shipped awcd daemon samples pressure, emits the awc_host_disk_pressure_state gauge, and gates new writable mounts on the reserve check on the host mount path.

Cache admission and eviction¶

The host cache uses a segmented policy that distinguishes one-hit streaming objects, repeatedly accessed hot objects, metadata objects, and explicitly pinned objects. A simple segmented LRU is the current policy; cache-policy learning is deferred.

Eviction order, cheapest first:

one-hit streaming objects (oldest access first);
hot objects (oldest access first);
metadata objects.

Pinned, installing, corrupt, and already-evicted objects are never selected.

Setting	Default	Purpose
Hard byte limit	512 MiB	Hard ceiling on total verified bytes. Admission past this triggers eviction.
Free-space reserve	2 GiB	Minimum free bytes preserved on the cache filesystem. Eviction frees space until this much is available.

Real deployments override both from cache.max_bytes and cache.min_free_bytes (defaults 1 TiB and 50 GiB; see configuration).

Minimum behavior the policy satisfies:

enforce a hard byte limit;
preserve the configured free-space reserve;
prefer evicting old one-hit objects before reused objects;
never evict a partially installed object as if it were valid;
permit deletion of an object that remains open (Unix file-handle semantics);
tombstone the row as Evicted; reconciliation later compacts tombstones.

Status: the policy exists in the library but no call site triggers eviction on admission in the shipped awcd daemon. Cache size is not actively reclaimed by this policy at runtime.

Remote-read backpressure¶

The fetch path bounds remote reads so it never creates unbounded tasks or connections. The admission layer is a concurrency limiter. Every bound is a tokio semaphore, so permit acquisition and drop are cancel-safe.

Setting	Default	Knob in `cache` config	Bound
Global concurrent	64	`max_remote_fetches` (default 128)	Total remote reads in flight across all tenants.
Per-tenant concurrent	16	`max_remote_fetches_per_tenant` (default 32)	Remote reads in flight for one tenant.
Prefetch concurrent	8	`max_prefetch_fetches` (default 8)	Sub-capacity only prefetch pays. Demand ignores it.
Bytes in flight	256 MiB	Library default	Hard ceiling on bytes admitted to flight at once.
Default size reservation	1 MiB	Library default	Bytes reserved when a fetch size is unknown.
Queued miss limit	256	Library default	Requests waiting for a permit before new ones are refused.

The config bridge maps the three operator concurrency knobs onto the limiter. A zero value (no remote reads permitted) would deadlock a semaphore, so it is clamped to one permit.

Interactive demand reads take priority over speculative prefetch. Demand waits for room; prefetch is refused (and should retry later) when its reserved concurrency or byte budget is unavailable.

Condition	When	Who is refused
Queue saturated	Distinct cold misses exceed the queued miss limit.	Any new waiter.
Prefetch throttled	Prefetch would exceed its reserved concurrency or byte budget.	Prefetch only. Demand is never refused this way.

Both conditions are transient. The caller backs off and retries.

Observability counters on the limiter: queued, in_flight, bytes_in_flight, demand_admitted, prefetch_admitted, prefetch_throttled, queued_refused. See metrics.

Status: the limiter exists in the library. The fetch coordinator acquires a permit on each leader fetch only when a limiter is attached. The shipped awcd daemon constructs the coordinator without a limiter, so this bound is dormant at runtime until the daemon attaches one.

Host quotas and denial-of-service limits¶

The host and API operational quota policy uses the limits below. A rate field set to 0 means unlimited.

Setting	Default	Class	Enforced by
`max_mounts_per_host`	256	Capacity	`awcd` active-mount count check on the host mount path.
`max_mounts_per_tenant`	64	Capacity	Not enforced by `awcd`. Deferred to the control-plane mount-ticket path.
`max_mount_rate_per_second`	32	Capacity	`awcd` host mount path rate check.
`max_mount_burst`	64	Capacity	`awcd` host mount path rate check.
`max_upload_bytes`	100 GiB	Capacity	Per-session ceiling.
`max_upload_concurrency`	8	Capacity	Declared. No enforcement helper.
`max_export_bytes`	50 GiB	Capacity	Export size check.
`api_rate_per_second`	0 (unlimited)	Capacity	Per-key token bucket.
`api_burst`	0 (unlimited)	Capacity	Per-key burst allowance.

A 0 for either mount dimension means unlimited for that dimension.

The rate limiter is a per-key (typically tenant) token bucket. Buckets are created lazily and start full. A backwards clock skew never robs tokens; the elapsed term is clamped at zero.

Status: the shipped awcd daemon enforces max_mounts_per_host (direct active-mount count check) and the mount-rate gate (max_mount_rate_per_second / max_mount_burst) on the host mount path. Per-tenant mount counting has no call site in the daemon; per-tenant mount isolation is deferred to the control-plane mount-ticket path.

Structural content limits¶

Structural limits bound the work and storage a single import can demand. They are intrinsic to the import operation. A breach is Structural, so it maps to INVALID_ARGUMENT and is permanent.

Setting	Default	Bound
Path component bytes	255 (POSIX `NAME_MAX`)	Bytes in one filename.
Path depth	64	Directory nesting depth. The root is depth 0.
Entries per directory tree	100000	Entries in one directory tree.
File bytes	64 GiB	Logical size of one regular file.
Snapshot logical bytes	1 TiB	Total logical size of the imported snapshot.
Objects per import	5000000	Distinct objects one import may produce.

Status: enforced during import. The depth and entry ceilings mirror the manifest graph budgets.

Checkpoint and lease timing¶

Checkpoint and lease limits are resolved and validated at startup. Startup rejects contradictory or unsafe settings.

Setting	Default	Validation
`max_concurrent`	4	Must be greater than zero.
`freeze_timeout_seconds`	10	Must be greater than zero.
`preserve_local_grace_seconds`	3600	Must be greater than zero.

Setting	Default	Validation
`renew_interval_seconds`	30	Must be strictly less than `expiry_seconds`.
`expiry_seconds`	90	Must be greater than `renew_interval_seconds`.
`disconnected_grace_seconds`	60	Must be greater than zero.

A writer lease carries a monotonically increasing generation fencing token. A host with an older generation may keep local bytes but must not publish a checkpoint. EXPIRED, REVOKED, and RELEASED leases cannot publish.

Status: lease lifecycle is owned by the control plane.

Idempotency-key retention¶

Every mutation API supports an Idempotency-Key header or field. Retrying publication with the same key returns the already published snapshot.

The control plane stores idempotency keys in the idempotency_keys table:

idempotency_keys(
    tenant_id, principal_id, operation, key,
    request_hash, response_blob, state, expires_at,
    primary key(tenant_id, principal_id, operation, key)
)

Each row carries an expires_at. Local checkpoint state also records phases and idempotency keys for interrupted-checkpoint recovery.

Status: no operator-facing retention knob is defined; the retention period is implementation-owned.

Per-tenant and per-project boundaries¶

Boundary	Mechanism	Status
Mount count per host	Active-mount count check (`max_mounts_per_host`).	Enforced by `awcd` on the host mount path.
Mount count per tenant	Per-tenant mount counter (`max_mounts_per_tenant`).	Not enforced by `awcd`; deferred to the control-plane mount-ticket path.
Remote reads per tenant	Per-tenant semaphore (`max_remote_fetches_per_tenant`).	Implemented; dormant in `awcd` (the limiter is not attached to the coordinator).
API request rate per tenant	Per-key token bucket (`api_rate_per_second`, `api_burst`).	Implemented; not wired per-tenant at runtime. Defaults to unlimited.
Upper-directory bytes and inodes	Filesystem project quotas or dedicated filesystem quotas.	Operator-provided. AWC recommends them and does not enforce a counter.
Cross-tenant cache deduplication	Forbidden. Objects are namespaced under `objects/<tenant>/`.	Enforced by the cache layout.

Per-tenant concurrency and bandwidth controls should be implemented in the daemon. A malicious workspace must not exhaust host disk and take down unrelated tenants.

Current scope¶

This page distinguishes what the code implements from what the shipped daemon enforces at runtime.

Implemented in library code (the enforcement logic exists):

The disk-pressure state machine, the reserve check, and the visible reserve error.
The segmented-LRU cache admission, eviction, and pinning policy.
The remote-read backpressure limiter and the config bridge.
The host quota primitives: mount counting, rate limiting, and upload and export size checks.
Checkpoint, lease, and cache limit resolution and validation.

Enforced at runtime:

Disk-pressure sampling, the awc_host_disk_pressure_state gauge, and the reserve gate on the host mount path.
max_mounts_per_host via a direct active-mount count check, and the mount-rate gate (max_mount_rate_per_second, max_mount_burst), both on the host mount path.
Structural content limits on the import path.

Implemented but dormant at runtime in awcd:

The eviction policy. No call site triggers eviction on admission, so the cache size is not actively reclaimed by this policy in the shipped daemon.
The remote-read concurrency limiter. The fetch coordinator acquires a permit only when a limiter is attached; awcd constructs the coordinator without one, so the bound is inactive.

Declared, with no library helper and no awcd call site:

max_upload_concurrency.
max_upload_bytes and max_export_bytes.
max_mounts_per_tenant. Per-tenant mount isolation is deferred to the control-plane mount-ticket path.

Not configurable in the documented configuration block (library defaults only):

Bytes in flight, default size reservation, and queued miss limit.
The idempotency-key retention period.

Specified, deferred (not implemented):

Cache-policy learning, content-defined chunking, and learned prefetch are deferred, not implemented.