Skip to content

Metadata normalization

AWC identifies a snapshot by a digest computed over its content. For two imports of the "same" tree to share storage, they must produce the same digest. Filesystems attach incidental metadata to every file: a modification time, an owner, a mode that drifts between checkouts. Left in, that metadata leaks into the digest and breaks deduplication. Normalization is the step that removes it before the manifests are hashed.

The digest depends on metadata, not just bytes

A FILE manifest hashes more than file content. Its canonical payload carries the POSIX mode and the optional mtime. A TREE manifest hashes the mode of every entry. So two files with identical bytes but different modes or timestamps get different FILE digests, and two otherwise equal directories then differ at the tree above them.

This is deliberate. Mode and timestamp are part of what a filesystem is, and a snapshot that ignored them could not reproduce a working tree at mount time. But it creates a problem. A fresh git clone of the same revision gives every file a new mtime. Without normalization, every clone would become a distinct snapshot, and the economic premise — that most agents share the same bytes — would not hold.

What normalization does

At import time, the builder captures each file's metadata, rewrites it according to a declared policy, and only then hashes the manifests. AWC ships two policies.

source-normalized rewrites metadata to the smallest set that still describes a usable source tree:

  • A regular file collapses to 0o755 if any execute bit is set, else 0o644.
  • A directory becomes 0o755. A symlink becomes 0o777.
  • File mtime is discarded.
  • setuid and setgid bits are dropped by the mode collapse.

This is the default for source repositories, where the only metadata that matters is whether a file runs. Two checkouts of the same revision — different mtimes, umask drift, a stray 0o600 — collapse to the same manifests and the same digest.

environment-preserving keeps what source-normalized throws away:

  • Mode bits are kept as captured, with setuid and setgid cleared by default.
  • File mtime is kept.

This policy exists for prepared dependency and build trees, where tools such as make and compilers read mtimes to decide what to rebuild. You trade some deduplication for metadata fidelity.

What is normalized and what is preserved

Normalization touches metadata. It does not touch content or structure.

Normalized: mode (under source-normalized), mtime (under source-normalized), and ownership. Owner and group normalize to zero under both policies. In the MVP this holds by omission: the FILE manifest schema has no owner field, so UID and GID are never recorded. Ownership reappears only at mount time, through the host UID and GID mapping, not from the snapshot.

Preserved: the file content bytes, the file type, the executable decision (under source-normalized) or the full mode (under environment-preserving), the symlink target bytes, and the filenames and directory shape. Filenames are kept as raw bytes and the tree entries are sorted lexicographically, so the path structure is deterministic without any rewriting of names.

The policy is part of the digest

The SNAPSHOT manifest carries the normalization_policy field inside its canonical bytes. The policy is therefore part of the snapshot digest. Two imports of the same tree under different policies produce different snapshot digests, on purpose.

This prevents an accidental cross-policy match. A source-normalized tree and an environment-preserving tree of the same files describe different logical states, because one has discarded mtimes. Deduplicating them would be wrong. Folding the policy into the digest keeps the two namespaces disjoint.

Choosing a policy

You pick the policy at import time with the --normalization flag:

awcctl snapshot create --from-dir /opt/prepared/backend \
  --normalization source-normalized

The accepted values are source-normalized (the default) and environment-preserving. The control plane defaults an unspecified policy to source-normalized.

The cost: provenance you cannot recover from metadata

Normalization buys determinism by discarding metadata. Once an import collapses a file to 0o644 and drops its mtime, the original mode and timestamp are gone. You cannot reconstruct who owned a file or when it was last touched by reading the snapshot.

AWC compensates by recording provenance that the metadata can no longer carry. The SNAPSHOT manifest has a source block (source type, locator, immutable revision) and a free-form labels map. The importer MUST record the normalization policy, the principal, and the source locator in provenance. The convention is to put the things normalization erases — a git revision, a build id, a base image digest — into labels, where they survive the collapse.

The split is the point. Content identity comes from normalized, deterministic metadata. Where the bytes came from comes from explicit labels. Mixing the two back into the digest would reintroduce the very drift normalization exists to remove.

Limits and what is not implemented

  • xattrs are not preserved in either policy. The spec says source-normalized removes unsupported xattrs and environment-preserving preserves only an allow-list. The MVP manifest schema has no xattr field, so xattrs are dropped by omission under both policies. ACLs and arbitrary xattrs are deferred. An environment-preserving import in the current build does not preserve xattrs.
  • UID and GID normalization is by omission. The spec normalizes owner to zero. The manifest carries no owner field, so this is satisfied trivially. Owner is not stored and cannot be recovered. It is applied fresh at mount time.
  • Determinism is unit-tested, not measured at scale. One test imports two directories whose files carry different modes (0o600 and 0o640) and asserts the snapshot digests are equal. There is no published measurement of deduplication ratios across real cross-host workloads.
  • Hard-link identity is not preserved. The spec permits hard links as independent entries. Link identity is not required.