Skip to content

Deploy on Proxmox VE

This is the supported first-test topology for AWC: one Debian 12 or Ubuntu 24.04 KVM VM, with PostgreSQL in Docker, the AWC binaries running natively under systemd, and the object store either loopback MinIO (default) or a Backblaze B2 bucket. Everything local binds to loopback. Plain HTTP for the VM-internal services is deliberate for this local pilot; do not expose its ports outside the VM. The B2 path is HTTPS end to end.

The pilot exercises the real product path: S3-compatible CAS, shared read-only FUSE base filesystem, per-workspace OverlayFS, signed mount tickets, writer leases, cooperative checkpoints, and remount from the published head.

Scope and security boundary

Included:

  • One KVM VM and one awcd host identity.
  • PostgreSQL 16 on 127.0.0.1 via Docker Compose, plus MinIO on 127.0.0.1 (default) or a Backblaze B2 bucket reached over HTTPS.
  • awc-control, awc-worker, and awcd as native systemd services.
  • Plain HTTP for REST, internal gRPC, and MinIO inside the VM.
  • Fixed development identity and signing key selected by AWC_DEV_MODE=1.
  • Cooperative checkpoints. They do not require an execution cgroup.

Deferred for a later deployment:

  • HTTPS/mTLS for the control plane, OIDC, S3 providers other than the supported B2 option, multi-host scheduling, HA, Kubernetes, sandbox namespace injection, and quiesced checkpoints.

AWC_DEV_MODE=1 is not a production security mode. Keep ports 5432, 8080, 8081, 9000, and 9001 on loopback and restrict SSH access to the VM.

VM requirements

Use a KVM VM, not an unprivileged LXC container. Recommended minimums are 4 vCPUs, 8 GiB RAM, and an 80 GiB system disk. The guest needs:

  • Linux kernel 6.1 or newer with cgroup v2, FUSE, and OverlayFS.
  • ext4 or xfs for /var/lib/awc.
  • /dev/fuse, fuse3, findmnt, Docker Engine, and Docker Compose v2.
  • Internet access during installation for packages and container images.

For the first deployment, keep AWC on the system disk. A passed-through cache disk is supported only after you explicitly format it and mount it at /var/lib/awc before installation; the bootstrap never formats disks.

1. Build the package

On a Linux build host with Rust, libfuse3-dev, pkg-config, and signing tools:

packaging/package.sh

The result is dist/awc-<version>-<target>.tar.gz. Copy the tarball and guest-bootstrap.sh to the VM, for example with scp.

2. Create the VM

On the Proxmox node, download or reuse a Debian 12 cloud image and run:

VMID=9001 \
NAME=awc-pilot \
BRIDGE=vmbr0 \
SSHKEYS=/root/.ssh/authorized_keys \
SYSTEM_DISK=80G \
bash deploy/proxmox/provision-vm.sh

The script configures DHCP, the QEMU guest agent channel, a cloud-init drive, virtio networking, and an 80 GiB default disk. Discover the address with:

qm guest cmd 9001 network-get-interfaces

If the guest agent has not started yet, use qm terminal 9001 or inspect DHCP.

3. Prepare the guest

Inside the VM, install Docker Engine and the Compose v2 plugin using Docker's Debian/Ubuntu instructions. Verify the exact command AWC uses:

docker info
docker compose version
test -e /dev/fuse
findmnt -n -o FSTYPE /sys/fs/cgroup

The last command should report cgroup2.

4. Install and configure the local pilot

Run the bootstrap against the copied package:

sudo LOCAL_PILOT=1 \
  CONTROL_ENDPOINT=http://127.0.0.1:8081 \
  PACKAGE_FILE=/tmp/awc-0.1.0-x86_64-unknown-linux-gnu.tar.gz \
  bash /tmp/guest-bootstrap.sh

CONTROL_ENDPOINT is required by the common bootstrap interface; local-pilot mode writes the loopback endpoint shown above. The configurator:

  1. Runs awc-pilot-preflight for kernel, cgroup, filesystem, xattr, FUSE, OverlayFS, and free-space compatibility.
  2. Creates /opt/awc-pilot and random local database, MinIO, and token secrets.
  3. Starts PostgreSQL (and MinIO on loopback, creating awc-cas, unless AWC_OBJECT_STORE=b2 selects the remote bucket instead).
  4. Writes mutually consistent control, worker, daemon, bind-root, and CAS config.
  5. Starts awc-control, applies the schema, and seeds the fixed pilot identity.
  6. Starts awc-worker and awcd and waits for a real FUSE base mount.
  7. Writes root-readable CLI settings to /opt/awc-pilot/awcctl.env.

It is safe to rerun: Docker volumes and generated secrets are retained.

Using Backblaze B2 instead of local MinIO

The pilot can store its CAS in a Backblaze B2 bucket so object data survives VM disposal and leaves the host disk. B2 speaks the S3 API over HTTPS only; AWC's transport signs with SigV4 and validates the webpki CA set, so no extra trust configuration is needed.

One-time operator setup in the B2 console:

  1. Create a private bucket (the pilot default is awc-test in eu-central-003).
  2. Set the bucket's file lifecycle to Keep only the last version. B2 does not implement conditional PutObject, so AWC's duplicate puts are idempotent overwrites; with "Keep all versions" those overwrites would stack versions and defeat GC reclamation.
  3. Create an application key restricted to that single bucket with only readFiles, writeFiles, deleteFiles, and listFiles capabilities (GC lists object versions, so listBuckets alone is not enough). If you also restrict the key to a file name prefix (awc/ is what the pilot expects), the configurator puts every CAS object under it via AWC_S3_KEY_PREFIX — B2_KEY_PREFIX in .b2.env defaults to awc; set it empty only for an unrestricted key, otherwise B2 rejects writes with AccessDenied: not entitled.

Then write the key to a root-owned file on the VM and bootstrap with AWC_OBJECT_STORE=b2:

mkdir -p /opt/awc-pilot
cat >/opt/awc-pilot/.b2.env <<'EOF'
B2_KEY_ID=<keyID>
B2_APPLICATION_KEY=<applicationKey>
EOF
chown root:root /opt/awc-pilot/.b2.env && chmod 0600 /opt/awc-pilot/.b2.env

sudo AWC_OBJECT_STORE=b2 LOCAL_PILOT=1 CONTROL_ENDPOINT=http://127.0.0.1:8081 \
  PACKAGE_FILE=/tmp/awc-0.1.0-x86_64-unknown-linux-gnu.tar.gz \
  bash /tmp/guest-bootstrap.sh

B2_ENDPOINT, B2_REGION, and B2_BUCKET may be set in .b2.env to target a different bucket; they default to https://s3.eu-central-003.backblazeb2.com, a region derived from the endpoint host (eu-central-003), and awc-test. The configurator probes the endpoint before starting anything so DNS/firewall failures surface immediately.

Rotate the application key once the pilot is verified, and treat the keyID and applicationKey like passwords: they never belong in the repository or on a command line that ends up in shell history. Switching an existing pilot from MinIO to B2 (or back) is a rerun of the configurator with the other AWC_OBJECT_STORE value; the CAS contents do not migrate.

5. Run the acceptance test

The bootstrap installs the acceptance command. Run:

sudo awc-pilot-acceptance

The test creates disposable AWC content and verifies all of the following:

  • all three systemd services are active and awcd is ready;
  • /var/lib/awc/basefs is a real FUSE mount;
  • an imported snapshot reaches READY through the worker;
  • a workspace target is a real writable OverlayFS mount;
  • lower content is readable and mutations/deletions are isolated in the upper;
  • a cooperative checkpoint publishes successfully;
  • unmount releases the writer lease; and
  • an immediate remount sees the checkpointed mutation and deletion; and
  • a full AWC/PostgreSQL/object-store service-stack restart retains the published head (MinIO restarts locally; a B2 bucket is remote and stays up, so only PostgreSQL restarts with the AWC services).

A passing run ends with AWC Proxmox pilot acceptance PASS.

After a VM reboot, rerun sudo awc-pilot-acceptance; this repeats the durable stack-restart check and creates a fresh end-to-end checkpoint cycle.

Operations and troubleshooting

Useful checks:

systemctl status awc-control awc-worker awcd --no-pager
journalctl -u awc-control -u awc-worker -u awcd -n 100 --no-pager
curl -fsS http://127.0.0.1:9100/healthz
curl -fsS http://127.0.0.1:9100/readyz
findmnt /var/lib/awc/basefs
findmnt /var/lib/awc/binds/pilot-acceptance
cd /opt/awc-pilot && docker compose ps

Load the CLI environment with:

sudo -s
set -a
. /opt/awc-pilot/awcctl.env
set +a
awcctl status

Common failures:

  • Docker Compose v2 is required: install the docker compose plugin; legacy docker-compose is not used.
  • /dev/fuse missing: use a KVM VM and install/load FUSE in the guest.
  • basefs is not mounted with FUSE: inspect journalctl -u awcd; startup now fails rather than reporting healthy if the FUSE worker exits or times out.
  • mount target rejected: both sides must use /var/lib/awc/binds; the pilot writes AWC_MOUNT_TARGET_ROOTS and allowed_bind_roots consistently.
  • checkpoint asks for a cgroup: use --mode cooperative in this pilot.
  • disk-pressure rejection: keep at least 5 GB free on the filesystem containing /var/lib/awc.

To restart without destroying pilot data:

systemctl restart awc-control awc-worker awcd

To stop services while retaining data:

systemctl stop awcd awc-worker awc-control
cd /opt/awc-pilot && docker compose stop

Do not remove Docker volumes or /var/lib/awc unless you explicitly intend to destroy the pilot's database, CAS objects, caches, and local uppers. On a B2 pilot the CAS objects live in the bucket instead of the miniodata volume; emptying the bucket is equally destructive.

Moving beyond the pilot

Before exposing AWC or adding hosts, replace development authentication with TLS/mTLS and managed signing keys, move PostgreSQL and the CAS to backed-up services, use unique registered host identities, validate firewall policy, and exercise quiesced checkpoints with delegated execution cgroups. Those changes are intentionally outside this first local deployment.