Skip to content

Rotate credentials

Use this runbook to rotate AWC credentials with no outage. Rotation covers four credential types (spec §13.2, §16.5):

Credential Used by Lifetime
Service-account token The external API caller (awcctl, platforms) Revocable, scoped
Host mTLS certificate Internal gRPC from awcd to the control plane Short-lived where supported
Mount-ticket signing key The control plane signs; the host verifies Rotated with overlap
Tenant-scoped object credential A host accesses the CAS for one tenant Short-lived, max 1 h

The rule that protects you during every rotation is overlap (spec §16.5). The old and the new credential MUST both be trusted until every in-flight use has expired. For signing keys, the overlap window MUST exceed the maximum mount-ticket lifetime.

Detect

Rotate on any of these signals:

  • A credential nears its not_after. mTLS certificates and signing keys carry explicit validity windows. Track expiry on a calendar.
  • The object-corruption runbook tells you to. Spec §25.3, step 9, says rotate or revoke when overwrite or unauthorized access is suspected.
  • A host fails to renew its lease over internal gRPC. The control plane returns Unauthenticated. This points at an expired or revoked mTLS certificate.
  • A new mount fails with ControlPlaneUnavailable and the text no active mount-ticket signing key when the control plane has no active signing key. An empty host verifying set, after a botched cut-over, instead fails ticket verification on the host. This ControlPlaneUnavailable error comes from the rotation-aware credential service, which is not wired yet (see Current scope). Today the single-key env path cannot emit it at run time; a missing or invalid AWC_MOUNT_TICKET_SEED stops the control plane from starting instead.
  • The audit log shows signing_key.revoke (see spec §16.11).

Procedure

Steps 2 through 9 assume the rotation API. That API is not wired yet; see Current scope for the env-var-and-restart path you use today.

  1. Inventory the credentials in use. List all four types from the table above. Confirm that no long-lived object-store credentials sit inside an agent sandbox. They MUST NOT (spec §16.5).
  2. Read the active signing key and its window. Check the audit trail for signing_key.rotate and signing_key.retire events. Note the retiring key id and its not_after.
  3. Stage the new credential alongside the old. For a signing key, add the new key whose validity opens now, and leave the retiring key trusted until its own not_after. Size the overlap to exceed the maximum mount-ticket lifetime. For an mTLS certificate, renew with a fresh fingerprint. The old certificate stays honored until its own not_after.
  4. Distribute the new verifying material to hosts. Publish the public verifying key set so hosts accept tickets signed during the overlap. Register the new mTLS identity. Hosts trust both old and new during the overlap window.
  5. Reload the control plane and each host. They must pick up the new trusted set. Do this inside the overlap window. See Deploy on Proxmox VE and Deploy a VM under systemd for the service and env files.
  6. Cut traffic over to the new credential. New tickets are signed with the new active key. New host connections present the new mTLS certificate.
  7. Retire the old credential. Set the old signing key not_after past the last in-flight ticket expiry. Revoke the old mTLS fingerprint. Revoke at once, do not retire, if you suspect compromise (spec §25.3).
  8. Confirm the audit trail. The log records signing_key.rotate, signing_key.retire, signing_key.revoke, and object_credential.issue against the signing_key and object_credential resource types. The detail holds only key ids and validity windows. No secret material is logged (spec §16.5).
  9. Keep rotating inside the overlap until every host confirms the new set. Do not let the overlap close before the last host reloads.

Verify

Confirm recovery when all of these hold:

  • A new mount ticket verifies on a host. A fresh awcctl workspace mount succeeds.
  • Existing mounts keep serving. Reads and writes continue. Leases renew over mTLS without Unauthenticated errors.
  • A checkpoint publishes successfully. See Checkpoint a workspace.
  • The old credential no longer verifies after its not_after. If you revoked it, it no longer verifies at once.
  • The audit log shows the rotate, retire, or revoke events.

Impact

State the risk plainly:

  • Overlap is the only thing that prevents an outage. If you cut over without overlap, or you outlive the window, new mount tickets cannot be issued or verified. New mounts and new lease renewals fail. Existing mounted reads and writes continue during a bounded lease grace period (spec §17.5), then the host fences publication and remounts the workspace read-only.
  • Rotation does not touch tenant content. No committed object, snapshot, or checkpoint is at risk. Data-loss risk is the same as a control-plane outage: uncheckpointed uppers are ephemeral (spec §17.1, §17.4). See Recover from a control-plane outage.
  • Secrets never reach logs or telemetry. Rotation audit events carry key ids and validity windows only (spec §16.5).

Current scope

The overlap rotation mechanism is implemented and unit-tested, but the operator-facing surface lags the spec. Be aware of these gaps:

  • No rotation API or CLI yet. The credential service in awc-control implements add, retire, revoke, issue, and the public verifying-set publication. The host credential registry implements renew and revoke with overlap. None of these are exposed over an HTTP route or an awcctl command. There is no /v1/signing-keys endpoint and no awcctl credentials subcommand.
  • Today you rotate through env vars and a restart. The control plane reads AWC_MOUNT_TICKET_SEED and AWC_SERVICE_TOKEN_KEY. Each host reads AWC_MOUNT_TICKET_PUBLIC_KEY or AWC_MOUNT_TICKET_PUBLIC_KEY_FILE. mTLS material is AWC_CONTROL_TLS_CERT, AWC_CONTROL_TLS_KEY, AWC_CONTROL_TLS_CLIENT_CA, and AWC_CONTROL_TLS_CA on the host. This path loads a single key, so it does NOT give overlap. Plan a maintenance window. Drain the host, restart, and re-verify. New mount-ticket issuance fails during the restart gap. See the configuration reference and environment variables.
  • Dev mode is not a production security mode. AWC_DEV_MODE=1 selects a fixed development signing seed and skips TLS (spec §9.11). It binds to loopback only. To rotate, leave dev mode and provision real mTLS and managed seeds. See Deploy on Proxmox VE.
  • Host credentials persist in awc-db. The host credential registry is an in-memory, tested core. A production deployment persists short-lived host credentials in the hosts table.