/var/lib/oximail. What protects a stolen or decommissioned disk is the volume: dm-crypt/LUKS under the data directory, set up by the operator, measured by the product. This is the model ADR-020 had named from the start, “LUKS, not end-to-end”, finally implemented where LUKS lives rather than re-implemented above it.
Why the change
Between April and September 2026 the server encrypted its content twice: the database with SQLCipher, every blob with a key wrapped for its account or organization. The cost was measured before the decision, not asserted: a factor of 6 to 20 on every request while the key memory was locked (strace -c, #1388), a per-page decryption cost the OS cache cannot absorb because it caches ciphertext, no standard tooling on a production database (sqlite3, EXPLAIN, VACUUM INTO, mmap), a key ceremony with escrow and paper key, and “key lost = database lost”, a data-loss risk the mechanism itself created.
And for what guarantee? On half the fleet the root key sat in clear next to the database, so a disk thief left with both; the search index was in clear on every box; no instance had volume encryption while five attested to it. The price was paid in full, without the tooling and without the guarantee.
The rule that decides the model: your server, your root. Someone who is already root on the box is outside the threat model. Security is played at the borders: who gets in (authentication, scoped app passwords), who sees what (per-node rights re-checked on every request, shares, the rights guard), where data leaves (egress guard, regional hosts, provenance). A mechanism that protects “the server against its own root” is suspect by construction, and must justify itself with a real threat and a measured cost. Hardening is weighed like an optimization.
What at-rest encryption really protects, application or volume: the disk that leaves the machine (theft, decommissioning, RMA) and the copy that leaves the box (backup, snapshot, an scp for debugging). Against the first, LUKS and SQLCipher are equivalent. Against the second, only one question matters: where does the key of that copy sleep? A key that travels on the same path as the data is not a key, it is one more copy.
What the server does, and what it measures
It writes in clear. A file under the blob root is its content, first byte to last, named by the SHA-256 of that content; a registry row says which account or organization it belongs to, and nothing else. A large file is served by positioned reads on the file itself, in one-megabyte windows, with the same range contract for a 10 KB message and a 27 GB archive. The database is plain SQLite, sosqlite3, EXPLAIN and VACUUM INTO work on production, and a blob is restored with cp.
It measures the volume, and says what it found. At boot, and on every oximail doctor run, a probe walks the real device-mapper stack under every path that holds content (the database, the blob root, and the search index, which has its own config key and defaults to a sibling of the blob directory). Three outcomes, none of them silent:
The probe sees the encryption through two layers of thin volumes; it does not see native ZFS encryption, which is why the reference configuration is LUKS under LVM. Logs and swap are not covered by the measurement: they are the operator’s, and the reference configuration covers them.
The boot never refuses (ADR-104): a bare volume is a warning in the journal, and that warning is the trace of the consent. The wizard does refuse: before writing anything,
oximail setup measures the data path exactly as the clock check does, continues if it is encrypted, continues without asserting anything if it cannot measure, and stops with an explanation if it is bare. oximail setup --accept-bare-volume lifts the refusal knowingly; the consent is written to the configuration and to the wizard state, and doctor keeps measuring:
The reference volume configuration
Established and measured on the fleet (September 2026), on a Proxmox VE host with LXC containers; the principles hold elsewhere. The wizard prints these six points when it measures a bare volume:- A LUKS2 volume scoped to the OxiMail instances, not the whole machine: a dedicated logical volume,
aes-xts-plain64with a 512-bit key,argon2id, 4096-byte sectors, on a dedicated volume group that carries only the OxiMail containers. The other guests of the host are untouched, and the encrypted perimeter is exactly the data to protect. The host’s own disk need not be encrypted: the unlock happens in the booted system (/etc/crypttab), not in the initramfs. - Unattended unlock: the key sealed in the TPM 2.0 (
systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=7) plus a recovery key on paper, off the box;crypttabwithtpm2-device=auto,discard,nofail,headless=true. A headless host must never wait for a passphrase; withnofail+headless, a TPM that does not unseal leaves the volume closed and the host up, and the paper key opens it by hand. Without the paper key, a motherboard change or a firmware update that moves PCR 7 makes the volume permanently inaccessible. - dm-crypt work queues off, persistently (
cryptsetup --perf-no_read_workqueue --perf-no_write_workqueue --allow-discards --persistent refresh): on NVMe the queues add latency without throughput. Measured withfioat QD1 on the random 4k 70/30 profile of a SQLite database: -11 % instead of -25 % with the queues on. - Host swap under dm-crypt with a random key (
/dev/urandomincrypttab), or zram only. Otherwise a page of the database or of a key that spills out of memory ends up in clear next to the encrypted volume. - Permissions:
/var/lib/oximailat700,data.db*at600, owned by the service user. The volume protects against the cold disk; the permissions protect against the other accounts of the same system, and since the database is in clear any UID of the container would read it otherwise. - Verification:
oximail doctor --jsonreportsat_rest=okmeasured, then a cold boot (a full power cut, not areboot) that brings the volume up with nobody at the console. A warm reboot proves nothing about the TPM seal across a power cycle.
What the root key still protects
server.key and everything around it stay: the key service, [security] key_source (systemd-creds on a host that can wrap it, plaintext with the consent flag otherwise), the tenant master keys, the escrow and the paper key, the rotation ceremony. They are not tooling for content encryption. The tenant key wraps secrets that are worth something elsewhere than on the box: the NT hash of SMB app passwords (a password equivalent), API keys of AI providers, billing certificates, the federation node identity, signed module management links, bearer tokens. A database copied in clear contains credentials valid elsewhere, which is exactly why the copy that leaves must still be encrypted, by the volume or by the backup.
What changes is what losing the key costs: re-provisionable secrets, never data. ADR-102 is therefore turned around rather than replaced: a backup may carry the key, and the escrow and the recovery drill remain as they were, because they still guard the identity root of the node.
A command that needs the key, on a box whose key is wrapped
Withkey_source = "systemd-creds", the key is readable by root and by a systemd unit that declares the credential, and by nobody else. A command run as the service account outside its unit cannot unwrap it. A command that needs the key then stops, says why, and prints its own command line placed in a transient unit, to run as root:
oximail doctor needs no key and runs as before.
scp command that takes the escrow off the box is printed. oximail key escrow --guided replays it on a box already installed, and doctor fails on a server without an escrow. The escrow also carries the instance secret, so a box restored from it on new hardware keeps its identity and its sessions; an older escrow is rewritten by oximail key upgrade-escrow --yes, and oximail key adopt-identity --yes accepts a secret that changed on purpose (backups).
- Rotation re-wraps the small key objects (the container secret, the tenant keys, the escrow) under a fresh
server.key; values are unchanged and no data is rewritten.rotateandescrowhard-gate on a recovery drill: an escrow you never tested is not an escrow. purge-pendingbuilds one list, the same in dry-run and for real: the previous root key in clear or wrapped, the stale escrow, the pre-rotation container and, on asystemd-credsbox, the previous wrapped key. A rotation keeps the value of the instance secret, so the old escrow, its old paper key and the old container are enough to rebuild the seed of the bearer tokens. One successful drill on the current escrow authorizes the whole list; a failed drill removes nothing;doctorcounts what remains (rotation_leftovers) and names the verb.- The container key (
container.key.wrapped, kept byfreeze-db-keyandrotate) is the instance secret that seeds the HMAC key of bearer tokens (ADR-071 Am. 1). Its name still says “db key”; that is the key it was until September, and the rename is deferred.
Integrity without a seal
The seal used to guarantee that a blob served was the blob stored. The guarantee moved to the name: a file’s name is the SHA-256 of its bytes, and the check is free.oximail blob verifycounts a file as CORRUPT when its hash no longer gives its name;--quarantinemarks it so read surfaces serve a placeholder instead of retrying and flooding the log, and sets the corrupt copy aside as<hash>.bad. The sample (1000 by default,--fullfor all) is drawn at random;--walkcounts the.badcopies and the vanished paths apart instead of failing.- A corrupt copy heals at the next write of its content. A stored copy is reused only when it is not quarantined and has the content’s length; otherwise it is set aside as
<hash>.bad, the content is written again and the quarantine is lifted. The same attachment arriving again therefore repairs it, where the server used to keep the corrupt copy (and a streamed upload discarded the healthy content it had just received). A registry row whose file is gone is repaired the same way. oximail blob verify --walkstarts from the disk rather than the registry: for every file under a tenant’s tree, does a registry row exist, and do the bytes hash to the name. Read-only; exit 1 on any unregistered file or mismatch;--jsonfor monitoring.doctor(blob_registry) does the same walk without hashing and flags files nobody knows how to read. Neither deletes: that is the garbage collector’s role.- A backup restore refuses a sealed blob. This binary has no reader for the
OXIMBLOBheader of earlier releases;oximail restorestops before writing anything and names the binary of the backup’s era. A backup that carries the binary of its time restores itself with that reader. - Two source guards replaced the former type-level guarantee: nothing writes under the blob root outside the blob store, and every direct reader is inventoried with its reason.
Erasure, and what a backup carries
Erasure deletes. There is no crypto-erasure any more: account erasure removes rows, blob files and index documents, and the purge service now triggers a rewrite of the search index segments that carried deletion marks at the end of a campaign, so the text of a destroyed message leaves the index files as well as the index. Deleting a blob whose attachments may be shared by other objects is checked in batches of five hundred, so a purge of a hundred thousand files frees its disk in seconds rather than days. The backup of record is the hypervisor’s, encrypted client-side. It restores the container whole, with the binary of its time, so there is no “old reader” to maintain and no format horizon beyond the snapshot retention. What the operator owns is the custody of the backup key (a single named copy, off the data path, known to a second person) and the retention (a dedicated namespace with a short purge, so an erased account does not survive for years in cold storage). The outbound relay queue is excluded from that backup by mount point (ADR-149): a snapshot taken between the acknowledgement and the garbage collection would otherwise carry relayed content for the whole retention.oximail backup remains for per-organization restoration, on another box, and for installations without a hypervisor. It writes on the box by default, hence under the encrypted volume; --encrypt wraps it under an Argon2id-derived passphrase for a copy that leaves. Its data.db is plain SQLite and needs no key to open; the operations page has the per-tenant mode. “No plaintext database leaves the box” is a runbook rule, not a code rule, because a network mount is indistinguishable from a local disk.
What is written, not silent
- A VPS at a third party is outside the model and says so at boot.
- A host whose unlock key is sealed in its TPM protects the disk that leaves the motherboard, not the whole server carried away.
- Naming blobs by the hash of their clear content is an existence oracle for a known document to whoever reads the disk. It was already true; it is now documented.
- A root ransomware is not stopped by backup encryption; it is stopped by a backup the box cannot delete (the box holds no backup credential; the second level is a pull that does not propagate deletions).
- Co-resident processes of a container (a proxy, a monitoring agent) read a clear blob without going through OxiMail: distinct UID and
0700on the blob root.
Retired, and what to do with a TOML that still says it
[storage] encrypted = true is a tombstone: the key still parses so an old file loads, and true is refused at boot with the migration instruction, never silently ignored, because a SQLCipher file would not open as SQLite. A box that skipped the migration runs oximail migrate-db --to-plaintext with the previous release first, then sets encrypted = false. The verbs migrate-db, encrypt-blobs, blob repack and account regenerate-key are gone, as are the JMAP key routes; the shipped binary embeds stock SQLite and depends on no system crypto, TLS or SQLite library, which the deploy path still checks.
What this is not
- Not end-to-end encryption. The server reads content for authenticated requests. E2E for specific surfaces is a separate concern with separate trade-offs (searchability, DAV bridges, server-side filtering all stop working under E2E).
- Not something the server can provide. It measures the volume and refuses to install on a bare one without consent; it cannot encrypt the volume for you.