Logs
The server logs to stdout and journald captures it — there is no/var/log/oximail:
tracing fields (account_id, tenant_id, method, error), and log messages describe what actually happened, not what was intended — a hard project rule. Diagnostic context is deliberately attributable: an IMAP session probing a nonexistent folder logs the account and peer address, a SQLITE_BUSY names the table and operation involved.
TLS handshake logs
The four listeners that terminate TLS — IMAPS, IMAP STARTTLS, SMTPS, SMTP STARTTLS — separate two events that used to share a single warning:- A peer that left before negotiating anything. A TCP availability probe, a port scan, a client that gave up. This is not a failure and is no longer logged as one: it drops a level, keeping its peer and error text, and
RUST_LOG=debugstill restores every byte of it. - A real negotiation failure. No cipher suite or group in common, a malformed message, an alert received, a certificate refused. This stays a warning, because it is one.
reason field. Key your alerting on that field rather than on the message text.
The reason this distinction earns its place: one production instance was emitting 480 of those warnings a day from a single monitoring probe that opened 993 every three minutes and hung up. A warning that fires 480 times a day perfectly harmlessly is not a cosmetic annoyance — it trains everyone to ignore the whole category, and a genuine handshake failure then hides inside it at the same port, with the same message, at the same level. That matters most right after a change to the negotiated groups, whose failure mode is exactly a handshake that stops completing with one incompatible client.
The two explicit-timeout branches (120 s on IMAPS, 30 s on STARTTLS) remain warnings: a peer that holds a connection for two minutes without concluding is rare and genuinely worth a look.
Metrics
A Prometheus endpoint is served on the admin surface at/metrics (bearer-authenticated like the rest of the admin API). Gauges worth alerting on:
Per-organization metrics can gain a
domain label ([metrics] config) — leave it off unless you need the breakdown; the cardinality cost is real.
The background workers
The binary runs its maintenance internally — there is no external cron to install:- Delivery queue worker — outbound retries with backoff, DSN generation (SMTP layer).
- Search index worker — batches Tantivy commits; commits on quiescence.
- WAL checkpoint worker — publishes size gauges,
TRUNCATE-checkpoints past 64 MiB. - Fail2ban cleanup — expires bans every few minutes.
- Reindex worker — converges the reindex queue; terminal conditions resolve entries instead of retrying forever.
- Relay domain re-verification — re-checks DNS proofs every six hours, suspends fail-closed on a definitive removal.
- Erasure maintenance — adopts and resumes erasure jobs whose driver died; boot-time orphan sweep.
- Audit retention — purges audit rows past the horizon.
- Alert schedulers — fire calendar and task reminders.
Backups
/etc/oximail, which holds the config, DKIM keys, and key material — without the key material, blobs are ciphertext). verify-backup checks a backup’s integrity before you need it. Test a restore on a scratch machine, not during an incident.
oximail backup --tenant <id> narrows a backup to one organization. Two properties of that mode are worth knowing, because both were once quietly untrue:
- It carries the organization’s attachment blobs, not only its rows. A per-organization archive whose messages restore without their attachments verifies clean while being unrestorable in the only sense that matters.
- The set of tables it filters is derived from the schema and guarded, not a list maintained by hand. A hand-maintained list goes stale the moment a table is added, and the failure is silent in the worst direction: rows belonging to other organizations riding along in a tenant-scoped export.
Upgrades
An upgrade is a binary swap; there is no separate migration step to run:oximail status, oximail health, and oximail doctor provide local checks (process, ports, config, storage) without the HTTP round-trip.
Local snapshots have a retention policy, and the deploy path applies it. After a swap is verified active, the two most recent .bak binaries survive and any database snapshot older than 14 days is removed, hand-placed one-offs included. A host whose swap failed keeps everything, because that is rollback material. Two things this is not: it is not your backup — the durable layer stays the off-box backups — and it never deletes a plaintext database copy. Copies predating encryption cancel the at-rest posture for their content; absorbing them silently would hide the fault instead of surfacing it. Left unmanaged this grows quietly: one box had accumulated 85 GB of rollback binaries, and a backup MX was at 89 % disk.
Graceful shutdown
systemctl stop (SIGTERM) triggers a drain: workers finish their in-flight items within a bounded drain budget, the search index flushes, and the WAL is checkpointed. Restarts normally complete in the time the workers actually need — typically near-instant. A “drain cap reached” warning in the logs means a genuinely slow worker and is worth investigating.
Security operations
The three-layer defense (kernel nftables, in-binary rate limiting + fail2ban, monitoring) is described in the security model; the operational surface is:[security] trusted_ips (CIDR supported) bypasses both the rate limiter and fail2ban for your own infrastructure. Do not install the fail2ban daemon next to OxiMail — the server tracks its own auth failures across every protocol, with the real client IP even behind a proxy.
When something is wrong
journalctl -u oximail --since -1h— the failure is loud by design; look forerror-level lines with context fields.oximail doctor— config, storage, and port diagnostics./metrics— is a queue growing, a worker stuck, a gauge stale?- For delivery problems, the DNS checklist and the queue CLI (
oximail queue …) come next.
If legitimate mail is landing in spam, check the clock.
oximail doctor reports clock health, and it is the second thing to look at after DNS. An inbound DKIM signature carrying a timestamp is refused when it arrives more than 300 seconds in our future, so a box whose clock lags starts rejecting signatures on perfectly good mail — which then scores as spam. The symptom looks like a DNS or DKIM problem and never looks like a clock. Note the exposure is one-directional: our own outbound signatures carry no timestamp, so a recipient has nothing of ours to judge, and nobody ever complains to us about it.