> ## Documentation Index > Fetch the complete documentation index at: https://docs.oximail.ch/llms.txt > Use this file to discover all available pages before exploring further. # Operations > Day-to-day operations: journald logs, Prometheus metrics, the background workers, backups, upgrades and deploy verification, health checks, and graceful shutdown. OxiMail is one binary, one systemd unit, one SQLite database, and a blob directory. Day-to-day operations are correspondingly small: read journald, scrape one metrics endpoint, keep backups, and swap the binary to upgrade. ## Logs The server logs to **stdout** and journald captures it — there is no `/var/log/oximail`: ```bash theme={null} journalctl -u oximail -f # follow journalctl -u oximail --since -1h # recent window ``` Logging uses structured `tracing` fields (`account_id`, `tenant_id`, `method`, `error`), and log messages describe what **actually** happened, not what was intended — a hard project rule. Diagnostic context is deliberately attributable: an IMAP session probing a nonexistent folder logs the account and peer address, a `SQLITE_BUSY` names the table and operation involved. ### TLS handshake logs The four listeners that terminate TLS — IMAPS, IMAP STARTTLS, SMTPS, SMTP STARTTLS — separate two events that used to share a single warning: * **A peer that left before negotiating anything.** A TCP availability probe, a port scan, a client that gave up. This is not a failure and is no longer logged as one: it drops a level, keeping its peer and error text, and `RUST_LOG=debug` still restores every byte of it. * **A real negotiation failure.** No cipher suite or group in common, a malformed message, an alert received, a certificate refused. This stays a warning, because it is one. Both branches carry a stable `reason` field. Key your alerting on that field rather than on the message text. The reason this distinction earns its place: one production instance was emitting **480 of those warnings a day** from a single monitoring probe that opened 993 every three minutes and hung up. A warning that fires 480 times a day perfectly harmlessly is not a cosmetic annoyance — it trains everyone to ignore the whole category, and a genuine handshake failure then hides inside it at the same port, with the same message, at the same level. That matters most right after a change to the negotiated groups, whose failure mode is *exactly* a handshake that stops completing with one incompatible client. The two explicit-timeout branches (120 s on IMAPS, 30 s on STARTTLS) remain warnings: a peer that holds a connection for two minutes without concluding is rare and genuinely worth a look. ## Metrics A Prometheus endpoint is served on the **admin surface** at `/metrics` (bearer-authenticated like the rest of the [admin API](../developer/admin-api)). Gauges worth alerting on: | Metric | Alert when | | ------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | `oximail_storage_db_bytes`, `oximail_storage_wal_bytes` | WAL grows past the database size — the checkpoint worker should keep it under \~64 MiB. | | `oximail_erasure_jobs_stuck` | `> 0` — an [erasure](./compliance) is parked on a durable failure and needs an operator. | | `oximail_smarthost_ip_listed{ip,list}` | `> 0` — your egress IP is on a DNSBL ([outbound relay](./outbound-relay)). | | audit-retention last-sweep gauge | stale — the retention worker stopped sweeping. | Per-organization metrics can gain a `domain` label (`[metrics]` config) — leave it off unless you need the breakdown; the cardinality cost is real. ## The background workers The binary runs its maintenance internally — there is no external cron to install: * **Delivery queue worker** — outbound retries with backoff, DSN generation ([SMTP layer](../architecture/oximail-smtp)). * **Search index worker** — batches Tantivy commits; commits on quiescence. * **WAL checkpoint worker** — publishes size gauges, `TRUNCATE`-checkpoints past 64 MiB. * **Fail2ban cleanup** — expires bans every few minutes. * **Reindex worker** — converges the reindex queue; terminal conditions resolve entries instead of retrying forever. * **Relay domain re-verification** — re-checks DNS proofs every six hours, suspends fail-closed on a definitive removal. * **Erasure maintenance** — adopts and resumes erasure jobs whose driver died; boot-time orphan sweep. * **Audit retention** — purges audit rows past the horizon. * **Alert schedulers** — fire calendar and task reminders. ## Backups ```bash theme={null} oximail backup --config /etc/oximail/oximail.toml ... oximail restore ... oximail verify-backup ... ``` A backup must capture **both** the SQLite database and the blob directory (and `/etc/oximail`, which holds the config, DKIM keys, and key material — without the key material, blobs are ciphertext). `verify-backup` checks a backup's integrity before you need it. Test a restore on a scratch machine, not during an incident. `oximail backup --tenant ` narrows a backup to one organization. Two properties of that mode are worth knowing, because both were once quietly untrue: * It carries the organization's **attachment blobs**, not only its rows. A per-organization archive whose messages restore without their attachments verifies clean while being unrestorable in the only sense that matters. * The set of tables it filters is **derived from the schema and guarded**, not a list maintained by hand. A hand-maintained list goes stale the moment a table is added, and the failure is silent in the worst direction: rows belonging to *other* organizations riding along in a tenant-scoped export. ## Upgrades An upgrade is a binary swap; there is no separate migration step to run: ```bash theme={null} systemctl stop oximail && mv oximail.new /usr/local/bin/oximail && systemctl start oximail ``` At boot, pending [schema migrations](../architecture/oximail-store#migrations-versioned-auto-applied-fail-loud) apply automatically in one exclusive transaction; a failure rolls everything back and the server refuses to start rather than running half-migrated. Verify the deploy afterwards: ```bash theme={null} curl -s -H "Authorization: Bearer $ADMIN_TOKEN" https://mail.example.com/admin/v1/version # → version + migrations_applied ``` `oximail status`, `oximail health`, and `oximail doctor` provide local checks (process, ports, config, storage) without the HTTP round-trip. **Local snapshots have a retention policy, and the deploy path applies it.** After a swap is verified active, the two most recent `.bak` binaries survive and any database snapshot older than 14 days is removed, hand-placed one-offs included. A host whose swap **failed** keeps everything, because that is rollback material. Two things this is not: it is not your backup — the durable layer stays the off-box [backups](#backups) — and it never deletes a plaintext database copy. Copies predating encryption cancel the at-rest posture for their content; absorbing them silently would hide the fault instead of surfacing it. Left unmanaged this grows quietly: one box had accumulated 85 GB of rollback binaries, and a backup MX was at 89 % disk. ## Graceful shutdown `systemctl stop` (SIGTERM) triggers a drain: workers finish their in-flight items within a bounded drain budget, the search index flushes, and the WAL is checkpointed. Restarts normally complete in the time the workers actually need — typically near-instant. A "drain cap reached" warning in the logs means a genuinely slow worker and is worth investigating. ## Security operations The three-layer defense (kernel nftables, in-binary rate limiting + fail2ban, monitoring) is described in the security model; the operational surface is: ```bash theme={null} oximail unban # lift a fail2ban ban oximail ban list|add ... # inspect and manage bans curl .../admin/v1/bans # the same over the admin API ``` `[security] trusted_ips` (CIDR supported) bypasses both the rate limiter and fail2ban for your own infrastructure. Do **not** install the fail2ban daemon next to OxiMail — the server tracks its own auth failures across every protocol, with the real client IP even behind a proxy. ## When something is wrong 1. `journalctl -u oximail --since -1h` — the failure is loud by design; look for `error`-level lines with context fields. 2. `oximail doctor` — config, storage, and port diagnostics. 3. `/metrics` — is a queue growing, a worker stuck, a gauge stale? 4. For delivery problems, the [DNS checklist](./email-auth-security#verifying-with-oximail-check-dns) and the queue CLI (`oximail queue …`) come next. **If legitimate mail is landing in spam, check the clock.** `oximail doctor` reports clock health, and it is the second thing to look at after DNS. An inbound DKIM signature carrying a timestamp is refused when it arrives more than 300 seconds in *our* future, so a box whose clock lags starts rejecting signatures on perfectly good mail — which then scores as spam. The symptom looks like a DNS or DKIM problem and never looks like a clock. Note the exposure is one-directional: our own outbound signatures carry no timestamp, so a recipient has nothing of ours to judge, and nobody ever complains to us about it.