--- schema_version: 1 id: home-server.operations type: reference scope: [jupiter, mercury, monitoring, troubleshooting] sensitivity: private-infrastructure last_reviewed: "2026-10-27" sources: - kind: owner-report reference: "24h outage tolerance 2026-10-06; Discord receipt 2026-10-23; Beszel/Kuma/Arr alert configuration 2026-10-26" - kind: live-inspection reference: "Traefik 3.7.13 startup health/log timing and subsequent route recovery, 2026-10-27" - kind: documentation reference: "Traefik v3.7 pkg/provider/docker/config.go health filtering; Beszel v0.21.0 alerts.ts/alerts_container.go health semantics" related: [home-server.backups, home-server.networking, home-server.authentication, home-server.downloads, home-server.host] update_triggers: [alerting-change, outage, recovery-test] unknowns: [exact-uptime-probes-and-timing, other-alert-thresholds, arr-alert-destinations-and-events, monitoring-review-frequency] --- # Operations ## Observability Owner-confirmed configuration; Discord server **Jupiter**, receipt confirmed 2026-10-23. | Source | Coverage / trigger | Destination | | --- | --- | --- | | Beszel hub on Jupiter | Jupiter + Mercury; reported systemd errors / container down >2 min | `#beszel-alerts` | | Mercury Beszel rules | CPU >90% for 10 min; RAM >90% for 10 min; failed systemd units | `#beszel-alerts` | | Uptime Kuma on Mercury | WireGuard path and Jupiter availability only; alerts Jay | `#uptimekuma-alerts` | | Backrest | Notifications enabled; exact hook events unknown | `#backrest-alerts` | | Sonarr / Radarr / Prowlarr | Settings -> Connect enabled; events/destinations unknown | Not recorded | | Dozzle | Container logs via Docker socket proxy | Web UI | - Beszel UI/database stores rules. `Failed Services` detects failed units, not every journal error; `Container Health` detects Docker `unhealthy`. Exact stopped/missing-container rules need inspection. - Beszel cannot alert when Jupiter/hub is down; Kuma is independent of Jupiter but cannot report its own complete outage. - Kuma has no application/container probes. Arr notifications do not establish empty-search, rejected-release or stalled-torrent coverage. - Backup notifications do not prove integrity, service restart or detection of a backup that never ran; see [Backups](backups.md). ### Proposed monitoring additions - Add authenticated-safe application health probes, not proxy/login-page checks; start at 60s / two retries. - Test outage **and recovery** delivery during maintenance; account for scheduled backup stops. - Add external Mercury monitoring and missed-backup detection. - Remote Docker checks require a secured connection; Mercury's socket cannot inspect Jupiter. Never expose an unauthenticated Docker API just for monitoring. ## Traefik startup troubleshooting - Symptom: `middleware "security-headers@docker" does not exist` immediately after restart. - Cause: shared middleware lives on Traefik's Docker labels; the provider filters its container while health is `starting`/unhealthy. - Observed 2026-10-27: errors **11:41:07 UTC**, first healthy check **11:41:11**; Homarr/Authentik routes recovered. - Action: inspect health + latest logs, then request affected routes. Persistent errors after healthy status require provider/label investigation. - Do **not** disable health checks or admit unhealthy services globally. - Possible fix: move shared middleware to the file provider and update references together; **not implemented**. - `aliasHeadersStrategy` and encoded-path warnings are separate; review backend compatibility before selecting policies. No warning-related settings changed. ## Outstanding work | Priority | Work / canonical detail | | --- | --- | | Recovery | Outage tolerance **24h**, not demonstrated. Define per-service RPO/order and exercise [restores](backups.md#recovery-limits). | | Startup | Ensure WireGuard owns `10.0.0.2` before Traefik binds; persistent ordering/reboot exercise unverified. | | VPN isolation | Complete [AirVPN failure/startup/DNS/IPv6 exercises](downloads.md#remaining-validation). | | Exposure | Private admin ingress is not implemented; public hostnames and Portainer's direct port still exist. | | Docker logs | Confirm [daemon default activation and container recreation](host.md#docker-log-rotation). | | Identity | Confirm [per-user bindings and Homarr linking](authentication.md); transport verification did not establish them. |