Files
notes/docs/home-server/operations.md
T
2026-10-27 12:21:10 +00:00

67 lines
4.3 KiB
Markdown

---
schema_version: 1
id: home-server.operations
type: reference
scope: [jupiter, mercury, monitoring, troubleshooting]
sensitivity: private-infrastructure
last_reviewed: "2026-10-27"
sources:
- kind: owner-report
reference: "24h outage tolerance 2026-10-06; Discord receipt 2026-10-23; Beszel/Kuma/Arr alert configuration 2026-10-26"
- kind: live-inspection
reference: "Traefik 3.7.13 startup health/log timing and subsequent route recovery, 2026-10-27"
- kind: documentation
reference: "Traefik v3.7 pkg/provider/docker/config.go health filtering; Beszel v0.21.0 alerts.ts/alerts_container.go health semantics"
related: [home-server.backups, home-server.networking, home-server.authentication, home-server.downloads, home-server.host]
update_triggers: [alerting-change, outage, recovery-test]
unknowns: [exact-uptime-probes-and-timing, other-alert-thresholds, arr-alert-destinations-and-events, monitoring-review-frequency]
---
# Operations
## Observability
Owner-confirmed configuration; Discord server **Jupiter**, receipt confirmed 2026-10-23.
| Source | Coverage / trigger | Destination |
| --- | --- | --- |
| Beszel hub on Jupiter | Jupiter + Mercury; reported systemd errors / container down >2 min | `#beszel-alerts` |
| Mercury Beszel rules | CPU >90% for 10 min; RAM >90% for 10 min; failed systemd units | `#beszel-alerts` |
| Uptime Kuma on Mercury | WireGuard path and Jupiter availability only; alerts Jay | `#uptimekuma-alerts` |
| Backrest | Notifications enabled; exact hook events unknown | `#backrest-alerts` |
| Sonarr / Radarr / Prowlarr | Settings -> Connect enabled; events/destinations unknown | Not recorded |
| Dozzle | Container logs via Docker socket proxy | Web UI |
- Beszel UI/database stores rules. `Failed Services` detects failed units, not every journal error; `Container Health` detects Docker `unhealthy`. Exact stopped/missing-container rules need inspection.
- Beszel cannot alert when Jupiter/hub is down; Kuma is independent of Jupiter but cannot report its own complete outage.
- Kuma has no application/container probes. Arr notifications do not establish empty-search, rejected-release or stalled-torrent coverage.
- Backup notifications do not prove integrity, service restart or detection of a backup that never ran; see [Backups](backups.md).
### Proposed monitoring additions
- Add authenticated-safe application health probes, not proxy/login-page checks; start at 60s / two retries.
- Test outage **and recovery** delivery during maintenance; account for scheduled backup stops.
- Add external Mercury monitoring and missed-backup detection.
- Remote Docker checks require a secured connection; Mercury's socket cannot inspect Jupiter. Never expose an unauthenticated Docker API just for monitoring.
## Traefik startup troubleshooting
- Symptom: `middleware "security-headers@docker" does not exist` immediately after restart.
- Cause: shared middleware lives on Traefik's Docker labels; the provider filters its container while health is `starting`/unhealthy.
- Observed 2026-10-27: errors **11:41:07 UTC**, first healthy check **11:41:11**; Homarr/Authentik routes recovered.
- Action: inspect health + latest logs, then request affected routes. Persistent errors after healthy status require provider/label investigation.
- Do **not** disable health checks or admit unhealthy services globally.
- Possible fix: move shared middleware to the file provider and update references together; **not implemented**.
- `aliasHeadersStrategy` and encoded-path warnings are separate; review backend compatibility before selecting policies. No warning-related settings changed.
## Outstanding work
| Priority | Work / canonical detail |
| --- | --- |
| Recovery | Outage tolerance **24h**, not demonstrated. Define per-service RPO/order and exercise [restores](backups.md#recovery-limits). |
| Startup | Ensure WireGuard owns `10.0.0.2` before Traefik binds; persistent ordering/reboot exercise unverified. |
| VPN isolation | Complete [AirVPN failure/startup/DNS/IPv6 exercises](downloads.md#remaining-validation). |
| Exposure | Private admin ingress is not implemented; public hostnames and Portainer's direct port still exist. |
| Docker logs | Confirm [daemon default activation and container recreation](host.md#docker-log-rotation). |
| Identity | Confirm [per-user bindings and Homarr linking](authentication.md); transport verification did not establish them. |