67 lines
4.3 KiB
Markdown
67 lines
4.3 KiB
Markdown
---
|
|
schema_version: 1
|
|
id: home-server.operations
|
|
type: reference
|
|
scope: [jupiter, mercury, monitoring, troubleshooting]
|
|
sensitivity: private-infrastructure
|
|
last_reviewed: "2026-10-27"
|
|
sources:
|
|
- kind: owner-report
|
|
reference: "24h outage tolerance 2026-10-06; Discord receipt 2026-10-23; Beszel/Kuma/Arr alert configuration 2026-10-26"
|
|
- kind: live-inspection
|
|
reference: "Traefik 3.7.13 startup health/log timing and subsequent route recovery, 2026-10-27"
|
|
- kind: documentation
|
|
reference: "Traefik v3.7 pkg/provider/docker/config.go health filtering; Beszel v0.21.0 alerts.ts/alerts_container.go health semantics"
|
|
related: [home-server.backups, home-server.networking, home-server.authentication, home-server.downloads, home-server.host]
|
|
update_triggers: [alerting-change, outage, recovery-test]
|
|
unknowns: [exact-uptime-probes-and-timing, other-alert-thresholds, arr-alert-destinations-and-events, monitoring-review-frequency]
|
|
---
|
|
|
|
# Operations
|
|
|
|
## Observability
|
|
|
|
Owner-confirmed configuration; Discord server **Jupiter**, receipt confirmed 2026-10-23.
|
|
|
|
| Source | Coverage / trigger | Destination |
|
|
| --- | --- | --- |
|
|
| Beszel hub on Jupiter | Jupiter + Mercury; reported systemd errors / container down >2 min | `#beszel-alerts` |
|
|
| Mercury Beszel rules | CPU >90% for 10 min; RAM >90% for 10 min; failed systemd units | `#beszel-alerts` |
|
|
| Uptime Kuma on Mercury | WireGuard path and Jupiter availability only; alerts Jay | `#uptimekuma-alerts` |
|
|
| Backrest | Notifications enabled; exact hook events unknown | `#backrest-alerts` |
|
|
| Sonarr / Radarr / Prowlarr | Settings -> Connect enabled; events/destinations unknown | Not recorded |
|
|
| Dozzle | Container logs via Docker socket proxy | Web UI |
|
|
|
|
- Beszel UI/database stores rules. `Failed Services` detects failed units, not every journal error; `Container Health` detects Docker `unhealthy`. Exact stopped/missing-container rules need inspection.
|
|
- Beszel cannot alert when Jupiter/hub is down; Kuma is independent of Jupiter but cannot report its own complete outage.
|
|
- Kuma has no application/container probes. Arr notifications do not establish empty-search, rejected-release or stalled-torrent coverage.
|
|
- Backup notifications do not prove integrity, service restart or detection of a backup that never ran; see [Backups](backups.md).
|
|
|
|
### Proposed monitoring additions
|
|
|
|
- Add authenticated-safe application health probes, not proxy/login-page checks; start at 60s / two retries.
|
|
- Test outage **and recovery** delivery during maintenance; account for scheduled backup stops.
|
|
- Add external Mercury monitoring and missed-backup detection.
|
|
- Remote Docker checks require a secured connection; Mercury's socket cannot inspect Jupiter. Never expose an unauthenticated Docker API just for monitoring.
|
|
|
|
## Traefik startup troubleshooting
|
|
|
|
- Symptom: `middleware "security-headers@docker" does not exist` immediately after restart.
|
|
- Cause: shared middleware lives on Traefik's Docker labels; the provider filters its container while health is `starting`/unhealthy.
|
|
- Observed 2026-10-27: errors **11:41:07 UTC**, first healthy check **11:41:11**; Homarr/Authentik routes recovered.
|
|
- Action: inspect health + latest logs, then request affected routes. Persistent errors after healthy status require provider/label investigation.
|
|
- Do **not** disable health checks or admit unhealthy services globally.
|
|
- Possible fix: move shared middleware to the file provider and update references together; **not implemented**.
|
|
- `aliasHeadersStrategy` and encoded-path warnings are separate; review backend compatibility before selecting policies. No warning-related settings changed.
|
|
|
|
## Outstanding work
|
|
|
|
| Priority | Work / canonical detail |
|
|
| --- | --- |
|
|
| Recovery | Outage tolerance **24h**, not demonstrated. Define per-service RPO/order and exercise [restores](backups.md#recovery-limits). |
|
|
| Startup | Ensure WireGuard owns `10.0.0.2` before Traefik binds; persistent ordering/reboot exercise unverified. |
|
|
| VPN isolation | Complete [AirVPN failure/startup/DNS/IPv6 exercises](downloads.md#remaining-validation). |
|
|
| Exposure | Private admin ingress is not implemented; public hostnames and Portainer's direct port still exist. |
|
|
| Docker logs | Confirm [daemon default activation and container recreation](host.md#docker-log-rotation). |
|
|
| Identity | Confirm [per-user bindings and Homarr linking](authentication.md); transport verification did not establish them. |
|