Files
notes/docs/home-server/operations.md
T
2026-10-27 12:21:10 +00:00

4.3 KiB

schema_version, id, type, scope, sensitivity, last_reviewed, sources, related, update_triggers, unknowns
schema_version id type scope sensitivity last_reviewed sources related update_triggers unknowns
1 home-server.operations reference
jupiter
mercury
monitoring
troubleshooting
private-infrastructure 2026-10-27
kind reference
owner-report 24h outage tolerance 2026-10-06; Discord receipt 2026-10-23; Beszel/Kuma/Arr alert configuration 2026-10-26
kind reference
live-inspection Traefik 3.7.13 startup health/log timing and subsequent route recovery, 2026-10-27
kind reference
documentation Traefik v3.7 pkg/provider/docker/config.go health filtering; Beszel v0.21.0 alerts.ts/alerts_container.go health semantics
home-server.backups
home-server.networking
home-server.authentication
home-server.downloads
home-server.host
alerting-change
outage
recovery-test
exact-uptime-probes-and-timing
other-alert-thresholds
arr-alert-destinations-and-events
monitoring-review-frequency

Operations

Observability

Owner-confirmed configuration; Discord server Jupiter, receipt confirmed 2026-10-23.

Source Coverage / trigger Destination
Beszel hub on Jupiter Jupiter + Mercury; reported systemd errors / container down >2 min #beszel-alerts
Mercury Beszel rules CPU >90% for 10 min; RAM >90% for 10 min; failed systemd units #beszel-alerts
Uptime Kuma on Mercury WireGuard path and Jupiter availability only; alerts Jay #uptimekuma-alerts
Backrest Notifications enabled; exact hook events unknown #backrest-alerts
Sonarr / Radarr / Prowlarr Settings -> Connect enabled; events/destinations unknown Not recorded
Dozzle Container logs via Docker socket proxy Web UI
  • Beszel UI/database stores rules. Failed Services detects failed units, not every journal error; Container Health detects Docker unhealthy. Exact stopped/missing-container rules need inspection.
  • Beszel cannot alert when Jupiter/hub is down; Kuma is independent of Jupiter but cannot report its own complete outage.
  • Kuma has no application/container probes. Arr notifications do not establish empty-search, rejected-release or stalled-torrent coverage.
  • Backup notifications do not prove integrity, service restart or detection of a backup that never ran; see Backups.

Proposed monitoring additions

  • Add authenticated-safe application health probes, not proxy/login-page checks; start at 60s / two retries.
  • Test outage and recovery delivery during maintenance; account for scheduled backup stops.
  • Add external Mercury monitoring and missed-backup detection.
  • Remote Docker checks require a secured connection; Mercury's socket cannot inspect Jupiter. Never expose an unauthenticated Docker API just for monitoring.

Traefik startup troubleshooting

  • Symptom: middleware "security-headers@docker" does not exist immediately after restart.
  • Cause: shared middleware lives on Traefik's Docker labels; the provider filters its container while health is starting/unhealthy.
  • Observed 2026-10-27: errors 11:41:07 UTC, first healthy check 11:41:11; Homarr/Authentik routes recovered.
  • Action: inspect health + latest logs, then request affected routes. Persistent errors after healthy status require provider/label investigation.
  • Do not disable health checks or admit unhealthy services globally.
  • Possible fix: move shared middleware to the file provider and update references together; not implemented.
  • aliasHeadersStrategy and encoded-path warnings are separate; review backend compatibility before selecting policies. No warning-related settings changed.

Outstanding work

Priority Work / canonical detail
Recovery Outage tolerance 24h, not demonstrated. Define per-service RPO/order and exercise restores.
Startup Ensure WireGuard owns 10.0.0.2 before Traefik binds; persistent ordering/reboot exercise unverified.
VPN isolation Complete AirVPN failure/startup/DNS/IPv6 exercises.
Exposure Private admin ingress is not implemented; public hostnames and Portainer's direct port still exist.
Docker logs Confirm daemon default activation and container recreation.
Identity Confirm per-user bindings and Homarr linking; transport verification did not establish them.