4.3 KiB
4.3 KiB
schema_version, id, type, scope, sensitivity, last_reviewed, sources, related, update_triggers, unknowns
| schema_version | id | type | scope | sensitivity | last_reviewed | sources | related | update_triggers | unknowns | |||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | home-server.operations | reference |
|
private-infrastructure | 2026-10-27 |
|
|
|
|
Operations
Observability
Owner-confirmed configuration; Discord server Jupiter, receipt confirmed 2026-10-23.
| Source | Coverage / trigger | Destination |
|---|---|---|
| Beszel hub on Jupiter | Jupiter + Mercury; reported systemd errors / container down >2 min | #beszel-alerts |
| Mercury Beszel rules | CPU >90% for 10 min; RAM >90% for 10 min; failed systemd units | #beszel-alerts |
| Uptime Kuma on Mercury | WireGuard path and Jupiter availability only; alerts Jay | #uptimekuma-alerts |
| Backrest | Notifications enabled; exact hook events unknown | #backrest-alerts |
| Sonarr / Radarr / Prowlarr | Settings -> Connect enabled; events/destinations unknown | Not recorded |
| Dozzle | Container logs via Docker socket proxy | Web UI |
- Beszel UI/database stores rules.
Failed Servicesdetects failed units, not every journal error;Container Healthdetects Dockerunhealthy. Exact stopped/missing-container rules need inspection. - Beszel cannot alert when Jupiter/hub is down; Kuma is independent of Jupiter but cannot report its own complete outage.
- Kuma has no application/container probes. Arr notifications do not establish empty-search, rejected-release or stalled-torrent coverage.
- Backup notifications do not prove integrity, service restart or detection of a backup that never ran; see Backups.
Proposed monitoring additions
- Add authenticated-safe application health probes, not proxy/login-page checks; start at 60s / two retries.
- Test outage and recovery delivery during maintenance; account for scheduled backup stops.
- Add external Mercury monitoring and missed-backup detection.
- Remote Docker checks require a secured connection; Mercury's socket cannot inspect Jupiter. Never expose an unauthenticated Docker API just for monitoring.
Traefik startup troubleshooting
- Symptom:
middleware "security-headers@docker" does not existimmediately after restart. - Cause: shared middleware lives on Traefik's Docker labels; the provider filters its container while health is
starting/unhealthy. - Observed 2026-10-27: errors 11:41:07 UTC, first healthy check 11:41:11; Homarr/Authentik routes recovered.
- Action: inspect health + latest logs, then request affected routes. Persistent errors after healthy status require provider/label investigation.
- Do not disable health checks or admit unhealthy services globally.
- Possible fix: move shared middleware to the file provider and update references together; not implemented.
aliasHeadersStrategyand encoded-path warnings are separate; review backend compatibility before selecting policies. No warning-related settings changed.
Outstanding work
| Priority | Work / canonical detail |
|---|---|
| Recovery | Outage tolerance 24h, not demonstrated. Define per-service RPO/order and exercise restores. |
| Startup | Ensure WireGuard owns 10.0.0.2 before Traefik binds; persistent ordering/reboot exercise unverified. |
| VPN isolation | Complete AirVPN failure/startup/DNS/IPv6 exercises. |
| Exposure | Private admin ingress is not implemented; public hostnames and Portainer's direct port still exist. |
| Docker logs | Confirm daemon default activation and container recreation. |
| Identity | Confirm per-user bindings and Homarr linking; transport verification did not establish them. |