Files
notes/docs/home-server/operations.md
T

11 KiB

schema_version, id, type, scope, sensitivity, last_reviewed, sources, related, update_triggers, unknowns
schema_version id type scope sensitivity last_reviewed sources related update_triggers unknowns
1 home-server.operations reference
jupiter
mercury
design-decisions
monitoring
recovery
private-infrastructure 2026-10-26
kind reference
owner-report Sonarr, Radarr and Prowlarr now have alerting enabled; destination and enabled events not supplied, 2026-10-26
kind reference
owner-report AirVPN local downloads live, Ultra.cc/rclone retired; cost saving and simpler architecture; GBP 56.06 three-year plan ends 2029-10-25; imports/hardlinks confirmed, full isolation checks and backups outstanding, 2026-10-26
kind reference
owner-report RAID/media tradeoff, CGNAT/public IPv4 rationale, no alerting and running monitoring/log containers, 2026-10-06
kind reference
owner-report Outages over one day catastrophic; no independent credentials except restic key, recovery evidence, host-config archive or operational rules, 2026-10-06
kind reference
owner-report OIDC and three-group motivation; Authentik fully deployed and Authelia retired; updated six-directory backup plan; independent recovery custody unchanged, 2026-10-16
kind reference
owner-report Notifications received in Discord server Jupiter: Beszel in #beszel-alerts, Backrest in #backrest-alerts, and Mercury Uptime Kuma in #uptimekuma-alerts, 2026-10-23
kind reference
owner-report Mercury Uptime Kuma only checks WireGuard tunnel health and Jupiter online status, notifying Jay on Discord when down; Kuma per-container monitoring not configured, 2026-10-26
kind reference
owner-report Beszel sends Discord alerts for any systemd error or any container down for more than two minutes, 2026-10-26
kind reference
owner-report Mercury monitored in Beszel; CPU above 90% for 10 minutes, memory above 90% for 10 minutes and any failed systemd services alerts configured, 2026-10-26
kind reference
documentation Beszel v0.21.0 upstream internal/site/src/lib/alerts.ts and internal/alerts/alerts_container.go: Container Health alerts detect Docker unhealthy status, 2026-10-26
home-server.reference
home-server.host
home-server.mercury
home-server.downloads
home-server.networking
home-server.storage
home-server.authentication
home-server.backups
home-server.deployment
data-priority-change
architecture-change
alerting-change
recovery-test
acceptable-data-loss-and-recovery-order-by-service
data-criticality-outside-the-six-backed-up-directories
tested-full-host-recovery-order
monitoring-review-frequency
other-alert-rules-thresholds-and-exact-uptime-probes
arr-alert-destinations-and-enabled-events

Operations / decisions

Owner decisions

Decision Reason / accepted tradeoff
RAID 0 for HDD data Capacity prioritized over disk redundancy; downloaded media can be reacquired
Back up important state Six application directories covered, subject to exclusions; media redundancy is not the objective
Mercury public edge Static public IPv4; home ISP uses CGNAT
No residential public ingress Owner preference to avoid exposing the home IP
AirVPN instead of Ultra.cc Live local downloads reduce cost and simplify architecture: no remote seedbox or SFTP/FUSE import dependency; GBP 56.06 prepaid for three years, expires Thursday, 25 October 2029
Authentik instead of Authelia Native OIDC rather than only forward-auth; retain forward-auth where needed
Application access 2026-10-26: replace the former Admin/Developers/Clients groups with manual per-user service bindings; keep jay as administrator with access to every configured application; repository prepared, deployment unconfirmed

RAID 0 still stores important state alongside media: either HDD failure loses the array; recovery depends on working backups. Owner intent is not evidence that every important dataset is covered.

The Authentik migration is complete by owner report; Authelia is retired. Exact repository group names, app assignments and MFA policy are in Authentication. Group access is not the same as application administrator privileges or Authentik superuser status.

Download migration

Local downloads are live by owner report on 2026-10-26. Direct home-ISP egress, both Arr hardlink imports, seeding/removal and the public UI's Authentik plus qBittorrent login are confirmed. Ultra.cc is cancelled/expired, retained files are handled and the old mount/service removed. The subscription saving is approximately GBP 35.31/year before incremental electricity, compared with the former GBP 4.50/month seedbox.

Remaining operational work: complete peer-ingress, DNS/IPv6 and VPN-failure/startup/reboot isolation checks, and back up qBittorrent state/defaults and AirVPN secrets. Normal successful downloads do not prove fail-closed behaviour. The retired seedbox must not be assumed available for rollback.

Observability

Facility Current configuration
Host/container monitoring Beszel monitors Jupiter and Mercury; existing systemd/container alerts plus Mercury CPU, memory and failed systemd service alerts (owner-confirmed)
Container logs Dozzle with Docker socket proxy present
Uptime monitoring Uptime Kuma on Mercury at uptime.umbra.mom; WireGuard tunnel health and Jupiter online status only
Automatic alerting Discord delivery configured for Beszel, Backrest and Uptime Kuma; receipt confirmed by owner. Sonarr, Radarr and Prowlarr alerting enabled by owner report; destinations/events not recorded

Arr application alerting

Sonarr, Radarr and Prowlarr all have alerting enabled, confirmed by the owner on 2026-10-26. Notification destinations, enabled events and delivery confirmation have not been recorded. These settings are application-managed state, not declared in the Compose files.

Do not infer that this covers searches returning no acceptable releases, missing media or stalled downloads: those conditions are not equivalent to application health issues or explicit download failures. Alerting for them requires separate confirmation of the configured events or additional monitoring. See Local downloads / AirVPN.

Discord notifications

The owner confirms receiving notifications in the Discord server Jupiter as of 2026-10-23:

Source Channel
Beszel #beszel-alerts
Backrest #backrest-alerts
Uptime Kuma on Mercury #uptimekuma-alerts

Mercury's Uptime Kuma alerts Jay in #uptimekuma-alerts when the WireGuard tunnel or Jupiter availability checks fail; this scope was confirmed on 2026-10-26. Individual containers and applications are not currently monitored by Kuma.

Beszel sends alerts to #beszel-alerts for any systemd error or any container down for more than two minutes, as confirmed by the owner on 2026-10-26. These are configured alerts, not proposed additions.

Mercury is also monitored in Beszel, with the following alerts configured by owner report on 2026-10-26:

Mercury alert Trigger
CPU usage Above 90% for 10 minutes
Memory usage Above 90% for 10 minutes
Failed Services Any failed systemd service

These settings are stored in the Beszel hub UI/database, not the agent Compose file. The failed-services rule covers failed systemd units, not all journal errors.

Exact Kuma probes and timing, other Beszel rules/thresholds and Backrest hook conditions have not been recorded. Monitoring of Mercury independent of Jupiter and missed-backup alerts remain unconfirmed. No webhook URLs or tokens are stored in these notes.

Alert coverage and optional additions

Beszel's systemd/container outage alerting above is owner-reported deployed state. The upstream Beszel 0.21.0 Container Health rule specifically checks Docker unhealthy status, and Failed Services checks failed systemd units, not every error-level journal entry. The exact live rules and stopped/missing container detection have not been independently inspected.

The following are optional additions, not configured Kuma monitors:

  1. In Mercury's Uptime Kuma, add an HTTP(s) monitor for each important application's health endpoint, or a TCP Port monitor where HTTP is not available. Enable the existing Discord notification on each monitor. Start with a 60-second interval and two retries to tolerate brief restarts. Check the application itself, not just a proxy or authentication login page. Use a stable health endpoint with expected status/body; do not disable authentication or expose private services solely for monitoring.
  2. For additional container-running-state checks, Kuma's Docker Container monitor needs a Docker host connection to the relevant machine. Mercury's Docker socket only describes Mercury, not Jupiter, and a Portainer environment connection does not automatically provide Kuma with Docker access. Remote Docker monitoring needs a separately secured connection; do not expose an unauthenticated Docker API or mount Jupiter's Docker socket into Kuma merely to add application availability checks.
  3. During a planned maintenance window, stop a non-critical monitored service, confirm Jay receives the outage alert, then restart it and confirm recovery. Account for scheduled backup stops before enabling noisy alerts.

Application probes detect service failures even when Docker still reports a container as running, but are not an inventory of every container's state. Beszel on Jupiter cannot deliver alerts while Jupiter or the Beszel hub is down; keep Mercury's independent tunnel/host checks.

The backup plan stops Authentik, interrupting new OIDC logins and forward-auth access. Authentik/Wardrowbe restart hooks ignore errors, so backup completion alone does not establish that authentication or applications recovered.

Recovery status

Item Current status
Jupiter outage tolerance At most 24 hours; longer is catastrophic to the owner
Recovery time Not measured; 24-hour objective is not demonstrated
Acceptable data loss / RPO Not defined
Independent recovery secrets Restic encryption key only
Backup/integrity evidence None supplied
Restore exercise Never performed
Independent host-config archive Not maintained
Operational rules / rollback policy Not established

Backup scope is documented in Backups. The six-directory plan now includes Authentik identity/MFA state and Wardrowbe data, but does not establish OS, broader host-configuration or named-volume recovery. There is still no independent Authentik admin/MFA or stack-secret recovery arrangement.

Proposed recovery storage

Not implemented:

  • Private Git repository: sanitized host configuration and bootstrap instructions.
  • Encrypted off-host/offline archive: required credentials and secret-bearing configuration.
  • Independent access instructions: account recovery, archive location and decryption.

Access/decryption must work without Jupiter or Portainer. A private Git repository is not a substitute for protecting secret values. Restic key custody alone does not provide Google Drive access.