Files
notes/docs/home-server/operations.md
T

187 lines
10 KiB
Markdown

---
schema_version: 1
id: home-server.operations
type: reference
scope: [jupiter, mercury, design-decisions, monitoring, recovery]
sensitivity: private-infrastructure
last_reviewed: "2026-10-26"
sources:
- kind: owner-report
reference: "AirVPN local downloads live, Ultra.cc/rclone retired; cost saving and simpler architecture; GBP 56.06 three-year plan ends 2029-10-25; imports/hardlinks confirmed, full isolation checks and backups outstanding, 2026-10-26"
- kind: owner-report
reference: "RAID/media tradeoff, CGNAT/public IPv4 rationale, no alerting and running monitoring/log containers, 2026-10-06"
- kind: owner-report
reference: "Outages over one day catastrophic; no independent credentials except restic key, recovery evidence, host-config archive or operational rules, 2026-10-06"
- kind: owner-report
reference: "OIDC and three-group motivation; Authentik fully deployed and Authelia retired; updated six-directory backup plan; independent recovery custody unchanged, 2026-10-16"
- kind: owner-report
reference: "Notifications received in Discord server Jupiter: Beszel in #beszel-alerts, Backrest in #backrest-alerts, and Mercury Uptime Kuma in #uptimekuma-alerts, 2026-10-23"
- kind: owner-report
reference: "Mercury Uptime Kuma only checks WireGuard tunnel health and Jupiter online status, notifying Jay on Discord when down; Kuma per-container monitoring not configured, 2026-10-26"
- kind: owner-report
reference: "Beszel sends Discord alerts for any systemd error or any container down for more than two minutes, 2026-10-26"
- kind: owner-report
reference: "Mercury monitored in Beszel; CPU above 90% for 10 minutes, memory above 90% for 10 minutes and any failed systemd services alerts configured, 2026-10-26"
- kind: documentation
reference: "Beszel v0.21.0 upstream internal/site/src/lib/alerts.ts and internal/alerts/alerts_container.go: Container Health alerts detect Docker unhealthy status, 2026-10-26"
related: [home-server.reference, home-server.host, home-server.mercury, home-server.downloads, home-server.networking, home-server.storage, home-server.authentication, home-server.backups, home-server.deployment]
update_triggers: [data-priority-change, architecture-change, alerting-change, recovery-test]
unknowns:
- acceptable-data-loss-and-recovery-order-by-service
- data-criticality-outside-the-six-backed-up-directories
- tested-full-host-recovery-order
- monitoring-review-frequency
- other-alert-rules-thresholds-and-exact-uptime-probes
---
# Operations / decisions
## Owner decisions
| Decision | Reason / accepted tradeoff |
| --- | --- |
| RAID 0 for HDD data | Capacity prioritized over disk redundancy; downloaded media can be reacquired |
| Back up important state | Six application directories covered, subject to exclusions; media redundancy is not the objective |
| Mercury public edge | Static public IPv4; home ISP uses CGNAT |
| No residential public ingress | Owner preference to avoid exposing the home IP |
| AirVPN instead of Ultra.cc | Live local downloads reduce cost and simplify architecture: no remote seedbox or SFTP/FUSE import dependency; GBP 56.06 prepaid for three years, expires Thursday, 25 October 2029 |
| Authentik instead of Authelia | Native OIDC rather than only forward-auth; retain forward-auth where needed |
| Application access | 2026-10-26: replace the former Admin/Developers/Clients groups with manual per-user service bindings; keep `jay` as administrator with access to every configured application; repository prepared, deployment unconfirmed |
RAID 0 still stores important state alongside media: either HDD failure loses
the array; recovery depends on working backups. Owner intent is not evidence
that every important dataset is covered.
The Authentik migration is complete by owner report; Authelia is retired.
Exact repository group names, app assignments and MFA policy are in
[Authentication](authentication.md). Group access is not the same as application
administrator privileges or Authentik superuser status.
## Download migration
[Local downloads](downloads.md) are live by owner report on 2026-10-26.
Direct home-ISP egress, both Arr hardlink imports, seeding/removal and the public
UI's Authentik plus qBittorrent login are confirmed. Ultra.cc is
cancelled/expired, retained files are handled and the old mount/service removed.
The subscription saving is approximately GBP 35.31/year before incremental
electricity, compared with the former GBP 4.50/month seedbox.
Remaining operational work: complete peer-ingress, DNS/IPv6 and
VPN-failure/startup/reboot isolation checks, and back up qBittorrent state/defaults
and AirVPN secrets. Normal successful downloads do not prove fail-closed behaviour.
The retired seedbox must not be assumed available for rollback.
## Observability
| Facility | Current configuration |
| --- | --- |
| Host/container monitoring | Beszel monitors Jupiter and Mercury; existing systemd/container alerts plus Mercury CPU, memory and failed systemd service alerts (owner-confirmed) |
| Container logs | Dozzle with Docker socket proxy present |
| Uptime monitoring | Uptime Kuma on Mercury at `uptime.umbra.mom`; WireGuard tunnel health and Jupiter online status only |
| Automatic alerting | Discord delivery configured for Beszel, Backrest and Uptime Kuma; receipt confirmed by owner |
### Discord notifications
The owner confirms receiving notifications in the Discord server **Jupiter**
as of 2026-10-23:
| Source | Channel |
| --- | --- |
| Beszel | `#beszel-alerts` |
| Backrest | `#backrest-alerts` |
| Uptime Kuma on Mercury | `#uptimekuma-alerts` |
Mercury's Uptime Kuma alerts **Jay** in `#uptimekuma-alerts` when the WireGuard
tunnel or Jupiter availability checks fail; this scope was confirmed on
2026-10-26. Individual containers and applications are not currently monitored
by Kuma.
Beszel sends alerts to **`#beszel-alerts`** for **any systemd error** or **any
container down for more than two minutes**, as confirmed by the owner on
2026-10-26. These are configured alerts, not proposed additions.
Mercury is also monitored in Beszel, with the following alerts configured by
owner report on 2026-10-26:
| Mercury alert | Trigger |
| --- | --- |
| CPU usage | Above 90% for 10 minutes |
| Memory usage | Above 90% for 10 minutes |
| Failed Services | Any failed systemd service |
These settings are stored in the Beszel hub UI/database, not the agent Compose
file. The failed-services rule covers failed systemd units, not all journal
errors.
Exact Kuma probes and timing, other Beszel rules/thresholds and Backrest hook
conditions have not been recorded. Monitoring of Mercury independent of Jupiter and
missed-backup alerts remain unconfirmed. No webhook URLs or tokens are stored
in these notes.
### Alert coverage and optional additions
Beszel's systemd/container outage alerting above is owner-reported deployed
state. The upstream Beszel `0.21.0` **Container Health** rule specifically checks
Docker `unhealthy` status, and **Failed Services** checks failed systemd units,
not every error-level journal entry. The exact live rules and stopped/missing
container detection have not been independently inspected.
The following are optional additions, not configured Kuma monitors:
1. In Mercury's Uptime Kuma, add an **HTTP(s)** monitor for each important
application's health endpoint, or a **TCP Port** monitor where HTTP is not
available. Enable the existing Discord notification on each monitor.
Start with a 60-second interval and two retries to tolerate brief restarts.
Check the application itself, not just a proxy or authentication login page.
Use a stable health endpoint with expected status/body; do not disable
authentication or expose private services solely for monitoring.
2. For additional container-running-state checks, Kuma's **Docker Container**
monitor needs a Docker host connection to the relevant machine. Mercury's
Docker socket only describes Mercury, not Jupiter, and a Portainer environment
connection does not automatically provide Kuma with Docker access. Remote
Docker monitoring needs a separately secured connection; do not expose an
unauthenticated Docker API or mount Jupiter's Docker socket into Kuma merely
to add application availability checks.
3. During a planned maintenance window, stop a non-critical monitored service,
confirm Jay receives the outage alert, then restart it and confirm recovery.
Account for scheduled backup stops before enabling noisy alerts.
Application probes detect service failures even when Docker still reports a
container as running, but are not an inventory of every container's state.
Beszel on Jupiter cannot deliver alerts while Jupiter or the Beszel hub is down;
keep Mercury's independent tunnel/host checks.
The backup plan stops Authentik, interrupting new OIDC logins and forward-auth
access. Authentik/Wardrowbe restart hooks ignore errors, so backup completion
alone does not establish that authentication or applications recovered.
## Recovery status
| Item | Current status |
| --- | --- |
| Jupiter outage tolerance | At most 24 hours; longer is catastrophic to the owner |
| Recovery time | Not measured; 24-hour objective is not demonstrated |
| Acceptable data loss / RPO | Not defined |
| Independent recovery secrets | Restic encryption key only |
| Backup/integrity evidence | None supplied |
| Restore exercise | Never performed |
| Independent host-config archive | Not maintained |
| Operational rules / rollback policy | Not established |
Backup scope is documented in [Backups](backups.md). The six-directory plan now
includes Authentik identity/MFA state and Wardrowbe data, but does not establish
OS, broader host-configuration or named-volume recovery. There is still no
independent Authentik admin/MFA or stack-secret recovery arrangement.
## Proposed recovery storage
Not implemented:
- Private Git repository: sanitized host configuration and bootstrap instructions.
- Encrypted off-host/offline archive: required credentials and secret-bearing configuration.
- Independent access instructions: account recovery, archive location and decryption.
Access/decryption must work without Jupiter or Portainer. A private Git
repository is not a substitute for protecting secret values. Restic key custody
alone does not provide Google Drive access.