187 lines
10 KiB
Markdown
187 lines
10 KiB
Markdown
---
|
|
schema_version: 1
|
|
id: home-server.operations
|
|
type: reference
|
|
scope: [jupiter, mercury, design-decisions, monitoring, recovery]
|
|
sensitivity: private-infrastructure
|
|
last_reviewed: "2026-10-26"
|
|
sources:
|
|
- kind: owner-report
|
|
reference: "AirVPN local downloads live, Ultra.cc/rclone retired; cost saving and simpler architecture; GBP 56.06 three-year plan ends 2029-10-25; imports/hardlinks confirmed, full isolation checks and backups outstanding, 2026-10-26"
|
|
- kind: owner-report
|
|
reference: "RAID/media tradeoff, CGNAT/public IPv4 rationale, no alerting and running monitoring/log containers, 2026-10-06"
|
|
- kind: owner-report
|
|
reference: "Outages over one day catastrophic; no independent credentials except restic key, recovery evidence, host-config archive or operational rules, 2026-10-06"
|
|
- kind: owner-report
|
|
reference: "OIDC and three-group motivation; Authentik fully deployed and Authelia retired; updated six-directory backup plan; independent recovery custody unchanged, 2026-10-16"
|
|
- kind: owner-report
|
|
reference: "Notifications received in Discord server Jupiter: Beszel in #beszel-alerts, Backrest in #backrest-alerts, and Mercury Uptime Kuma in #uptimekuma-alerts, 2026-10-23"
|
|
- kind: owner-report
|
|
reference: "Mercury Uptime Kuma only checks WireGuard tunnel health and Jupiter online status, notifying Jay on Discord when down; Kuma per-container monitoring not configured, 2026-10-26"
|
|
- kind: owner-report
|
|
reference: "Beszel sends Discord alerts for any systemd error or any container down for more than two minutes, 2026-10-26"
|
|
- kind: owner-report
|
|
reference: "Mercury monitored in Beszel; CPU above 90% for 10 minutes, memory above 90% for 10 minutes and any failed systemd services alerts configured, 2026-10-26"
|
|
- kind: documentation
|
|
reference: "Beszel v0.21.0 upstream internal/site/src/lib/alerts.ts and internal/alerts/alerts_container.go: Container Health alerts detect Docker unhealthy status, 2026-10-26"
|
|
related: [home-server.reference, home-server.host, home-server.mercury, home-server.downloads, home-server.networking, home-server.storage, home-server.authentication, home-server.backups, home-server.deployment]
|
|
update_triggers: [data-priority-change, architecture-change, alerting-change, recovery-test]
|
|
unknowns:
|
|
- acceptable-data-loss-and-recovery-order-by-service
|
|
- data-criticality-outside-the-six-backed-up-directories
|
|
- tested-full-host-recovery-order
|
|
- monitoring-review-frequency
|
|
- other-alert-rules-thresholds-and-exact-uptime-probes
|
|
---
|
|
|
|
# Operations / decisions
|
|
|
|
## Owner decisions
|
|
|
|
| Decision | Reason / accepted tradeoff |
|
|
| --- | --- |
|
|
| RAID 0 for HDD data | Capacity prioritized over disk redundancy; downloaded media can be reacquired |
|
|
| Back up important state | Six application directories covered, subject to exclusions; media redundancy is not the objective |
|
|
| Mercury public edge | Static public IPv4; home ISP uses CGNAT |
|
|
| No residential public ingress | Owner preference to avoid exposing the home IP |
|
|
| AirVPN instead of Ultra.cc | Live local downloads reduce cost and simplify architecture: no remote seedbox or SFTP/FUSE import dependency; GBP 56.06 prepaid for three years, expires Thursday, 25 October 2029 |
|
|
| Authentik instead of Authelia | Native OIDC rather than only forward-auth; retain forward-auth where needed |
|
|
| Application access | 2026-10-26: replace the former Admin/Developers/Clients groups with manual per-user service bindings; keep `jay` as administrator with access to every configured application; repository prepared, deployment unconfirmed |
|
|
|
|
RAID 0 still stores important state alongside media: either HDD failure loses
|
|
the array; recovery depends on working backups. Owner intent is not evidence
|
|
that every important dataset is covered.
|
|
|
|
The Authentik migration is complete by owner report; Authelia is retired.
|
|
Exact repository group names, app assignments and MFA policy are in
|
|
[Authentication](authentication.md). Group access is not the same as application
|
|
administrator privileges or Authentik superuser status.
|
|
|
|
## Download migration
|
|
|
|
[Local downloads](downloads.md) are live by owner report on 2026-10-26.
|
|
Direct home-ISP egress, both Arr hardlink imports, seeding/removal and the public
|
|
UI's Authentik plus qBittorrent login are confirmed. Ultra.cc is
|
|
cancelled/expired, retained files are handled and the old mount/service removed.
|
|
The subscription saving is approximately GBP 35.31/year before incremental
|
|
electricity, compared with the former GBP 4.50/month seedbox.
|
|
|
|
Remaining operational work: complete peer-ingress, DNS/IPv6 and
|
|
VPN-failure/startup/reboot isolation checks, and back up qBittorrent state/defaults
|
|
and AirVPN secrets. Normal successful downloads do not prove fail-closed behaviour.
|
|
The retired seedbox must not be assumed available for rollback.
|
|
|
|
## Observability
|
|
|
|
| Facility | Current configuration |
|
|
| --- | --- |
|
|
| Host/container monitoring | Beszel monitors Jupiter and Mercury; existing systemd/container alerts plus Mercury CPU, memory and failed systemd service alerts (owner-confirmed) |
|
|
| Container logs | Dozzle with Docker socket proxy present |
|
|
| Uptime monitoring | Uptime Kuma on Mercury at `uptime.umbra.mom`; WireGuard tunnel health and Jupiter online status only |
|
|
| Automatic alerting | Discord delivery configured for Beszel, Backrest and Uptime Kuma; receipt confirmed by owner |
|
|
|
|
### Discord notifications
|
|
|
|
The owner confirms receiving notifications in the Discord server **Jupiter**
|
|
as of 2026-10-23:
|
|
|
|
| Source | Channel |
|
|
| --- | --- |
|
|
| Beszel | `#beszel-alerts` |
|
|
| Backrest | `#backrest-alerts` |
|
|
| Uptime Kuma on Mercury | `#uptimekuma-alerts` |
|
|
|
|
Mercury's Uptime Kuma alerts **Jay** in `#uptimekuma-alerts` when the WireGuard
|
|
tunnel or Jupiter availability checks fail; this scope was confirmed on
|
|
2026-10-26. Individual containers and applications are not currently monitored
|
|
by Kuma.
|
|
|
|
Beszel sends alerts to **`#beszel-alerts`** for **any systemd error** or **any
|
|
container down for more than two minutes**, as confirmed by the owner on
|
|
2026-10-26. These are configured alerts, not proposed additions.
|
|
|
|
Mercury is also monitored in Beszel, with the following alerts configured by
|
|
owner report on 2026-10-26:
|
|
|
|
| Mercury alert | Trigger |
|
|
| --- | --- |
|
|
| CPU usage | Above 90% for 10 minutes |
|
|
| Memory usage | Above 90% for 10 minutes |
|
|
| Failed Services | Any failed systemd service |
|
|
|
|
These settings are stored in the Beszel hub UI/database, not the agent Compose
|
|
file. The failed-services rule covers failed systemd units, not all journal
|
|
errors.
|
|
|
|
Exact Kuma probes and timing, other Beszel rules/thresholds and Backrest hook
|
|
conditions have not been recorded. Monitoring of Mercury independent of Jupiter and
|
|
missed-backup alerts remain unconfirmed. No webhook URLs or tokens are stored
|
|
in these notes.
|
|
|
|
### Alert coverage and optional additions
|
|
|
|
Beszel's systemd/container outage alerting above is owner-reported deployed
|
|
state. The upstream Beszel `0.21.0` **Container Health** rule specifically checks
|
|
Docker `unhealthy` status, and **Failed Services** checks failed systemd units,
|
|
not every error-level journal entry. The exact live rules and stopped/missing
|
|
container detection have not been independently inspected.
|
|
|
|
The following are optional additions, not configured Kuma monitors:
|
|
|
|
1. In Mercury's Uptime Kuma, add an **HTTP(s)** monitor for each important
|
|
application's health endpoint, or a **TCP Port** monitor where HTTP is not
|
|
available. Enable the existing Discord notification on each monitor.
|
|
Start with a 60-second interval and two retries to tolerate brief restarts.
|
|
Check the application itself, not just a proxy or authentication login page.
|
|
Use a stable health endpoint with expected status/body; do not disable
|
|
authentication or expose private services solely for monitoring.
|
|
2. For additional container-running-state checks, Kuma's **Docker Container**
|
|
monitor needs a Docker host connection to the relevant machine. Mercury's
|
|
Docker socket only describes Mercury, not Jupiter, and a Portainer environment
|
|
connection does not automatically provide Kuma with Docker access. Remote
|
|
Docker monitoring needs a separately secured connection; do not expose an
|
|
unauthenticated Docker API or mount Jupiter's Docker socket into Kuma merely
|
|
to add application availability checks.
|
|
3. During a planned maintenance window, stop a non-critical monitored service,
|
|
confirm Jay receives the outage alert, then restart it and confirm recovery.
|
|
Account for scheduled backup stops before enabling noisy alerts.
|
|
|
|
Application probes detect service failures even when Docker still reports a
|
|
container as running, but are not an inventory of every container's state.
|
|
Beszel on Jupiter cannot deliver alerts while Jupiter or the Beszel hub is down;
|
|
keep Mercury's independent tunnel/host checks.
|
|
|
|
The backup plan stops Authentik, interrupting new OIDC logins and forward-auth
|
|
access. Authentik/Wardrowbe restart hooks ignore errors, so backup completion
|
|
alone does not establish that authentication or applications recovered.
|
|
|
|
## Recovery status
|
|
|
|
| Item | Current status |
|
|
| --- | --- |
|
|
| Jupiter outage tolerance | At most 24 hours; longer is catastrophic to the owner |
|
|
| Recovery time | Not measured; 24-hour objective is not demonstrated |
|
|
| Acceptable data loss / RPO | Not defined |
|
|
| Independent recovery secrets | Restic encryption key only |
|
|
| Backup/integrity evidence | None supplied |
|
|
| Restore exercise | Never performed |
|
|
| Independent host-config archive | Not maintained |
|
|
| Operational rules / rollback policy | Not established |
|
|
|
|
Backup scope is documented in [Backups](backups.md). The six-directory plan now
|
|
includes Authentik identity/MFA state and Wardrowbe data, but does not establish
|
|
OS, broader host-configuration or named-volume recovery. There is still no
|
|
independent Authentik admin/MFA or stack-secret recovery arrangement.
|
|
|
|
## Proposed recovery storage
|
|
|
|
Not implemented:
|
|
|
|
- Private Git repository: sanitized host configuration and bootstrap instructions.
|
|
- Encrypted off-host/offline archive: required credentials and secret-bearing configuration.
|
|
- Independent access instructions: account recovery, archive location and decryption.
|
|
|
|
Access/decryption must work without Jupiter or Portainer. A private Git
|
|
repository is not a substitute for protecting secret values. Restic key custody
|
|
alone does not provide Google Drive access.
|