Files
notes/docs/home-server/operations.md
T
2026-10-26 12:10:24 +00:00

125 lines
6.5 KiB
Markdown

---
schema_version: 1
id: home-server.operations
type: reference
scope: [jupiter, mercury, design-decisions, monitoring, recovery]
sensitivity: private-infrastructure
last_reviewed: "2026-10-26"
sources:
- kind: owner-report
reference: "AirVPN local downloads live, Ultra.cc/rclone retired; cost saving and simpler architecture; GBP 56.06 three-year plan ends 2029-10-25; imports/hardlinks confirmed, full isolation checks and backups outstanding, 2026-10-26"
- kind: owner-report
reference: "RAID/media tradeoff, CGNAT/public IPv4 rationale, no alerting and running monitoring/log containers, 2026-10-06"
- kind: owner-report
reference: "Outages over one day catastrophic; no independent credentials except restic key, recovery evidence, host-config archive or operational rules, 2026-10-06"
- kind: owner-report
reference: "OIDC and three-group motivation; Authentik fully deployed and Authelia retired; updated six-directory backup plan; independent recovery custody unchanged, 2026-10-16"
- kind: owner-report
reference: "Notifications received in Discord server Jupiter: Beszel in #beszel-alerts, Backrest in #backrest-alerts, and Mercury Uptime Kuma in #uptimekuma-alerts, 2026-10-23"
related: [home-server.reference, home-server.host, home-server.mercury, home-server.downloads, home-server.networking, home-server.storage, home-server.authentication, home-server.backups, home-server.deployment]
update_triggers: [data-priority-change, architecture-change, alerting-change, recovery-test]
unknowns:
- acceptable-data-loss-and-recovery-order-by-service
- data-criticality-outside-the-six-backed-up-directories
- tested-full-host-recovery-order
- monitoring-review-frequency
- configured-alert-rules-and-uptime-monitor-coverage
---
# Operations / decisions
## Owner decisions
| Decision | Reason / accepted tradeoff |
| --- | --- |
| RAID 0 for HDD data | Capacity prioritized over disk redundancy; downloaded media can be reacquired |
| Back up important state | Six application directories covered, subject to exclusions; media redundancy is not the objective |
| Mercury public edge | Static public IPv4; home ISP uses CGNAT |
| No residential public ingress | Owner preference to avoid exposing the home IP |
| AirVPN instead of Ultra.cc | Live local downloads reduce cost and simplify architecture: no remote seedbox or SFTP/FUSE import dependency; GBP 56.06 prepaid for three years, expires Thursday, 25 October 2029 |
| Authentik instead of Authelia | Native OIDC rather than only forward-auth; retain forward-auth where needed |
| Application access | 2026-10-26: replace the former Admin/Developers/Clients groups with manual per-user service bindings; keep `jay` as administrator with access to every configured application; repository prepared, deployment unconfirmed |
RAID 0 still stores important state alongside media: either HDD failure loses
the array; recovery depends on working backups. Owner intent is not evidence
that every important dataset is covered.
The Authentik migration is complete by owner report; Authelia is retired.
Exact repository group names, app assignments and MFA policy are in
[Authentication](authentication.md). Group access is not the same as application
administrator privileges or Authentik superuser status.
## Download migration
[Local downloads](downloads.md) are live by owner report on 2026-10-26.
Direct home-ISP egress, both Arr hardlink imports, seeding/removal and the public
UI's Authentik plus qBittorrent login are confirmed. Ultra.cc is
cancelled/expired, retained files are handled and the old mount/service removed.
The subscription saving is approximately GBP 35.31/year before incremental
electricity, compared with the former GBP 4.50/month seedbox.
Remaining operational work: complete peer-ingress, DNS/IPv6 and
VPN-failure/startup/reboot isolation checks, and back up qBittorrent state/defaults
and AirVPN secrets. Normal successful downloads do not prove fail-closed behaviour.
The retired seedbox must not be assumed available for rollback.
## Observability
| Facility | Current configuration |
| --- | --- |
| Host/container monitoring | Beszel hub and host-network agent present |
| Container logs | Dozzle with Docker socket proxy present |
| Uptime monitoring | Uptime Kuma on Mercury at `uptime.umbra.mom` |
| Automatic alerting | Discord delivery configured for Beszel, Backrest and Uptime Kuma; receipt confirmed by owner |
### Discord notifications
The owner confirms receiving notifications in the Discord server **Jupiter**
as of 2026-10-23:
| Source | Channel |
| --- | --- |
| Beszel | `#beszel-alerts` |
| Backrest | `#backrest-alerts` |
| Uptime Kuma on Mercury | `#uptimekuma-alerts` |
Notification receipt does not establish complete monitoring coverage. Exact
enabled alert rules, thresholds, Backrest hook conditions and Uptime Kuma
monitor targets have not been recorded. Per-container outage detection,
whole-Jupiter outage detection, external monitoring of Mercury and missed-backup
alerts are not yet confirmed. No webhook URLs or tokens are stored in these notes.
The backup plan stops Authentik, interrupting new OIDC logins and forward-auth
access. Authentik/Wardrowbe restart hooks ignore errors, so backup completion
alone does not establish that authentication or applications recovered.
## Recovery status
| Item | Current status |
| --- | --- |
| Jupiter outage tolerance | At most 24 hours; longer is catastrophic to the owner |
| Recovery time | Not measured; 24-hour objective is not demonstrated |
| Acceptable data loss / RPO | Not defined |
| Independent recovery secrets | Restic encryption key only |
| Backup/integrity evidence | None supplied |
| Restore exercise | Never performed |
| Independent host-config archive | Not maintained |
| Operational rules / rollback policy | Not established |
Backup scope is documented in [Backups](backups.md). The six-directory plan now
includes Authentik identity/MFA state and Wardrowbe data, but does not establish
OS, broader host-configuration or named-volume recovery. There is still no
independent Authentik admin/MFA or stack-secret recovery arrangement.
## Proposed recovery storage
Not implemented:
- Private Git repository: sanitized host configuration and bootstrap instructions.
- Encrypted off-host/offline archive: required credentials and secret-bearing configuration.
- Independent access instructions: account recovery, archive location and decryption.
Access/decryption must work without Jupiter or Portainer. A private Git
repository is not a substitute for protecting secret values. Restic key custody
alone does not provide Google Drive access.