125 lines
6.5 KiB
Markdown
125 lines
6.5 KiB
Markdown
---
|
|
schema_version: 1
|
|
id: home-server.operations
|
|
type: reference
|
|
scope: [jupiter, mercury, design-decisions, monitoring, recovery]
|
|
sensitivity: private-infrastructure
|
|
last_reviewed: "2026-10-26"
|
|
sources:
|
|
- kind: owner-report
|
|
reference: "AirVPN local downloads live, Ultra.cc/rclone retired; cost saving and simpler architecture; GBP 56.06 three-year plan ends 2029-10-25; imports/hardlinks confirmed, full isolation checks and backups outstanding, 2026-10-26"
|
|
- kind: owner-report
|
|
reference: "RAID/media tradeoff, CGNAT/public IPv4 rationale, no alerting and running monitoring/log containers, 2026-10-06"
|
|
- kind: owner-report
|
|
reference: "Outages over one day catastrophic; no independent credentials except restic key, recovery evidence, host-config archive or operational rules, 2026-10-06"
|
|
- kind: owner-report
|
|
reference: "OIDC and three-group motivation; Authentik fully deployed and Authelia retired; updated six-directory backup plan; independent recovery custody unchanged, 2026-10-16"
|
|
- kind: owner-report
|
|
reference: "Notifications received in Discord server Jupiter: Beszel in #beszel-alerts, Backrest in #backrest-alerts, and Mercury Uptime Kuma in #uptimekuma-alerts, 2026-10-23"
|
|
related: [home-server.reference, home-server.host, home-server.mercury, home-server.downloads, home-server.networking, home-server.storage, home-server.authentication, home-server.backups, home-server.deployment]
|
|
update_triggers: [data-priority-change, architecture-change, alerting-change, recovery-test]
|
|
unknowns:
|
|
- acceptable-data-loss-and-recovery-order-by-service
|
|
- data-criticality-outside-the-six-backed-up-directories
|
|
- tested-full-host-recovery-order
|
|
- monitoring-review-frequency
|
|
- configured-alert-rules-and-uptime-monitor-coverage
|
|
---
|
|
|
|
# Operations / decisions
|
|
|
|
## Owner decisions
|
|
|
|
| Decision | Reason / accepted tradeoff |
|
|
| --- | --- |
|
|
| RAID 0 for HDD data | Capacity prioritized over disk redundancy; downloaded media can be reacquired |
|
|
| Back up important state | Six application directories covered, subject to exclusions; media redundancy is not the objective |
|
|
| Mercury public edge | Static public IPv4; home ISP uses CGNAT |
|
|
| No residential public ingress | Owner preference to avoid exposing the home IP |
|
|
| AirVPN instead of Ultra.cc | Live local downloads reduce cost and simplify architecture: no remote seedbox or SFTP/FUSE import dependency; GBP 56.06 prepaid for three years, expires Thursday, 25 October 2029 |
|
|
| Authentik instead of Authelia | Native OIDC rather than only forward-auth; retain forward-auth where needed |
|
|
| Application access | 2026-10-26: replace the former Admin/Developers/Clients groups with manual per-user service bindings; keep `jay` as administrator with access to every configured application; repository prepared, deployment unconfirmed |
|
|
|
|
RAID 0 still stores important state alongside media: either HDD failure loses
|
|
the array; recovery depends on working backups. Owner intent is not evidence
|
|
that every important dataset is covered.
|
|
|
|
The Authentik migration is complete by owner report; Authelia is retired.
|
|
Exact repository group names, app assignments and MFA policy are in
|
|
[Authentication](authentication.md). Group access is not the same as application
|
|
administrator privileges or Authentik superuser status.
|
|
|
|
## Download migration
|
|
|
|
[Local downloads](downloads.md) are live by owner report on 2026-10-26.
|
|
Direct home-ISP egress, both Arr hardlink imports, seeding/removal and the public
|
|
UI's Authentik plus qBittorrent login are confirmed. Ultra.cc is
|
|
cancelled/expired, retained files are handled and the old mount/service removed.
|
|
The subscription saving is approximately GBP 35.31/year before incremental
|
|
electricity, compared with the former GBP 4.50/month seedbox.
|
|
|
|
Remaining operational work: complete peer-ingress, DNS/IPv6 and
|
|
VPN-failure/startup/reboot isolation checks, and back up qBittorrent state/defaults
|
|
and AirVPN secrets. Normal successful downloads do not prove fail-closed behaviour.
|
|
The retired seedbox must not be assumed available for rollback.
|
|
|
|
## Observability
|
|
|
|
| Facility | Current configuration |
|
|
| --- | --- |
|
|
| Host/container monitoring | Beszel hub and host-network agent present |
|
|
| Container logs | Dozzle with Docker socket proxy present |
|
|
| Uptime monitoring | Uptime Kuma on Mercury at `uptime.umbra.mom` |
|
|
| Automatic alerting | Discord delivery configured for Beszel, Backrest and Uptime Kuma; receipt confirmed by owner |
|
|
|
|
### Discord notifications
|
|
|
|
The owner confirms receiving notifications in the Discord server **Jupiter**
|
|
as of 2026-10-23:
|
|
|
|
| Source | Channel |
|
|
| --- | --- |
|
|
| Beszel | `#beszel-alerts` |
|
|
| Backrest | `#backrest-alerts` |
|
|
| Uptime Kuma on Mercury | `#uptimekuma-alerts` |
|
|
|
|
Notification receipt does not establish complete monitoring coverage. Exact
|
|
enabled alert rules, thresholds, Backrest hook conditions and Uptime Kuma
|
|
monitor targets have not been recorded. Per-container outage detection,
|
|
whole-Jupiter outage detection, external monitoring of Mercury and missed-backup
|
|
alerts are not yet confirmed. No webhook URLs or tokens are stored in these notes.
|
|
|
|
The backup plan stops Authentik, interrupting new OIDC logins and forward-auth
|
|
access. Authentik/Wardrowbe restart hooks ignore errors, so backup completion
|
|
alone does not establish that authentication or applications recovered.
|
|
|
|
## Recovery status
|
|
|
|
| Item | Current status |
|
|
| --- | --- |
|
|
| Jupiter outage tolerance | At most 24 hours; longer is catastrophic to the owner |
|
|
| Recovery time | Not measured; 24-hour objective is not demonstrated |
|
|
| Acceptable data loss / RPO | Not defined |
|
|
| Independent recovery secrets | Restic encryption key only |
|
|
| Backup/integrity evidence | None supplied |
|
|
| Restore exercise | Never performed |
|
|
| Independent host-config archive | Not maintained |
|
|
| Operational rules / rollback policy | Not established |
|
|
|
|
Backup scope is documented in [Backups](backups.md). The six-directory plan now
|
|
includes Authentik identity/MFA state and Wardrowbe data, but does not establish
|
|
OS, broader host-configuration or named-volume recovery. There is still no
|
|
independent Authentik admin/MFA or stack-secret recovery arrangement.
|
|
|
|
## Proposed recovery storage
|
|
|
|
Not implemented:
|
|
|
|
- Private Git repository: sanitized host configuration and bootstrap instructions.
|
|
- Encrypted off-host/offline archive: required credentials and secret-bearing configuration.
|
|
- Independent access instructions: account recovery, archive location and decryption.
|
|
|
|
Access/decryption must work without Jupiter or Portainer. A private Git
|
|
repository is not a substitute for protecting secret values. Restic key custody
|
|
alone does not provide Google Drive access.
|