--- schema_version: 1 id: home-server.operations type: reference scope: [jupiter, mercury, design-decisions, monitoring, recovery] sensitivity: private-infrastructure last_reviewed: "2026-10-23" sources: - kind: owner-report reference: "RAID/media tradeoff, CGNAT/public IPv4 rationale, no alerting and running monitoring/log containers, 2026-10-06" - kind: owner-report reference: "Outages over one day catastrophic; no independent credentials except restic key, recovery evidence, host-config archive or operational rules, 2026-10-06" - kind: owner-report reference: "OIDC and three-group motivation; Authentik fully deployed and Authelia retired; updated six-directory backup plan; independent recovery custody unchanged, 2026-10-16" - kind: owner-report reference: "Notifications received in Discord server Jupiter: Beszel in #beszel-alerts, Backrest in #backrest-alerts, and Mercury Uptime Kuma in #uptimekuma-alerts, 2026-10-23" related: [home-server.reference, home-server.host, home-server.mercury, home-server.networking, home-server.storage, home-server.authentication, home-server.backups, home-server.deployment] update_triggers: [data-priority-change, architecture-change, alerting-change, recovery-test] unknowns: - acceptable-data-loss-and-recovery-order-by-service - data-criticality-outside-the-six-backed-up-directories - tested-full-host-recovery-order - monitoring-review-frequency - configured-alert-rules-and-uptime-monitor-coverage --- # Operations / decisions ## Owner decisions | Decision | Reason / accepted tradeoff | | --- | --- | | RAID 0 for HDD data | Capacity prioritized over disk redundancy; downloaded media can be reacquired | | Back up important state | Six application directories covered, subject to exclusions; media redundancy is not the objective | | Mercury public edge | Static public IPv4; home ISP uses CGNAT | | No residential public ingress | Owner preference to avoid exposing the home IP | | Authentik instead of Authelia | Native OIDC rather than only forward-auth; retain forward-auth where needed | | Three access categories | Admin: owner only, server health/operations; Developers: development services such as Gitea/Portainer; Clients: useful end-user services such as Wardrowbe/Paperless | RAID 0 still stores important state alongside media: either HDD failure loses the array; recovery depends on working backups. Owner intent is not evidence that every important dataset is covered. The Authentik migration is complete by owner report; Authelia is retired. Exact repository group names, app assignments and MFA policy are in [Authentication](authentication.md). Group access is not the same as application administrator privileges or Authentik superuser status. ## Observability | Facility | Current configuration | | --- | --- | | Host/container monitoring | Beszel hub and host-network agent present | | Container logs | Dozzle with Docker socket proxy present | | Uptime monitoring | Uptime Kuma on Mercury at `uptime.umbra.mom` | | Automatic alerting | Discord delivery configured for Beszel, Backrest and Uptime Kuma; receipt confirmed by owner | ### Discord notifications The owner confirms receiving notifications in the Discord server **Jupiter** as of 2026-10-23: | Source | Channel | | --- | --- | | Beszel | `#beszel-alerts` | | Backrest | `#backrest-alerts` | | Uptime Kuma on Mercury | `#uptimekuma-alerts` | Notification receipt does not establish complete monitoring coverage. Exact enabled alert rules, thresholds, Backrest hook conditions and Uptime Kuma monitor targets have not been recorded. Per-container outage detection, whole-Jupiter outage detection, external monitoring of Mercury and missed-backup alerts are not yet confirmed. No webhook URLs or tokens are stored in these notes. The backup plan stops Authentik, interrupting new OIDC logins and forward-auth access. Authentik/Wardrowbe restart hooks ignore errors, so backup completion alone does not establish that authentication or applications recovered. ## Recovery status | Item | Current status | | --- | --- | | Jupiter outage tolerance | At most 24 hours; longer is catastrophic to the owner | | Recovery time | Not measured; 24-hour objective is not demonstrated | | Acceptable data loss / RPO | Not defined | | Independent recovery secrets | Restic encryption key only | | Backup/integrity evidence | None supplied | | Restore exercise | Never performed | | Independent host-config archive | Not maintained | | Operational rules / rollback policy | Not established | Backup scope is documented in [Backups](backups.md). The six-directory plan now includes Authentik identity/MFA state and Wardrowbe data, but does not establish OS, broader host-configuration or named-volume recovery. There is still no independent Authentik admin/MFA or stack-secret recovery arrangement. ## Proposed recovery storage Not implemented: - Private Git repository: sanitized host configuration and bootstrap instructions. - Encrypted off-host/offline archive: required credentials and secret-bearing configuration. - Independent access instructions: account recovery, archive location and decryption. Access/decryption must work without Jupiter or Portainer. A private Git repository is not a substitute for protecting secret values. Restic key custody alone does not provide Google Drive access.