update merucry monitoring docs

This commit is contained in:
John Gatward committed 2026-10-26 15:51:15 +00:00
1 parent 699f81436b
commit 6f9f7e7e6c
2 files changed
+120 -19

No files matched your search

+50 -11
View File
@@ -4,7 +4,7 @@ id: home-server.mercury
type: reference type: reference
scope: [mercury, vps, hosting, nginx, wireguard, firewall] scope: [mercury, vps, hosting, nginx, wireguard, firewall]
sensitivity: private-infrastructure sensitivity: private-infrastructure
last_reviewed: "2026-10-23" last_reviewed: "2026-10-26"
sources: sources:
- kind: owner-report - kind: owner-report
reference: "Fasthosts plan, fastfetch, nginx sites, iptables and fail2ban output, 2026-10-06" reference: "Fasthosts plan, fastfetch, nginx sites, iptables and fail2ban output, 2026-10-06"
@@ -12,6 +12,18 @@ sources:
reference: "nginx/WireGuard apt management and public IPv4/CGNAT rationale, 2026-10-06" reference: "nginx/WireGuard apt management and public IPv4/CGNAT rationale, 2026-10-06"
- kind: owner-report - kind: owner-report
reference: "Uptime service confirmed as Uptime Kuma; notifications received in Discord server Jupiter, channel #uptimekuma-alerts, 2026-10-23" reference: "Uptime service confirmed as Uptime Kuma; notifications received in Discord server Jupiter, channel #uptimekuma-alerts, 2026-10-23"
- kind: owner-report
reference: "Mercury uptime-kuma.sh supplied with bind mount, resource limits and log rotation, 2026-10-26"
- kind: owner-report
reference: "Mercury Uptime Kuma only checks WireGuard tunnel health and whether Jupiter is online; alerts Jay on Discord when down, 2026-10-26"
- kind: owner-report
reference: "Mercury monitored in Beszel; CPU above 90% for 10 minutes, memory above 90% for 10 minutes and any failed systemd services alerts configured, 2026-10-26"
- kind: repository
repository: jupiter-stacks
revision: working-tree
paths:
- mercury/stacks/uptime-kuma/docker-compose.yml
- mercury/stacks/uptime-kuma/README.md
related: [home-server.reference, home-server.host, home-server.networking, home-server.operations] related: [home-server.reference, home-server.host, home-server.networking, home-server.operations]
update_triggers: [hosting-renewal, vps-resize, os-upgrade, proxy-change, firewall-change, certificate-change, alerting-change] update_triggers: [hosting-renewal, vps-resize, os-upgrade, proxy-change, firewall-change, certificate-change, alerting-change]
unknowns: unknowns:
@@ -20,7 +32,7 @@ unknowns:
- certificate-renewal-method - certificate-renewal-method
- wireguard-allowedips-and-keepalive - wireguard-allowedips-and-keepalive
- backup-and-restore-policy - backup-and-restore-policy
- uptime-kuma-monitor-targets-and-outage-coverage - uptime-kuma-exact-probes-and-alert-timing
--- ---
# Mercury VPS # Mercury VPS
@@ -69,16 +81,43 @@ home connection uses CGNAT. Residential-IP exposure is avoided by design.
## Monitoring / notifications ## Monitoring / notifications
Uptime Kuma runs on Mercury, served at `https://uptime.umbra.mom` through Mercury is now monitored by the **Beszel hub on Jupiter**, confirmed by the
the local upstream `http://127.0.0.1:3001`. Notifications are delivered to owner on 2026-10-26. The following Mercury alerts are configured:
**Jupiter** (Discord server) in `#uptimekuma-alerts`; receipt confirmed by
the owner on 2026-10-23.
Monitor targets and outage coverage have not been recorded. Running Kuma on | Alert | Trigger |
Mercury allows monitoring independently of Jupiter, but does not establish that | --- | --- |
Jupiter or its containers are currently monitored. Mercury cannot report its | CPU usage | Above 90% for 10 minutes |
own complete outage through this instance; an external monitor is not yet | Memory usage | Above 90% for 10 minutes |
confirmed. See [Observability](operations.md#observability). | Failed Services | Any failed systemd service |
These are deployed alert settings, not proposed additions. Beszel's Failed
Services rule detects failed systemd units, not every error-level journal entry.
Alert settings are stored in the Beszel hub UI/database, not the agent Compose
file. Beszel cannot deliver these alerts while Jupiter or its hub is down.
Uptime Kuma runs on Mercury, served at `https://uptime.umbra.mom` through
the local upstream `http://127.0.0.1:3001`. Its current monitoring scope is
**only the health of the WireGuard tunnel and whether Jupiter is online**.
It notifies **Jay on Discord when either is down**, using the **Jupiter**
Discord server's `#uptimekuma-alerts` channel. Coverage confirmed by the owner
on 2026-10-26; notification receipt previously confirmed on 2026-10-23.
It does not currently monitor individual containers or application health.
The supplied `~/uptime-kuma.sh` is now represented by
`mercury/stacks/uptime-kuma/docker-compose.yml` in `jupiter-stacks`.
It retains image `louislam/uptime-kuma:2-slim`, container name `uptime-kuma`,
restart policy `unless-stopped`, loopback port `3001`, and bind mount
`/opt/uptime-kuma:/app/data`. Limits remain 300 MiB RAM and 0.5 CPU, with
10 MiB / 3-file log rotation (Compose explicitly selects `json-file`).
The adjacent README documents a stopped-data backup and migration into the
Mercury Portainer environment. Migration has not been confirmed; the repository
change alone does not replace the running container. nginx remains host-managed.
Exact probe types, targets and alert timing have not been recorded. A failed
check over WireGuard can indicate a tunnel/path failure rather than a powered-off
Jupiter. Mercury monitors independently of Jupiter, but cannot report its own
complete outage through this instance; an external monitor is not yet confirmed.
See [Observability](operations.md#observability) for container-alerting guidance.
## Firewall / fail2ban ## Firewall / fail2ban
+70 -8
View File
@@ -16,6 +16,14 @@ sources:
reference: "OIDC and three-group motivation; Authentik fully deployed and Authelia retired; updated six-directory backup plan; independent recovery custody unchanged, 2026-10-16" reference: "OIDC and three-group motivation; Authentik fully deployed and Authelia retired; updated six-directory backup plan; independent recovery custody unchanged, 2026-10-16"
- kind: owner-report - kind: owner-report
reference: "Notifications received in Discord server Jupiter: Beszel in #beszel-alerts, Backrest in #backrest-alerts, and Mercury Uptime Kuma in #uptimekuma-alerts, 2026-10-23" reference: "Notifications received in Discord server Jupiter: Beszel in #beszel-alerts, Backrest in #backrest-alerts, and Mercury Uptime Kuma in #uptimekuma-alerts, 2026-10-23"
- kind: owner-report
reference: "Mercury Uptime Kuma only checks WireGuard tunnel health and Jupiter online status, notifying Jay on Discord when down; Kuma per-container monitoring not configured, 2026-10-26"
- kind: owner-report
reference: "Beszel sends Discord alerts for any systemd error or any container down for more than two minutes, 2026-10-26"
- kind: owner-report
reference: "Mercury monitored in Beszel; CPU above 90% for 10 minutes, memory above 90% for 10 minutes and any failed systemd services alerts configured, 2026-10-26"
- kind: documentation
reference: "Beszel v0.21.0 upstream internal/site/src/lib/alerts.ts and internal/alerts/alerts_container.go: Container Health alerts detect Docker unhealthy status, 2026-10-26"
related: [home-server.reference, home-server.host, home-server.mercury, home-server.downloads, home-server.networking, home-server.storage, home-server.authentication, home-server.backups, home-server.deployment] related: [home-server.reference, home-server.host, home-server.mercury, home-server.downloads, home-server.networking, home-server.storage, home-server.authentication, home-server.backups, home-server.deployment]
update_triggers: [data-priority-change, architecture-change, alerting-change, recovery-test] update_triggers: [data-priority-change, architecture-change, alerting-change, recovery-test]
unknowns: unknowns:
@@ -23,7 +31,7 @@ unknowns:
- data-criticality-outside-the-six-backed-up-directories - data-criticality-outside-the-six-backed-up-directories
- tested-full-host-recovery-order - tested-full-host-recovery-order
- monitoring-review-frequency - monitoring-review-frequency
- configured-alert-rules-and-uptime-monitor-coverage - other-alert-rules-thresholds-and-exact-uptime-probes
--- ---
# Operations / decisions # Operations / decisions
@@ -67,9 +75,9 @@ The retired seedbox must not be assumed available for rollback.
| Facility | Current configuration | | Facility | Current configuration |
| --- | --- | | --- | --- |
| Host/container monitoring | Beszel hub and host-network agent present | | Host/container monitoring | Beszel monitors Jupiter and Mercury; existing systemd/container alerts plus Mercury CPU, memory and failed systemd service alerts (owner-confirmed) |
| Container logs | Dozzle with Docker socket proxy present | | Container logs | Dozzle with Docker socket proxy present |
| Uptime monitoring | Uptime Kuma on Mercury at `uptime.umbra.mom` | | Uptime monitoring | Uptime Kuma on Mercury at `uptime.umbra.mom`; WireGuard tunnel health and Jupiter online status only |
| Automatic alerting | Discord delivery configured for Beszel, Backrest and Uptime Kuma; receipt confirmed by owner | | Automatic alerting | Discord delivery configured for Beszel, Backrest and Uptime Kuma; receipt confirmed by owner |
### Discord notifications ### Discord notifications
@@ -83,11 +91,65 @@ as of 2026-10-23:
| Backrest | `#backrest-alerts` | | Backrest | `#backrest-alerts` |
| Uptime Kuma on Mercury | `#uptimekuma-alerts` | | Uptime Kuma on Mercury | `#uptimekuma-alerts` |
Notification receipt does not establish complete monitoring coverage. Exact Mercury's Uptime Kuma alerts **Jay** in `#uptimekuma-alerts` when the WireGuard
enabled alert rules, thresholds, Backrest hook conditions and Uptime Kuma tunnel or Jupiter availability checks fail; this scope was confirmed on
monitor targets have not been recorded. Per-container outage detection, 2026-10-26. Individual containers and applications are not currently monitored
whole-Jupiter outage detection, external monitoring of Mercury and missed-backup by Kuma.
alerts are not yet confirmed. No webhook URLs or tokens are stored in these notes.
Beszel sends alerts to **`#beszel-alerts`** for **any systemd error** or **any
container down for more than two minutes**, as confirmed by the owner on
2026-10-26. These are configured alerts, not proposed additions.
Mercury is also monitored in Beszel, with the following alerts configured by
owner report on 2026-10-26:
| Mercury alert | Trigger |
| --- | --- |
| CPU usage | Above 90% for 10 minutes |
| Memory usage | Above 90% for 10 minutes |
| Failed Services | Any failed systemd service |
These settings are stored in the Beszel hub UI/database, not the agent Compose
file. The failed-services rule covers failed systemd units, not all journal
errors.
Exact Kuma probes and timing, other Beszel rules/thresholds and Backrest hook
conditions have not been recorded. Monitoring of Mercury independent of Jupiter and
missed-backup alerts remain unconfirmed. No webhook URLs or tokens are stored
in these notes.
### Alert coverage and optional additions
Beszel's systemd/container outage alerting above is owner-reported deployed
state. The upstream Beszel `0.21.0` **Container Health** rule specifically checks
Docker `unhealthy` status, and **Failed Services** checks failed systemd units,
not every error-level journal entry. The exact live rules and stopped/missing
container detection have not been independently inspected.
The following are optional additions, not configured Kuma monitors:
1. In Mercury's Uptime Kuma, add an **HTTP(s)** monitor for each important
application's health endpoint, or a **TCP Port** monitor where HTTP is not
available. Enable the existing Discord notification on each monitor.
Start with a 60-second interval and two retries to tolerate brief restarts.
Check the application itself, not just a proxy or authentication login page.
Use a stable health endpoint with expected status/body; do not disable
authentication or expose private services solely for monitoring.
2. For additional container-running-state checks, Kuma's **Docker Container**
monitor needs a Docker host connection to the relevant machine. Mercury's
Docker socket only describes Mercury, not Jupiter, and a Portainer environment
connection does not automatically provide Kuma with Docker access. Remote
Docker monitoring needs a separately secured connection; do not expose an
unauthenticated Docker API or mount Jupiter's Docker socket into Kuma merely
to add application availability checks.
3. During a planned maintenance window, stop a non-critical monitored service,
confirm Jay receives the outage alert, then restart it and confirm recovery.
Account for scheduled backup stops before enabling noisy alerts.
Application probes detect service failures even when Docker still reports a
container as running, but are not an inventory of every container's state.
Beszel on Jupiter cannot deliver alerts while Jupiter or the Beszel hub is down;
keep Mercury's independent tunnel/host checks.
The backup plan stops Authentik, interrupting new OIDC logins and forward-auth The backup plan stops Authentik, interrupting new OIDC logins and forward-auth
access. Authentik/Wardrowbe restart hooks ignore errors, so backup completion access. Authentik/Wardrowbe restart hooks ignore errors, so backup completion