From 6f9f7e7e6c82080ee093ed1ab09d4d0ba51f6ada Mon Sep 17 00:00:00 2001 From: John Gatward Date: Mon, 26 Oct 2026 15:51:15 +0000 Subject: [PATCH] update merucry monitoring docs --- docs/home-server/mercury.md | 61 +++++++++++++++++++++----- docs/home-server/operations.md | 78 ++++++++++++++++++++++++++++++---- 2 files changed, 120 insertions(+), 19 deletions(-) diff --git a/docs/home-server/mercury.md b/docs/home-server/mercury.md index 2923c4a..c2d6c8c 100644 --- a/docs/home-server/mercury.md +++ b/docs/home-server/mercury.md @@ -4,7 +4,7 @@ id: home-server.mercury type: reference scope: [mercury, vps, hosting, nginx, wireguard, firewall] sensitivity: private-infrastructure -last_reviewed: "2026-10-23" +last_reviewed: "2026-10-26" sources: - kind: owner-report reference: "Fasthosts plan, fastfetch, nginx sites, iptables and fail2ban output, 2026-10-06" @@ -12,6 +12,18 @@ sources: reference: "nginx/WireGuard apt management and public IPv4/CGNAT rationale, 2026-10-06" - kind: owner-report reference: "Uptime service confirmed as Uptime Kuma; notifications received in Discord server Jupiter, channel #uptimekuma-alerts, 2026-10-23" + - kind: owner-report + reference: "Mercury uptime-kuma.sh supplied with bind mount, resource limits and log rotation, 2026-10-26" + - kind: owner-report + reference: "Mercury Uptime Kuma only checks WireGuard tunnel health and whether Jupiter is online; alerts Jay on Discord when down, 2026-10-26" + - kind: owner-report + reference: "Mercury monitored in Beszel; CPU above 90% for 10 minutes, memory above 90% for 10 minutes and any failed systemd services alerts configured, 2026-10-26" + - kind: repository + repository: jupiter-stacks + revision: working-tree + paths: + - mercury/stacks/uptime-kuma/docker-compose.yml + - mercury/stacks/uptime-kuma/README.md related: [home-server.reference, home-server.host, home-server.networking, home-server.operations] update_triggers: [hosting-renewal, vps-resize, os-upgrade, proxy-change, firewall-change, certificate-change, alerting-change] unknowns: @@ -20,7 +32,7 @@ unknowns: - certificate-renewal-method - wireguard-allowedips-and-keepalive - backup-and-restore-policy - - uptime-kuma-monitor-targets-and-outage-coverage + - uptime-kuma-exact-probes-and-alert-timing --- # Mercury VPS @@ -69,16 +81,43 @@ home connection uses CGNAT. Residential-IP exposure is avoided by design. ## Monitoring / notifications -Uptime Kuma runs on Mercury, served at `https://uptime.umbra.mom` through -the local upstream `http://127.0.0.1:3001`. Notifications are delivered to -**Jupiter** (Discord server) in `#uptimekuma-alerts`; receipt confirmed by -the owner on 2026-10-23. +Mercury is now monitored by the **Beszel hub on Jupiter**, confirmed by the +owner on 2026-10-26. The following Mercury alerts are configured: -Monitor targets and outage coverage have not been recorded. Running Kuma on -Mercury allows monitoring independently of Jupiter, but does not establish that -Jupiter or its containers are currently monitored. Mercury cannot report its -own complete outage through this instance; an external monitor is not yet -confirmed. See [Observability](operations.md#observability). +| Alert | Trigger | +| --- | --- | +| CPU usage | Above 90% for 10 minutes | +| Memory usage | Above 90% for 10 minutes | +| Failed Services | Any failed systemd service | + +These are deployed alert settings, not proposed additions. Beszel's Failed +Services rule detects failed systemd units, not every error-level journal entry. +Alert settings are stored in the Beszel hub UI/database, not the agent Compose +file. Beszel cannot deliver these alerts while Jupiter or its hub is down. + +Uptime Kuma runs on Mercury, served at `https://uptime.umbra.mom` through +the local upstream `http://127.0.0.1:3001`. Its current monitoring scope is +**only the health of the WireGuard tunnel and whether Jupiter is online**. +It notifies **Jay on Discord when either is down**, using the **Jupiter** +Discord server's `#uptimekuma-alerts` channel. Coverage confirmed by the owner +on 2026-10-26; notification receipt previously confirmed on 2026-10-23. +It does not currently monitor individual containers or application health. + +The supplied `~/uptime-kuma.sh` is now represented by +`mercury/stacks/uptime-kuma/docker-compose.yml` in `jupiter-stacks`. +It retains image `louislam/uptime-kuma:2-slim`, container name `uptime-kuma`, +restart policy `unless-stopped`, loopback port `3001`, and bind mount +`/opt/uptime-kuma:/app/data`. Limits remain 300 MiB RAM and 0.5 CPU, with +10 MiB / 3-file log rotation (Compose explicitly selects `json-file`). +The adjacent README documents a stopped-data backup and migration into the +Mercury Portainer environment. Migration has not been confirmed; the repository +change alone does not replace the running container. nginx remains host-managed. + +Exact probe types, targets and alert timing have not been recorded. A failed +check over WireGuard can indicate a tunnel/path failure rather than a powered-off +Jupiter. Mercury monitors independently of Jupiter, but cannot report its own +complete outage through this instance; an external monitor is not yet confirmed. +See [Observability](operations.md#observability) for container-alerting guidance. ## Firewall / fail2ban diff --git a/docs/home-server/operations.md b/docs/home-server/operations.md index 11865f2..e99fdd5 100644 --- a/docs/home-server/operations.md +++ b/docs/home-server/operations.md @@ -16,6 +16,14 @@ sources: reference: "OIDC and three-group motivation; Authentik fully deployed and Authelia retired; updated six-directory backup plan; independent recovery custody unchanged, 2026-10-16" - kind: owner-report reference: "Notifications received in Discord server Jupiter: Beszel in #beszel-alerts, Backrest in #backrest-alerts, and Mercury Uptime Kuma in #uptimekuma-alerts, 2026-10-23" + - kind: owner-report + reference: "Mercury Uptime Kuma only checks WireGuard tunnel health and Jupiter online status, notifying Jay on Discord when down; Kuma per-container monitoring not configured, 2026-10-26" + - kind: owner-report + reference: "Beszel sends Discord alerts for any systemd error or any container down for more than two minutes, 2026-10-26" + - kind: owner-report + reference: "Mercury monitored in Beszel; CPU above 90% for 10 minutes, memory above 90% for 10 minutes and any failed systemd services alerts configured, 2026-10-26" + - kind: documentation + reference: "Beszel v0.21.0 upstream internal/site/src/lib/alerts.ts and internal/alerts/alerts_container.go: Container Health alerts detect Docker unhealthy status, 2026-10-26" related: [home-server.reference, home-server.host, home-server.mercury, home-server.downloads, home-server.networking, home-server.storage, home-server.authentication, home-server.backups, home-server.deployment] update_triggers: [data-priority-change, architecture-change, alerting-change, recovery-test] unknowns: @@ -23,7 +31,7 @@ unknowns: - data-criticality-outside-the-six-backed-up-directories - tested-full-host-recovery-order - monitoring-review-frequency - - configured-alert-rules-and-uptime-monitor-coverage + - other-alert-rules-thresholds-and-exact-uptime-probes --- # Operations / decisions @@ -67,9 +75,9 @@ The retired seedbox must not be assumed available for rollback. | Facility | Current configuration | | --- | --- | -| Host/container monitoring | Beszel hub and host-network agent present | +| Host/container monitoring | Beszel monitors Jupiter and Mercury; existing systemd/container alerts plus Mercury CPU, memory and failed systemd service alerts (owner-confirmed) | | Container logs | Dozzle with Docker socket proxy present | -| Uptime monitoring | Uptime Kuma on Mercury at `uptime.umbra.mom` | +| Uptime monitoring | Uptime Kuma on Mercury at `uptime.umbra.mom`; WireGuard tunnel health and Jupiter online status only | | Automatic alerting | Discord delivery configured for Beszel, Backrest and Uptime Kuma; receipt confirmed by owner | ### Discord notifications @@ -83,11 +91,65 @@ as of 2026-10-23: | Backrest | `#backrest-alerts` | | Uptime Kuma on Mercury | `#uptimekuma-alerts` | -Notification receipt does not establish complete monitoring coverage. Exact -enabled alert rules, thresholds, Backrest hook conditions and Uptime Kuma -monitor targets have not been recorded. Per-container outage detection, -whole-Jupiter outage detection, external monitoring of Mercury and missed-backup -alerts are not yet confirmed. No webhook URLs or tokens are stored in these notes. +Mercury's Uptime Kuma alerts **Jay** in `#uptimekuma-alerts` when the WireGuard +tunnel or Jupiter availability checks fail; this scope was confirmed on +2026-10-26. Individual containers and applications are not currently monitored +by Kuma. + +Beszel sends alerts to **`#beszel-alerts`** for **any systemd error** or **any +container down for more than two minutes**, as confirmed by the owner on +2026-10-26. These are configured alerts, not proposed additions. + +Mercury is also monitored in Beszel, with the following alerts configured by +owner report on 2026-10-26: + +| Mercury alert | Trigger | +| --- | --- | +| CPU usage | Above 90% for 10 minutes | +| Memory usage | Above 90% for 10 minutes | +| Failed Services | Any failed systemd service | + +These settings are stored in the Beszel hub UI/database, not the agent Compose +file. The failed-services rule covers failed systemd units, not all journal +errors. + +Exact Kuma probes and timing, other Beszel rules/thresholds and Backrest hook +conditions have not been recorded. Monitoring of Mercury independent of Jupiter and +missed-backup alerts remain unconfirmed. No webhook URLs or tokens are stored +in these notes. + +### Alert coverage and optional additions + +Beszel's systemd/container outage alerting above is owner-reported deployed +state. The upstream Beszel `0.21.0` **Container Health** rule specifically checks +Docker `unhealthy` status, and **Failed Services** checks failed systemd units, +not every error-level journal entry. The exact live rules and stopped/missing +container detection have not been independently inspected. + +The following are optional additions, not configured Kuma monitors: + +1. In Mercury's Uptime Kuma, add an **HTTP(s)** monitor for each important + application's health endpoint, or a **TCP Port** monitor where HTTP is not + available. Enable the existing Discord notification on each monitor. + Start with a 60-second interval and two retries to tolerate brief restarts. + Check the application itself, not just a proxy or authentication login page. + Use a stable health endpoint with expected status/body; do not disable + authentication or expose private services solely for monitoring. +2. For additional container-running-state checks, Kuma's **Docker Container** + monitor needs a Docker host connection to the relevant machine. Mercury's + Docker socket only describes Mercury, not Jupiter, and a Portainer environment + connection does not automatically provide Kuma with Docker access. Remote + Docker monitoring needs a separately secured connection; do not expose an + unauthenticated Docker API or mount Jupiter's Docker socket into Kuma merely + to add application availability checks. +3. During a planned maintenance window, stop a non-critical monitored service, + confirm Jay receives the outage alert, then restart it and confirm recovery. + Account for scheduled backup stops before enabling noisy alerts. + +Application probes detect service failures even when Docker still reports a +container as running, but are not an inventory of every container's state. +Beszel on Jupiter cannot deliver alerts while Jupiter or the Beszel hub is down; +keep Mercury's independent tunnel/host checks. The backup plan stops Authentik, interrupting new OIDC logins and forward-auth access. Authentik/Wardrowbe restart hooks ignore errors, so backup completion