update merucry monitoring docs
This commit is contained in:
2 files changed
+120
-19
No files matched your search
+50
-11
@@ -4,7 +4,7 @@ id: home-server.mercury
|
||||
type: reference
|
||||
scope: [mercury, vps, hosting, nginx, wireguard, firewall]
|
||||
sensitivity: private-infrastructure
|
||||
last_reviewed: "2026-10-23"
|
||||
last_reviewed: "2026-10-26"
|
||||
sources:
|
||||
- kind: owner-report
|
||||
reference: "Fasthosts plan, fastfetch, nginx sites, iptables and fail2ban output, 2026-10-06"
|
||||
@@ -12,6 +12,18 @@ sources:
|
||||
reference: "nginx/WireGuard apt management and public IPv4/CGNAT rationale, 2026-10-06"
|
||||
- kind: owner-report
|
||||
reference: "Uptime service confirmed as Uptime Kuma; notifications received in Discord server Jupiter, channel #uptimekuma-alerts, 2026-10-23"
|
||||
- kind: owner-report
|
||||
reference: "Mercury uptime-kuma.sh supplied with bind mount, resource limits and log rotation, 2026-10-26"
|
||||
- kind: owner-report
|
||||
reference: "Mercury Uptime Kuma only checks WireGuard tunnel health and whether Jupiter is online; alerts Jay on Discord when down, 2026-10-26"
|
||||
- kind: owner-report
|
||||
reference: "Mercury monitored in Beszel; CPU above 90% for 10 minutes, memory above 90% for 10 minutes and any failed systemd services alerts configured, 2026-10-26"
|
||||
- kind: repository
|
||||
repository: jupiter-stacks
|
||||
revision: working-tree
|
||||
paths:
|
||||
- mercury/stacks/uptime-kuma/docker-compose.yml
|
||||
- mercury/stacks/uptime-kuma/README.md
|
||||
related: [home-server.reference, home-server.host, home-server.networking, home-server.operations]
|
||||
update_triggers: [hosting-renewal, vps-resize, os-upgrade, proxy-change, firewall-change, certificate-change, alerting-change]
|
||||
unknowns:
|
||||
@@ -20,7 +32,7 @@ unknowns:
|
||||
- certificate-renewal-method
|
||||
- wireguard-allowedips-and-keepalive
|
||||
- backup-and-restore-policy
|
||||
- uptime-kuma-monitor-targets-and-outage-coverage
|
||||
- uptime-kuma-exact-probes-and-alert-timing
|
||||
---
|
||||
|
||||
# Mercury VPS
|
||||
@@ -69,16 +81,43 @@ home connection uses CGNAT. Residential-IP exposure is avoided by design.
|
||||
|
||||
## Monitoring / notifications
|
||||
|
||||
Uptime Kuma runs on Mercury, served at `https://uptime.umbra.mom` through
|
||||
the local upstream `http://127.0.0.1:3001`. Notifications are delivered to
|
||||
**Jupiter** (Discord server) in `#uptimekuma-alerts`; receipt confirmed by
|
||||
the owner on 2026-10-23.
|
||||
Mercury is now monitored by the **Beszel hub on Jupiter**, confirmed by the
|
||||
owner on 2026-10-26. The following Mercury alerts are configured:
|
||||
|
||||
Monitor targets and outage coverage have not been recorded. Running Kuma on
|
||||
Mercury allows monitoring independently of Jupiter, but does not establish that
|
||||
Jupiter or its containers are currently monitored. Mercury cannot report its
|
||||
own complete outage through this instance; an external monitor is not yet
|
||||
confirmed. See [Observability](operations.md#observability).
|
||||
| Alert | Trigger |
|
||||
| --- | --- |
|
||||
| CPU usage | Above 90% for 10 minutes |
|
||||
| Memory usage | Above 90% for 10 minutes |
|
||||
| Failed Services | Any failed systemd service |
|
||||
|
||||
These are deployed alert settings, not proposed additions. Beszel's Failed
|
||||
Services rule detects failed systemd units, not every error-level journal entry.
|
||||
Alert settings are stored in the Beszel hub UI/database, not the agent Compose
|
||||
file. Beszel cannot deliver these alerts while Jupiter or its hub is down.
|
||||
|
||||
Uptime Kuma runs on Mercury, served at `https://uptime.umbra.mom` through
|
||||
the local upstream `http://127.0.0.1:3001`. Its current monitoring scope is
|
||||
**only the health of the WireGuard tunnel and whether Jupiter is online**.
|
||||
It notifies **Jay on Discord when either is down**, using the **Jupiter**
|
||||
Discord server's `#uptimekuma-alerts` channel. Coverage confirmed by the owner
|
||||
on 2026-10-26; notification receipt previously confirmed on 2026-10-23.
|
||||
It does not currently monitor individual containers or application health.
|
||||
|
||||
The supplied `~/uptime-kuma.sh` is now represented by
|
||||
`mercury/stacks/uptime-kuma/docker-compose.yml` in `jupiter-stacks`.
|
||||
It retains image `louislam/uptime-kuma:2-slim`, container name `uptime-kuma`,
|
||||
restart policy `unless-stopped`, loopback port `3001`, and bind mount
|
||||
`/opt/uptime-kuma:/app/data`. Limits remain 300 MiB RAM and 0.5 CPU, with
|
||||
10 MiB / 3-file log rotation (Compose explicitly selects `json-file`).
|
||||
The adjacent README documents a stopped-data backup and migration into the
|
||||
Mercury Portainer environment. Migration has not been confirmed; the repository
|
||||
change alone does not replace the running container. nginx remains host-managed.
|
||||
|
||||
Exact probe types, targets and alert timing have not been recorded. A failed
|
||||
check over WireGuard can indicate a tunnel/path failure rather than a powered-off
|
||||
Jupiter. Mercury monitors independently of Jupiter, but cannot report its own
|
||||
complete outage through this instance; an external monitor is not yet confirmed.
|
||||
See [Observability](operations.md#observability) for container-alerting guidance.
|
||||
|
||||
## Firewall / fail2ban
|
||||
|
||||
|
||||
@@ -16,6 +16,14 @@ sources:
|
||||
reference: "OIDC and three-group motivation; Authentik fully deployed and Authelia retired; updated six-directory backup plan; independent recovery custody unchanged, 2026-10-16"
|
||||
- kind: owner-report
|
||||
reference: "Notifications received in Discord server Jupiter: Beszel in #beszel-alerts, Backrest in #backrest-alerts, and Mercury Uptime Kuma in #uptimekuma-alerts, 2026-10-23"
|
||||
- kind: owner-report
|
||||
reference: "Mercury Uptime Kuma only checks WireGuard tunnel health and Jupiter online status, notifying Jay on Discord when down; Kuma per-container monitoring not configured, 2026-10-26"
|
||||
- kind: owner-report
|
||||
reference: "Beszel sends Discord alerts for any systemd error or any container down for more than two minutes, 2026-10-26"
|
||||
- kind: owner-report
|
||||
reference: "Mercury monitored in Beszel; CPU above 90% for 10 minutes, memory above 90% for 10 minutes and any failed systemd services alerts configured, 2026-10-26"
|
||||
- kind: documentation
|
||||
reference: "Beszel v0.21.0 upstream internal/site/src/lib/alerts.ts and internal/alerts/alerts_container.go: Container Health alerts detect Docker unhealthy status, 2026-10-26"
|
||||
related: [home-server.reference, home-server.host, home-server.mercury, home-server.downloads, home-server.networking, home-server.storage, home-server.authentication, home-server.backups, home-server.deployment]
|
||||
update_triggers: [data-priority-change, architecture-change, alerting-change, recovery-test]
|
||||
unknowns:
|
||||
@@ -23,7 +31,7 @@ unknowns:
|
||||
- data-criticality-outside-the-six-backed-up-directories
|
||||
- tested-full-host-recovery-order
|
||||
- monitoring-review-frequency
|
||||
- configured-alert-rules-and-uptime-monitor-coverage
|
||||
- other-alert-rules-thresholds-and-exact-uptime-probes
|
||||
---
|
||||
|
||||
# Operations / decisions
|
||||
@@ -67,9 +75,9 @@ The retired seedbox must not be assumed available for rollback.
|
||||
|
||||
| Facility | Current configuration |
|
||||
| --- | --- |
|
||||
| Host/container monitoring | Beszel hub and host-network agent present |
|
||||
| Host/container monitoring | Beszel monitors Jupiter and Mercury; existing systemd/container alerts plus Mercury CPU, memory and failed systemd service alerts (owner-confirmed) |
|
||||
| Container logs | Dozzle with Docker socket proxy present |
|
||||
| Uptime monitoring | Uptime Kuma on Mercury at `uptime.umbra.mom` |
|
||||
| Uptime monitoring | Uptime Kuma on Mercury at `uptime.umbra.mom`; WireGuard tunnel health and Jupiter online status only |
|
||||
| Automatic alerting | Discord delivery configured for Beszel, Backrest and Uptime Kuma; receipt confirmed by owner |
|
||||
|
||||
### Discord notifications
|
||||
@@ -83,11 +91,65 @@ as of 2026-10-23:
|
||||
| Backrest | `#backrest-alerts` |
|
||||
| Uptime Kuma on Mercury | `#uptimekuma-alerts` |
|
||||
|
||||
Notification receipt does not establish complete monitoring coverage. Exact
|
||||
enabled alert rules, thresholds, Backrest hook conditions and Uptime Kuma
|
||||
monitor targets have not been recorded. Per-container outage detection,
|
||||
whole-Jupiter outage detection, external monitoring of Mercury and missed-backup
|
||||
alerts are not yet confirmed. No webhook URLs or tokens are stored in these notes.
|
||||
Mercury's Uptime Kuma alerts **Jay** in `#uptimekuma-alerts` when the WireGuard
|
||||
tunnel or Jupiter availability checks fail; this scope was confirmed on
|
||||
2026-10-26. Individual containers and applications are not currently monitored
|
||||
by Kuma.
|
||||
|
||||
Beszel sends alerts to **`#beszel-alerts`** for **any systemd error** or **any
|
||||
container down for more than two minutes**, as confirmed by the owner on
|
||||
2026-10-26. These are configured alerts, not proposed additions.
|
||||
|
||||
Mercury is also monitored in Beszel, with the following alerts configured by
|
||||
owner report on 2026-10-26:
|
||||
|
||||
| Mercury alert | Trigger |
|
||||
| --- | --- |
|
||||
| CPU usage | Above 90% for 10 minutes |
|
||||
| Memory usage | Above 90% for 10 minutes |
|
||||
| Failed Services | Any failed systemd service |
|
||||
|
||||
These settings are stored in the Beszel hub UI/database, not the agent Compose
|
||||
file. The failed-services rule covers failed systemd units, not all journal
|
||||
errors.
|
||||
|
||||
Exact Kuma probes and timing, other Beszel rules/thresholds and Backrest hook
|
||||
conditions have not been recorded. Monitoring of Mercury independent of Jupiter and
|
||||
missed-backup alerts remain unconfirmed. No webhook URLs or tokens are stored
|
||||
in these notes.
|
||||
|
||||
### Alert coverage and optional additions
|
||||
|
||||
Beszel's systemd/container outage alerting above is owner-reported deployed
|
||||
state. The upstream Beszel `0.21.0` **Container Health** rule specifically checks
|
||||
Docker `unhealthy` status, and **Failed Services** checks failed systemd units,
|
||||
not every error-level journal entry. The exact live rules and stopped/missing
|
||||
container detection have not been independently inspected.
|
||||
|
||||
The following are optional additions, not configured Kuma monitors:
|
||||
|
||||
1. In Mercury's Uptime Kuma, add an **HTTP(s)** monitor for each important
|
||||
application's health endpoint, or a **TCP Port** monitor where HTTP is not
|
||||
available. Enable the existing Discord notification on each monitor.
|
||||
Start with a 60-second interval and two retries to tolerate brief restarts.
|
||||
Check the application itself, not just a proxy or authentication login page.
|
||||
Use a stable health endpoint with expected status/body; do not disable
|
||||
authentication or expose private services solely for monitoring.
|
||||
2. For additional container-running-state checks, Kuma's **Docker Container**
|
||||
monitor needs a Docker host connection to the relevant machine. Mercury's
|
||||
Docker socket only describes Mercury, not Jupiter, and a Portainer environment
|
||||
connection does not automatically provide Kuma with Docker access. Remote
|
||||
Docker monitoring needs a separately secured connection; do not expose an
|
||||
unauthenticated Docker API or mount Jupiter's Docker socket into Kuma merely
|
||||
to add application availability checks.
|
||||
3. During a planned maintenance window, stop a non-critical monitored service,
|
||||
confirm Jay receives the outage alert, then restart it and confirm recovery.
|
||||
Account for scheduled backup stops before enabling noisy alerts.
|
||||
|
||||
Application probes detect service failures even when Docker still reports a
|
||||
container as running, but are not an inventory of every container's state.
|
||||
Beszel on Jupiter cannot deliver alerts while Jupiter or the Beszel hub is down;
|
||||
keep Mercury's independent tunnel/host checks.
|
||||
|
||||
The backup plan stops Authentik, interrupting new OIDC logins and forward-auth
|
||||
access. Authentik/Wardrowbe restart hooks ignore errors, so backup completion
|
||||
|
||||
Reference in new issue
Block a user