update merucry monitoring docs
This commit is contained in:
2 files changed
+120
-19
No files matched your search
+50
-11
@@ -4,7 +4,7 @@ id: home-server.mercury
|
|||||||
type: reference
|
type: reference
|
||||||
scope: [mercury, vps, hosting, nginx, wireguard, firewall]
|
scope: [mercury, vps, hosting, nginx, wireguard, firewall]
|
||||||
sensitivity: private-infrastructure
|
sensitivity: private-infrastructure
|
||||||
last_reviewed: "2026-10-23"
|
last_reviewed: "2026-10-26"
|
||||||
sources:
|
sources:
|
||||||
- kind: owner-report
|
- kind: owner-report
|
||||||
reference: "Fasthosts plan, fastfetch, nginx sites, iptables and fail2ban output, 2026-10-06"
|
reference: "Fasthosts plan, fastfetch, nginx sites, iptables and fail2ban output, 2026-10-06"
|
||||||
@@ -12,6 +12,18 @@ sources:
|
|||||||
reference: "nginx/WireGuard apt management and public IPv4/CGNAT rationale, 2026-10-06"
|
reference: "nginx/WireGuard apt management and public IPv4/CGNAT rationale, 2026-10-06"
|
||||||
- kind: owner-report
|
- kind: owner-report
|
||||||
reference: "Uptime service confirmed as Uptime Kuma; notifications received in Discord server Jupiter, channel #uptimekuma-alerts, 2026-10-23"
|
reference: "Uptime service confirmed as Uptime Kuma; notifications received in Discord server Jupiter, channel #uptimekuma-alerts, 2026-10-23"
|
||||||
|
- kind: owner-report
|
||||||
|
reference: "Mercury uptime-kuma.sh supplied with bind mount, resource limits and log rotation, 2026-10-26"
|
||||||
|
- kind: owner-report
|
||||||
|
reference: "Mercury Uptime Kuma only checks WireGuard tunnel health and whether Jupiter is online; alerts Jay on Discord when down, 2026-10-26"
|
||||||
|
- kind: owner-report
|
||||||
|
reference: "Mercury monitored in Beszel; CPU above 90% for 10 minutes, memory above 90% for 10 minutes and any failed systemd services alerts configured, 2026-10-26"
|
||||||
|
- kind: repository
|
||||||
|
repository: jupiter-stacks
|
||||||
|
revision: working-tree
|
||||||
|
paths:
|
||||||
|
- mercury/stacks/uptime-kuma/docker-compose.yml
|
||||||
|
- mercury/stacks/uptime-kuma/README.md
|
||||||
related: [home-server.reference, home-server.host, home-server.networking, home-server.operations]
|
related: [home-server.reference, home-server.host, home-server.networking, home-server.operations]
|
||||||
update_triggers: [hosting-renewal, vps-resize, os-upgrade, proxy-change, firewall-change, certificate-change, alerting-change]
|
update_triggers: [hosting-renewal, vps-resize, os-upgrade, proxy-change, firewall-change, certificate-change, alerting-change]
|
||||||
unknowns:
|
unknowns:
|
||||||
@@ -20,7 +32,7 @@ unknowns:
|
|||||||
- certificate-renewal-method
|
- certificate-renewal-method
|
||||||
- wireguard-allowedips-and-keepalive
|
- wireguard-allowedips-and-keepalive
|
||||||
- backup-and-restore-policy
|
- backup-and-restore-policy
|
||||||
- uptime-kuma-monitor-targets-and-outage-coverage
|
- uptime-kuma-exact-probes-and-alert-timing
|
||||||
---
|
---
|
||||||
|
|
||||||
# Mercury VPS
|
# Mercury VPS
|
||||||
@@ -69,16 +81,43 @@ home connection uses CGNAT. Residential-IP exposure is avoided by design.
|
|||||||
|
|
||||||
## Monitoring / notifications
|
## Monitoring / notifications
|
||||||
|
|
||||||
Uptime Kuma runs on Mercury, served at `https://uptime.umbra.mom` through
|
Mercury is now monitored by the **Beszel hub on Jupiter**, confirmed by the
|
||||||
the local upstream `http://127.0.0.1:3001`. Notifications are delivered to
|
owner on 2026-10-26. The following Mercury alerts are configured:
|
||||||
**Jupiter** (Discord server) in `#uptimekuma-alerts`; receipt confirmed by
|
|
||||||
the owner on 2026-10-23.
|
|
||||||
|
|
||||||
Monitor targets and outage coverage have not been recorded. Running Kuma on
|
| Alert | Trigger |
|
||||||
Mercury allows monitoring independently of Jupiter, but does not establish that
|
| --- | --- |
|
||||||
Jupiter or its containers are currently monitored. Mercury cannot report its
|
| CPU usage | Above 90% for 10 minutes |
|
||||||
own complete outage through this instance; an external monitor is not yet
|
| Memory usage | Above 90% for 10 minutes |
|
||||||
confirmed. See [Observability](operations.md#observability).
|
| Failed Services | Any failed systemd service |
|
||||||
|
|
||||||
|
These are deployed alert settings, not proposed additions. Beszel's Failed
|
||||||
|
Services rule detects failed systemd units, not every error-level journal entry.
|
||||||
|
Alert settings are stored in the Beszel hub UI/database, not the agent Compose
|
||||||
|
file. Beszel cannot deliver these alerts while Jupiter or its hub is down.
|
||||||
|
|
||||||
|
Uptime Kuma runs on Mercury, served at `https://uptime.umbra.mom` through
|
||||||
|
the local upstream `http://127.0.0.1:3001`. Its current monitoring scope is
|
||||||
|
**only the health of the WireGuard tunnel and whether Jupiter is online**.
|
||||||
|
It notifies **Jay on Discord when either is down**, using the **Jupiter**
|
||||||
|
Discord server's `#uptimekuma-alerts` channel. Coverage confirmed by the owner
|
||||||
|
on 2026-10-26; notification receipt previously confirmed on 2026-10-23.
|
||||||
|
It does not currently monitor individual containers or application health.
|
||||||
|
|
||||||
|
The supplied `~/uptime-kuma.sh` is now represented by
|
||||||
|
`mercury/stacks/uptime-kuma/docker-compose.yml` in `jupiter-stacks`.
|
||||||
|
It retains image `louislam/uptime-kuma:2-slim`, container name `uptime-kuma`,
|
||||||
|
restart policy `unless-stopped`, loopback port `3001`, and bind mount
|
||||||
|
`/opt/uptime-kuma:/app/data`. Limits remain 300 MiB RAM and 0.5 CPU, with
|
||||||
|
10 MiB / 3-file log rotation (Compose explicitly selects `json-file`).
|
||||||
|
The adjacent README documents a stopped-data backup and migration into the
|
||||||
|
Mercury Portainer environment. Migration has not been confirmed; the repository
|
||||||
|
change alone does not replace the running container. nginx remains host-managed.
|
||||||
|
|
||||||
|
Exact probe types, targets and alert timing have not been recorded. A failed
|
||||||
|
check over WireGuard can indicate a tunnel/path failure rather than a powered-off
|
||||||
|
Jupiter. Mercury monitors independently of Jupiter, but cannot report its own
|
||||||
|
complete outage through this instance; an external monitor is not yet confirmed.
|
||||||
|
See [Observability](operations.md#observability) for container-alerting guidance.
|
||||||
|
|
||||||
## Firewall / fail2ban
|
## Firewall / fail2ban
|
||||||
|
|
||||||
|
|||||||
@@ -16,6 +16,14 @@ sources:
|
|||||||
reference: "OIDC and three-group motivation; Authentik fully deployed and Authelia retired; updated six-directory backup plan; independent recovery custody unchanged, 2026-10-16"
|
reference: "OIDC and three-group motivation; Authentik fully deployed and Authelia retired; updated six-directory backup plan; independent recovery custody unchanged, 2026-10-16"
|
||||||
- kind: owner-report
|
- kind: owner-report
|
||||||
reference: "Notifications received in Discord server Jupiter: Beszel in #beszel-alerts, Backrest in #backrest-alerts, and Mercury Uptime Kuma in #uptimekuma-alerts, 2026-10-23"
|
reference: "Notifications received in Discord server Jupiter: Beszel in #beszel-alerts, Backrest in #backrest-alerts, and Mercury Uptime Kuma in #uptimekuma-alerts, 2026-10-23"
|
||||||
|
- kind: owner-report
|
||||||
|
reference: "Mercury Uptime Kuma only checks WireGuard tunnel health and Jupiter online status, notifying Jay on Discord when down; Kuma per-container monitoring not configured, 2026-10-26"
|
||||||
|
- kind: owner-report
|
||||||
|
reference: "Beszel sends Discord alerts for any systemd error or any container down for more than two minutes, 2026-10-26"
|
||||||
|
- kind: owner-report
|
||||||
|
reference: "Mercury monitored in Beszel; CPU above 90% for 10 minutes, memory above 90% for 10 minutes and any failed systemd services alerts configured, 2026-10-26"
|
||||||
|
- kind: documentation
|
||||||
|
reference: "Beszel v0.21.0 upstream internal/site/src/lib/alerts.ts and internal/alerts/alerts_container.go: Container Health alerts detect Docker unhealthy status, 2026-10-26"
|
||||||
related: [home-server.reference, home-server.host, home-server.mercury, home-server.downloads, home-server.networking, home-server.storage, home-server.authentication, home-server.backups, home-server.deployment]
|
related: [home-server.reference, home-server.host, home-server.mercury, home-server.downloads, home-server.networking, home-server.storage, home-server.authentication, home-server.backups, home-server.deployment]
|
||||||
update_triggers: [data-priority-change, architecture-change, alerting-change, recovery-test]
|
update_triggers: [data-priority-change, architecture-change, alerting-change, recovery-test]
|
||||||
unknowns:
|
unknowns:
|
||||||
@@ -23,7 +31,7 @@ unknowns:
|
|||||||
- data-criticality-outside-the-six-backed-up-directories
|
- data-criticality-outside-the-six-backed-up-directories
|
||||||
- tested-full-host-recovery-order
|
- tested-full-host-recovery-order
|
||||||
- monitoring-review-frequency
|
- monitoring-review-frequency
|
||||||
- configured-alert-rules-and-uptime-monitor-coverage
|
- other-alert-rules-thresholds-and-exact-uptime-probes
|
||||||
---
|
---
|
||||||
|
|
||||||
# Operations / decisions
|
# Operations / decisions
|
||||||
@@ -67,9 +75,9 @@ The retired seedbox must not be assumed available for rollback.
|
|||||||
|
|
||||||
| Facility | Current configuration |
|
| Facility | Current configuration |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| Host/container monitoring | Beszel hub and host-network agent present |
|
| Host/container monitoring | Beszel monitors Jupiter and Mercury; existing systemd/container alerts plus Mercury CPU, memory and failed systemd service alerts (owner-confirmed) |
|
||||||
| Container logs | Dozzle with Docker socket proxy present |
|
| Container logs | Dozzle with Docker socket proxy present |
|
||||||
| Uptime monitoring | Uptime Kuma on Mercury at `uptime.umbra.mom` |
|
| Uptime monitoring | Uptime Kuma on Mercury at `uptime.umbra.mom`; WireGuard tunnel health and Jupiter online status only |
|
||||||
| Automatic alerting | Discord delivery configured for Beszel, Backrest and Uptime Kuma; receipt confirmed by owner |
|
| Automatic alerting | Discord delivery configured for Beszel, Backrest and Uptime Kuma; receipt confirmed by owner |
|
||||||
|
|
||||||
### Discord notifications
|
### Discord notifications
|
||||||
@@ -83,11 +91,65 @@ as of 2026-10-23:
|
|||||||
| Backrest | `#backrest-alerts` |
|
| Backrest | `#backrest-alerts` |
|
||||||
| Uptime Kuma on Mercury | `#uptimekuma-alerts` |
|
| Uptime Kuma on Mercury | `#uptimekuma-alerts` |
|
||||||
|
|
||||||
Notification receipt does not establish complete monitoring coverage. Exact
|
Mercury's Uptime Kuma alerts **Jay** in `#uptimekuma-alerts` when the WireGuard
|
||||||
enabled alert rules, thresholds, Backrest hook conditions and Uptime Kuma
|
tunnel or Jupiter availability checks fail; this scope was confirmed on
|
||||||
monitor targets have not been recorded. Per-container outage detection,
|
2026-10-26. Individual containers and applications are not currently monitored
|
||||||
whole-Jupiter outage detection, external monitoring of Mercury and missed-backup
|
by Kuma.
|
||||||
alerts are not yet confirmed. No webhook URLs or tokens are stored in these notes.
|
|
||||||
|
Beszel sends alerts to **`#beszel-alerts`** for **any systemd error** or **any
|
||||||
|
container down for more than two minutes**, as confirmed by the owner on
|
||||||
|
2026-10-26. These are configured alerts, not proposed additions.
|
||||||
|
|
||||||
|
Mercury is also monitored in Beszel, with the following alerts configured by
|
||||||
|
owner report on 2026-10-26:
|
||||||
|
|
||||||
|
| Mercury alert | Trigger |
|
||||||
|
| --- | --- |
|
||||||
|
| CPU usage | Above 90% for 10 minutes |
|
||||||
|
| Memory usage | Above 90% for 10 minutes |
|
||||||
|
| Failed Services | Any failed systemd service |
|
||||||
|
|
||||||
|
These settings are stored in the Beszel hub UI/database, not the agent Compose
|
||||||
|
file. The failed-services rule covers failed systemd units, not all journal
|
||||||
|
errors.
|
||||||
|
|
||||||
|
Exact Kuma probes and timing, other Beszel rules/thresholds and Backrest hook
|
||||||
|
conditions have not been recorded. Monitoring of Mercury independent of Jupiter and
|
||||||
|
missed-backup alerts remain unconfirmed. No webhook URLs or tokens are stored
|
||||||
|
in these notes.
|
||||||
|
|
||||||
|
### Alert coverage and optional additions
|
||||||
|
|
||||||
|
Beszel's systemd/container outage alerting above is owner-reported deployed
|
||||||
|
state. The upstream Beszel `0.21.0` **Container Health** rule specifically checks
|
||||||
|
Docker `unhealthy` status, and **Failed Services** checks failed systemd units,
|
||||||
|
not every error-level journal entry. The exact live rules and stopped/missing
|
||||||
|
container detection have not been independently inspected.
|
||||||
|
|
||||||
|
The following are optional additions, not configured Kuma monitors:
|
||||||
|
|
||||||
|
1. In Mercury's Uptime Kuma, add an **HTTP(s)** monitor for each important
|
||||||
|
application's health endpoint, or a **TCP Port** monitor where HTTP is not
|
||||||
|
available. Enable the existing Discord notification on each monitor.
|
||||||
|
Start with a 60-second interval and two retries to tolerate brief restarts.
|
||||||
|
Check the application itself, not just a proxy or authentication login page.
|
||||||
|
Use a stable health endpoint with expected status/body; do not disable
|
||||||
|
authentication or expose private services solely for monitoring.
|
||||||
|
2. For additional container-running-state checks, Kuma's **Docker Container**
|
||||||
|
monitor needs a Docker host connection to the relevant machine. Mercury's
|
||||||
|
Docker socket only describes Mercury, not Jupiter, and a Portainer environment
|
||||||
|
connection does not automatically provide Kuma with Docker access. Remote
|
||||||
|
Docker monitoring needs a separately secured connection; do not expose an
|
||||||
|
unauthenticated Docker API or mount Jupiter's Docker socket into Kuma merely
|
||||||
|
to add application availability checks.
|
||||||
|
3. During a planned maintenance window, stop a non-critical monitored service,
|
||||||
|
confirm Jay receives the outage alert, then restart it and confirm recovery.
|
||||||
|
Account for scheduled backup stops before enabling noisy alerts.
|
||||||
|
|
||||||
|
Application probes detect service failures even when Docker still reports a
|
||||||
|
container as running, but are not an inventory of every container's state.
|
||||||
|
Beszel on Jupiter cannot deliver alerts while Jupiter or the Beszel hub is down;
|
||||||
|
keep Mercury's independent tunnel/host checks.
|
||||||
|
|
||||||
The backup plan stops Authentik, interrupting new OIDC logins and forward-auth
|
The backup plan stops Authentik, interrupting new OIDC logins and forward-auth
|
||||||
access. Authentik/Wardrowbe restart hooks ignore errors, so backup completion
|
access. Authentik/Wardrowbe restart hooks ignore errors, so backup completion
|
||||||
|
|||||||
Reference in new issue
Block a user