adamksmith/jbod-monitor
1
0
Fork 0
Code Issues 3 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Split into dedicated hardware poller + read-only consumers

Browse Source 
Rearchitect so a single poller process is the sole owner of all SAS/SMART/
SES hardware I/O, serialized behind one gate and paced, writing results to
Redis. The API/web and MQTT publisher become pure Redis readers — they no
longer issue any subprocess and can restart freely without touching the bus.

This addresses backplane/expander stress from concurrent + restart-triggered
SMART/SES storms (the prior model re-ran a hardware sweep on every container
start, and polled sg_ses 0x02+0x07 every 60s; 0x07 errored on the IOM6/
Xyratex expanders).

- poller.py: paced sweep (inventory, SMART per-drive w/ gap, SES 0x02, ZFS,
  host/MegaRAID), startup jitter, single-instance Redis lock, LED-queue worker
- services/hwgate.py: global serialization semaphore (POLL_CONCURRENCY=1)
- services/store.py: Redis as the only producer<->consumer interface + LED queue
- services/health.py: shared drive-health classifier (fixes overview double-count)
- gate smartctl/sg_ses/zpool/ledctl; drop sg_ses 0x07 from the hot path
- routers + temps read-only from the store; main.py drops the in-process poller
- compose: 3 services (privileged poller + unprivileged app + redis) w/ pacing knobs
This commit is contained in:
adam
2026-06-16 15:12:34 +00:00
parent 7a50d1261f
commit 0f829e0380
19 changed files with 717 additions and 464 deletions
Download Patch File Download Diff File Expand all files Collapse all files

135
README.md
View File

@@ -1,107 +1,118 @@
# JBOD Monitor
REST API for monitoring drive health in JBOD enclosures on Linux.
Drive-health monitoring for JBOD enclosures on Linux, with a REST API, web UI,
and optional Home Assistant MQTT publishing.
Auto-discovers SES enclosures via sysfs, maps drives to physical slots, and exposes SMART health data.
Auto-discovers SES enclosures via sysfs, maps drives to physical slots, reads
SMART, and aggregates per-enclosure temperatures (named SES sensors + hotspot).
## Architecture
Two processes, with **Redis as the only interface between them**:
```
poller (privileged, pid:host) ──smartctl/sg_ses/zpool/ledctl──> hardware
│ serialized + paced
▼
Redis ◀── reads ── app (REST API + web UI + MQTT) [unprivileged]
▲
└── locate-LED command queue (app enqueues, poller runs)
```
- **`poller`** is the *only* process that touches the SAS bus. All hardware
commands run behind a single gate (`POLL_CONCURRENCY`, default 1 = fully
serialized) with a gap between drives, so it never storms fragile SAS
expanders. It sweeps on an interval and writes results to Redis.
- **`app`** (and the MQTT publisher) are pure Redis readers — unprivileged, no
`/dev`. They can restart freely without issuing a single hardware command.
- **Locate LEDs**: the API enqueues a request in Redis; the poller executes it
serialized with everything else.
This split exists for a reason: concurrent/bursty SMART+SES traffic — especially
repeated on every container restart — can drop SAS expanders and suspend pools.
One paced owner makes that structurally impossible.
## Prerequisites
- Linux with SAS/SATA JBODs connected via HBA
- `smartmontools` — for `smartctl` (SMART data)
- `sg3-utils` — for `sg_ses` (SES enclosure data)
- `smartmontools` (`smartctl`), `sg3-utils` (`sg_ses`), `ledmon` (`ledctl`)
- Redis
- Python 3.11+
```bash
# Debian/Ubuntu
apt install smartmontools sg3-utils
# RHEL/Fedora
dnf install smartmontools sg3_utils
apt install smartmontools sg3-utils ledmon # Debian/Ubuntu
```
## Install
## Run (docker compose)
```bash
pip install -r requirements.txt
docker compose up -d --build
```
## Run
The API needs root access for `smartctl` to query drives:
Brings up `poller` (privileged), `app` (port 8000), and `redis`. To run the two
processes by hand instead:
```bash
sudo uvicorn main:app --host 0.0.0.0 --port 8000
REDIS_HOST=localhost sudo -E python poller.py # producer (needs root)
REDIS_HOST=localhost uvicorn main:app --port 8000 # consumer
```
## API Endpoints
| Endpoint | Description |
|---|---|
| `GET /api/health` | Service health + tool availability |
| `GET /api/enclosures` | List all discovered SES enclosures |
| `GET /api/enclosures/{id}/drives` | List drive slots for an enclosure |
| `GET /api/health` | Service health + `redis`/`mqtt`/`poller_fresh` status |
| `GET /api/enclosures` | List discovered SES enclosures |
| `GET /api/enclosures/{id}/drives` | Drive slots for an enclosure |
| `GET /api/drives/{device}` | SMART detail for a block device |
| `GET /api/overview` | Aggregate enclosure + drive health |
| `GET /api/temps` | Per-enclosure temperatures: named SES sensors + hotspot |
| `GET /docs` | Interactive API docs (Swagger UI) |
| `POST /api/drives/{device}/led` | Queue a locate-LED change (`state`: `locate`/`off`) |
| `GET /docs` | Swagger UI |
Read endpoints serve the poller's last sweep; `X-Poll-Age` (seconds) headers and
the `poller_fresh` health flag surface staleness.
## Poller tuning (env)
| Variable | Default | Description |
|---|---|---|
| `POLL_SWEEP_INTERVAL` | `300` | Seconds between full sweeps |
| `POLL_DRIVE_GAP` | `0.5` | Seconds between each drive's SMART query |
| `POLL_CONCURRENCY` | `1` | Max concurrent hardware ops (1 = serialized) |
| `POLL_STARTUP_JITTER` | `10` | Random startup delay to avoid restart storms |
| `POLL_DATA_TTL` | `900` | Redis TTL for poller data (must exceed sweep interval) |
| `ZFS_USE_NSENTER` | — | `true` to run `zpool` in the host namespace (containers) |
## Home Assistant (MQTT)
The monitor can publish per-enclosure temperatures to Home Assistant over MQTT
using [HA discovery](https://www.home-assistant.io/integrations/mqtt/#mqtt-discovery) —
no HA YAML required. Each enclosure becomes a device with:
- a **Hotspot** sensor — the hottest reading across all SES temperature
sensors *and* the drives housed in that enclosure (drive temps are usually
the real hotspot); `drive_max_c` and `drive_temp_count` are exposed as
attributes.
- one sensor per named SES temperature element (e.g. *Temp Inlet*,
*Temp Outlet*), labelled from the SES element descriptor page.
Publishing is **opt-in**: set `MQTT_HOST` to enable it.
The `app` publishes per-enclosure temperatures via
[MQTT discovery](https://www.home-assistant.io/integrations/mqtt/#mqtt-discovery)
— no HA YAML. Each enclosure becomes a device with a **Hotspot** sensor (max
across SES sensors + housed drive temps; `drive_max_c`/`drive_temp_count` as
attributes) and one sensor per named SES temperature element. Opt-in via
`MQTT_HOST`; availability is tracked via an MQTT LWT.
| Variable | Default | Description |
|---|---|---|
| `MQTT_HOST` | _(unset)_ | Broker host. Unset → MQTT disabled. |
| `MQTT_PORT` | `1883` | Broker port |
| `MQTT_USERNAME` / `MQTT_PASSWORD` | _(unset)_ | Broker credentials (fallback if OpenBao unused/unavailable) |
| `MQTT_USERNAME` / `MQTT_PASSWORD` | _(unset)_ | Broker credentials |
| `MQTT_DISCOVERY_PREFIX` | `homeassistant` | HA discovery topic prefix |
| `MQTT_BASE_TOPIC` | `jbod-monitor` | State/availability topic root |
| `MQTT_PUBLISH_INTERVAL` | `60` | Seconds between publishes |
| `MQTT_NODE_ID` | _(hostname)_ | Stable id for this monitor (disambiguates multiple hosts) |
| `MQTT_CLIENT_ID` | `jbod-monitor-<node>` | MQTT client id |
| `MQTT_NODE_ID` | _(hostname)_ | Stable id (disambiguates multiple hosts) |
### Credentials via OpenBao
Broker credentials are sourced from OpenBao at startup rather than stored in
plaintext (matching the homelab runtime-secret pattern). Set `MQTT_SECRET_PATH`
to a KV v2 path holding `username`/`password`; the app reads
`<OPENBAO_ADDR>/v1/<OPENBAO_MOUNT>/data/<MQTT_SECRET_PATH>` with the token from
`OPENBAO_TOKEN_FILE` (or `OPENBAO_TOKEN`). If the read fails, it falls back to
the `MQTT_USERNAME`/`MQTT_PASSWORD` env vars.
| Variable | Default | Description |
|---|---|---|
| `MQTT_SECRET_PATH` | _(unset)_ | KV v2 path holding the broker creds (e.g. `home_assistant`). Unset → use env creds. |
| `MQTT_SECRET_USER_KEY` | `username` | Key within the secret for the broker username |
| `MQTT_SECRET_PASS_KEY` | `password` | Key within the secret for the broker password |
| `OPENBAO_ADDR` | `https://vault.adamksmith.xyz` | OpenBao API base URL |
| `OPENBAO_MOUNT` | `secret` | KV v2 mount point |
| `OPENBAO_TOKEN_FILE` | _(unset)_ | Path to a file containing the OpenBao token (preferred; mount read-only) |
| `OPENBAO_TOKEN` | _(unset)_ | OpenBao token (used if no token file) |
The deployed compose reads the broker user/pass from `secret/home_assistant`
(keys `mqtt_user`/`mqtt_pass`) using the read-only `claude-read` token mounted
at `/run/secrets/openbao-token`.
Topics (defaults):
**Credentials**: simplest is a root-owned `/etc/jbod-monitor/mqtt.env` with
`MQTT_USERNAME=`/`MQTT_PASSWORD=`, loaded via compose `env_file`. Optionally the
app can read them from OpenBao at runtime (`MQTT_SECRET_PATH` + `OPENBAO_*`,
keys via `MQTT_SECRET_USER_KEY`/`MQTT_SECRET_PASS_KEY`) — see
`services/secrets.py`; it falls back to the env vars if the read fails.
Topics:
```
homeassistant/sensor/<node>_enc<id>/hotspot/config # retained discovery
homeassistant/sensor/<node>_enc<id>/temp<n>/config # retained discovery
jbod-monitor/<node>/status # online | offline (LWT)
jbod-monitor/<node>/enclosure/<id>/state # {"hotspot_c":42.0, "sensor_0":28.5, ...}
jbod-monitor/<node>/enclosure/<id>/state # {"hotspot_c":42.0, ...}
```
Availability is tracked via an MQTT LWT, so entities show *unavailable* in HA
if the monitor stops.