adam
0f829e0380
Rearchitect so a single poller process is the sole owner of all SAS/SMART/ SES hardware I/O, serialized behind one gate and paced, writing results to Redis. The API/web and MQTT publisher become pure Redis readers — they no longer issue any subprocess and can restart freely without touching the bus. This addresses backplane/expander stress from concurrent + restart-triggered SMART/SES storms (the prior model re-ran a hardware sweep on every container start, and polled sg_ses 0x02+0x07 every 60s; 0x07 errored on the IOM6/ Xyratex expanders). - poller.py: paced sweep (inventory, SMART per-drive w/ gap, SES 0x02, ZFS, host/MegaRAID), startup jitter, single-instance Redis lock, LED-queue worker - services/hwgate.py: global serialization semaphore (POLL_CONCURRENCY=1) - services/store.py: Redis as the only producer<->consumer interface + LED queue - services/health.py: shared drive-health classifier (fixes overview double-count) - gate smartctl/sg_ses/zpool/ledctl; drop sg_ses 0x07 from the hot path - routers + temps read-only from the store; main.py drops the in-process poller - compose: 3 services (privileged poller + unprivileged app + redis) w/ pacing knobs
119 lines
4.8 KiB
Markdown
119 lines
4.8 KiB
Markdown
# JBOD Monitor
|
|
|
|
Drive-health monitoring for JBOD enclosures on Linux, with a REST API, web UI,
|
|
and optional Home Assistant MQTT publishing.
|
|
|
|
Auto-discovers SES enclosures via sysfs, maps drives to physical slots, reads
|
|
SMART, and aggregates per-enclosure temperatures (named SES sensors + hotspot).
|
|
|
|
## Architecture
|
|
|
|
Two processes, with **Redis as the only interface between them**:
|
|
|
|
```
|
|
poller (privileged, pid:host) ──smartctl/sg_ses/zpool/ledctl──> hardware
|
|
│ serialized + paced
|
|
▼
|
|
Redis ◀── reads ── app (REST API + web UI + MQTT) [unprivileged]
|
|
▲
|
|
└── locate-LED command queue (app enqueues, poller runs)
|
|
```
|
|
|
|
- **`poller`** is the *only* process that touches the SAS bus. All hardware
|
|
commands run behind a single gate (`POLL_CONCURRENCY`, default 1 = fully
|
|
serialized) with a gap between drives, so it never storms fragile SAS
|
|
expanders. It sweeps on an interval and writes results to Redis.
|
|
- **`app`** (and the MQTT publisher) are pure Redis readers — unprivileged, no
|
|
`/dev`. They can restart freely without issuing a single hardware command.
|
|
- **Locate LEDs**: the API enqueues a request in Redis; the poller executes it
|
|
serialized with everything else.
|
|
|
|
This split exists for a reason: concurrent/bursty SMART+SES traffic — especially
|
|
repeated on every container restart — can drop SAS expanders and suspend pools.
|
|
One paced owner makes that structurally impossible.
|
|
|
|
## Prerequisites
|
|
|
|
- Linux with SAS/SATA JBODs connected via HBA
|
|
- `smartmontools` (`smartctl`), `sg3-utils` (`sg_ses`), `ledmon` (`ledctl`)
|
|
- Redis
|
|
- Python 3.11+
|
|
|
|
```bash
|
|
apt install smartmontools sg3-utils ledmon # Debian/Ubuntu
|
|
```
|
|
|
|
## Run (docker compose)
|
|
|
|
```bash
|
|
docker compose up -d --build
|
|
```
|
|
|
|
Brings up `poller` (privileged), `app` (port 8000), and `redis`. To run the two
|
|
processes by hand instead:
|
|
|
|
```bash
|
|
REDIS_HOST=localhost sudo -E python poller.py # producer (needs root)
|
|
REDIS_HOST=localhost uvicorn main:app --port 8000 # consumer
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
| Endpoint | Description |
|
|
|---|---|
|
|
| `GET /api/health` | Service health + `redis`/`mqtt`/`poller_fresh` status |
|
|
| `GET /api/enclosures` | List discovered SES enclosures |
|
|
| `GET /api/enclosures/{id}/drives` | Drive slots for an enclosure |
|
|
| `GET /api/drives/{device}` | SMART detail for a block device |
|
|
| `GET /api/overview` | Aggregate enclosure + drive health |
|
|
| `GET /api/temps` | Per-enclosure temperatures: named SES sensors + hotspot |
|
|
| `POST /api/drives/{device}/led` | Queue a locate-LED change (`state`: `locate`/`off`) |
|
|
| `GET /docs` | Swagger UI |
|
|
|
|
Read endpoints serve the poller's last sweep; `X-Poll-Age` (seconds) headers and
|
|
the `poller_fresh` health flag surface staleness.
|
|
|
|
## Poller tuning (env)
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `POLL_SWEEP_INTERVAL` | `300` | Seconds between full sweeps |
|
|
| `POLL_DRIVE_GAP` | `0.5` | Seconds between each drive's SMART query |
|
|
| `POLL_CONCURRENCY` | `1` | Max concurrent hardware ops (1 = serialized) |
|
|
| `POLL_STARTUP_JITTER` | `10` | Random startup delay to avoid restart storms |
|
|
| `POLL_DATA_TTL` | `900` | Redis TTL for poller data (must exceed sweep interval) |
|
|
| `ZFS_USE_NSENTER` | — | `true` to run `zpool` in the host namespace (containers) |
|
|
|
|
## Home Assistant (MQTT)
|
|
|
|
The `app` publishes per-enclosure temperatures via
|
|
[MQTT discovery](https://www.home-assistant.io/integrations/mqtt/#mqtt-discovery)
|
|
— no HA YAML. Each enclosure becomes a device with a **Hotspot** sensor (max
|
|
across SES sensors + housed drive temps; `drive_max_c`/`drive_temp_count` as
|
|
attributes) and one sensor per named SES temperature element. Opt-in via
|
|
`MQTT_HOST`; availability is tracked via an MQTT LWT.
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `MQTT_HOST` | _(unset)_ | Broker host. Unset → MQTT disabled. |
|
|
| `MQTT_PORT` | `1883` | Broker port |
|
|
| `MQTT_USERNAME` / `MQTT_PASSWORD` | _(unset)_ | Broker credentials |
|
|
| `MQTT_DISCOVERY_PREFIX` | `homeassistant` | HA discovery topic prefix |
|
|
| `MQTT_BASE_TOPIC` | `jbod-monitor` | State/availability topic root |
|
|
| `MQTT_PUBLISH_INTERVAL` | `60` | Seconds between publishes |
|
|
| `MQTT_NODE_ID` | _(hostname)_ | Stable id (disambiguates multiple hosts) |
|
|
|
|
**Credentials**: simplest is a root-owned `/etc/jbod-monitor/mqtt.env` with
|
|
`MQTT_USERNAME=`/`MQTT_PASSWORD=`, loaded via compose `env_file`. Optionally the
|
|
app can read them from OpenBao at runtime (`MQTT_SECRET_PATH` + `OPENBAO_*`,
|
|
keys via `MQTT_SECRET_USER_KEY`/`MQTT_SECRET_PASS_KEY`) — see
|
|
`services/secrets.py`; it falls back to the env vars if the read fails.
|
|
|
|
Topics:
|
|
```
|
|
homeassistant/sensor/<node>_enc<id>/hotspot/config # retained discovery
|
|
homeassistant/sensor/<node>_enc<id>/temp<n>/config # retained discovery
|
|
jbod-monitor/<node>/status # online | offline (LWT)
|
|
jbod-monitor/<node>/enclosure/<id>/state # {"hotspot_c":42.0, ...}
|
|
```
|