adam
cccc2bc004
Decouple the privileged hardware poller from web iterations so a web
redeploy can never recreate/restart the poller (the backplane-touching
container). Two images from one Dockerfile via build targets:
- jbod-poller: privileged producer, ships smartmontools/sg3-utils/ledmon,
Redis-only deps (~197MB)
- jbod-web: unprivileged read-only consumer (REST/UI/MQTT), no hardware
tools, no /dev (~147MB)
Two compose projects sharing Redis over host networking:
- compose.infra.yml (jbod-infra): poller + redis — stable substrate
- compose.web.yml (jbod-web): app — 'up -d --build' touches only app
build.sh builds/pushes both images; split requirements-{poller,web}.txt;
dropped the combined docker-compose.yml; .dockerignore excludes tmp/ (keeps
the venv and the mqtt.env creds out of the build context).
128 lines
5.4 KiB
Markdown
128 lines
5.4 KiB
Markdown
# JBOD Monitor
|
|
|
|
Drive-health monitoring for JBOD enclosures on Linux, with a REST API, web UI,
|
|
and optional Home Assistant MQTT publishing.
|
|
|
|
Auto-discovers SES enclosures via sysfs, maps drives to physical slots, reads
|
|
SMART, and aggregates per-enclosure temperatures (named SES sensors + hotspot).
|
|
|
|
## Architecture
|
|
|
|
Two processes, with **Redis as the only interface between them**:
|
|
|
|
```
|
|
poller (privileged, pid:host) ──smartctl/sg_ses/zpool/ledctl──> hardware
|
|
│ serialized + paced
|
|
▼
|
|
Redis ◀── reads ── app (REST API + web UI + MQTT) [unprivileged]
|
|
▲
|
|
└── locate-LED command queue (app enqueues, poller runs)
|
|
```
|
|
|
|
- **`poller`** is the *only* process that touches the SAS bus. All hardware
|
|
commands run behind a single gate (`POLL_CONCURRENCY`, default 1 = fully
|
|
serialized) with a gap between drives, so it never storms fragile SAS
|
|
expanders. It sweeps on an interval and writes results to Redis.
|
|
- **`app`** (and the MQTT publisher) are pure Redis readers — unprivileged, no
|
|
`/dev`. They can restart freely without issuing a single hardware command.
|
|
- **Locate LEDs**: the API enqueues a request in Redis; the poller executes it
|
|
serialized with everything else.
|
|
|
|
This split exists for a reason: concurrent/bursty SMART+SES traffic — especially
|
|
repeated on every container restart — can drop SAS expanders and suspend pools.
|
|
One paced owner makes that structurally impossible.
|
|
|
|
## Prerequisites
|
|
|
|
- Linux with SAS/SATA JBODs connected via HBA
|
|
- `smartmontools` (`smartctl`), `sg3-utils` (`sg_ses`), `ledmon` (`ledctl`)
|
|
- Redis
|
|
- Python 3.11+
|
|
|
|
```bash
|
|
apt install smartmontools sg3-utils ledmon # Debian/Ubuntu
|
|
```
|
|
|
|
## Run (docker compose)
|
|
|
|
Two **independent** stacks so web iterations never recreate the privileged
|
|
poller (which touches the SAS bus). They're built as two images —
|
|
`jbod-poller` (privileged, has the SAS/SMART tooling) and `jbod-web` (slim,
|
|
unprivileged) — and share Redis over host networking.
|
|
|
|
```bash
|
|
# 1. Infra: poller + redis — the stable substrate. Deploy once; touch deliberately.
|
|
docker compose -f compose.infra.yml up -d
|
|
|
|
# 2. Web: REST/UI/MQTT — iterate freely; this only ever recreates `app`.
|
|
docker compose -f compose.web.yml up -d --build
|
|
```
|
|
|
|
Pin both images to a known-good SHA in deploy (`./build.sh` tags `:<sha>` and
|
|
`:latest` for both). To run the two processes by hand instead:
|
|
|
|
```bash
|
|
REDIS_HOST=localhost sudo -E python poller.py # producer (needs root)
|
|
REDIS_HOST=localhost uvicorn main:app --port 8000 # consumer
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
| Endpoint | Description |
|
|
|---|---|
|
|
| `GET /api/health` | Service health + `redis`/`mqtt`/`poller_fresh` status |
|
|
| `GET /api/enclosures` | List discovered SES enclosures |
|
|
| `GET /api/enclosures/{id}/drives` | Drive slots for an enclosure |
|
|
| `GET /api/drives/{device}` | SMART detail for a block device |
|
|
| `GET /api/overview` | Aggregate enclosure + drive health |
|
|
| `GET /api/temps` | Per-enclosure temperatures: named SES sensors + hotspot |
|
|
| `POST /api/drives/{device}/led` | Queue a locate-LED change (`state`: `locate`/`off`) |
|
|
| `GET /docs` | Swagger UI |
|
|
|
|
Read endpoints serve the poller's last sweep; `X-Poll-Age` (seconds) headers and
|
|
the `poller_fresh` health flag surface staleness.
|
|
|
|
## Poller tuning (env)
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `POLL_SWEEP_INTERVAL` | `300` | Seconds between full sweeps |
|
|
| `POLL_DRIVE_GAP` | `0.5` | Seconds between each drive's SMART query |
|
|
| `POLL_CONCURRENCY` | `1` | Max concurrent hardware ops (1 = serialized) |
|
|
| `POLL_STARTUP_JITTER` | `10` | Random startup delay to avoid restart storms |
|
|
| `POLL_DATA_TTL` | `900` | Redis TTL for poller data (must exceed sweep interval) |
|
|
| `ZFS_USE_NSENTER` | — | `true` to run `zpool` in the host namespace (containers) |
|
|
|
|
## Home Assistant (MQTT)
|
|
|
|
The `app` publishes per-enclosure temperatures via
|
|
[MQTT discovery](https://www.home-assistant.io/integrations/mqtt/#mqtt-discovery)
|
|
— no HA YAML. Each enclosure becomes a device with a **Hotspot** sensor (max
|
|
across SES sensors + housed drive temps; `drive_max_c`/`drive_temp_count` as
|
|
attributes) and one sensor per named SES temperature element. Opt-in via
|
|
`MQTT_HOST`; availability is tracked via an MQTT LWT.
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `MQTT_HOST` | _(unset)_ | Broker host. Unset → MQTT disabled. |
|
|
| `MQTT_PORT` | `1883` | Broker port |
|
|
| `MQTT_USERNAME` / `MQTT_PASSWORD` | _(unset)_ | Broker credentials |
|
|
| `MQTT_DISCOVERY_PREFIX` | `homeassistant` | HA discovery topic prefix |
|
|
| `MQTT_BASE_TOPIC` | `jbod-monitor` | State/availability topic root |
|
|
| `MQTT_PUBLISH_INTERVAL` | `60` | Seconds between publishes |
|
|
| `MQTT_NODE_ID` | _(hostname)_ | Stable id (disambiguates multiple hosts) |
|
|
|
|
**Credentials**: simplest is a root-owned `/etc/jbod-monitor/mqtt.env` with
|
|
`MQTT_USERNAME=`/`MQTT_PASSWORD=`, loaded via compose `env_file`. Optionally the
|
|
app can read them from OpenBao at runtime (`MQTT_SECRET_PATH` + `OPENBAO_*`,
|
|
keys via `MQTT_SECRET_USER_KEY`/`MQTT_SECRET_PASS_KEY`) — see
|
|
`services/secrets.py`; it falls back to the env vars if the read fails.
|
|
|
|
Topics:
|
|
```
|
|
homeassistant/sensor/<node>_enc<id>/hotspot/config # retained discovery
|
|
homeassistant/sensor/<node>_enc<id>/temp<n>/config # retained discovery
|
|
jbod-monitor/<node>/status # online | offline (LWT)
|
|
jbod-monitor/<node>/enclosure/<id>/state # {"hotspot_c":42.0, ...}
|
|
```
|