# JBOD Monitor

Drive-health monitoring for JBOD enclosures on Linux, with a REST API, web UI,
and optional Home Assistant MQTT publishing.

Auto-discovers SES enclosures via sysfs, maps drives to physical slots, reads
SMART, and aggregates per-enclosure temperatures (named SES sensors + hotspot).

## Architecture

Two processes, with **Redis as the only interface between them**:

```
poller (privileged, pid:host)  ──smartctl/sg_ses/zpool/ledctl──>  hardware
        │  serialized + paced
        ▼
      Redis  ◀── reads ──  app (REST API + web UI + MQTT)   [unprivileged]
        ▲
        └── locate-LED command queue (app enqueues, poller runs)
```

- **`poller`** is the *only* process that touches the SAS bus. All hardware
  commands run behind a single gate (`POLL_CONCURRENCY`, default 1 = fully
  serialized) with a gap between drives, so it never storms fragile SAS
  expanders. It sweeps on an interval and writes results to Redis.
- **`app`** (and the MQTT publisher) are pure Redis readers — unprivileged, no
  `/dev`. They can restart freely without issuing a single hardware command.
- **Locate LEDs**: the API enqueues a request in Redis; the poller executes it
  serialized with everything else.

This split exists for a reason: concurrent/bursty SMART+SES traffic — especially
repeated on every container restart — can drop SAS expanders and suspend pools.
One paced owner makes that structurally impossible.

## Prerequisites

- Linux with SAS/SATA JBODs connected via HBA
- `smartmontools` (`smartctl`), `sg3-utils` (`sg_ses`), `ledmon` (`ledctl`)
- Redis
- Python 3.11+

```bash
apt install smartmontools sg3-utils ledmon   # Debian/Ubuntu
```

## Run (docker compose)

```bash
docker compose up -d --build
```

Brings up `poller` (privileged), `app` (port 8000), and `redis`. To run the two
processes by hand instead:

```bash
REDIS_HOST=localhost sudo -E python poller.py            # producer (needs root)
REDIS_HOST=localhost uvicorn main:app --port 8000        # consumer
```

## API Endpoints

| Endpoint | Description |
|---|---|
| `GET /api/health` | Service health + `redis`/`mqtt`/`poller_fresh` status |
| `GET /api/enclosures` | List discovered SES enclosures |
| `GET /api/enclosures/{id}/drives` | Drive slots for an enclosure |
| `GET /api/drives/{device}` | SMART detail for a block device |
| `GET /api/overview` | Aggregate enclosure + drive health |
| `GET /api/temps` | Per-enclosure temperatures: named SES sensors + hotspot |
| `POST /api/drives/{device}/led` | Queue a locate-LED change (`state`: `locate`/`off`) |
| `GET /docs` | Swagger UI |

Read endpoints serve the poller's last sweep; `X-Poll-Age` (seconds) headers and
the `poller_fresh` health flag surface staleness.

## Poller tuning (env)

| Variable | Default | Description |
|---|---|---|
| `POLL_SWEEP_INTERVAL` | `300` | Seconds between full sweeps |
| `POLL_DRIVE_GAP` | `0.5` | Seconds between each drive's SMART query |
| `POLL_CONCURRENCY` | `1` | Max concurrent hardware ops (1 = serialized) |
| `POLL_STARTUP_JITTER` | `10` | Random startup delay to avoid restart storms |
| `POLL_DATA_TTL` | `900` | Redis TTL for poller data (must exceed sweep interval) |
| `ZFS_USE_NSENTER` | — | `true` to run `zpool` in the host namespace (containers) |

## Home Assistant (MQTT)

The `app` publishes per-enclosure temperatures via
[MQTT discovery](https://www.home-assistant.io/integrations/mqtt/#mqtt-discovery)
— no HA YAML. Each enclosure becomes a device with a **Hotspot** sensor (max
across SES sensors + housed drive temps; `drive_max_c`/`drive_temp_count` as
attributes) and one sensor per named SES temperature element. Opt-in via
`MQTT_HOST`; availability is tracked via an MQTT LWT.

| Variable | Default | Description |
|---|---|---|
| `MQTT_HOST` | _(unset)_ | Broker host. Unset → MQTT disabled. |
| `MQTT_PORT` | `1883` | Broker port |
| `MQTT_USERNAME` / `MQTT_PASSWORD` | _(unset)_ | Broker credentials |
| `MQTT_DISCOVERY_PREFIX` | `homeassistant` | HA discovery topic prefix |
| `MQTT_BASE_TOPIC` | `jbod-monitor` | State/availability topic root |
| `MQTT_PUBLISH_INTERVAL` | `60` | Seconds between publishes |
| `MQTT_NODE_ID` | _(hostname)_ | Stable id (disambiguates multiple hosts) |

**Credentials**: simplest is a root-owned `/etc/jbod-monitor/mqtt.env` with
`MQTT_USERNAME=`/`MQTT_PASSWORD=`, loaded via compose `env_file`. Optionally the
app can read them from OpenBao at runtime (`MQTT_SECRET_PATH` + `OPENBAO_*`,
keys via `MQTT_SECRET_USER_KEY`/`MQTT_SECRET_PASS_KEY`) — see
`services/secrets.py`; it falls back to the env vars if the read fails.

Topics:
```
homeassistant/sensor/<node>_enc<id>/hotspot/config   # retained discovery
homeassistant/sensor/<node>_enc<id>/temp<n>/config    # retained discovery
jbod-monitor/<node>/status                            # online | offline (LWT)
jbod-monitor/<node>/enclosure/<id>/state              # {"hotspot_c":42.0, ...}
```