Dedicated hardware poller + read-only consumers #4

Open
adamksmith wants to merge 13 commits from feat/dedicated-poller into main
Owner

Why

Backplane/expander stress. The prior model re-ran a full hardware sweep on every container start, polled sg_ses 0x02+0x07 every 60s (the 0x07 page errored on the IOM6/Xyratex expanders — couldn't read config page, res=35), and fanned out smartctl concurrently. Rapid deploy cycling multiplied those storms and crashed backplanes on a host whose SAS expanders have a documented history of dropping out and suspending pools.

What

A single poller process is now the sole owner of all SAS/SMART/SES hardware I/O — serialized behind one gate, paced, writing to Redis. The API/web and MQTT publisher become pure Redis readers: zero subprocesses, so they can restart freely without ever touching the bus.

Producer

  • poller.py — paced sweep (inventory → SMART per-drive with a gap → SES 0x02 → ZFS → host/MegaRAID), startup jitter, single-instance Redis lock, responsive LED-queue worker.
  • services/hwgate.py — global serialization semaphore (POLL_CONCURRENCY, default 1); every smartctl/sg_ses/zpool/ledctl runs behind it.
  • services/store.py — Redis as the only producer↔consumer interface (+ LED command queue + single-instance lock).
  • services/health.py — shared drive-health classifier (also fixes the old overview double-count).

Consumers (read-only)

  • routers/{overview,enclosures,drives,leds} + services/temps.py read exclusively from the store. LED POSTs enqueue to Redis; the poller executes them.
  • main.py drops the in-process poll loop; keeps API + MQTT. /api/health now reports poller_fresh; read endpoints expose X-Poll-Age.

Hardware-stress fixes

  • sg_ses 0x07 removed from the hot path.
  • SMART fan-out serialized + paced (POLL_DRIVE_GAP).
  • Only the poller container is privileged/pid:host; app is unprivileged with no /dev.

Deploy

  • docker-compose.yml: 3 services (poller + app + redis) with pacing knobs (POLL_SWEEP_INTERVAL, POLL_DRIVE_GAP, POLL_CONCURRENCY, POLL_STARTUP_JITTER, POLL_DATA_TTL).

Testing

  • Compiles + imports clean; no stale references.
  • End-to-end consumer read-path (in-memory store): /temps hotspot = max(SES, drive), /overview health/zfs/counts, /enclosures, /drives verified.
  • LED queue + single-instance lock unit-checked.
  • docker compose config valid.

Not yet deployed — hold until backplanes/pools are confirmed healthy.

## Why Backplane/expander stress. The prior model re-ran a full hardware sweep on **every container start**, polled `sg_ses 0x02+0x07` every 60s (the `0x07` page errored on the IOM6/Xyratex expanders — `couldn't read config page, res=35`), and fanned out `smartctl` concurrently. Rapid deploy cycling multiplied those storms and crashed backplanes on a host whose SAS expanders have a documented history of dropping out and suspending pools. ## What A single **poller** process is now the *sole* owner of all SAS/SMART/SES hardware I/O — serialized behind one gate, paced, writing to Redis. The API/web and MQTT publisher become **pure Redis readers**: zero subprocesses, so they can restart freely without ever touching the bus. ### Producer - `poller.py` — paced sweep (inventory → SMART per-drive with a gap → SES `0x02` → ZFS → host/MegaRAID), startup jitter, single-instance Redis lock, responsive LED-queue worker. - `services/hwgate.py` — global serialization semaphore (`POLL_CONCURRENCY`, default **1**); every `smartctl`/`sg_ses`/`zpool`/`ledctl` runs behind it. - `services/store.py` — Redis as the only producer↔consumer interface (+ LED command queue + single-instance lock). - `services/health.py` — shared drive-health classifier (also fixes the old overview double-count). ### Consumers (read-only) - `routers/{overview,enclosures,drives,leds}` + `services/temps.py` read exclusively from the store. LED POSTs enqueue to Redis; the poller executes them. - `main.py` drops the in-process poll loop; keeps API + MQTT. `/api/health` now reports `poller_fresh`; read endpoints expose `X-Poll-Age`. ### Hardware-stress fixes - `sg_ses 0x07` removed from the hot path. - SMART fan-out serialized + paced (`POLL_DRIVE_GAP`). - Only the `poller` container is privileged/`pid:host`; `app` is unprivileged with no `/dev`. ### Deploy - `docker-compose.yml`: 3 services (`poller` + `app` + `redis`) with pacing knobs (`POLL_SWEEP_INTERVAL`, `POLL_DRIVE_GAP`, `POLL_CONCURRENCY`, `POLL_STARTUP_JITTER`, `POLL_DATA_TTL`). ## Testing - Compiles + imports clean; no stale references. - End-to-end consumer read-path (in-memory store): `/temps` hotspot = max(SES, drive), `/overview` health/zfs/counts, `/enclosures`, `/drives` verified. - LED queue + single-instance lock unit-checked. - `docker compose config` valid. > Not yet deployed — hold until backplanes/pools are confirmed healthy.
adamksmith added 1 commit 2026-06-16 15:13:26 +00:00
Rearchitect so a single poller process is the sole owner of all SAS/SMART/
SES hardware I/O, serialized behind one gate and paced, writing results to
Redis. The API/web and MQTT publisher become pure Redis readers — they no
longer issue any subprocess and can restart freely without touching the bus.

This addresses backplane/expander stress from concurrent + restart-triggered
SMART/SES storms (the prior model re-ran a hardware sweep on every container
start, and polled sg_ses 0x02+0x07 every 60s; 0x07 errored on the IOM6/
Xyratex expanders).

- poller.py: paced sweep (inventory, SMART per-drive w/ gap, SES 0x02, ZFS,
  host/MegaRAID), startup jitter, single-instance Redis lock, LED-queue worker
- services/hwgate.py: global serialization semaphore (POLL_CONCURRENCY=1)
- services/store.py: Redis as the only producer<->consumer interface + LED queue
- services/health.py: shared drive-health classifier (fixes overview double-count)
- gate smartctl/sg_ses/zpool/ledctl; drop sg_ses 0x07 from the hot path
- routers + temps read-only from the store; main.py drops the in-process poller
- compose: 3 services (privileged poller + unprivileged app + redis) w/ pacing knobs
adamksmith added 1 commit 2026-06-16 15:23:46 +00:00
Decouple the privileged hardware poller from web iterations so a web
redeploy can never recreate/restart the poller (the backplane-touching
container). Two images from one Dockerfile via build targets:

- jbod-poller: privileged producer, ships smartmontools/sg3-utils/ledmon,
  Redis-only deps (~197MB)
- jbod-web: unprivileged read-only consumer (REST/UI/MQTT), no hardware
  tools, no /dev (~147MB)

Two compose projects sharing Redis over host networking:
- compose.infra.yml (jbod-infra): poller + redis — stable substrate
- compose.web.yml   (jbod-web):   app — 'up -d --build' touches only app

build.sh builds/pushes both images; split requirements-{poller,web}.txt;
dropped the combined docker-compose.yml; .dockerignore excludes tmp/ (keeps
the venv and the mqtt.env creds out of the build context).
Author
Owner

Added a second commit: independent poller and web deploy stacks so a web redeploy can never recreate/restart the privileged poller.

  • Two images from one Dockerfile (build targets): jbod-poller (privileged, ships smartmontools/sg3-utils/ledmon, Redis-only deps, ~197MB) and jbod-web (unprivileged read-only consumer, no hardware tools, ~147MB).
  • Two compose projects sharing Redis over host networking: compose.infra.yml (jbod-infra: poller + redis — stable substrate) and compose.web.yml (jbod-web: app — up -d --build touches only app).
  • build.sh builds/pushes both; split requirements-{poller,web}.txt; dropped the combined docker-compose.yml; .dockerignore now excludes tmp/.

Verified: both targets build; imports succeed under each image's slim dep set; web image has no smartctl/sg_ses/ledctl, poller has all three.

Deploy workflow:

docker compose -f compose.infra.yml up -d           # poller + redis (rarely)
docker compose -f compose.web.yml  up -d --build    # iterate the web app
Added a second commit: **independent poller and web deploy stacks** so a web redeploy can never recreate/restart the privileged poller. - Two images from one Dockerfile (build targets): `jbod-poller` (privileged, ships smartmontools/sg3-utils/ledmon, Redis-only deps, ~197MB) and `jbod-web` (unprivileged read-only consumer, no hardware tools, ~147MB). - Two compose projects sharing Redis over host networking: `compose.infra.yml` (`jbod-infra`: poller + redis — stable substrate) and `compose.web.yml` (`jbod-web`: app — `up -d --build` touches only `app`). - `build.sh` builds/pushes both; split `requirements-{poller,web}.txt`; dropped the combined `docker-compose.yml`; `.dockerignore` now excludes `tmp/`. Verified: both targets build; imports succeed under each image's slim dep set; web image has no `smartctl`/`sg_ses`/`ledctl`, poller has all three. Deploy workflow: ``` docker compose -f compose.infra.yml up -d # poller + redis (rarely) docker compose -f compose.web.yml up -d --build # iterate the web app ```
adamksmith added 1 commit 2026-06-16 15:58:46 +00:00
Adversarial review (gpt-5.5) of the new architecture surfaced real gaps:

Hardware-safety / storm prevention:
- Restart-storm guard: on startup, defer the first sweep if a sweep ran
  < POLL_SWEEP_INTERVAL ago (persisted in Redis), so deploy-cycling can no
  longer trigger back-to-back full hardware sweeps (poller.py).
- Centralize pacing in the hardware gate: POLL_DRIVE_GAP is now held after
  EVERY hardware op (SMART, SES, host, MegaRAID, ledctl), not just the
  enclosure-drive loop (hwgate.py); removed the now-redundant per-loop sleeps.
- Single-instance lock made atomic (Lua compare-and-set / compare-and-expire)
  and the refresher is now fatal on any error + has a done-callback, so a
  dropped lock can never leave two pollers sweeping concurrently (store.py,
  poller.py).
- Warn loudly when POLL_CONCURRENCY > 1.

Correctness:
- Heartbeat now actually writes: cache_set omits EX when ttl<=0 (Redis
  rejects EX 0), so poller meta/last_sweep_ts persists — this also enables
  the restart-storm guard and the poller_fresh health flag (cache.py).
- Web image runs a single uvicorn worker so the MQTT publisher is a true
  singleton (two workers shared a client_id and flapped the broker) (Dockerfile).
- LED enqueue catches Redis errors -> API returns 503, not 500 (store.py).

Deploy independence:
- build.sh takes a target (web|poller|all) so web iterations never rebuild
  or repush the poller image.

Nits: drop dead SMART_CACHE_TTL constant.
Author
Owner

Third commit: hardening from an adversarial Codex (gpt-5.5) review focused on whether anything can still storm the SAS bus. Real gaps it caught (vetted against the code):

Storm prevention

  • Restart-storm guard — on startup the poller now defers its first sweep if a sweep ran < POLL_SWEEP_INTERVAL ago (persisted in Redis). Previously jitter (≤10s) then an immediate sweep meant deploy-cycling could still fire back-to-back full sweeps — the exact failure mode that crashed the backplanes.
  • Centralized pacingPOLL_DRIVE_GAP is now enforced inside the hardware gate after every op (SMART, SES, host, MegaRAID, ledctl), not just the enclosure loop. Host/MegaRAID/LED were previously serialized but gap-less.
  • Atomic single-instance lock — Lua compare-and-set / compare-and-expire (was GET-then-EXPIRE, racy); refresher is now fatal on any error + has a done-callback, so a dropped lock can't leave two pollers sweeping concurrently.

Correctness

  • Heartbeat actually writes nowset_meta used ttl=0 → Redis rejects EX 0 → meta never persisted → poller_fresh always false and the restart guard had nothing to read. cache_set now omits EX when ttl<=0.
  • MQTT singleton — web image runs --workers 1; two workers shared a client_id and flapped the broker.
  • LED enqueue returns 503 (not 500) on Redis failure.

Deploy independence

  • build.sh web|poller|all so a web iteration never rebuilds/repushes the poller image.

Vetted-but-declined: healthy=True with warnings is by-design (errors only); Pydantic v2 handles mutable defaults; concurrency now warns instead of hard-clamping.

Verified: compiles/imports; gate serializes + paces (no overlap); consumer read-path, heartbeat persistence, and Lua lock semantics all pass; both images build + import.

Third commit: **hardening from an adversarial Codex (gpt-5.5) review** focused on whether anything can still storm the SAS bus. Real gaps it caught (vetted against the code): **Storm prevention** - **Restart-storm guard** — on startup the poller now defers its first sweep if a sweep ran < `POLL_SWEEP_INTERVAL` ago (persisted in Redis). Previously jitter (≤10s) then an *immediate* sweep meant deploy-cycling could still fire back-to-back full sweeps — the exact failure mode that crashed the backplanes. - **Centralized pacing** — `POLL_DRIVE_GAP` is now enforced inside the hardware gate after *every* op (SMART, SES, host, MegaRAID, ledctl), not just the enclosure loop. Host/MegaRAID/LED were previously serialized but gap-less. - **Atomic single-instance lock** — Lua compare-and-set / compare-and-expire (was GET-then-EXPIRE, racy); refresher is now fatal on any error + has a done-callback, so a dropped lock can't leave two pollers sweeping concurrently. **Correctness** - **Heartbeat actually writes now** — `set_meta` used `ttl=0` → Redis rejects `EX 0` → meta never persisted → `poller_fresh` always false *and* the restart guard had nothing to read. `cache_set` now omits `EX` when `ttl<=0`. - **MQTT singleton** — web image runs `--workers 1`; two workers shared a `client_id` and flapped the broker. - **LED enqueue** returns 503 (not 500) on Redis failure. **Deploy independence** - `build.sh web|poller|all` so a web iteration never rebuilds/repushes the poller image. Vetted-but-declined: `healthy=True` with warnings is by-design (errors only); Pydantic v2 handles mutable defaults; concurrency now warns instead of hard-clamping. Verified: compiles/imports; gate serializes + paces (no overlap); consumer read-path, heartbeat persistence, and Lua lock semantics all pass; both images build + import.
adamksmith added 1 commit 2026-06-16 16:26:40 +00:00
- poller.py: start lock_refresher() IMMEDIATELY after acquiring the lock,
  before the startup jitter and restart-storm deferral. Those sleeps can
  together exceed LOCK_TTL (restart defer is up to POLL_SWEEP_INTERVAL), so
  with the refresher started afterward the lock could expire mid-deferral and
  a second poller could begin sweeping concurrently — the exact hardware-storm
  risk this design prevents.
- hwgate.py: clamp POLL_CONCURRENCY to 1 unless POLL_ALLOW_UNSAFE_CONCURRENCY
  is set (fragile SAS hardware; >1 can drop expanders). Warn either way.
- build.sh: keep backward-compatible — a non-target first arg is treated as
  the commit message (./build.sh "msg" still works), known targets select
  web|poller|all.
Author
Owner

Fourth commit: fixes from a Codex second pass, which caught a critical regression the first round's restart-storm guard introduced.

Critical (was a no-go): the restart-storm guard could let the poller lock expire before the refresher started. Flow was: acquire_lock (30s TTL) → jitter (≤10s) → restart-defer (up to POLL_SWEEP_INTERVAL ≈ 300s) → then start lock_refresher. During a long defer the lock expired, so a second poller (deploy overlap) could acquire it and both sweep together — the exact storm this prevents. Fixed: lock_refresher() now starts immediately after acquiring the lock, before any pre-sweep sleep.

Also:

  • POLL_CONCURRENCY now clamps to 1 unless POLL_ALLOW_UNSAFE_CONCURRENCY=1 (fragile expanders); warns either way.
  • build.sh kept backward-compatible — a non-target first arg is treated as the commit message, so ./build.sh "msg" still works.

Codex confirmed correct on second pass: the Lua lock scripts, cache_set(ttl<=0) heartbeat fix, gate serialize+pace (incl. gathered SMART / MegaRAID), and _critical_task_done shutdown handling. Verified locally: compiles/imports; refresher now precedes jitter+defer; clamp works (1 without override, N with); build.sh parsing covers all cases.

Codex's go/no-go: no-go until the lock-timing fix — which this commit makes. Remaining deploy conditions it noted: POLL_CONCURRENCY=1, pinned poller image SHA, web-only build/deploy path. (Still holding actual deploy until the backplanes are confirmed healthy.)

Fourth commit: fixes from a **Codex second pass**, which caught a critical regression the first round's restart-storm guard introduced. **Critical (was a no-go):** the restart-storm guard could let the poller lock expire *before* the refresher started. Flow was: `acquire_lock` (30s TTL) → jitter (≤10s) → restart-defer (**up to `POLL_SWEEP_INTERVAL` ≈ 300s**) → *then* start `lock_refresher`. During a long defer the lock expired, so a second poller (deploy overlap) could acquire it and both sweep together — the exact storm this prevents. Fixed: `lock_refresher()` now starts **immediately after acquiring the lock**, before any pre-sweep sleep. Also: - `POLL_CONCURRENCY` now clamps to 1 unless `POLL_ALLOW_UNSAFE_CONCURRENCY=1` (fragile expanders); warns either way. - `build.sh` kept backward-compatible — a non-target first arg is treated as the commit message, so `./build.sh "msg"` still works. Codex confirmed correct on second pass: the Lua lock scripts, `cache_set(ttl<=0)` heartbeat fix, gate serialize+pace (incl. gathered SMART / MegaRAID), and `_critical_task_done` shutdown handling. Verified locally: compiles/imports; refresher now precedes jitter+defer; clamp works (1 without override, N with); build.sh parsing covers all cases. Codex's go/no-go: no-go **until** the lock-timing fix — which this commit makes. Remaining deploy conditions it noted: `POLL_CONCURRENCY=1`, pinned poller image SHA, web-only build/deploy path. (Still holding actual deploy until the backplanes are confirmed healthy.)
adamksmith added 2 commits 2026-06-16 18:32:32 +00:00
Enclosure entities set via_device to a parent hub id that nothing defined,
so Home Assistant showed an 'Unnamed Device'. Publish a hub connectivity
binary_sensor ('Status', online/offline from the LWT topic) that defines the
parent device with a proper name, so the per-enclosure devices nest cleanly
under 'JBOD Monitor (<node>)'.
New third producer container (host-poller) owning the server chassis
subsystem, separate from the SAS-shelf poller:
- services/host_sensors.py: in-band IPMI (ipmitool sdr) for Inlet/Exhaust/etc.
  + CPU package temps from coretemp hwmon
- host_poller.py: own single-instance lock + heartbeat (host_poll keys),
  restart-storm guard, paced; writes jbod:host_sensors + jbod:host_drives
- move host/on-metal drive collection off the SAS poller to host-poller
  (reads jbod:zfs_map for annotation)
- store: generalized lock/meta with a key param; host_sensors + host-poll keys
- mqtt_publisher: publish env/CPU/host-drive temps as entities on the hub
  device (JBOD Monitor)
- Dockerfile host-poller target (ipmitool+smartmontools); compose.infra adds
  host-poller; build.sh gains host-poller/all
Author
Owner

Fifth commit: host-poller — a third producer container for server-chassis temps, kept separate from the SAS-shelf poller (different subsystem, shouldn't share its hardware gate).

  • services/host_sensors.py: in-band IPMI (ipmitool sdr type temperature) for Inlet/Exhaust + CPU coretemp from hwmon.
  • host_poller.py: own single-instance lock + heartbeat (host_poll keys), restart-storm guard, paced; writes jbod:host_sensors + jbod:host_drives. On-metal drive collection moved off the SAS poller (reads jbod:zfs_map for annotation).
  • services/store.py: lock/meta generalized with a key param; host keys added.
  • services/mqtt_publisher.py: publishes env (Inlet/Exhaust), CPU, and on-metal drive temps as entities on the hub device.
  • Dockerfile: host-poller target (ipmitool + smartmontools); compose.infra.yml adds the service; build.sh gains host-poller/all.

Deployed to nas01 (1839e9d, all four containers healthy). HA hub "JBOD Monitor (usco-dc-nas01)" now shows Inlet, Exhaust, CPU 0, and Drive nvme0n1/megaraid:0/megaraid:1 — plus the 5 nested enclosure devices. (R620 is single-CPU, so one CPU temp is correct; coretemp vs the iDRAC CPU sensor differ by ~10°C — currently publishing coretemp.)

Fifth commit: **host-poller** — a third producer container for server-chassis temps, kept separate from the SAS-shelf poller (different subsystem, shouldn't share its hardware gate). - `services/host_sensors.py`: in-band IPMI (`ipmitool sdr type temperature`) for Inlet/Exhaust + CPU coretemp from hwmon. - `host_poller.py`: own single-instance lock + heartbeat (`host_poll` keys), restart-storm guard, paced; writes `jbod:host_sensors` + `jbod:host_drives`. On-metal drive collection moved off the SAS poller (reads `jbod:zfs_map` for annotation). - `services/store.py`: lock/meta generalized with a `key` param; host keys added. - `services/mqtt_publisher.py`: publishes env (Inlet/Exhaust), CPU, and on-metal drive temps as entities on the **hub** device. - `Dockerfile`: `host-poller` target (ipmitool + smartmontools); `compose.infra.yml` adds the service; `build.sh` gains `host-poller`/`all`. Deployed to nas01 (`1839e9d`, all four containers healthy). HA hub "JBOD Monitor (usco-dc-nas01)" now shows Inlet, Exhaust, CPU 0, and Drive nvme0n1/megaraid:0/megaraid:1 — plus the 5 nested enclosure devices. (R620 is single-CPU, so one CPU temp is correct; coretemp vs the iDRAC CPU sensor differ by ~10°C — currently publishing coretemp.)
adamksmith added 1 commit 2026-06-16 18:45:35 +00:00
adamksmith added 1 commit 2026-06-16 18:50:20 +00:00
Two concurrent loops in one process sharing the lock + hardware gate (IPMI and
SMART never run concurrently) but on separate schedules: env every
HOST_ENV_INTERVAL (60s, responsive inlet/CPU) and drives every
HOST_DRIVE_INTERVAL (300s, gentle SMART). Each loop has its own heartbeat
(env_meta/drive_meta) + restart-storm guard.
adamksmith added 1 commit 2026-06-16 19:05:14 +00:00
- skip IPMI '*Margin' sensors (thermal margin / distance-to-throttle reads in
  degrees C but is not an absolute temperature, e.g. robin's P1 Therm Margin)
- coretemp: when a CPU has no 'Package id' sensor (older Xeons like robin's
  X3430), fall back to the hottest Core as the CPU temp
adamksmith added 1 commit 2026-06-16 19:41:54 +00:00
Design artifact (NOT applied). Per-node pod = host-poller + redis + web sharing
pod localhost; only host-poller privileged with hostPath /dev+/sys. No SAS
poller (no enclosures). MQTT_NODE_ID from spec.nodeName; MQTT creds via VSO
VaultStaticSecret reading secret/home_assistant (mqtt_user/pass -> MQTT_USERNAME/
PASSWORD). nodeSelector jbod-monitor=enabled to pin to pascal/currie/fermi.
adamksmith added 1 commit 2026-06-16 19:46:41 +00:00
Boxes without IPMI (Dell workstations like dev-linux) expose CPU/ambient/
SODIMM/board temps via the BIOS SMM interface (dell_smm hwmon). Add
read_hwmon_env(): labelled temps from chips in HOST_HWMON_CHIPS (default
'dell_smm') as env sensors, named '<chip> <label>' with index suffix for
repeated labels. Merged into the host-poller env sweep. No-op on hosts without
the configured chips.
adamksmith added 1 commit 2026-06-16 19:56:04 +00:00
Prereqs verified on the cluster: VSO 1.3.0 (transformation supported,
openbao-kubernetes VaultAuth healthy), MQTT egress to 10.5.30.3:1883 reachable
from a pod, target nodes untainted, registry requires auth. Add
registry-cred-vaultstaticsecret.yaml (secret/registry/nexus -> dockerconfigjson,
mirroring existing cluster pattern) and wire imagePullSecrets into the DaemonSet.
README updated with verified prereqs + apply order.
adamksmith added 1 commit 2026-06-16 20:09:22 +00:00
excludeRaw alone left all home_assistant source keys (api_key, vantage_*, etc.)
in the synced secret, so envFrom injected them into the web container. Add
transformation.excludes ['.*'] so only the templated MQTT_USERNAME/MQTT_PASSWORD
remain.
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/dedicated-poller:feat/dedicated-poller
git checkout feat/dedicated-poller
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: adamksmith/jbod-monitor#4