Dedicated hardware poller + read-only consumers #4
Reference in New Issue
Block a user
Delete Branch "feat/dedicated-poller"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
Backplane/expander stress. The prior model re-ran a full hardware sweep on every container start, polled
sg_ses 0x02+0x07every 60s (the0x07page errored on the IOM6/Xyratex expanders —couldn't read config page, res=35), and fanned outsmartctlconcurrently. Rapid deploy cycling multiplied those storms and crashed backplanes on a host whose SAS expanders have a documented history of dropping out and suspending pools.What
A single poller process is now the sole owner of all SAS/SMART/SES hardware I/O — serialized behind one gate, paced, writing to Redis. The API/web and MQTT publisher become pure Redis readers: zero subprocesses, so they can restart freely without ever touching the bus.
Producer
poller.py— paced sweep (inventory → SMART per-drive with a gap → SES0x02→ ZFS → host/MegaRAID), startup jitter, single-instance Redis lock, responsive LED-queue worker.services/hwgate.py— global serialization semaphore (POLL_CONCURRENCY, default 1); everysmartctl/sg_ses/zpool/ledctlruns behind it.services/store.py— Redis as the only producer↔consumer interface (+ LED command queue + single-instance lock).services/health.py— shared drive-health classifier (also fixes the old overview double-count).Consumers (read-only)
routers/{overview,enclosures,drives,leds}+services/temps.pyread exclusively from the store. LED POSTs enqueue to Redis; the poller executes them.main.pydrops the in-process poll loop; keeps API + MQTT./api/healthnow reportspoller_fresh; read endpoints exposeX-Poll-Age.Hardware-stress fixes
sg_ses 0x07removed from the hot path.POLL_DRIVE_GAP).pollercontainer is privileged/pid:host;appis unprivileged with no/dev.Deploy
docker-compose.yml: 3 services (poller+app+redis) with pacing knobs (POLL_SWEEP_INTERVAL,POLL_DRIVE_GAP,POLL_CONCURRENCY,POLL_STARTUP_JITTER,POLL_DATA_TTL).Testing
/tempshotspot = max(SES, drive),/overviewhealth/zfs/counts,/enclosures,/drivesverified.docker compose configvalid.Decouple the privileged hardware poller from web iterations so a web redeploy can never recreate/restart the poller (the backplane-touching container). Two images from one Dockerfile via build targets: - jbod-poller: privileged producer, ships smartmontools/sg3-utils/ledmon, Redis-only deps (~197MB) - jbod-web: unprivileged read-only consumer (REST/UI/MQTT), no hardware tools, no /dev (~147MB) Two compose projects sharing Redis over host networking: - compose.infra.yml (jbod-infra): poller + redis — stable substrate - compose.web.yml (jbod-web): app — 'up -d --build' touches only app build.sh builds/pushes both images; split requirements-{poller,web}.txt; dropped the combined docker-compose.yml; .dockerignore excludes tmp/ (keeps the venv and the mqtt.env creds out of the build context).Added a second commit: independent poller and web deploy stacks so a web redeploy can never recreate/restart the privileged poller.
jbod-poller(privileged, ships smartmontools/sg3-utils/ledmon, Redis-only deps, ~197MB) andjbod-web(unprivileged read-only consumer, no hardware tools, ~147MB).compose.infra.yml(jbod-infra: poller + redis — stable substrate) andcompose.web.yml(jbod-web: app —up -d --buildtouches onlyapp).build.shbuilds/pushes both; splitrequirements-{poller,web}.txt; dropped the combineddocker-compose.yml;.dockerignorenow excludestmp/.Verified: both targets build; imports succeed under each image's slim dep set; web image has no
smartctl/sg_ses/ledctl, poller has all three.Deploy workflow:
Third commit: hardening from an adversarial Codex (gpt-5.5) review focused on whether anything can still storm the SAS bus. Real gaps it caught (vetted against the code):
Storm prevention
POLL_SWEEP_INTERVALago (persisted in Redis). Previously jitter (≤10s) then an immediate sweep meant deploy-cycling could still fire back-to-back full sweeps — the exact failure mode that crashed the backplanes.POLL_DRIVE_GAPis now enforced inside the hardware gate after every op (SMART, SES, host, MegaRAID, ledctl), not just the enclosure loop. Host/MegaRAID/LED were previously serialized but gap-less.Correctness
set_metausedttl=0→ Redis rejectsEX 0→ meta never persisted →poller_freshalways false and the restart guard had nothing to read.cache_setnow omitsEXwhenttl<=0.--workers 1; two workers shared aclient_idand flapped the broker.Deploy independence
build.sh web|poller|allso a web iteration never rebuilds/repushes the poller image.Vetted-but-declined:
healthy=Truewith warnings is by-design (errors only); Pydantic v2 handles mutable defaults; concurrency now warns instead of hard-clamping.Verified: compiles/imports; gate serializes + paces (no overlap); consumer read-path, heartbeat persistence, and Lua lock semantics all pass; both images build + import.
Fourth commit: fixes from a Codex second pass, which caught a critical regression the first round's restart-storm guard introduced.
Critical (was a no-go): the restart-storm guard could let the poller lock expire before the refresher started. Flow was:
acquire_lock(30s TTL) → jitter (≤10s) → restart-defer (up toPOLL_SWEEP_INTERVAL≈ 300s) → then startlock_refresher. During a long defer the lock expired, so a second poller (deploy overlap) could acquire it and both sweep together — the exact storm this prevents. Fixed:lock_refresher()now starts immediately after acquiring the lock, before any pre-sweep sleep.Also:
POLL_CONCURRENCYnow clamps to 1 unlessPOLL_ALLOW_UNSAFE_CONCURRENCY=1(fragile expanders); warns either way.build.shkept backward-compatible — a non-target first arg is treated as the commit message, so./build.sh "msg"still works.Codex confirmed correct on second pass: the Lua lock scripts,
cache_set(ttl<=0)heartbeat fix, gate serialize+pace (incl. gathered SMART / MegaRAID), and_critical_task_doneshutdown handling. Verified locally: compiles/imports; refresher now precedes jitter+defer; clamp works (1 without override, N with); build.sh parsing covers all cases.Codex's go/no-go: no-go until the lock-timing fix — which this commit makes. Remaining deploy conditions it noted:
POLL_CONCURRENCY=1, pinned poller image SHA, web-only build/deploy path. (Still holding actual deploy until the backplanes are confirmed healthy.)Enclosure entities set via_device to a parent hub id that nothing defined, so Home Assistant showed an 'Unnamed Device'. Publish a hub connectivity binary_sensor ('Status', online/offline from the LWT topic) that defines the parent device with a proper name, so the per-enclosure devices nest cleanly under 'JBOD Monitor (<node>)'.Fifth commit: host-poller — a third producer container for server-chassis temps, kept separate from the SAS-shelf poller (different subsystem, shouldn't share its hardware gate).
services/host_sensors.py: in-band IPMI (ipmitool sdr type temperature) for Inlet/Exhaust + CPU coretemp from hwmon.host_poller.py: own single-instance lock + heartbeat (host_pollkeys), restart-storm guard, paced; writesjbod:host_sensors+jbod:host_drives. On-metal drive collection moved off the SAS poller (readsjbod:zfs_mapfor annotation).services/store.py: lock/meta generalized with akeyparam; host keys added.services/mqtt_publisher.py: publishes env (Inlet/Exhaust), CPU, and on-metal drive temps as entities on the hub device.Dockerfile:host-pollertarget (ipmitool + smartmontools);compose.infra.ymladds the service;build.shgainshost-poller/all.Deployed to nas01 (
1839e9d, all four containers healthy). HA hub "JBOD Monitor (usco-dc-nas01)" now shows Inlet, Exhaust, CPU 0, and Drive nvme0n1/megaraid:0/megaraid:1 — plus the 5 nested enclosure devices. (R620 is single-CPU, so one CPU temp is correct; coretemp vs the iDRAC CPU sensor differ by ~10°C — currently publishing coretemp.)View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.