Prereqs verified on the cluster: VSO 1.3.0 (transformation supported,
openbao-kubernetes VaultAuth healthy), MQTT egress to 10.5.30.3:1883 reachable
from a pod, target nodes untainted, registry requires auth. Add
registry-cred-vaultstaticsecret.yaml (secret/registry/nexus -> dockerconfigjson,
mirroring existing cluster pattern) and wire imagePullSecrets into the DaemonSet.
README updated with verified prereqs + apply order.
Boxes without IPMI (Dell workstations like dev-linux) expose CPU/ambient/
SODIMM/board temps via the BIOS SMM interface (dell_smm hwmon). Add
read_hwmon_env(): labelled temps from chips in HOST_HWMON_CHIPS (default
'dell_smm') as env sensors, named '<chip> <label>' with index suffix for
repeated labels. Merged into the host-poller env sweep. No-op on hosts without
the configured chips.
Design artifact (NOT applied). Per-node pod = host-poller + redis + web sharing
pod localhost; only host-poller privileged with hostPath /dev+/sys. No SAS
poller (no enclosures). MQTT_NODE_ID from spec.nodeName; MQTT creds via VSO
VaultStaticSecret reading secret/home_assistant (mqtt_user/pass -> MQTT_USERNAME/
PASSWORD). nodeSelector jbod-monitor=enabled to pin to pascal/currie/fermi.
- skip IPMI '*Margin' sensors (thermal margin / distance-to-throttle reads in
degrees C but is not an absolute temperature, e.g. robin's P1 Therm Margin)
- coretemp: when a CPU has no 'Package id' sensor (older Xeons like robin's
X3430), fall back to the hottest Core as the CPU temp
Two concurrent loops in one process sharing the lock + hardware gate (IPMI and
SMART never run concurrently) but on separate schedules: env every
HOST_ENV_INTERVAL (60s, responsive inlet/CPU) and drives every
HOST_DRIVE_INTERVAL (300s, gentle SMART). Each loop has its own heartbeat
(env_meta/drive_meta) + restart-storm guard.
New third producer container (host-poller) owning the server chassis
subsystem, separate from the SAS-shelf poller:
- services/host_sensors.py: in-band IPMI (ipmitool sdr) for Inlet/Exhaust/etc.
+ CPU package temps from coretemp hwmon
- host_poller.py: own single-instance lock + heartbeat (host_poll keys),
restart-storm guard, paced; writes jbod:host_sensors + jbod:host_drives
- move host/on-metal drive collection off the SAS poller to host-poller
(reads jbod:zfs_map for annotation)
- store: generalized lock/meta with a key param; host_sensors + host-poll keys
- mqtt_publisher: publish env/CPU/host-drive temps as entities on the hub
device (JBOD Monitor)
- Dockerfile host-poller target (ipmitool+smartmontools); compose.infra adds
host-poller; build.sh gains host-poller/all
Enclosure entities set via_device to a parent hub id that nothing defined,
so Home Assistant showed an 'Unnamed Device'. Publish a hub connectivity
binary_sensor ('Status', online/offline from the LWT topic) that defines the
parent device with a proper name, so the per-enclosure devices nest cleanly
under 'JBOD Monitor (<node>)'.
- poller.py: start lock_refresher() IMMEDIATELY after acquiring the lock,
before the startup jitter and restart-storm deferral. Those sleeps can
together exceed LOCK_TTL (restart defer is up to POLL_SWEEP_INTERVAL), so
with the refresher started afterward the lock could expire mid-deferral and
a second poller could begin sweeping concurrently — the exact hardware-storm
risk this design prevents.
- hwgate.py: clamp POLL_CONCURRENCY to 1 unless POLL_ALLOW_UNSAFE_CONCURRENCY
is set (fragile SAS hardware; >1 can drop expanders). Warn either way.
- build.sh: keep backward-compatible — a non-target first arg is treated as
the commit message (./build.sh "msg" still works), known targets select
web|poller|all.
Adversarial review (gpt-5.5) of the new architecture surfaced real gaps:
Hardware-safety / storm prevention:
- Restart-storm guard: on startup, defer the first sweep if a sweep ran
< POLL_SWEEP_INTERVAL ago (persisted in Redis), so deploy-cycling can no
longer trigger back-to-back full hardware sweeps (poller.py).
- Centralize pacing in the hardware gate: POLL_DRIVE_GAP is now held after
EVERY hardware op (SMART, SES, host, MegaRAID, ledctl), not just the
enclosure-drive loop (hwgate.py); removed the now-redundant per-loop sleeps.
- Single-instance lock made atomic (Lua compare-and-set / compare-and-expire)
and the refresher is now fatal on any error + has a done-callback, so a
dropped lock can never leave two pollers sweeping concurrently (store.py,
poller.py).
- Warn loudly when POLL_CONCURRENCY > 1.
Correctness:
- Heartbeat now actually writes: cache_set omits EX when ttl<=0 (Redis
rejects EX 0), so poller meta/last_sweep_ts persists — this also enables
the restart-storm guard and the poller_fresh health flag (cache.py).
- Web image runs a single uvicorn worker so the MQTT publisher is a true
singleton (two workers shared a client_id and flapped the broker) (Dockerfile).
- LED enqueue catches Redis errors -> API returns 503, not 500 (store.py).
Deploy independence:
- build.sh takes a target (web|poller|all) so web iterations never rebuild
or repush the poller image.
Nits: drop dead SMART_CACHE_TTL constant.
Decouple the privileged hardware poller from web iterations so a web
redeploy can never recreate/restart the poller (the backplane-touching
container). Two images from one Dockerfile via build targets:
- jbod-poller: privileged producer, ships smartmontools/sg3-utils/ledmon,
Redis-only deps (~197MB)
- jbod-web: unprivileged read-only consumer (REST/UI/MQTT), no hardware
tools, no /dev (~147MB)
Two compose projects sharing Redis over host networking:
- compose.infra.yml (jbod-infra): poller + redis — stable substrate
- compose.web.yml (jbod-web): app — 'up -d --build' touches only app
build.sh builds/pushes both images; split requirements-{poller,web}.txt;
dropped the combined docker-compose.yml; .dockerignore excludes tmp/ (keeps
the venv and the mqtt.env creds out of the build context).
Rearchitect so a single poller process is the sole owner of all SAS/SMART/
SES hardware I/O, serialized behind one gate and paced, writing results to
Redis. The API/web and MQTT publisher become pure Redis readers — they no
longer issue any subprocess and can restart freely without touching the bus.
This addresses backplane/expander stress from concurrent + restart-triggered
SMART/SES storms (the prior model re-ran a hardware sweep on every container
start, and polled sg_ses 0x02+0x07 every 60s; 0x07 errored on the IOM6/
Xyratex expanders).
- poller.py: paced sweep (inventory, SMART per-drive w/ gap, SES 0x02, ZFS,
host/MegaRAID), startup jitter, single-instance Redis lock, LED-queue worker
- services/hwgate.py: global serialization semaphore (POLL_CONCURRENCY=1)
- services/store.py: Redis as the only producer<->consumer interface + LED queue
- services/health.py: shared drive-health classifier (fixes overview double-count)
- gate smartctl/sg_ses/zpool/ledctl; drop sg_ses 0x07 from the hot path
- routers + temps read-only from the store; main.py drops the in-process poller
- compose: 3 services (privileged poller + unprivileged app + redis) w/ pacing knobs
Some enclosures (Xyratex/NetApp shelves) leave SES element descriptor
strings blank; sg_ses prints the literal '<empty>'. Skip it so temp
sensors fall back to a generic 'Temp N' label instead of '<empty>'.
- enclosure: fetch SES element descriptor page (0x07) and label temp/fan/
psu/voltage elements with human-readable names
- temps service + /api/temps: per-enclosure hotspot (max across SES sensors
and housed drive temps) plus named SES sensor list
- mqtt_publisher: paho-mqtt with HA discovery (device per enclosure, hotspot
+ per-sensor entities), LWT availability, opt-in via MQTT_HOST
- secrets: OpenBao KV v2 reader; MQTT creds sourced from secret/home_assistant
with env fallback
- compose/requirements/README updated
Xyratex HB-1235 enclosures have non-functional temperature sensor
slots that report Unrecoverable/Unknown at 0°C. The SES header
rolls these into UNRECOV=1, causing a false CRITICAL flag. Now we
skip dead sensors (0°C + non-OK status) and derive overall_status
from the actual parsed elements instead of the raw header counts.
Dual-IOM shelves (e.g. NetApp DS2246 with two IOM6 modules) create
two sysfs enclosure entries. The secondary path reports populated
slots via SES but the kernel only links block devices through the
primary path. Filter these out during discovery to avoid showing
ghost enclosures with no drive data.
Parse sg_ses --page=0x02 output to surface enclosure-level health data
including power supply status, fan RPMs, temperature sensors, and voltage
rails. Failed/critical components are reflected in the overview totals
and shown as status pills in the enclosure card header with an expandable
detail panel.
- Vite + React frontend with dark-themed dashboard, slot grid per
enclosure, and SMART detail overlay
- ZFS pool membership via zpool status -P
- Multi-stage Dockerfile (Node build + Python runtime)
- Updated docker-compose with network_mode host and healthcheck
