excludeRaw alone left all home_assistant source keys (api_key, vantage_*, etc.)
in the synced secret, so envFrom injected them into the web container. Add
transformation.excludes ['.*'] so only the templated MQTT_USERNAME/MQTT_PASSWORD
remain.
Prereqs verified on the cluster: VSO 1.3.0 (transformation supported,
openbao-kubernetes VaultAuth healthy), MQTT egress to 10.5.30.3:1883 reachable
from a pod, target nodes untainted, registry requires auth. Add
registry-cred-vaultstaticsecret.yaml (secret/registry/nexus -> dockerconfigjson,
mirroring existing cluster pattern) and wire imagePullSecrets into the DaemonSet.
README updated with verified prereqs + apply order.
Boxes without IPMI (Dell workstations like dev-linux) expose CPU/ambient/
SODIMM/board temps via the BIOS SMM interface (dell_smm hwmon). Add
read_hwmon_env(): labelled temps from chips in HOST_HWMON_CHIPS (default
'dell_smm') as env sensors, named '<chip> <label>' with index suffix for
repeated labels. Merged into the host-poller env sweep. No-op on hosts without
the configured chips.
Design artifact (NOT applied). Per-node pod = host-poller + redis + web sharing
pod localhost; only host-poller privileged with hostPath /dev+/sys. No SAS
poller (no enclosures). MQTT_NODE_ID from spec.nodeName; MQTT creds via VSO
VaultStaticSecret reading secret/home_assistant (mqtt_user/pass -> MQTT_USERNAME/
PASSWORD). nodeSelector jbod-monitor=enabled to pin to pascal/currie/fermi.
- skip IPMI '*Margin' sensors (thermal margin / distance-to-throttle reads in
degrees C but is not an absolute temperature, e.g. robin's P1 Therm Margin)
- coretemp: when a CPU has no 'Package id' sensor (older Xeons like robin's
X3430), fall back to the hottest Core as the CPU temp
Two concurrent loops in one process sharing the lock + hardware gate (IPMI and
SMART never run concurrently) but on separate schedules: env every
HOST_ENV_INTERVAL (60s, responsive inlet/CPU) and drives every
HOST_DRIVE_INTERVAL (300s, gentle SMART). Each loop has its own heartbeat
(env_meta/drive_meta) + restart-storm guard.
New third producer container (host-poller) owning the server chassis
subsystem, separate from the SAS-shelf poller:
- services/host_sensors.py: in-band IPMI (ipmitool sdr) for Inlet/Exhaust/etc.
+ CPU package temps from coretemp hwmon
- host_poller.py: own single-instance lock + heartbeat (host_poll keys),
restart-storm guard, paced; writes jbod:host_sensors + jbod:host_drives
- move host/on-metal drive collection off the SAS poller to host-poller
(reads jbod:zfs_map for annotation)
- store: generalized lock/meta with a key param; host_sensors + host-poll keys
- mqtt_publisher: publish env/CPU/host-drive temps as entities on the hub
device (JBOD Monitor)
- Dockerfile host-poller target (ipmitool+smartmontools); compose.infra adds
host-poller; build.sh gains host-poller/all
Enclosure entities set via_device to a parent hub id that nothing defined,
so Home Assistant showed an 'Unnamed Device'. Publish a hub connectivity
binary_sensor ('Status', online/offline from the LWT topic) that defines the
parent device with a proper name, so the per-enclosure devices nest cleanly
under 'JBOD Monitor (<node>)'.
- poller.py: start lock_refresher() IMMEDIATELY after acquiring the lock,
before the startup jitter and restart-storm deferral. Those sleeps can
together exceed LOCK_TTL (restart defer is up to POLL_SWEEP_INTERVAL), so
with the refresher started afterward the lock could expire mid-deferral and
a second poller could begin sweeping concurrently — the exact hardware-storm
risk this design prevents.
- hwgate.py: clamp POLL_CONCURRENCY to 1 unless POLL_ALLOW_UNSAFE_CONCURRENCY
is set (fragile SAS hardware; >1 can drop expanders). Warn either way.
- build.sh: keep backward-compatible — a non-target first arg is treated as
the commit message (./build.sh "msg" still works), known targets select
web|poller|all.
Adversarial review (gpt-5.5) of the new architecture surfaced real gaps:
Hardware-safety / storm prevention:
- Restart-storm guard: on startup, defer the first sweep if a sweep ran
< POLL_SWEEP_INTERVAL ago (persisted in Redis), so deploy-cycling can no
longer trigger back-to-back full hardware sweeps (poller.py).
- Centralize pacing in the hardware gate: POLL_DRIVE_GAP is now held after
EVERY hardware op (SMART, SES, host, MegaRAID, ledctl), not just the
enclosure-drive loop (hwgate.py); removed the now-redundant per-loop sleeps.
- Single-instance lock made atomic (Lua compare-and-set / compare-and-expire)
and the refresher is now fatal on any error + has a done-callback, so a
dropped lock can never leave two pollers sweeping concurrently (store.py,
poller.py).
- Warn loudly when POLL_CONCURRENCY > 1.
Correctness:
- Heartbeat now actually writes: cache_set omits EX when ttl<=0 (Redis
rejects EX 0), so poller meta/last_sweep_ts persists — this also enables
the restart-storm guard and the poller_fresh health flag (cache.py).
- Web image runs a single uvicorn worker so the MQTT publisher is a true
singleton (two workers shared a client_id and flapped the broker) (Dockerfile).
- LED enqueue catches Redis errors -> API returns 503, not 500 (store.py).
Deploy independence:
- build.sh takes a target (web|poller|all) so web iterations never rebuild
or repush the poller image.
Nits: drop dead SMART_CACHE_TTL constant.
Decouple the privileged hardware poller from web iterations so a web
redeploy can never recreate/restart the poller (the backplane-touching
container). Two images from one Dockerfile via build targets:
- jbod-poller: privileged producer, ships smartmontools/sg3-utils/ledmon,
Redis-only deps (~197MB)
- jbod-web: unprivileged read-only consumer (REST/UI/MQTT), no hardware
tools, no /dev (~147MB)
Two compose projects sharing Redis over host networking:
- compose.infra.yml (jbod-infra): poller + redis — stable substrate
- compose.web.yml (jbod-web): app — 'up -d --build' touches only app
build.sh builds/pushes both images; split requirements-{poller,web}.txt;
dropped the combined docker-compose.yml; .dockerignore excludes tmp/ (keeps
the venv and the mqtt.env creds out of the build context).
Rearchitect so a single poller process is the sole owner of all SAS/SMART/
SES hardware I/O, serialized behind one gate and paced, writing results to
Redis. The API/web and MQTT publisher become pure Redis readers — they no
longer issue any subprocess and can restart freely without touching the bus.
This addresses backplane/expander stress from concurrent + restart-triggered
SMART/SES storms (the prior model re-ran a hardware sweep on every container
start, and polled sg_ses 0x02+0x07 every 60s; 0x07 errored on the IOM6/
Xyratex expanders).
- poller.py: paced sweep (inventory, SMART per-drive w/ gap, SES 0x02, ZFS,
host/MegaRAID), startup jitter, single-instance Redis lock, LED-queue worker
- services/hwgate.py: global serialization semaphore (POLL_CONCURRENCY=1)
- services/store.py: Redis as the only producer<->consumer interface + LED queue
- services/health.py: shared drive-health classifier (fixes overview double-count)
- gate smartctl/sg_ses/zpool/ledctl; drop sg_ses 0x07 from the hot path
- routers + temps read-only from the store; main.py drops the in-process poller
- compose: 3 services (privileged poller + unprivileged app + redis) w/ pacing knobs
