Adversarial review (gpt-5.5) of the new architecture surfaced real gaps:
Hardware-safety / storm prevention:
- Restart-storm guard: on startup, defer the first sweep if a sweep ran
< POLL_SWEEP_INTERVAL ago (persisted in Redis), so deploy-cycling can no
longer trigger back-to-back full hardware sweeps (poller.py).
- Centralize pacing in the hardware gate: POLL_DRIVE_GAP is now held after
EVERY hardware op (SMART, SES, host, MegaRAID, ledctl), not just the
enclosure-drive loop (hwgate.py); removed the now-redundant per-loop sleeps.
- Single-instance lock made atomic (Lua compare-and-set / compare-and-expire)
and the refresher is now fatal on any error + has a done-callback, so a
dropped lock can never leave two pollers sweeping concurrently (store.py,
poller.py).
- Warn loudly when POLL_CONCURRENCY > 1.
Correctness:
- Heartbeat now actually writes: cache_set omits EX when ttl<=0 (Redis
rejects EX 0), so poller meta/last_sweep_ts persists — this also enables
the restart-storm guard and the poller_fresh health flag (cache.py).
- Web image runs a single uvicorn worker so the MQTT publisher is a true
singleton (two workers shared a client_id and flapped the broker) (Dockerfile).
- LED enqueue catches Redis errors -> API returns 503, not 500 (store.py).
Deploy independence:
- build.sh takes a target (web|poller|all) so web iterations never rebuild
or repush the poller image.
Nits: drop dead SMART_CACHE_TTL constant.
adam
7878e17e55