Harden poller from Codex review: close residual storm/robustness gaps

Adversarial review (gpt-5.5) of the new architecture surfaced real gaps:

Hardware-safety / storm prevention:
- Restart-storm guard: on startup, defer the first sweep if a sweep ran
  < POLL_SWEEP_INTERVAL ago (persisted in Redis), so deploy-cycling can no
  longer trigger back-to-back full hardware sweeps (poller.py).
- Centralize pacing in the hardware gate: POLL_DRIVE_GAP is now held after
  EVERY hardware op (SMART, SES, host, MegaRAID, ledctl), not just the
  enclosure-drive loop (hwgate.py); removed the now-redundant per-loop sleeps.
- Single-instance lock made atomic (Lua compare-and-set / compare-and-expire)
  and the refresher is now fatal on any error + has a done-callback, so a
  dropped lock can never leave two pollers sweeping concurrently (store.py,
  poller.py).
- Warn loudly when POLL_CONCURRENCY > 1.

Correctness:
- Heartbeat now actually writes: cache_set omits EX when ttl<=0 (Redis
  rejects EX 0), so poller meta/last_sweep_ts persists — this also enables
  the restart-storm guard and the poller_fresh health flag (cache.py).
- Web image runs a single uvicorn worker so the MQTT publisher is a true
  singleton (two workers shared a client_id and flapped the broker) (Dockerfile).
- LED enqueue catches Redis errors -> API returns 503, not 500 (store.py).

Deploy independence:
- build.sh takes a target (web|poller|all) so web iterations never rebuild
  or repush the poller image.

Nits: drop dead SMART_CACHE_TTL constant.
This commit is contained in:
2026-06-16 15:58:43 +00:00
parent cccc2bc004
commit 7878e17e55
7 changed files with 123 additions and 44 deletions

View File

@@ -1,24 +1,38 @@
#!/usr/bin/env bash
set -euo pipefail
# Usage: ./build.sh [poller|web|all] [commit message]
# web — build/push ONLY the web image (use this for web iterations so the
# poller image is never rebuilt/repushed)
# poller— build/push ONLY the poller image (touch deliberately)
# all — both (default)
#
# Deploy pins images by SHA (compose.*.yml ship :latest as a convenience only).
REG="docker.adamksmith.xyz"
cd "$(dirname "$0")"
TARGET="${1:-all}"
case "$TARGET" in
poller) TARGETS="poller" ;;
web) TARGETS="web" ;;
all) TARGETS="poller web" ;;
*) echo "usage: $0 [poller|web|all] [message]" >&2; exit 1 ;;
esac
MSG="${2:-Build and push images}"
# Stage and commit all changes
git add -A
if git diff --cached --quiet; then
echo "No changes to commit, using HEAD"
else
git commit -m "${1:-Build and push images}"
git commit -m "${MSG}"
fi
SHA=$(git rev-parse --short HEAD)
# Two images from one Dockerfile (shared frontend build stage is cached):
# jbod-poller — privileged hardware producer
# jbod-web — unprivileged read-only consumer (REST/UI/MQTT)
for target in poller web; do
for target in ${TARGETS}; do
img="${REG}/jbod-${target}"
echo "Building ${img}:${SHA}"
docker build --target "${target}" -t "${img}:${SHA}" -t "${img}:latest" .
@@ -27,4 +41,4 @@ for target in poller web; do
docker push "${img}:latest"
done
echo "Done: ${REG}/jbod-{poller,web}:${SHA}"
echo "Done (${TARGET}): ${REG}/jbod-{${TARGETS// /,}}:${SHA}"