Harden poller from Codex review: close residual storm/robustness gaps
Adversarial review (gpt-5.5) of the new architecture surfaced real gaps: Hardware-safety / storm prevention: - Restart-storm guard: on startup, defer the first sweep if a sweep ran < POLL_SWEEP_INTERVAL ago (persisted in Redis), so deploy-cycling can no longer trigger back-to-back full hardware sweeps (poller.py). - Centralize pacing in the hardware gate: POLL_DRIVE_GAP is now held after EVERY hardware op (SMART, SES, host, MegaRAID, ledctl), not just the enclosure-drive loop (hwgate.py); removed the now-redundant per-loop sleeps. - Single-instance lock made atomic (Lua compare-and-set / compare-and-expire) and the refresher is now fatal on any error + has a done-callback, so a dropped lock can never leave two pollers sweeping concurrently (store.py, poller.py). - Warn loudly when POLL_CONCURRENCY > 1. Correctness: - Heartbeat now actually writes: cache_set omits EX when ttl<=0 (Redis rejects EX 0), so poller meta/last_sweep_ts persists — this also enables the restart-storm guard and the poller_fresh health flag (cache.py). - Web image runs a single uvicorn worker so the MQTT publisher is a true singleton (two workers shared a client_id and flapped the broker) (Dockerfile). - LED enqueue catches Redis errors -> API returns 503, not 500 (store.py). Deploy independence: - build.sh takes a target (web|poller|all) so web iterations never rebuild or repush the poller image. Nits: drop dead SMART_CACHE_TTL constant.
This commit is contained in:
26
build.sh
26
build.sh
@@ -1,24 +1,38 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Usage: ./build.sh [poller|web|all] [commit message]
|
||||
# web — build/push ONLY the web image (use this for web iterations so the
|
||||
# poller image is never rebuilt/repushed)
|
||||
# poller— build/push ONLY the poller image (touch deliberately)
|
||||
# all — both (default)
|
||||
#
|
||||
# Deploy pins images by SHA (compose.*.yml ship :latest as a convenience only).
|
||||
|
||||
REG="docker.adamksmith.xyz"
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
TARGET="${1:-all}"
|
||||
case "$TARGET" in
|
||||
poller) TARGETS="poller" ;;
|
||||
web) TARGETS="web" ;;
|
||||
all) TARGETS="poller web" ;;
|
||||
*) echo "usage: $0 [poller|web|all] [message]" >&2; exit 1 ;;
|
||||
esac
|
||||
MSG="${2:-Build and push images}"
|
||||
|
||||
# Stage and commit all changes
|
||||
git add -A
|
||||
if git diff --cached --quiet; then
|
||||
echo "No changes to commit, using HEAD"
|
||||
else
|
||||
git commit -m "${1:-Build and push images}"
|
||||
git commit -m "${MSG}"
|
||||
fi
|
||||
|
||||
SHA=$(git rev-parse --short HEAD)
|
||||
|
||||
# Two images from one Dockerfile (shared frontend build stage is cached):
|
||||
# jbod-poller — privileged hardware producer
|
||||
# jbod-web — unprivileged read-only consumer (REST/UI/MQTT)
|
||||
for target in poller web; do
|
||||
for target in ${TARGETS}; do
|
||||
img="${REG}/jbod-${target}"
|
||||
echo "Building ${img}:${SHA}"
|
||||
docker build --target "${target}" -t "${img}:${SHA}" -t "${img}:latest" .
|
||||
@@ -27,4 +41,4 @@ for target in poller web; do
|
||||
docker push "${img}:latest"
|
||||
done
|
||||
|
||||
echo "Done: ${REG}/jbod-{poller,web}:${SHA}"
|
||||
echo "Done (${TARGET}): ${REG}/jbod-{${TARGETS// /,}}:${SHA}"
|
||||
|
||||
Reference in New Issue
Block a user