Harden poller from Codex review: close residual storm/robustness gaps
Adversarial review (gpt-5.5) of the new architecture surfaced real gaps: Hardware-safety / storm prevention: - Restart-storm guard: on startup, defer the first sweep if a sweep ran < POLL_SWEEP_INTERVAL ago (persisted in Redis), so deploy-cycling can no longer trigger back-to-back full hardware sweeps (poller.py). - Centralize pacing in the hardware gate: POLL_DRIVE_GAP is now held after EVERY hardware op (SMART, SES, host, MegaRAID, ledctl), not just the enclosure-drive loop (hwgate.py); removed the now-redundant per-loop sleeps. - Single-instance lock made atomic (Lua compare-and-set / compare-and-expire) and the refresher is now fatal on any error + has a done-callback, so a dropped lock can never leave two pollers sweeping concurrently (store.py, poller.py). - Warn loudly when POLL_CONCURRENCY > 1. Correctness: - Heartbeat now actually writes: cache_set omits EX when ttl<=0 (Redis rejects EX 0), so poller meta/last_sweep_ts persists — this also enables the restart-storm guard and the poller_fresh health flag (cache.py). - Web image runs a single uvicorn worker so the MQTT publisher is a true singleton (two workers shared a client_id and flapped the broker) (Dockerfile). - LED enqueue catches Redis errors -> API returns 503, not 500 (store.py). Deploy independence: - build.sh takes a target (web|poller|all) so web iterations never rebuild or repush the poller image. Nits: drop dead SMART_CACHE_TTL constant.
This commit is contained in:
@@ -18,8 +18,6 @@ ATTR_PENDING = 197
|
||||
ATTR_UNCORRECTABLE = 198
|
||||
ATTR_WEAR_LEVELING = 177 # SSD wear leveling
|
||||
|
||||
SMART_CACHE_TTL = int(os.environ.get("SMART_CACHE_TTL", "120"))
|
||||
|
||||
|
||||
def smartctl_available() -> bool:
|
||||
return shutil.which("smartctl") is not None
|
||||
|
||||
Reference in New Issue
Block a user