Harden poller from Codex review: close residual storm/robustness gaps

Adversarial review (gpt-5.5) of the new architecture surfaced real gaps:

Hardware-safety / storm prevention:
- Restart-storm guard: on startup, defer the first sweep if a sweep ran
  < POLL_SWEEP_INTERVAL ago (persisted in Redis), so deploy-cycling can no
  longer trigger back-to-back full hardware sweeps (poller.py).
- Centralize pacing in the hardware gate: POLL_DRIVE_GAP is now held after
  EVERY hardware op (SMART, SES, host, MegaRAID, ledctl), not just the
  enclosure-drive loop (hwgate.py); removed the now-redundant per-loop sleeps.
- Single-instance lock made atomic (Lua compare-and-set / compare-and-expire)
  and the refresher is now fatal on any error + has a done-callback, so a
  dropped lock can never leave two pollers sweeping concurrently (store.py,
  poller.py).
- Warn loudly when POLL_CONCURRENCY > 1.

Correctness:
- Heartbeat now actually writes: cache_set omits EX when ttl<=0 (Redis
  rejects EX 0), so poller meta/last_sweep_ts persists — this also enables
  the restart-storm guard and the poller_fresh health flag (cache.py).
- Web image runs a single uvicorn worker so the MQTT publisher is a true
  singleton (two workers shared a client_id and flapped the broker) (Dockerfile).
- LED enqueue catches Redis errors -> API returns 503, not 500 (store.py).

Deploy independence:
- build.sh takes a target (web|poller|all) so web iterations never rebuild
  or repush the poller image.

Nits: drop dead SMART_CACHE_TTL constant.

poller.py

@@ -35,9 +35,10 @@ logging.basicConfig(
 logger = logging.getLogger("poller")
 
 SWEEP_INTERVAL = int(os.environ.get("POLL_SWEEP_INTERVAL", "300"))
-DRIVE_GAP = float(os.environ.get("POLL_DRIVE_GAP", "0.5"))
 STARTUP_JITTER = float(os.environ.get("POLL_STARTUP_JITTER", "10"))
 LOCK_TTL = 30
+# Pacing between hardware ops (POLL_DRIVE_GAP) is enforced centrally by the
+# hardware gate, so every op — SMART, SES, host, MegaRAID, ledctl — is spaced.
 
 TOKEN = f"{socket.gethostname()}-{os.getpid()}"
 
@@ -66,16 +67,15 @@ async def sweep() -> None:
     except Exception as e:
         errors.append(f"zfs:{e}")
 
-    # 3. Per-drive SMART, one at a time with a gap between each.
+    # 3. Per-drive SMART (gate serializes + paces each call).
     for dev in drive_devices:
         try:
             data = await _run_smartctl(dev)
             await store.set_smart(dev, data)
         except Exception as e:
             errors.append(f"smart:{dev}:{e}")
-        await asyncio.sleep(DRIVE_GAP)
 
-    # 4. Per-enclosure SES status (gated), paced.
+    # 4. Per-enclosure SES status (gate serializes + paces each call).
     for enc in enclosures:
         if not enc.get("sg_device"):
             continue
@@ -85,7 +85,6 @@ async def sweep() -> None:
                 await store.set_ses(enc["id"], ses)
         except Exception as e:
             errors.append(f"ses:{enc['id']}:{e}")
-        await asyncio.sleep(DRIVE_GAP)
 
     # 5. Host (non-enclosure) drives + MegaRAID (gated, reuses pool_map).
     try:
@@ -132,11 +131,27 @@ async def led_consumer() -> None:
 async def lock_refresher() -> None:
     while True:
         await asyncio.sleep(LOCK_TTL / 3)
-        if not await store.refresh_lock(TOKEN, LOCK_TTL):
+        # Any failure to confirm we still hold the lock is fatal — keeping the
+        # lock alive is what prevents a second poller from sweeping concurrently.
+        try:
+            ok = await store.refresh_lock(TOKEN, LOCK_TTL)
+        except Exception as e:
+            logger.error("lock refresh error (%s) — exiting to avoid double-polling", e)
+            os._exit(1)
+        if not ok:
             logger.error("lost poller lock — exiting to avoid double-polling")
             os._exit(1)
 
 
+def _critical_task_done(task: asyncio.Task) -> None:
+    """Fail-safe: if a critical task ends for any reason other than a clean
+    cancel (shutdown), exit so we never run unguarded."""
+    if task.cancelled():
+        return
+    logger.error("critical task ended unexpectedly — exiting")
+    os._exit(1)
+
+
 async def main() -> None:
     await init_cache()
     while not redis_available():
@@ -162,7 +177,19 @@ async def main() -> None:
     logger.info("startup jitter %.1fs", jitter)
     await asyncio.sleep(jitter)
 
+    # Restart-storm guard: if a sweep ran recently (persisted in Redis across
+    # restarts), wait out the remainder of the interval instead of immediately
+    # re-sweeping. This is the key protection against deploy-cycling storms.
+    meta = await store.get_meta()
+    if meta and meta.get("last_sweep_ts"):
+        since = store.now() - meta["last_sweep_ts"]
+        if 0 <= since < SWEEP_INTERVAL:
+            wait = SWEEP_INTERVAL - since
+            logger.info("last sweep %.0fs ago — deferring first sweep %.0fs", since, wait)
+            await asyncio.sleep(wait)
+
     refresher = asyncio.create_task(lock_refresher())
+    refresher.add_done_callback(_critical_task_done)
     leds = asyncio.create_task(led_consumer())
     try:
         while True: