Split into dedicated hardware poller + read-only consumers

Rearchitect so a single poller process is the sole owner of all SAS/SMART/
SES hardware I/O, serialized behind one gate and paced, writing results to
Redis. The API/web and MQTT publisher become pure Redis readers — they no
longer issue any subprocess and can restart freely without touching the bus.

This addresses backplane/expander stress from concurrent + restart-triggered
SMART/SES storms (the prior model re-ran a hardware sweep on every container
start, and polled sg_ses 0x02+0x07 every 60s; 0x07 errored on the IOM6/
Xyratex expanders).

- poller.py: paced sweep (inventory, SMART per-drive w/ gap, SES 0x02, ZFS,
  host/MegaRAID), startup jitter, single-instance Redis lock, LED-queue worker
- services/hwgate.py: global serialization semaphore (POLL_CONCURRENCY=1)
- services/store.py: Redis as the only producer<->consumer interface + LED queue
- services/health.py: shared drive-health classifier (fixes overview double-count)
- gate smartctl/sg_ses/zpool/ledctl; drop sg_ses 0x07 from the hot path
- routers + temps read-only from the store; main.py drops the in-process poller
- compose: 3 services (privileged poller + unprivileged app + redis) w/ pacing knobs
This commit is contained in:
2026-06-16 15:12:34 +00:00
parent 7a50d1261f
commit 0f829e0380
19 changed files with 717 additions and 464 deletions

View File

@@ -1,9 +1,12 @@
import asyncio
import re
from services.hwgate import gate
async def set_led(device: str, state: str) -> dict:
"""Set the locate LED on a drive via ledctl.
async def run_led(device: str, state: str) -> dict:
"""Set the locate LED on a drive via ledctl. Hardware-gated; runs in the
poller, fed by the Redis LED queue. The API enqueues, never calls this.
Args:
device: Block device name (e.g. "sda"). Must be alphanumeric.
@@ -14,13 +17,14 @@ async def set_led(device: str, state: str) -> dict:
if state not in ("locate", "off"):
raise ValueError(f"Invalid state: {state}")
proc = await asyncio.create_subprocess_exec(
"ledctl",
f"{state}=/dev/{device}",
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, stderr = await proc.communicate()
async with gate():
proc = await asyncio.create_subprocess_exec(
"ledctl",
f"{state}=/dev/{device}",
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, stderr = await proc.communicate()
if proc.returncode != 0:
err = stderr.decode().strip() or stdout.decode().strip()