Split into dedicated hardware poller + read-only consumers

Rearchitect so a single poller process is the sole owner of all SAS/SMART/
SES hardware I/O, serialized behind one gate and paced, writing results to
Redis. The API/web and MQTT publisher become pure Redis readers — they no
longer issue any subprocess and can restart freely without touching the bus.

This addresses backplane/expander stress from concurrent + restart-triggered
SMART/SES storms (the prior model re-ran a hardware sweep on every container
start, and polled sg_ses 0x02+0x07 every 60s; 0x07 errored on the IOM6/
Xyratex expanders).

- poller.py: paced sweep (inventory, SMART per-drive w/ gap, SES 0x02, ZFS,
  host/MegaRAID), startup jitter, single-instance Redis lock, LED-queue worker
- services/hwgate.py: global serialization semaphore (POLL_CONCURRENCY=1)
- services/store.py: Redis as the only producer<->consumer interface + LED queue
- services/health.py: shared drive-health classifier (fixes overview double-count)
- gate smartctl/sg_ses/zpool/ledctl; drop sg_ses 0x07 from the hot path
- routers + temps read-only from the store; main.py drops the in-process poller
- compose: 3 services (privileged poller + unprivileged app + redis) w/ pacing knobs
This commit is contained in:
2026-06-16 15:12:34 +00:00
parent 7a50d1261f
commit 0f829e0380
19 changed files with 717 additions and 464 deletions

View File

@@ -1,30 +1,37 @@
import re
from fastapi import APIRouter, HTTPException, Response
from models.schemas import DriveDetail
from services.smart import get_smart_data
from services.zfs import get_zfs_pool_map
from services import store
router = APIRouter(prefix="/api/drives", tags=["drives"])
@router.get("/{device}", response_model=DriveDetail)
async def get_drive_detail(device: str, response: Response):
"""Get SMART detail for a specific block device."""
try:
data, cache_hit = await get_smart_data(device)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
"""Get SMART detail for a specific block device (from the store)."""
if not re.match(r"^[a-zA-Z0-9\-]+$", device):
raise HTTPException(status_code=400, detail=f"Invalid device name: {device}")
data = await store.get_smart(device)
if data is None:
raise HTTPException(
status_code=404,
detail=f"No SMART data for '{device}' yet (poller may not have swept it)",
)
if "error" in data:
raise HTTPException(status_code=502, detail=data["error"])
pool_map = await get_zfs_pool_map()
pool_map = await store.get_zfs_map()
zfs_info = pool_map.get(device)
if zfs_info:
data["zfs_pool"] = zfs_info["pool"]
data["zfs_vdev"] = zfs_info["vdev"]
data["zfs_state"] = zfs_info.get("state")
response.headers["X-Cache"] = "HIT" if cache_hit else "MISS"
meta = await store.get_meta()
if meta and meta.get("last_sweep_ts"):
response.headers["X-Poll-Age"] = str(int(store.now() - meta["last_sweep_ts"]))
return DriveDetail(**data)