Split into dedicated hardware poller + read-only consumers
Rearchitect so a single poller process is the sole owner of all SAS/SMART/ SES hardware I/O, serialized behind one gate and paced, writing results to Redis. The API/web and MQTT publisher become pure Redis readers — they no longer issue any subprocess and can restart freely without touching the bus. This addresses backplane/expander stress from concurrent + restart-triggered SMART/SES storms (the prior model re-ran a hardware sweep on every container start, and polled sg_ses 0x02+0x07 every 60s; 0x07 errored on the IOM6/ Xyratex expanders). - poller.py: paced sweep (inventory, SMART per-drive w/ gap, SES 0x02, ZFS, host/MegaRAID), startup jitter, single-instance Redis lock, LED-queue worker - services/hwgate.py: global serialization semaphore (POLL_CONCURRENCY=1) - services/store.py: Redis as the only producer<->consumer interface + LED queue - services/health.py: shared drive-health classifier (fixes overview double-count) - gate smartctl/sg_ses/zpool/ledctl; drop sg_ses 0x07 from the hot path - routers + temps read-only from the store; main.py drops the in-process poller - compose: 3 services (privileged poller + unprivileged app + redis) w/ pacing knobs
This commit is contained in:
27
services/hwgate.py
Normal file
27
services/hwgate.py
Normal file
@@ -0,0 +1,27 @@
|
||||
"""Global hardware-access gate.
|
||||
|
||||
Every command that touches the SAS bus — smartctl, sg_ses, zpool, ledctl —
|
||||
must run inside this gate. It serializes hardware I/O so the poller can
|
||||
never fire a thundering herd of subprocesses at fragile SAS expanders
|
||||
(which on some shelves drop out and suspend pools under concurrent load).
|
||||
|
||||
Concurrency defaults to 1 (fully serialized). It can be raised via
|
||||
POLL_CONCURRENCY for healthier backplanes, but 1 is the safe default.
|
||||
|
||||
The gate is a per-process asyncio.Semaphore — only the poller process
|
||||
calls hardware, so a per-process gate is sufficient. The semaphore is
|
||||
created lazily on first use so it binds to the running event loop.
|
||||
"""
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
_sem: asyncio.Semaphore | None = None
|
||||
|
||||
|
||||
def gate() -> asyncio.Semaphore:
|
||||
"""Return the shared hardware semaphore (use as `async with gate():`)."""
|
||||
global _sem
|
||||
if _sem is None:
|
||||
n = max(1, int(os.environ.get("POLL_CONCURRENCY", "1")))
|
||||
_sem = asyncio.Semaphore(n)
|
||||
return _sem
|
||||
Reference in New Issue
Block a user