Split into dedicated hardware poller + read-only consumers
Rearchitect so a single poller process is the sole owner of all SAS/SMART/ SES hardware I/O, serialized behind one gate and paced, writing results to Redis. The API/web and MQTT publisher become pure Redis readers — they no longer issue any subprocess and can restart freely without touching the bus. This addresses backplane/expander stress from concurrent + restart-triggered SMART/SES storms (the prior model re-ran a hardware sweep on every container start, and polled sg_ses 0x02+0x07 every 60s; 0x07 errored on the IOM6/ Xyratex expanders). - poller.py: paced sweep (inventory, SMART per-drive w/ gap, SES 0x02, ZFS, host/MegaRAID), startup jitter, single-instance Redis lock, LED-queue worker - services/hwgate.py: global serialization semaphore (POLL_CONCURRENCY=1) - services/store.py: Redis as the only producer<->consumer interface + LED queue - services/health.py: shared drive-health classifier (fixes overview double-count) - gate smartctl/sg_ses/zpool/ledctl; drop sg_ses 0x07 from the hot path - routers + temps read-only from the store; main.py drops the in-process poller - compose: 3 services (privileged poller + unprivileged app + redis) w/ pacing knobs
This commit is contained in:
@@ -4,26 +4,23 @@ import logging
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from services.cache import cache_get, cache_set
|
||||
from services.hwgate import gate
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Allow overriding the zpool binary path via env (for bind-mounted host tools)
|
||||
ZPOOL_BIN = os.environ.get("ZPOOL_BIN", "zpool")
|
||||
|
||||
ZFS_CACHE_TTL = 300
|
||||
|
||||
|
||||
async def get_zfs_pool_map() -> dict[str, dict]:
|
||||
"""Return a dict mapping device names to ZFS pool and vdev info.
|
||||
|
||||
e.g. {"sda": {"pool": "tank", "vdev": "raidz2-0"},
|
||||
"sdb": {"pool": "fast", "vdev": "mirror-0"}}
|
||||
"""
|
||||
cached = await cache_get("jbod:zfs_map")
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
Hardware-gated producer; the poller persists the result to the store
|
||||
and consumers read it from there.
|
||||
"""
|
||||
pool_map = {}
|
||||
try:
|
||||
# When running in a container with pid:host, use nsenter to run
|
||||
@@ -34,12 +31,13 @@ async def get_zfs_pool_map() -> dict[str, dict]:
|
||||
else:
|
||||
cmd = [ZPOOL_BIN, "status", "-P"]
|
||||
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, _ = await proc.communicate()
|
||||
async with gate():
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, _ = await proc.communicate()
|
||||
if proc.returncode != 0:
|
||||
return pool_map
|
||||
|
||||
@@ -103,7 +101,6 @@ async def get_zfs_pool_map() -> dict[str, dict]:
|
||||
except FileNotFoundError:
|
||||
logger.debug("zpool not available")
|
||||
|
||||
await cache_set("jbod:zfs_map", pool_map, ZFS_CACHE_TTL)
|
||||
return pool_map
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user