Compare commits

...

3 Commits

Author SHA1 Message Date
d41e3838d5 Add per-enclosure temperature metrics + Home Assistant MQTT publisher
- enclosure: fetch SES element descriptor page (0x07) and label temp/fan/
  psu/voltage elements with human-readable names
- temps service + /api/temps: per-enclosure hotspot (max across SES sensors
  and housed drive temps) plus named SES sensor list
- mqtt_publisher: paho-mqtt with HA discovery (device per enclosure, hotspot
  + per-sensor entities), LWT availability, opt-in via MQTT_HOST
- secrets: OpenBao KV v2 reader; MQTT creds sourced from secret/home_assistant
  with env fallback
- compose/requirements/README updated
2026-06-16 04:45:32 +00:00
50dd2a631e Filter dead temp sensors from SES health and recompute overall status
Xyratex HB-1235 enclosures have non-functional temperature sensor
slots that report Unrecoverable/Unknown at 0°C. The SES header
rolls these into UNRECOV=1, causing a false CRITICAL flag. Now we
skip dead sensors (0°C + non-OK status) and derive overall_status
from the actual parsed elements instead of the raw header counts.
2026-03-16 22:19:58 +00:00
3185acbb24 Skip secondary IOM enclosures with no device-linked slots
Dual-IOM shelves (e.g. NetApp DS2246 with two IOM6 modules) create
two sysfs enclosure entries. The secondary path reports populated
slots via SES but the kernel only links block devices through the
primary path. Filter these out during discovery to avoid showing
ghost enclosures with no drive data.
2026-03-16 22:12:48 +00:00
11 changed files with 626 additions and 25 deletions

2
.gitignore vendored
View File

@@ -1,3 +1,5 @@
__pycache__/ __pycache__/
*.pyc *.pyc
.venv/ .venv/
tmp/
jbod-monitor.jsx*

View File

@@ -42,4 +42,66 @@ sudo uvicorn main:app --host 0.0.0.0 --port 8000
| `GET /api/enclosures/{id}/drives` | List drive slots for an enclosure | | `GET /api/enclosures/{id}/drives` | List drive slots for an enclosure |
| `GET /api/drives/{device}` | SMART detail for a block device | | `GET /api/drives/{device}` | SMART detail for a block device |
| `GET /api/overview` | Aggregate enclosure + drive health | | `GET /api/overview` | Aggregate enclosure + drive health |
| `GET /api/temps` | Per-enclosure temperatures: named SES sensors + hotspot |
| `GET /docs` | Interactive API docs (Swagger UI) | | `GET /docs` | Interactive API docs (Swagger UI) |
## Home Assistant (MQTT)
The monitor can publish per-enclosure temperatures to Home Assistant over MQTT
using [HA discovery](https://www.home-assistant.io/integrations/mqtt/#mqtt-discovery) —
no HA YAML required. Each enclosure becomes a device with:
- a **Hotspot** sensor — the hottest reading across all SES temperature
sensors *and* the drives housed in that enclosure (drive temps are usually
the real hotspot); `drive_max_c` and `drive_temp_count` are exposed as
attributes.
- one sensor per named SES temperature element (e.g. *Temp Inlet*,
*Temp Outlet*), labelled from the SES element descriptor page.
Publishing is **opt-in**: set `MQTT_HOST` to enable it.
| Variable | Default | Description |
|---|---|---|
| `MQTT_HOST` | _(unset)_ | Broker host. Unset → MQTT disabled. |
| `MQTT_PORT` | `1883` | Broker port |
| `MQTT_USERNAME` / `MQTT_PASSWORD` | _(unset)_ | Broker credentials (fallback if OpenBao unused/unavailable) |
| `MQTT_DISCOVERY_PREFIX` | `homeassistant` | HA discovery topic prefix |
| `MQTT_BASE_TOPIC` | `jbod-monitor` | State/availability topic root |
| `MQTT_PUBLISH_INTERVAL` | `60` | Seconds between publishes |
| `MQTT_NODE_ID` | _(hostname)_ | Stable id for this monitor (disambiguates multiple hosts) |
| `MQTT_CLIENT_ID` | `jbod-monitor-<node>` | MQTT client id |
### Credentials via OpenBao
Broker credentials are sourced from OpenBao at startup rather than stored in
plaintext (matching the homelab runtime-secret pattern). Set `MQTT_SECRET_PATH`
to a KV v2 path holding `username`/`password`; the app reads
`<OPENBAO_ADDR>/v1/<OPENBAO_MOUNT>/data/<MQTT_SECRET_PATH>` with the token from
`OPENBAO_TOKEN_FILE` (or `OPENBAO_TOKEN`). If the read fails, it falls back to
the `MQTT_USERNAME`/`MQTT_PASSWORD` env vars.
| Variable | Default | Description |
|---|---|---|
| `MQTT_SECRET_PATH` | _(unset)_ | KV v2 path holding the broker creds (e.g. `home_assistant`). Unset → use env creds. |
| `MQTT_SECRET_USER_KEY` | `username` | Key within the secret for the broker username |
| `MQTT_SECRET_PASS_KEY` | `password` | Key within the secret for the broker password |
| `OPENBAO_ADDR` | `https://vault.adamksmith.xyz` | OpenBao API base URL |
| `OPENBAO_MOUNT` | `secret` | KV v2 mount point |
| `OPENBAO_TOKEN_FILE` | _(unset)_ | Path to a file containing the OpenBao token (preferred; mount read-only) |
| `OPENBAO_TOKEN` | _(unset)_ | OpenBao token (used if no token file) |
The deployed compose reads the broker user/pass from `secret/home_assistant`
(keys `mqtt_user`/`mqtt_pass`) using the read-only `claude-read` token mounted
at `/run/secrets/openbao-token`.
Topics (defaults):
```
homeassistant/sensor/<node>_enc<id>/hotspot/config # retained discovery
homeassistant/sensor/<node>_enc<id>/temp<n>/config # retained discovery
jbod-monitor/<node>/status # online | offline (LWT)
jbod-monitor/<node>/enclosure/<id>/state # {"hotspot_c":42.0, "sensor_0":28.5, ...}
```
Availability is tracked via an MQTT LWT, so entities show *unavailable* in HA
if the monitor stops.

View File

@@ -11,6 +11,8 @@ services:
- /dev:/dev - /dev:/dev
- /sys:/sys:ro - /sys:/sys:ro
- /run/udev:/run/udev:ro - /run/udev:/run/udev:ro
# OpenBao RO token (claude-read) mounted read-only; never bake into image.
- /etc/jbod-monitor/openbao-token:/run/secrets/openbao-token:ro
environment: environment:
- TZ=America/Denver - TZ=America/Denver
- UVICORN_LOG_LEVEL=info - UVICORN_LOG_LEVEL=info
@@ -20,6 +22,22 @@ services:
- REDIS_DB=0 - REDIS_DB=0
- SMART_CACHE_TTL=120 - SMART_CACHE_TTL=120
- SMART_POLL_INTERVAL=90 - SMART_POLL_INTERVAL=90
# Home Assistant MQTT publishing (opt-in: leave MQTT_HOST unset to disable).
# EMQX at 10.5.30.3 is reachable by IP and bridged to the Mosquitto HA
# add-on, so discovery reaches Home Assistant either way.
- MQTT_HOST=10.5.30.3
- MQTT_PORT=1883
- MQTT_DISCOVERY_PREFIX=homeassistant
- MQTT_BASE_TOPIC=jbod-monitor
- MQTT_PUBLISH_INTERVAL=60
# Broker credentials sourced from OpenBao (secret/home_assistant,
# keys: mqtt_user/mqtt_pass). Falls back to MQTT_USERNAME/MQTT_PASSWORD
# env if the read fails. Token is the read-only claude-read token.
- OPENBAO_ADDR=https://vault.adamksmith.xyz
- OPENBAO_TOKEN_FILE=/run/secrets/openbao-token
- MQTT_SECRET_PATH=home_assistant
- MQTT_SECRET_USER_KEY=mqtt_user
- MQTT_SECRET_PASS_KEY=mqtt_pass
depends_on: depends_on:
- redis - redis

30
main.py
View File

@@ -10,9 +10,10 @@ from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles from fastapi.staticfiles import StaticFiles
from models.schemas import HealthCheck from models.schemas import HealthCheck
from routers import drives, enclosures, leds, overview from routers import drives, enclosures, leds, overview, temps
from services.cache import cache_set, close_cache, init_cache, redis_available from services.cache import cache_set, close_cache, init_cache, redis_available
from services.enclosure import discover_enclosures, list_slots from services.enclosure import discover_enclosures, list_slots
from services.mqtt_publisher import MqttPublisher, _PAHO_AVAILABLE, mqtt_enabled
from services.smart import SMART_CACHE_TTL, _run_smartctl, sg_ses_available, smartctl_available from services.smart import SMART_CACHE_TTL, _run_smartctl, sg_ses_available, smartctl_available
from services.zfs import get_zfs_pool_map from services.zfs import get_zfs_pool_map
@@ -26,6 +27,8 @@ SMART_POLL_INTERVAL = int(os.environ.get("SMART_POLL_INTERVAL", "90"))
_tool_status: dict[str, bool] = {} _tool_status: dict[str, bool] = {}
_poll_task: asyncio.Task | None = None _poll_task: asyncio.Task | None = None
_mqtt_task: asyncio.Task | None = None
_mqtt_publisher: MqttPublisher | None = None
async def smart_poll_loop(): async def smart_poll_loop():
@@ -77,7 +80,7 @@ async def smart_poll_loop():
@asynccontextmanager @asynccontextmanager
async def lifespan(app: FastAPI): async def lifespan(app: FastAPI):
global _poll_task global _poll_task, _mqtt_task, _mqtt_publisher
# Startup # Startup
_tool_status["smartctl"] = smartctl_available() _tool_status["smartctl"] = smartctl_available()
_tool_status["sg_ses"] = sg_ses_available() _tool_status["sg_ses"] = sg_ses_available()
@@ -95,9 +98,31 @@ async def lifespan(app: FastAPI):
if redis_available(): if redis_available():
_poll_task = asyncio.create_task(smart_poll_loop()) _poll_task = asyncio.create_task(smart_poll_loop())
# Optional MQTT publisher for Home Assistant (opt-in via MQTT_HOST).
_tool_status["mqtt"] = False
if mqtt_enabled():
if not _PAHO_AVAILABLE:
logger.warning("MQTT_HOST set but paho-mqtt not installed — MQTT disabled")
else:
try:
_mqtt_publisher = MqttPublisher()
_mqtt_publisher.start()
_mqtt_task = asyncio.create_task(_mqtt_publisher.run())
_tool_status["mqtt"] = True
except Exception as e:
logger.error("Failed to start MQTT publisher: %s", e)
yield yield
# Shutdown # Shutdown
if _mqtt_task is not None:
_mqtt_task.cancel()
try:
await _mqtt_task
except asyncio.CancelledError:
pass
if _mqtt_publisher is not None:
_mqtt_publisher.stop()
if _poll_task is not None: if _poll_task is not None:
_poll_task.cancel() _poll_task.cancel()
try: try:
@@ -125,6 +150,7 @@ app.include_router(enclosures.router)
app.include_router(drives.router) app.include_router(drives.router)
app.include_router(leds.router) app.include_router(leds.router)
app.include_router(overview.router) app.include_router(overview.router)
app.include_router(temps.router)
@app.get("/api/health", response_model=HealthCheck, tags=["health"]) @app.get("/api/health", response_model=HealthCheck, tags=["health"])

View File

@@ -67,6 +67,7 @@ class SlotWithDrive(BaseModel):
class PsuStatus(BaseModel): class PsuStatus(BaseModel):
index: int index: int
name: str | None = None
status: str status: str
fail: bool = False fail: bool = False
ac_fail: bool = False ac_fail: bool = False
@@ -75,6 +76,7 @@ class PsuStatus(BaseModel):
class FanStatus(BaseModel): class FanStatus(BaseModel):
index: int index: int
name: str | None = None
status: str status: str
rpm: int | None = None rpm: int | None = None
fail: bool = False fail: bool = False
@@ -82,12 +84,14 @@ class FanStatus(BaseModel):
class TempSensor(BaseModel): class TempSensor(BaseModel):
index: int index: int
name: str | None = None
status: str status: str
temperature_c: float | None = None temperature_c: float | None = None
class VoltageSensor(BaseModel): class VoltageSensor(BaseModel):
index: int index: int
name: str | None = None
status: str status: str
voltage: float | None = None voltage: float | None = None
@@ -147,3 +151,25 @@ class Overview(BaseModel):
class HealthCheck(BaseModel): class HealthCheck(BaseModel):
status: str status: str
tools: dict[str, bool] tools: dict[str, bool]
class EnclosureTempSensor(BaseModel):
index: int
name: str | None = None
status: str
temperature_c: float | None = None
class EnclosureTemps(BaseModel):
id: str
vendor: str
model: str
hotspot_c: float | None = None
drive_max_c: float | None = None
drive_temp_count: int = 0
sensors: list[EnclosureTempSensor] = []
class TempSnapshot(BaseModel):
node: str
enclosures: list[EnclosureTemps] = []

View File

@@ -2,3 +2,4 @@ fastapi>=0.115.0
uvicorn>=0.34.0 uvicorn>=0.34.0
pydantic>=2.10.0 pydantic>=2.10.0
redis>=5.0.0 redis>=5.0.0
paho-mqtt>=2.1.0

12
routers/temps.py Normal file
View File

@@ -0,0 +1,12 @@
from fastapi import APIRouter
from models.schemas import TempSnapshot
from services.temps import build_temp_snapshot
router = APIRouter(prefix="/api/temps", tags=["temps"])
@router.get("", response_model=TempSnapshot)
async def get_temps():
"""Per-enclosure temperature snapshot: named SES sensors + hotspot."""
return await build_temp_snapshot()

View File

@@ -51,6 +51,18 @@ def discover_enclosures() -> list[dict]:
slots = list_slots(enc_id) slots = list_slots(enc_id)
total = len(slots) total = len(slots)
populated = sum(1 for s in slots if s["populated"]) populated = sum(1 for s in slots if s["populated"])
has_devices = any(s["device"] for s in slots)
# Skip secondary/passive IOM paths: enclosures where SES reports
# populated slots but the kernel has no device symlinks for any of
# them (the drives are owned by the primary IOM enclosure entry).
if populated > 0 and not has_devices:
logger.info(
"Skipping enclosure %s (%s %s) — %d populated slots but no "
"device links (likely a secondary IOM path)",
enc_id, vendor, model, populated,
)
continue
enclosures.append({ enclosures.append({
"id": enc_id, "id": enc_id,
@@ -140,29 +152,88 @@ def _parse_slot_number(entry: Path) -> int | None:
return None return None
async def get_enclosure_status(sg_device: str) -> dict | None: async def _run_sg_ses(sg_device: str, page: str) -> str | None:
"""Run sg_ses --page=0x02 and parse enclosure health data.""" """Run sg_ses for a given page, returning decoded stdout or None."""
try: try:
proc = await asyncio.create_subprocess_exec( proc = await asyncio.create_subprocess_exec(
"sg_ses", "--page=0x02", sg_device, "sg_ses", f"--page={page}", sg_device,
stdout=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
) )
stdout, stderr = await proc.communicate() stdout, stderr = await proc.communicate()
if proc.returncode != 0: if proc.returncode != 0:
logger.warning("sg_ses failed for %s: %s", sg_device, stderr.decode().strip()) logger.warning(
"sg_ses page %s failed for %s: %s",
page, sg_device, stderr.decode().strip(),
)
return None return None
return _parse_ses_page02(stdout.decode(errors="replace")) return stdout.decode(errors="replace")
except FileNotFoundError: except FileNotFoundError:
logger.warning("sg_ses not found") logger.warning("sg_ses not found")
return None return None
except Exception as e: except Exception as e:
logger.warning("sg_ses error for %s: %s", sg_device, e) logger.warning("sg_ses page %s error for %s: %s", page, sg_device, e)
return None return None
def _parse_ses_page02(text: str) -> dict: async def get_enclosure_status(sg_device: str) -> dict | None:
"""Parse sg_ses --page=0x02 text output into structured health data.""" """Run sg_ses status (0x02) + element descriptor (0x07) pages and parse.
The element descriptor page provides human-readable names for each
element (e.g. "Temp Inlet", "Fan 1"); these are merged into the status
elements so callers get labelled sensors. Descriptor lookup is
best-effort — if page 0x07 is unsupported, elements fall back to bare
indices.
"""
status_text, desc_text = await asyncio.gather(
_run_sg_ses(sg_device, "0x02"),
_run_sg_ses(sg_device, "0x07"),
)
if status_text is None:
return None
names = _parse_element_descriptors(desc_text) if desc_text else {}
return _parse_ses_page02(status_text, names)
def _norm_type(element_type: str) -> str:
"""Normalize an SES element type line to a stable key.
sg_ses appends qualifiers like ", subenclosure id: 0 [ti=2]" to the
type line; strip those so the same element type matches across the
status (0x02) and element descriptor (0x07) pages.
"""
return element_type.split(",")[0].strip().lower()
def _parse_element_descriptors(text: str) -> dict[tuple[str, int], str]:
"""Parse sg_ses --page=0x07 into {(element_type, index): name}."""
names: dict[tuple[str, int], str] = {}
sections = re.split(r"(?=\s*Element type:)", text)
for section in sections:
type_match = re.match(r"\s*Element type:\s*(.+)", section)
if not type_match:
continue
etype = _norm_type(type_match.group(1).strip().rstrip(","))
for line in section.splitlines():
m = re.match(r"\s*Element (\d+) descriptor:\s*(.*)", line)
if not m:
continue
idx = int(m.group(1))
name = m.group(2).strip()
if name:
names[(etype, idx)] = name
return names
def _parse_ses_page02(
text: str, names: dict[tuple[str, int], str] | None = None
) -> dict:
"""Parse sg_ses --page=0x02 text output into structured health data.
``names`` maps (normalized element type, index) -> descriptor name from
the element descriptor page (0x07); when present it labels each element.
"""
names = names or {}
result = { result = {
"overall_status": "OK", "overall_status": "OK",
"psus": [], "psus": [],
@@ -171,21 +242,6 @@ def _parse_ses_page02(text: str) -> dict:
"voltages": [], "voltages": [],
} }
# Parse header line for overall status:
# INVOP=0, INFO=0, NON-CRIT=0, CRIT=1, UNRECOV=0
header_match = re.search(
r"INVOP=\d+,\s*INFO=\d+,\s*NON-CRIT=(\d+),\s*CRIT=(\d+),\s*UNRECOV=(\d+)",
text,
)
if header_match:
non_crit = int(header_match.group(1))
crit = int(header_match.group(2))
unrecov = int(header_match.group(3))
if crit > 0 or unrecov > 0:
result["overall_status"] = "CRITICAL"
elif non_crit > 0:
result["overall_status"] = "WARNING"
# Split into element type sections. # Split into element type sections.
# Each section starts with "Element type: <type>" # Each section starts with "Element type: <type>"
sections = re.split(r"(?=\s*Element type:)", text) sections = re.split(r"(?=\s*Element type:)", text)
@@ -195,6 +251,7 @@ def _parse_ses_page02(text: str) -> dict:
if not type_match: if not type_match:
continue continue
element_type = type_match.group(1).strip().rstrip(",").lower() element_type = type_match.group(1).strip().rstrip(",").lower()
etype_key = _norm_type(element_type)
# Find individual element blocks (skip "Overall descriptor") # Find individual element blocks (skip "Overall descriptor")
elements = re.split(r"(?=\s*Element \d+ descriptor:)", section) elements = re.split(r"(?=\s*Element \d+ descriptor:)", section)
@@ -212,12 +269,15 @@ def _parse_ses_page02(text: str) -> dict:
if status.lower() == "not installed": if status.lower() == "not installed":
continue continue
name = names.get((etype_key, idx))
if "power supply" in element_type: if "power supply" in element_type:
fail = "Fail=1" in elem_text fail = "Fail=1" in elem_text
ac_fail = "AC fail=1" in elem_text ac_fail = "AC fail=1" in elem_text
dc_fail = "DC fail=1" in elem_text dc_fail = "DC fail=1" in elem_text
result["psus"].append({ result["psus"].append({
"index": idx, "index": idx,
"name": name,
"status": status, "status": status,
"fail": fail, "fail": fail,
"ac_fail": ac_fail, "ac_fail": ac_fail,
@@ -230,6 +290,7 @@ def _parse_ses_page02(text: str) -> dict:
rpm = int(rpm_match.group(1)) if rpm_match else None rpm = int(rpm_match.group(1)) if rpm_match else None
result["fans"].append({ result["fans"].append({
"index": idx, "index": idx,
"name": name,
"status": status, "status": status,
"rpm": rpm, "rpm": rpm,
"fail": fail, "fail": fail,
@@ -238,8 +299,15 @@ def _parse_ses_page02(text: str) -> dict:
elif "temperature" in element_type: elif "temperature" in element_type:
temp_match = re.search(r"Temperature=\s*([\d.]+)\s*C", elem_text) temp_match = re.search(r"Temperature=\s*([\d.]+)\s*C", elem_text)
temp = float(temp_match.group(1)) if temp_match else None temp = float(temp_match.group(1)) if temp_match else None
# Skip dead/disconnected sensors: 0°C with a non-OK status
# is a non-functional sensor slot, not an actual reading.
if (temp is None or temp == 0) and status.lower() in (
"unrecoverable", "unknown", "not available",
):
continue
result["temps"].append({ result["temps"].append({
"index": idx, "index": idx,
"name": name,
"status": status, "status": status,
"temperature_c": temp, "temperature_c": temp,
}) })
@@ -251,8 +319,26 @@ def _parse_ses_page02(text: str) -> dict:
voltage = float(volt_match.group(1)) if volt_match else None voltage = float(volt_match.group(1)) if volt_match else None
result["voltages"].append({ result["voltages"].append({
"index": idx, "index": idx,
"name": name,
"status": status, "status": status,
"voltage": voltage, "voltage": voltage,
}) })
# Derive overall_status from the actual parsed elements rather than
# the raw SES header, which counts dead/disconnected sensors as
# UNRECOV and inflates severity.
all_statuses = (
[e["status"] for e in result["psus"]]
+ [e["status"] for e in result["fans"]]
+ [e["status"] for e in result["temps"]]
+ [e["status"] for e in result["voltages"]]
)
status_lower = [s.lower() for s in all_statuses]
if any(s in ("unrecoverable", "critical") for s in status_lower):
result["overall_status"] = "CRITICAL"
elif any(s in ("noncritical", "non-critical", "warning") for s in status_lower):
result["overall_status"] = "WARNING"
elif any(s not in ("ok", "unknown") for s in status_lower):
result["overall_status"] = "WARNING"
return result return result

226
services/mqtt_publisher.py Normal file
View File

@@ -0,0 +1,226 @@
"""MQTT publisher with Home Assistant discovery for enclosure temperatures.
Opt-in: only starts when ``MQTT_HOST`` is set. Publishes one retained
discovery config per HA entity (a device per enclosure, with a hotspot
sensor plus one sensor per named SES temperature element) and pushes a
single JSON state message per enclosure on each poll. Availability is
tracked via an LWT so HA marks entities unavailable if the monitor dies.
Topic layout (defaults)::
homeassistant/sensor/<node>_enc<id>/hotspot/config (retained discovery)
homeassistant/sensor/<node>_enc<id>/temp<n>/config (retained discovery)
jbod-monitor/<node>/status online | offline (LWT)
jbod-monitor/<node>/enclosure/<id>/state {"hotspot_c":.., "sensor_0":..}
"""
import asyncio
import json
import logging
import os
import re
from services.secrets import read_kv2
from services.temps import build_temp_snapshot, get_node_id
logger = logging.getLogger(__name__)
try:
import paho.mqtt.client as mqtt
from paho.mqtt.enums import CallbackAPIVersion
_PAHO_AVAILABLE = True
except ImportError: # pragma: no cover - exercised only without the dep
mqtt = None
CallbackAPIVersion = None
_PAHO_AVAILABLE = False
def mqtt_enabled() -> bool:
"""MQTT publishing is opt-in via MQTT_HOST."""
return bool(os.environ.get("MQTT_HOST"))
def _slug(value: str) -> str:
"""Sanitize a string for use in MQTT topics / HA unique_ids."""
return re.sub(r"[^a-zA-Z0-9_-]", "_", value).strip("_") or "x"
def _resolve_credentials() -> tuple[str | None, str | None]:
"""Resolve broker credentials, preferring OpenBao over plaintext env.
If ``MQTT_SECRET_PATH`` is set, read username/password from that OpenBao
KV v2 path (keys ``username``/``password``). Falls back to the
``MQTT_USERNAME``/``MQTT_PASSWORD`` env vars when unset or on failure.
"""
username = os.environ.get("MQTT_USERNAME") or None
password = os.environ.get("MQTT_PASSWORD") or None
secret_path = os.environ.get("MQTT_SECRET_PATH")
if secret_path:
user_key = os.environ.get("MQTT_SECRET_USER_KEY", "username")
pass_key = os.environ.get("MQTT_SECRET_PASS_KEY", "password")
data = read_kv2(secret_path)
if data:
username = data.get(user_key, username)
password = data.get(pass_key, password)
logger.info("Loaded MQTT credentials from OpenBao (%s)", secret_path)
else:
logger.warning(
"MQTT_SECRET_PATH set (%s) but OpenBao read failed — "
"falling back to env credentials", secret_path,
)
return username, password
class MqttPublisher:
def __init__(self) -> None:
self.host = os.environ["MQTT_HOST"]
self.port = int(os.environ.get("MQTT_PORT", "1883"))
self.username, self.password = _resolve_credentials()
self.discovery_prefix = os.environ.get("MQTT_DISCOVERY_PREFIX", "homeassistant")
self.base_topic = os.environ.get("MQTT_BASE_TOPIC", "jbod-monitor")
self.interval = int(os.environ.get("MQTT_PUBLISH_INTERVAL", "60"))
self.node = _slug(get_node_id())
self.availability_topic = f"{self.base_topic}/{self.node}/status"
# unique_ids for which we've already published discovery this connection.
self._discovered: set[str] = set()
self._client = mqtt.Client(
CallbackAPIVersion.VERSION2,
client_id=os.environ.get("MQTT_CLIENT_ID", f"jbod-monitor-{self.node}"),
)
if self.username:
self._client.username_pw_set(self.username, self.password)
self._client.will_set(self.availability_topic, "offline", qos=1, retain=True)
self._client.reconnect_delay_set(min_delay=1, max_delay=60)
self._client.on_connect = self._on_connect
self._client.on_disconnect = self._on_disconnect
def start(self) -> None:
logger.info("MQTT connecting to %s:%d", self.host, self.port)
self._client.connect_async(self.host, self.port, keepalive=60)
self._client.loop_start()
def stop(self) -> None:
try:
self._client.publish(self.availability_topic, "offline", qos=1, retain=True)
except Exception:
pass
self._client.loop_stop()
try:
self._client.disconnect()
except Exception:
pass
# paho callbacks (run in the network thread) -----------------------------
def _on_connect(self, client, userdata, flags, reason_code, properties=None):
if reason_code != 0:
logger.warning("MQTT connect failed: %s", reason_code)
return
logger.info("MQTT connected")
# Re-announce discovery + availability on every (re)connect.
self._discovered.clear()
client.publish(self.availability_topic, "online", qos=1, retain=True)
def _on_disconnect(self, client, userdata, flags, reason_code, properties=None):
logger.warning("MQTT disconnected: %s", reason_code)
# discovery / state ------------------------------------------------------
def _device_block(self, enc: dict) -> dict:
enc_id = enc["id"]
vendor = enc.get("vendor", "").strip()
model = enc.get("model", "").strip()
name = " ".join(p for p in [vendor, model] if p) or "JBOD enclosure"
return {
"identifiers": [f"{self.node}_enc_{_slug(enc_id)}"],
"name": f"JBOD {name} ({enc_id})",
"manufacturer": vendor or None,
"model": model or None,
"via_device": f"{self.node}_jbod_monitor",
}
def _publish_discovery(self, enc: dict) -> None:
enc_id = enc["id"]
enc_slug = _slug(enc_id)
state_topic = f"{self.base_topic}/{self.node}/enclosure/{enc_slug}/state"
device = self._device_block(enc)
def announce(object_id: str, payload: dict) -> None:
uid = payload["unique_id"]
if uid in self._discovered:
return
topic = (
f"{self.discovery_prefix}/sensor/"
f"{self.node}_enc{enc_slug}/{object_id}/config"
)
self._client.publish(topic, json.dumps(payload), qos=1, retain=True)
self._discovered.add(uid)
common = {
"device_class": "temperature",
"unit_of_measurement": "°C",
"state_class": "measurement",
"state_topic": state_topic,
"availability_topic": self.availability_topic,
"device": device,
}
# Hotspot (max across SES sensors + drive temps).
announce("hotspot", {
**common,
"name": "Hotspot",
"unique_id": f"{self.node}_enc{enc_slug}_hotspot",
"value_template": "{{ value_json.hotspot_c }}",
"icon": "mdi:thermometer-high",
"json_attributes_topic": state_topic,
"json_attributes_template":
'{"drive_max_c": {{ value_json.drive_max_c | default("null") }}, '
'"drive_temp_count": {{ value_json.drive_temp_count | default(0) }}}',
})
# One sensor per named SES temperature element.
for s in enc.get("sensors", []):
idx = s["index"]
label = s.get("name") or f"Temp {idx}"
announce(f"temp{idx}", {
**common,
"name": label,
"unique_id": f"{self.node}_enc{enc_slug}_temp{idx}",
"value_template": f"{{{{ value_json.sensor_{idx} }}}}",
})
def _publish_state(self, enc: dict) -> None:
enc_slug = _slug(enc["id"])
state_topic = f"{self.base_topic}/{self.node}/enclosure/{enc_slug}/state"
payload: dict = {
"hotspot_c": enc.get("hotspot_c"),
"drive_max_c": enc.get("drive_max_c"),
"drive_temp_count": enc.get("drive_temp_count", 0),
}
for s in enc.get("sensors", []):
payload[f"sensor_{s['index']}"] = s.get("temperature_c")
payload[f"sensor_{s['index']}_status"] = s.get("status")
self._client.publish(state_topic, json.dumps(payload), qos=0, retain=True)
def publish_snapshot(self, snapshot: dict) -> None:
for enc in snapshot.get("enclosures", []):
self._publish_discovery(enc)
self._publish_state(enc)
async def run(self) -> None:
"""Background loop: build a snapshot and publish on an interval."""
await asyncio.sleep(3) # let the first SMART poll populate the cache
while True:
try:
snapshot = await build_temp_snapshot()
self.publish_snapshot(snapshot)
logger.info(
"MQTT published temps for %d enclosure(s)",
len(snapshot.get("enclosures", [])),
)
except asyncio.CancelledError:
raise
except Exception as e:
logger.error("MQTT publish loop error: %s", e)
await asyncio.sleep(self.interval)

58
services/secrets.py Normal file
View File

@@ -0,0 +1,58 @@
"""Minimal OpenBao (Vault) KV v2 reader for runtime secret injection.
Mirrors the homelab pattern used by other non-k8s services: read
``<addr>/v1/<mount>/data/<path>`` with an ``X-Vault-Token``. Stdlib only —
no extra dependency. Token comes from ``OPENBAO_TOKEN`` or, preferably, a
mounted token file via ``OPENBAO_TOKEN_FILE`` (Docker/host secret).
"""
import json
import logging
import os
import urllib.error
import urllib.request
logger = logging.getLogger(__name__)
DEFAULT_ADDR = "https://vault.adamksmith.xyz"
DEFAULT_MOUNT = "secret"
def _resolve_token() -> str | None:
token = os.environ.get("OPENBAO_TOKEN")
if token:
return token.strip()
token_file = os.environ.get("OPENBAO_TOKEN_FILE")
if token_file:
try:
return open(token_file).read().strip()
except OSError as e:
logger.warning("Could not read OPENBAO_TOKEN_FILE %s: %s", token_file, e)
return None
def read_kv2(path: str) -> dict | None:
"""Read a KV v2 secret. ``path`` is relative to the mount (e.g.
"jbod-monitor/mqtt"). Returns the inner data dict, or None on failure.
"""
token = _resolve_token()
if not token:
logger.warning("OpenBao read requested for %s but no token configured", path)
return None
addr = os.environ.get("OPENBAO_ADDR", DEFAULT_ADDR).rstrip("/")
mount = os.environ.get("OPENBAO_MOUNT", DEFAULT_MOUNT).strip("/")
url = f"{addr}/v1/{mount}/data/{path.strip('/')}"
req = urllib.request.Request(url, headers={"X-Vault-Token": token})
try:
with urllib.request.urlopen(req, timeout=10) as resp:
body = json.loads(resp.read())
except urllib.error.HTTPError as e:
logger.warning("OpenBao read %s failed: HTTP %s", path, e.code)
return None
except Exception as e:
logger.warning("OpenBao read %s failed: %s", path, e)
return None
# KV v2 wraps the secret in data.data.
return body.get("data", {}).get("data")

84
services/temps.py Normal file
View File

@@ -0,0 +1,84 @@
"""Per-enclosure temperature aggregation.
Builds a compact snapshot of enclosure temperatures suitable for Home
Assistant: each named SES temperature sensor plus a derived per-enclosure
hotspot (the hottest reading across SES sensors *and* the drives housed in
that enclosure — drive temps are usually the real hotspot signal).
"""
import asyncio
import logging
import socket
from services.enclosure import discover_enclosures, get_enclosure_status, list_slots
from services.smart import get_smart_data
logger = logging.getLogger(__name__)
def get_node_id() -> str:
"""Stable identifier for this monitor instance (defaults to hostname)."""
import os
return os.environ.get("MQTT_NODE_ID") or socket.gethostname() or "jbod-monitor"
async def build_temp_snapshot() -> dict:
"""Collect per-enclosure temperature metrics for all enclosures."""
enclosures = discover_enclosures()
async def _health(enc):
if enc.get("sg_device"):
return await get_enclosure_status(enc["sg_device"])
return None
health_results = await asyncio.gather(
*[_health(enc) for enc in enclosures], return_exceptions=True
)
out_enclosures: list[dict] = []
for enc, health in zip(enclosures, health_results):
if isinstance(health, Exception):
logger.warning("SES health failed for %s: %s", enc["id"], health)
health = None
sensors: list[dict] = []
sensor_temps: list[float] = []
if isinstance(health, dict):
for t in health.get("temps", []):
temp = t.get("temperature_c")
sensors.append({
"index": t["index"],
"name": t.get("name"),
"status": t.get("status", "Unknown"),
"temperature_c": temp,
})
if isinstance(temp, (int, float)) and temp > 0:
sensor_temps.append(float(temp))
# Drive temperatures for drives housed in this enclosure.
drive_temps: list[float] = []
devices = [s["device"] for s in list_slots(enc["id"]) if s.get("device")]
if devices:
smart_results = await asyncio.gather(
*[get_smart_data(d) for d in devices], return_exceptions=True
)
for res in smart_results:
if isinstance(res, Exception):
continue
data, _hit = res
temp = data.get("temperature_c")
if isinstance(temp, (int, float)) and temp > 0:
drive_temps.append(float(temp))
all_temps = sensor_temps + drive_temps
out_enclosures.append({
"id": enc["id"],
"vendor": enc.get("vendor", ""),
"model": enc.get("model", ""),
"hotspot_c": max(all_temps) if all_temps else None,
"drive_max_c": max(drive_temps) if drive_temps else None,
"drive_temp_count": len(drive_temps),
"sensors": sensors,
})
return {"node": get_node_id(), "enclosures": out_enclosures}