Quartz 4

design

7 min read

SigenStor Control — Design (Python 3)

Single-user home-energy controller (Mick). k8s-hosted. Monitors + controls one Sigenergy SigenStor (battery + PV) using local Modbus TCP primary, Amber Electric for price/forecast, and a solar-forecast provider (Solcast preferred, Open-Meteo fallback).

Architecture, Modbus register map, Amber + solar integrations, 60s controller logic, safety guardrails, k8s deploy, observability, testing, and phasing are APPROVED from the Go draft and unchanged below. ONLY the stack section is rewritten in Python 3.

Stack (Python 3 — replaces Go)

  • Runtime: Python 3.13 (latest stable at build time; pin python:3.13-slim base image by digest). Verify newest patched 3.12/3.13 at build.
  • Web framework: FastAPI (async) + uvicorn (uvloop loop). Pin fastapi==<latest>, uvicorn[standard]==<latest>. Templates via Jinja2 (server-rendered HTML).
  • Modbus: pymodbus (latest, async client). Pin exact version. Hold the register map in sigenstor/modbus/registers.py (port of HA sigenergy integration constants; do not invent addresses).
  • HTTP clients: httpx (async, latest pinned). Amber + Solcast + Open-Meteo clients.
  • DB: SQLite via sqlmodel (SQLAlchemy 2.x core) + aiosqlite driver. WAL mode. One file at /var/lib/sigenstor/sigenstor.db on PVC. Pin sqlmodel==<latest>, sqlalchemy==<latest>, aiosqlite==<latest>.
  • Scheduler: APScheduler AsyncIOScheduler for fixed-tick jobs (Modbus 5s, Amber 60s, Solcast 30min/2h, controller 60s). Pin apscheduler==<latest>. Alternative: pure asyncio.create_task loops — APScheduler chosen for crash-recovery + job-coalescing.
  • Validation: pydantic v2 latest (pydantic==<latest>, pydantic-settings==<latest> for env).
  • Auth: fastapi-users (HTTP bearer cookie session, single user) OR minimal Basic-Auth + itsdangerous signed cookies — pick minimal Basic + signed cookie (matches single-user scope). Pin itsdangerous==<latest>, bcrypt==<latest> (or argon2-cffi==<latest>).
  • Metrics: prometheus-client (latest pinned). /metrics endpoint.
  • Logging: structlog JSON to stdout (latest pinned).
  • Frontend (unchanged): htmx + Alpine.js + Chart.js — vendored under /static, no CDN.
  • Packaging: single uvicorn process, multi-stage Dockerfile, non-root user, distroless-free but python:3.13-slim base (slim is the patched, minimal Debian; pin by digest in FROM). Rebuild on base-image CVE / new patch.
  • Dependency mgmt: uv (or pip-tools) lockfile — uv.lock committed. All runtime deps pinned exact in pyproject.toml + lockfile. Dev deps separate.
  • Type/lint/format: mypy --strict, ruff, pytest+pytest-asyncio+httpx test client. Pin all.
  • Secrets: pydantic-settings reads env from k8s Secret sigenstor-secrets mounted as env vars. NEVER hardcoded.
  • NOTE: “verify newest patched at build time” — at every container build, run pip-audit (or uv pip install --upgrade review) and check python:3.13-slim digest for CVEs.

Architecture (one process, three internal layers — unchanged)

  • poller — APScheduler jobs: SigenStor every 5s, Amber every 60s, Solcast every 30min (04:00–22:00) / 2h overnight, price-forecast refresh every 5min. All write to DB and in-memory state dataclass.
  • controller — runs every 60s. Reads state + DB (schedules, mode, override, prices, forecast, SoC). Decides target charge/discharge kW and EMS mode. Idempotent.
  • api — FastAPI app: JSON /api/v1/..., HTML / (Jinja2 templates), /metrics, /healthz//readyz.

Data model (SQLite, single file — unchanged)

  • telemetry (ts INT PK, soc_pct REAL, bat_kw REAL, pv_kw REAL, load_kw REAL, grid_kw REAL, mode TEXT, online INT) — 30d retention.
  • prices (ts INT PK, buy_c_per_kwh REAL, feed_in_c_per_kwh REAL, source TEXT, fetched_at INT) — 30-min slots.
  • forecast_solar (ts INT PK, kwh REAL, source TEXT).
  • forecast_price (ts INT PK, predicted_c_per_kwh REAL, interval_type TEXT).
  • schedules (id, name, days_mask, start_min, end_min, action TEXT, kw REAL, priority INT, enabled INT).
  • mode_state (key TEXT PK, value TEXT) — current mode, manual override expiry, last decision reason, controller enabled flag.
  • events (ts, kind, payload TEXT) — 90d retention.
  • kv (key TEXT PK, value TEXT) — modbus host/port/unit-id/poll interval; modbus write-password AES-GCM-encrypted with key from secret.

Sigenergy Modbus integration (unchanged)

  • Reference: HA sigenergy integration register map, ported verbatim.
  • Transport: Modbus TCP, port 502 (or 5020 — confirm at commissioning; default 502). Unit ID 1 default.
  • Reads: device master EMS status, SoC, battery power, PV power, load, grid power, energy totals, fault bits.
  • Writes: EMS mode (self-consume / forced charge / forced discharge / remote schedule), forced charge/discharge power limit, remote schedule enable/disable.
  • Auth: installer/admin password for writes (HA integration prompts); passed per-session to pymodbus client.
  • Failure: exponential backoff 1/2/4/8s, max 3, then event + revert safe mode (self-consume). Never leave battery in forced-charge indefinitely.
  • Polling: 5s reads in single task; writes serialized through asyncio.Lock (or asyncio.Queue writer task) — no interleaved register updates.
  • Discovery: hardcode IP via k8s ConfigMap SIGENSTOR_MODBUS_HOST env var; override from kv.

Amber Electric (unchanged)

  • Base URL https://api.amber.com.au/v1. Bearer token from secret.
  • Endpoints: GET /sites, GET /sites/{id}/prices/current, GET /sites/{id}/prices/forecasts, GET /sites/{id}/prices/next.
  • Store buy + feed-in at 30-min granularity, general price type.
  • Refresh: current 60s, forecast 5min. 5s in-memory cache.

Solar forecast (unchanged)

  • Primary Solcast (free tier, 7d horizon). Fallback Open-Meteo (no key, 11d).
  • Refresh: 30min 04:00–22:00, 2h overnight. UPSERT to forecast_solar.
  • Use rolling 4h PV estimate vs SoC for pre-charge headroom.

Controller logic (every 60s — unchanged)

  • Decision order: Safety (SoC ≤10% force charge, ≥95% force discharge) → Manual override (if active, not expired) → Manual schedule (highest priority enabled covering now) → Auto PRICE (cheap window ≤60min + SoC<80% → charge; expensive now + SoC>30% → discharge capped) → Auto WEATHER (tomorrow solar <2 kWh + tonight prices low → pre-charge) → Auto USAGE (learn 14d baseline, top-up before high-load evening) → self-consume default.
  • Single coherent EMS-mode write per tick. Log decision reason to events.

UI (htmx + Alpine + Chart.js — unchanged)

  • Pages: / Dashboard, /schedules, /automations, /history, /settings, /events.
  • Big mode badge, one-click override buttons (“Charge now 30 min” etc.), banners on stale data.
  • All charts 30-min resolution.

API (JSON — unchanged)

  • GET /api/v1/state, /api/v1/telemetry, /api/v1/prices, /api/v1/forecast/solar, /api/v1/schedules (GET/POST/PATCH/DELETE), POST /api/v1/mode, POST/DELETE /api/v1/override, GET /api/v1/events, GET /metrics.
  • All state-changing require CSRF + Basic-auth session.

k8s deployment

  • Namespace sigenstor. Helm chart sigenstor-control. 1 replica, no HPA.
  • Resources: req 50m/64Mi, lim 500m/256Mi (Python ~50–80Mi idle).
  • PVC sigenstor-data 1Gi RWO (sqlite + logs).
  • Secret sigenstor-secrets mounted as env vars (keys used):
    • AMBER_API_TOKEN — Amber client.
    • AMBER_SITE_ID — Amber client.
    • SIGENSTOR_MODBUS_HOST / _PORT / _UNIT_ID — Modbus client.
    • SOLCAST_API_KEY — Solcast client (optional; Open-Meteo needs none).
    • MYSIGEN_USERNAME / MYSIGEN_PASSWORD — only if cloud path used (NOT primary; local Modbus is).
    • BASIC_AUTH_HTPASSWD — UI auth.
    • SESSION_SECRET — cookie signing.
    • SQLITE_ENCRYPTION_KEY — encrypts modbus password in kv.
  • ConfigMap sigenstor-config (non-secret env: LOG_LEVEL, LISTEN_ADDR=:8080, DB_PATH, POLL_INTERVAL_S).
  • Service ClusterIP 8080. Ingress via existing nginx ingress, TLS via cert-manager, host sigenstor.home.<domain>.
  • Probes: liveness GET /healthz (DB pingable + last poll <60s); readiness same minus modbus (allow degraded).
  • PDB min 1 max 1.
  • NetworkPolicy: egress only to LAN CIDR (SigenStor), api.amber.com.au, api.solcast.com, api.open-meteo.com.
  • Backup: nightly CronJob sqlite3 .backup to PVC, retain 14, push to S3-compatible bucket via restic.

Security

  • All secrets via k8s secret env-injected, never logged. pydantic-settings reads them at startup; secret values masked in logs (structlog processor).
  • Modbus write password AES-GCM-encrypted in kv, decrypted in-memory only.
  • HTTPS at ingress; HTTP→HTTPS redirect.
  • CSP: default-src 'self'; connect-src 'self' api.amber.com.au api.solcast.com; chart.js + alpine.js vendored under /static (no CDN).
  • CSRF token on all POST/PATCH/DELETE; signed cookie session.
  • Non-root container user (USER app).

Observability (unchanged)

  • Prometheus /metrics: sigenstor_soc_percent, sigenstor_battery_kw, sigenstor_pv_kw, sigenstor_grid_kw, sigenstor_amber_buy_cents, sigenstor_amber_feed_in_cents, sigenstor_controller_mode, sigenstor_modbus_errors_total, sigenstor_decision_duration_seconds.
  • structlog JSON to stdout. INFO/WARN/ERROR per Go draft.
  • Grafana dashboard JSON committed.

Failure modes + safety (unchanged)

  • Modbus unreachable >120s → safe self-consume + banner, no writes.
  • Amber 401 → cache last, disable PRICE mode + banner.
  • Solcast+Open-Meteo both fail >2h → disable WEATHER, PRICE+USAGE still work.
  • DB write failure → controller pauses, ERROR log, banner.
  • Pod restart → controller re-asserts last mode from mode_state on startup.
  • Battery fault bit → force self-consume + alert.
  • All writes idempotent.

Testing strategy

  • Unit: pytest table-driven on controller decision tree (synthetic inputs).
  • Integration: docker-compose with pymodbus.server (async Modbus fake), respx for httpx (Amber/Solcast mocks), fixtures.
  • E2e manual: deploy to k8s, point at real SigenStor, verify one diurnal cycle.
  • Regression: golden-file test for controller output given canned input.
  • Lint/type: ruff, mypy --strict in CI.
  • Security: pip-audit, trivy image scan, kubeconform chart lint.

Out of scope (v1) — unchanged

  • Multi-site/multi-user, VPP, EV charger, HA add-on packaging, mobile app, per-string PV.

Risks (additions for Python stack)

  • Python base image CVEs → rebuild on python:3.13-slim digest updates; pin digest.
  • pymodbus async API changes between minor versions → pin exact, integration tests catch regressions.
  • APScheduler in async loop — use AsyncIOScheduler; ensure DB sessions closed per tick to avoid leaks.
  • uv/pip-tools lockfile drift → CI fails builds on unpinned deps.

Decisions summary (locked, stack swap noted)

  • Local Modbus primary, no cloud mySigen dependency.
  • Python 3.13 single uvicorn process, python:3.13-slim multi-stage non-root image.
  • SQLite single-file DB, no external DB.
  • htmx server-rendered UI, no SPA.
  • Single replica, no HPA.
  • PRICE / WEATHER / USAGE composable; safety overrides all.
  • Decision logged every tick.
  • Controller idempotent + restart-safe.

Graph View