Skip to content

progress.md

Latest commit

 

History

History
425 lines (353 loc) · 21.5 KB

progress.md

File metadata and controls

425 lines (353 loc) · 21.5 KB
Raw

Flashbee — storm-detection debug, in progress

2026-05-27 — Operating profiles (Phase A) done

After the storm, addressed the false-positive drift it exposed (the frozen storm-chase config flooded phantom "overhead" strikes once the storm passed / indoors). Re-read the full AS3935 datasheet (spec: docs/superpowers/specs/2026-05-27-operating-profiles-design.md; plan: docs/superpowers/plans/2026-05-27-operating-profiles.md) and built an operating-profile system replacing the indoor/outdoor "mode":

  • Profiles (src/profiles.{h,cpp}): STATION / PORTABLE / STORMCHASE, all AFE=Outdoor. Auto-selected by WiFi association (Station when associated, Portable otherwise; 60 s boot-transient, 30 s debounce). STORMCHASE is manual-only. Applying a profile drives the chip via the settings:: setters; leaving STORMCHASE auto-restores a safe preset, so the post-storm flood cannot recur.
  • MIN_NUM_LIGHTNING as the false-alarm lever (the datasheet knob we'd never used): STATION runs chip MNL=1 (=5 strikes/15 min) so isolated noise never raises INT_L → HA stays clean. PORTABLE runs MNL=0 (every strike) + a firmware two-stage alert: quiet ACTIVITY indicator on early strikes, escalate to the SHELTER overlay on a close strike or ≥5 strikes/15 min.
  • Control surface: profile auto|station|portable|stormchase over CLI and the MQTT cmd topic; profile/profile_mode in the state JSON; an HA select entity; the TFT indoor/outdoor button repurposed into a profile selector (on-device control when off-WiFi). A simstrike N debug command injects synthetic strikes for testing.

Verified on hardware (OTA-flashed): boot auto-resolves Station (MNL=1 WD=2 SR=2 MASK=1 PIN=0); simstrike ×4 → ACTIVITY, 5th → SHELTER; Station ignores simstrikes (two-stage inactive); STORMCHASE = WD=1 SR=0 MNL=0 MASK=0 PIN=1; profile auto after stormchase restores the Station preset; state JSON carries profile/profile_mode; the HA select config is published retained. Built clean on C3 and C6. Remaining to confirm with eyes-on: the TFT profile button + main-screen profile label. Phase B (VBUS-gated power saving) not yet started.

Implemented via subagent-driven development; spec/quality review caught and fixed: a setMode(auto) creds-vs-association bug, an unvalidated NVS enum cast, a const external-linkage issue on AFE_OUTDOOR_GB, and a two-stage window-empty stage-reset / countdown bug.

Most recent session: 2026-05-27 — six real [L] from a cell over Graz (Wave 0 validated under a live storm), then the entire remote-ops refactor — Waves 1–5 — landed in one sitting between cells: WiFi, OTA, log mux, MQTT, and HA discovery. The device now publishes strikes + telemetry to the home broker and auto-registers in Home Assistant. Earlier sessions: 2026-05-22 (Wave 0), 2026-05-18 (first [L], single strike ~17 km), and before.

2026-05-27 — Wave 5 done (HA discovery); remote-ops COMPLETE

src/net/ha_discovery.{h,cpp}: publishes 5 retained Home Assistant discovery payloads (homeassistant/sensor|binary_sensor/<obj>/config) for strikes, last distance (km), last energy, RSSI (dBm), and a noise-fault binary sensor — all grouped under one HA device "Flashbee ", referencing flashbee/<id>/state + /availability. Published from mqtt tryConnect() once connected, gated by ha_disc on|off (NVS). Verified live: all 5 configs landed on broker 192.168.80.7 as valid retained JSON under device "Flashbee 8f9a50".

Remote-ops feature complete (Waves 0–5). End-to-end on the C6: USB-flash → wifi/ota_pw/mqtt/ha_disc over serial → joins WiFi, OTA-flashable over the air, publishes availability/state/event/log to MQTT, executes commands from the cmd topic, and auto-discovers in HA. All five post-Wave-0 waves were built, OTA/USB-flashed, and hardware-verified in this session, several of them while a storm cell sat directly overhead (so the very telemetry being wired up was carrying real [L] events to the broker as it was built).

Deferred (per spec / noted follow-ups): HA number/select entities for adjusting knobs from the dashboard (v1.1); captive-portal provisioning; TLS to broker; per-[D] event rate-limiting (a storm emits hundreds). Platforms still aren't version-pinned — pin both espressif32 (C3) and the pioarduino platform (C6) to stop Arduino-core drift (the stale C3 "stays on 2.x" comment is the warning sign).

2026-05-27 — Wave 4 done (MQTT) — IT PUSHES NOW

src/net/mqtt.{h,cpp} via PubSubClient. Publishes availability (LWT retained), retained state JSON every 10 s, per-event event JSON (L/D/NH), and a log topic mirroring netlog::LOG (with ring-buffer replay on connect). Subscribes to cmd and feeds it through cli::parseAndExec(line, LOG), so every serial command is reachable over MQTT and its output streams back on log. Broker is runtime- configurable via mqtt HOST [PORT] [USER] [PASS] → NVS; nothing hardcoded. Topic prefix flashbee/<id> (e.g. flashbee/8f9a50).

Two bugs fixed vs plan draft: (1) the extern globals (strikeCount/lastStrikeMs/lastEnergy/noiseFault/outdoorMode) must be at GLOBAL scope, not inside namespace netmqtt, or they name netmqtt::* and fail to link; (2) reused the existing global lastDistRaw for last_dist_km instead of the plan's redundant new lastDistanceKm.

Live verified against broker 192.168.80.7 (anonymous; the home broker, set at runtime — not hardcoded): availability online, retained state JSON, and live log+event streams during an active overhead cell. mosquitto_pub .../cmd -m 'stat' and -m 'maskon' executed on-device and round-tripped — the maskon flip showed up in the next retained state (mask:true). Broker-down resilience tested on a local throwaway broker (did NOT kill the production broker): device rode through a broker outage (uptime continuous), backed off, and auto-reconnected when the broker returned, resuming telemetry. LWT is registered (retained offline observed); it can't be seen by killing the broker itself (a dead broker can't deliver the will).

Watch-outs: one event is published per [D] disturber — during a storm that's hundreds of messages; fine for Mosquitto but consider rate-limiting/aggregation before it clutters HA. Broker creds for .7 are anonymous here; mqtt HOST PORT USER PASS supports auth when needed.

2026-05-27 — Wave 3 done (log multiplexer)

netlog::LOG (src/net/remote_log.{h,cpp}): a Print subclass that tees every write to Serial AND a 16 KB ring buffer, flushing one publish-hook callback per complete line (wired to MQTT in Wave 4) and offering replay() to walk the ring oldest→newest for connect-time history. Migrated the chip-event print sites — [L] strike, [D], [NH], [WDT] re-init, [afe] mode — from Serial.* to LOG.*. UI/touch/sleep debug stays Serial-only (noisy, not useful remotely); the CLOSE STRIKE alert also stays on Serial (the strike line carries [CLOSE]).

Bug fixed vs plan draft: the ring constant was named LINE_MAX, which is a POSIX <limits.h> macro on the C6 toolchain — renamed LOG_LINE_MAX. Verified after an OTA flash mid-storm: [D], strike #N, and [afe] lines still print identically on Serial (LOG tees), so the ring now collects them silently. Firmware ~440 KB, far under the 1.9 MB app slot.

2026-05-27 — Wave 2 done (OTA)

ArduinoOTA server in src/net/ota.{h,cpp}, gated by an NVS-stored password (ota_pw). Once WiFi is up it listens on flashbee-<id>.local:3232; empty password = OTA disabled. Wired into setup()/loop() after netwifi. CLI: ota_pw PW, ota_pw off. Added [env:*_ota] espota upload envs — the OTA password is kept OUT of the repo (placeholder --auth=CHANGEME; override per run with PLATFORMIO_UPLOAD_FLAGS=--auth=...). ArduinoOTA needed no extra build fix beyond Wave 1's Network -I/lib_deps.

Two bugs fixed vs the plan draft: (1) ota_pw off must be matched before the generic ota_pw prefix or it sets the password to the literal "off"; (2) loop() reprinted "no password set" every iteration on a connected device with no pw — added a one-shot s_disabled latch.

Live verified on C6: USB-flashed, ota_pw set, [ota] listening confirmed; an OTA upload over WiFi ran to 100 %, the device rebooted (SW_CPU) into the new image and re-listened; empty-pw logs the disabled line exactly once; a wrong --auth is rejected (Authenticating… FAIL). The USB-CDC serial monitor survives the OTA SW_CPU reboot.

2026-05-27 — Wave 1 done (WiFi)

Non-blocking STA state machine in src/net/wifi.{h,cpp}: reads creds from settings::, exponential backoff to 60 s on failure, mDNS flashbee-<id>.local on connect. Wired into setup()/loop(); netwifi::loop() runs before the sensor-down early-return so WiFi keeps servicing even while the AS3935 is being re-init. New CLI: wifi SSID PASS, wifi off, id. No creds → State::Disabled → behaves exactly as before (verified: a creds-less boot logs wifi_ssid NOT_FOUND and the detector runs untouched). Builds clean on both C3 and C6.

Build gotcha worth remembering. Arduino-ESP32 3.x split networking into a separate bundled Networking library (libraries/Network). WiFi.h includes Network.h unconditionally but declares no depends, so PlatformIO does nothing automatically. Two changes were both needed: lib_deps = Networking (compiles + links Network's .cpp) and -I .../libraries/Network/src (PIO compiles each lib against only its own declared deps' include paths, so WiFi's TUs otherwise can't find the header). Also discovered: the stock espressif32 platform the C3 uses is now on Arduino 3.x too — it hit the identical error and builds STA.cpp (3.x naming). The platformio.ini comment claiming "C3 stays on stock 2.x" is stale; neither platform is version-pinned, so they drift. Worth pinning both in a future cleanup.

Live verification on C6 (device 8f9a50, IP 192.168.80.179, RSSI ~−66): provisioned over serial, id → state=2; host pinged both the IP and flashbee-8f9a50.local (mDNS) at 0 % loss; reboot auto-rejoined from NVS; wifi off cleared creds and rebooted offline with the detector still working; a wrong password stayed out of Connected (4WAY_HANDSHAKE_TIMEOUT, state=1) and the correct one recovered to state=2. The AP-down/backoff acceptance item was substituted by the wrong-creds path (same failure→recovery FSM) — the home AP couldn't be cycled mid-storm. Incidental finding: a software ESP.restart() keeps the C6 USB-CDC enumerated, so the serial monitor survives a reboot.

Known nit (left as-is): wifi.cpp loop()'s 30 s per-attempt timeout reads s_nextTryMs, which startConnect() never sets to now — the window is measured from a stale reference. Functionally fine (first connect and reconnect both verified); just imprecise timing.

2026-05-27 — six [L] in one cell; Wave 0 pin fix validated

Resumed after a host crash with the bee still on USB. Device came up on its saved NVS state from Wave 0 (AFE=OUT WD=1 PIN=1); re-applied sr 0 + maskoff over serial and saved. Working profile for the session: NF=2 WD=1 SR=0 MASK=0 AFE=OUT PIN=1.

A storm cell tracked directly overhead. Six [L] in ~7 minutes, the project's first multi-strike capture and first under Wave 0 fw:

# dist energy
1 14 km 32653
2 12 km 43313
3 5 km 207564
4 1 km 336605
5 1 km 90081
6 1 km 4967

Distance/energy matched the on-device display to the digit (user confirmed 14 km/33k and 12 km/43k against the TFT) — AFE well calibrated at TUN_CAP=9.

The headline: 540 [D] disturbers during the cell and WD never moved off 1. Pre-Wave-0 the tightenFilters() ratchet would have walked WD to 10 within seconds of that flood and gone silent — exactly the failure mode of the 2026-05-09/10 session. pin on held it. This is the second storm (n>1) confirming OUTDOOR + WD=1 + SR=0, so the "bake WD=1/SR=0 into OUTDOOR init defaults" item below now has the evidence it was waiting for. No I²C glitches, no panics this session (contrast 2026-05-18's one-off ESP_ERR_INVALID_STATE).

Capture tooling: tools/bee_monitor.py opens the USB-CDC console without asserting DTR/RTS (one reset on open is unavoidable on the C6 USB-serial-JTAG), timestamps + tees to a logfile, and reads commands from a /tmp/bee_cmd FIFO so filters can be tuned live without reopening the port. This serial log is the de-facto remote log until Wave 4 (MQTT) lands. Session log: storm-2026-05-27.log.

2026-05-22 — Wave 0 done (settings registry + CLI extraction)

Plan: docs/superpowers/plans/2026-05-22-remote-ops.md. Spec: docs/superpowers/specs/2026-05-19-remote-ops-design.md.

Added a typed NVS-backed settings registry (src/settings.{h,cpp}) that owns NF/WD/SR/MNL/MASK/TUN/WDT/PIN. AFE_GB stays on the existing setAfeMode()/NVS_KEY_AFE path to avoid key collision. Extracted serial CLI dispatch into src/cli.{h,cpp} (cli::parseAndExec(line, Print&)), reused later by the MQTT cmd topic.

New CLI commands: nf N, mnl N, afe in|out, tuncap N, wdt N, pin on|off, save, reset. Existing checkint, maskoff, maskon, wd N, sr N, stat work identically.

pin on is the resolution of the progress.md ratchet bug. Verified live: with wd 1 / sr 0 / pin on / maskoff, four real [D] events arrived without WD getting bumped — first time the user-set filter values survive a storm. tightenFilters() / loosenFilters() / raise+lowerNoiseFloor() all short-circuit when pin=1. The 10-min software watchdog re-init via initAS3935() also re-applies settings::live instead of hardcoded NF=2/WD=2/SR=2, so saved values survive that path too.

SENSOR_WDT_MS (was a #define) is now settings::live.wdt_sec * 1000 so the re-init period is runtime-tunable via wdt N (60..3600 s).

settings::reset() preserves tuncap — it's antenna calibration, not a behavioural default.

Both C3 and C6 PIO envs build clean. Verified on C6 only.

2026-05-18 session — first real [L] of the project

Storm directly over Graz, audible thunder, visible lightning. Started the session matching the state in progress.md: AFE_GB_OUTDOOR, NF=2, WD=2, SR=2, MASK_DIST=1. Multi-hour silence pattern still present.

Live experiments, in order:

  1. maskoff → [D] flood within seconds. AFE was receiving RF the whole time, just suppressed.
  2. AFE → INDOOR via settings UI. [D] rate increased; still zero [L] despite strikes within ~5 km. Conclusion: indoor over-amplifies in this RF environment — chip classifier rejects every pulse as disturber.
  3. Flashed new CLI commands wd N / sr N (see below). Tried wd 1 + sr 0 + maskoff indoors — still no [L], only [D].
  4. AFE → OUTDOOR + WD=1 + SR=0 + MASK_DIST=1. First [L] of the session within ~minutes: strike #1 d=0x11 e=17022 (≈17 km).
  5. Subsequent visible flash (21:06:59) produced no [L] and no [D] — RF profile of any given strike is variable. Sample size too small to claim "OUTDOOR + WD=1/SR=0" is the setting, but it's the first combination that ever produced [L] from this hardware.

Key revision to prior hypothesis: the hours-of-silence in the 2026-05-09/10 entry was not AFE_GB_OUTDOOR being insensitive — it was OUTDOOR plus WD=2/SR=2 plus a not-quite-overhead storm. One notch looser on each filter (WD=1, SR=0) with OUTDOOR gave a real detection. INDOOR is the wrong direction for this location.

tightenFilters() ratchet, still alive. With MASK_DIST=0, every [D] runs the ratchet and bumps WD back up toward 10 (SR toward 11). That immediately undoes any user-set wd 1. Workaround during the session: maskon so disturbers are suppressed and tighten doesn't fire. Proper fix is a "filters pinned" flag set by wd/sr commands and respected by tightenFilters, but that's a follow-up.

One-off I2C glitch during the AFE-mode switch in settings:

[esp32-hal-i2c-ng.c:321] i2cRead(): i2c_master_receive failed:
  [259] ESP_ERR_INVALID_STATE
[Wire.cpp:527] requestFrom(): i2cRead returned Error 259

Recovered immediately (strike #1 came in seconds later). Single event — not yet a pattern. Touch (0x2E) and AS3935 (0x03) share the bus, contention plausible. Note in case it shows up again.

Touchscreen note: short flicks register as taps. Hit the swipe gesture more reliably with slower drags > ~150 ms. Not blocking; pre- existing UX issue.

Original symptom

Audible lightning happening, but the device only fired INT_L once shortly after power-on, then went silent for the rest of the cell. Touch / UI / I²C to the AS3935 all healthy throughout.

What's been done (chronological, in commits)

  1. f16747b as3935: mask disturbers, floor filters, add INT-path check

    • MASK_DIST=1 in normal operation (datasheet §8.5 — INT_D is informational, chip already rejects disturbers internally). Stops per-disturber tightenFilters() ratchet that walks WD/SR to 10/11 in seconds during real activity.
    • loosenFilters / lowerNoiseFloor floor at the AS3935 init defaults (2/2/2). Without the floor and with disturbers masked, they walked NF/WD/SR all the way to 0/0/0 — observed live.
    • Added checkint serial command: routes LCO to INT for one tune-window, prints Hz. Non-destructive (no NVS write). Proved live during the storm that the chip + INT line are healthy (3935 Hz, antenna in tune at TUN_CAP=9).

  2. afa8dcd cli: add maskoff/maskon/stat for live AS3935 debug

    • maskoff / maskon flip MASK_DIST at runtime. Note: maskoff is reverted by the 10-min initAS3935() watchdog re-init — this is intentional for now (it's a probe, not a setting).
    • stat dumps regs 0x00–0x03, 0x07, 0x08 + runtime state.

Current state of the diagnosis

Chip is alive and tuned but completely silent in normal operation in the current location. Multi-hour log on 2026-05-10: zero INT_L, zero INT_D, zero INT_NH — just the 10-min watchdog re-init pattern firing in a loop.

Key signal: INT_NH is not firing at NF=2. The chip isn't picking up enough RF to reach the noise threshold at all. If it were classifying real strikes as disturbers, we'd at least see [D] flooding during the maskoff window — we don't.

stat confirms the chip is configured exactly as intended:

[stat] AFE=0x1C NF_WDG=0x22 CLSTAT_SREJ=0xC2 LCO_INT=0x20 DIST=0x3F TUN=0x09
[stat] runtime: NF=2 WD=2 SR=2 AFE=OUT strikes=0

checkint consistently reports ~3935 Hz (expected 3906 Hz, well inside ±3.5% tolerance) — INT path good, antenna tuned, chip responsive.

Leading hypothesis (superseded — see 2026-05-18 above)

Old reading: AFE gain too low → flip to INDOOR. Updated reading: INDOOR over-amplifies in this RF environment and puts every pulse into the disturber bucket. OUTDOOR + WD=1/SR=0 is the first combination that produced [L]. The 2026-05-09/10 silence was the filter settings (WD=2/SR=2), not the AFE gain.

To try next

  1. Fix the tightenFilters() ratchet so wd/sr actually pin the filter level for the duration of a debug probe (see Stuff-to-clean- up). Currently wd 1 survives only until the next [D] when MASK_DIST=0. Workaround during a session: maskon + wd 1 + sr 0.
  2. Bake WD=1, SR=0 into the OUTDOOR init defaults? Tempting after tonight, but n=1 strike. Need at least one more storm to validate before changing the defaults.
  3. Antenna placement experiment — move the device near a window or outside the enclosure during a future storm and compare hit rate.

Stuff to clean up eventually (not blocking)

  • tightenFilters ratchet undoes user wd/sr commands. Need a filtersPinned flag set by the CLI commands and respected by tightenFilters / loosenFilters. Cleared on initAS3935().
  • maskoff reverts on watchdog re-init. Fine for a probe, but if we want it to stick we'd need a runtime flag honored by initAS3935().
  • The 10-min watchdog re-init also re-applies all init regs every time the chip is silent. That's correct for a wedged chip but spammy when the chip is just genuinely quiet. Possibly distinguish "chip not responding to I²C" from "chip responding but no IRQs".
  • Transient I²C ESP_ERR_INVALID_STATE during settings UI on 2026-05-18 — shared bus with touch (0x2E). Single event, watch for recurrence.
  • critical-issues.md is deleted in the working tree; decide whether to keep that deletion (was a planning doc, probably stale now).
  • resume/ directory is untracked — unrelated to this work.

Runtime serial commands (current)

cmd effect
checkint Verify INT line + antenna tune. Non-destructive.
maskoff Set MASK_DIST=0 (disturbers visible). Reverts on re-init.
maskon Set MASK_DIST=1 (default; disturbers suppressed).
wd N Set WDTH=N (0..15). Reverts on re-init AND on next [D]
if MASK_DIST=0 (tightenFilters ratchet — see above).
sr N Set SREJ=N (0..15). Same revert behaviour as wd.
stat Dump AS3935 regs + runtime state.

Aliases: checkint also accepts i.

Build / flash

pio run -e seeed_xiao_esp32c6 -t upload --upload-port /dev/ttyACM0

USB-CDC native — DTR/RTS reset works via pyserial. By-id path is /dev/serial/by-id/usb-Espressif_USB_JTAG_serial_debug_unit_98:A3:16:8F:9A:50-if00 (pinning logger to this avoids grabbing the wrong device when the port enumerates ahead of a Morserino-like sibling).