Repository navigation
perf(state): retain bounded telemetry aggregates for EMS #1287
Description
Activity
The implementation has merged:
- feat(state): store bounded scalar history aggregates #1290: bounded 10-second scalar summaries, verified 1-minute/5-minute Parquet, restartable conversion, actual-resolution metadata and durable latest samples.
- feat(state): aggregate dashboard history after counting interval energy #1294: original-interval energy accounting before dashboard compaction, separate signed flows and explicit gaps, quarter-hour retention of older energy records, and backup/restore of the added tables.
- fix(state): keep maintenance bounded and retain forecast observations #1283: bounded maintenance and persisted forecast observations.
- fix(control): keep EV and model disk writes outside safety ticks #1300: EV/session/model persistence outside control locks and committed identity reads without per-tick SQL.
Full local
make verify, focused race tests and required CI passed. Isolated ARM64 tests on the Pi passed for the archive, energy accounting, vehicle goals and control writes. Existing backups and sources remain intact.Keep this issue open for acceptance step 5 and the installed transition. The final synthetic backup test still failed #1246 at 2.228 s against the unchanged 2 s durable-goal limit. No new beta or production install has taken place. A new candidate must still demonstrate growth, archive/backup load, the installed history transition and control behavior under that load. Code completion is not this target acceptance.
FTW should retain the history needed for local energy management, charging, forecasts and support without storing every driver poll indefinitely. Live telemetry and safety must keep their current freshness and cadence.
Current code buffers every scalar emission in
telemetry.Storeand persists it throughpersistTelemetryTick.RecentRetentionkeeps those rows for 14 days;RolloffToParquetcopies the same resolution to daily files. Dashboard JSON also keeps 30 days of hot snapshots. This amplifies live database growth, archive work and backup IO. See #1246, #1280 and #1259.Proposed fixed defaults for scalar/chart history:
Do not reduce the control or driver polling rate. Aggregate before writing scalar/chart history. Keep energy accounting separate: meter deltas, power integration, import/export direction, gaps and resets must use the original measurements before compaction. Forecast training uses its existing quarter-hour observations. Preserve settings, stable identities, session goals, command outcomes and fault transitions.
Gauge buckets need count, sum, min/max, last value/time, interval width and coverage; gap intervals stay gaps. Never average a cumulative meter into energy or a status code into a new state. Merge summaries by counts/coverage, not by averaging averages. Only retain fields that have a concrete product consumer; raw export to a user-owned store stays optional and must not become a dependency of Core.
Implement in stages:
Acceptance tests: irregular polling, one-second power spike, sign changes, cumulative counter reset, outage/reconnect, late correction, duplicate retry, restart mid-bucket, partial archive failure, mixed old/new archive reads and backup/restore across the transition. Energy and EV delivered totals must match the original-measurement reference. Diagnostics must retain the spike and transitions while empty periods remain missing.
This issue records the owner's direction and a proposed retention policy. The new aggregation policy has not shipped. Existing storage reliability fixes remain necessary during the transition.