Repository navigation

60 smart meter analysis pricing simulation - #61

Open

griffinsharps wants to merge 149 commits into

mainfrom

60-smart-meter-analysis-pricing-simulation

Commits on Jan 26, 2026
- WIP Wide-to-long transformation code and accompanying runner and validation.
  Griffin Sharps
  committed
  971096f
Commits on Feb 4, 2026
- WIP on transform CSV scripts
  
  Griffin Sharps
  and
  cursoragent
  committed
  296cb57
Commits on Feb 5, 2026
- Add duplicate detection, row counts, and run artifact validation to month validator
  
  Griffin Sharps
  and
  claude
  committed
  a6758d9
- Fix Polars 1.38 compatibility: Expr.is_sorted() removed, use Series.is_sorted()
  
  Griffin Sharps
  and
  claude
  committed
  304e49a
- Fix sortedness bug: sink_parquet ignores sort, use collect+write_parquet
  Griffin Sharps
  committed
  49ae5b2
Commits on Feb 6, 2026
- Add migrate-month recipe for EC2 CSV-to-Parquet production runs
  
  Griffin Sharps
  and
  claude
  committed
  05356e5
- Add .txt and archive dirs to .gitignore, add secrets pre-commit guards
  
  Griffin Sharps
  and
  claude
  committed
  157db07
- Refactor validate_month_output.py for O(batch_size) memory streaming
  
  Griffin Sharps
  and
  claude
  committed
  0342d90
- Fix false sortedness failures in sample mode due to overlapping slices
  
  Griffin Sharps
  and
  claude
  committed
  9c50745
- Add migrate-month recipe to Justfile (based on main branch)
  
  Griffin Sharps
  and
  claude
  committed
  4890400
- Add professional annotations to CSV-to-Parquet pipeline files
  
  Griffin Sharps
  and
  claude
  committed
  a6a59dc
- Merge remote-tracking branch 'origin/main' into 43-convert-coned-meter-data-from-csv-to-parquet
  Griffin Sharps
  committed
  012e9cf
- Fix trailing whitespace in README files from main merge
  
  Griffin Sharps
  and
  claude
  committed
  c5d7157
Commits on Feb 9, 2026
- Add multi-month orchestration recipes to Justfile
  
  Griffin Sharps
  and
  claude
  committed
  da5843a
Commits on Feb 10, 2026
- Add rate_structures directory for tariff definitions
  Griffin Sharps
  committed
  31d37d3
- Added STOU YAML file to begin simulation testing
  Griffin Sharps
  committed
  a91613c
- Generalize billing to compare two generic hourly tariffs (A vs B)
  
  Griffin Sharps
  and
  claude
  committed
  a79be5f
- Add hourly tariff price calendar builder from YAML rate structure
  
  Griffin Sharps
  and
  claude
  committed
  a370788
- Add builder for hourly flat-rate price calendar
  Griffin Sharps
  committed
  fe68a1a
Commits on Feb 11, 2026
- Add end-to-end DST roll-in validator for RTP billing pipeline
  
  Griffin Sharps
  and
  claude
  committed
  1cc14ab
- Add BG regression dataset builder and multi-month billing orchestrator
  
  Griffin Sharps
  and
  claude
  committed
  49dfc60
- Add pricing simulation E2E tests and reviewer test plan; allow tracking tests/test_*.py
  Griffin Sharps
  committed
  64eb66b
- Add reviewer "why" annotations to pricing simulation pipeline
  
  Griffin Sharps
  and
  claude
  committed
  743e88f
Commits on Feb 16, 2026
- Update E2E tests for month-scoped bills and expanded regression outputs
  
  Griffin Sharps
  and
  claude
  committed
  d13a802
Commits on Feb 17, 2026
- Add deterministic month-level compaction stage with atomic swap and validation
  
  Griffin Sharps
  and
  cursoragent
  committed
  7285174
- Filter migrate-month inputs by filename month to tolerate mixed-month S3 prefixes
  Griffin Sharps
  committed
  4e0f3e4
- Run migrate_month_runner via uv-managed environment
  Griffin Sharps
  committed
  35ff63e
- Add --compact-no-swap mode; perform full compaction+validation without atomic swap
  Griffin Sharps
  committed
  286c4f7
Commits on Feb 18, 2026
- update .gitignore to explicitly avoid committing test parquets
  Griffin Sharps
  committed
  8f18d52
- Fix adjacent-key validation to avoid to_list() memory blowup
  Griffin Sharps
  committed
  9cc5234
- update .gitignore to explicitly avoid committing test parquets
  Griffin Sharps
  committed
  e0acf58
- rtp: support glob/directory input and multi-file parquet scan in compute_hourly_loads
  
  Griffin Sharps
  and
  cursoragent
  committed
  d84455b
Commits on Feb 19, 2026
- add BG correlation check, class-aware GeoJSON export, and account chain-of-custody audit
  
  Griffin Sharps
  and
  cursoragent
  committed
  f310997
- fix code-review issues in pricing_pilot analysis scripts
  
  Griffin Sharps
  and
  claude
  committed
  66a7531
- fix B007 lint error in validate_mf_esh_bg_count.py
  
  Griffin Sharps
  and
  claude
  committed
  5d594f2
- add wide_to_long unit tests; sync ruff to v0.14.4
  
  Griffin Sharps
  and
  claude
  committed
  6fc476b
Commits on Feb 20, 2026
- add build_account_bg_map.py: reproducible account→BG map builder
  
  Griffin Sharps
  and
  claude
  committed
  a2dc7b5
- add TOU window audit script and alignment unit tests
  
  Griffin Sharps
  and
  claude
  committed
  ccd166e
- enforce STOU/DTOU TOU window alignment at YAML and CI level
  
  Griffin Sharps
  and
  claude
  committed
  03772f6
- add per-class scatterplots and fix axis labeling in pricing pilot regression script
  
  Griffin Sharps
  and
  claude
  committed
  de16ecd
- cosmetic fixes to pricing pilot scatterplots
  
  Griffin Sharps
  and
  claude
  committed
  7379134
- fix stats box redundancy and Times New Roman font in scatterplots
  
  Griffin Sharps
  and
  claude
  committed
  98a4089
Commits on Feb 21, 2026
- include all block groups in GeoJSON export (left join, null-fill for no-data BGs)
  
  Griffin Sharps
  and
  claude
  committed
  d4265d3
- add --pct flag for percentage-change y-axis in pricing pilot regressions
  
  Griffin Sharps
  and
  claude
  committed
  e6bdbef
Commits on Feb 23, 2026
- GeoJSON export: global symmetric cap + fail-loud validation
  Griffin Sharps
  committed
  b7a22d6
- fix GeoJSON exporter sign convention: prefer bill_b - bill_a over bill_diff_dollars
  
  Griffin Sharps
  and
  claude
  committed
  2f35612
- filter GeoJSON export to Chicago ZIP-based BG universe
  
  Griffin Sharps
  and
  claude
  committed
  d00c135
- fix CI: restore wide_to_long module and update regression x-axis scale
  
  Griffin Sharps
  and
  claude
  committed
  3cd2331
Commits on Feb 24, 2026
- add 95% CI band to regression scatterplots; rename Rate BEST -> STOU
  
  Griffin Sharps
  and
  claude
  committed
  84ade66
- Replace _validate_adjacent_keys with streaming PyArrow iter_batches validator
  
  Griffin Sharps
  and
  claude
  committed
  b0a7480
Commits on Feb 26, 2026
- Cap rows_per_chunk at 50M to prevent OOM on dense months
  
  Griffin Sharps
  and
  claude
  committed
  4ef97da
- Refactor _stream_write_chunks to write multi-row-group files targeting 1 GiB
  
  Griffin Sharps
  and
  claude
  committed
  0937618
Commits on Feb 27, 2026
- Switch to plain YYYY/MM partition dirs, Spark part- naming, and add export restructure script
  
  Griffin Sharps
  and
  claude
  committed
  50416e6
Commits on Mar 2, 2026
- Commit AGENTS.md and ignore .cursor/ in .gitignore
  Griffin Sharps
  committed
  044dfdd
- Move edit_geojson.py from root to scripts/
  Griffin Sharps
  committed
  fef0ea6
- Restore code quality targets to Justfile from main
  Griffin Sharps
  committed
  6fcf966
- Archive deprecated Chicago visualization scripts
  
  Griffin Sharps
  and
  claude
  committed
  8e39d88
- Merge into csv/parquet to get new agent code.
  Griffin Sharps
  committed
  9557d48
- Fix sign convention docs in geojson export
  Griffin Sharps
  committed
  be03d98
- Fix delta fallback to match Flat − Alt convention
  Griffin Sharps
  committed
  4618690
- Add HC1 robust standard errors to BG OLS fit
  Griffin Sharps
  committed
  f16d757
- Use lazy scan for bills parquet load
  Griffin Sharps
  committed
  78f4321
- Use lazy scan for aggregate BG bill loading
  Griffin Sharps
  committed
  58bd196
Commits on Mar 3, 2026
- Fix pre-existing ruff lint warnings
  Griffin Sharps
  committed
  adf98ed
- Make ZIP filter optional in account-BG mapper
  Griffin Sharps
  committed
  c77e621
- Parameterize geo scope and add domain anchors in GeoJSON exporter
  Griffin Sharps
  committed
  08b8a27
- Archive superseded GeoJSON exporter
  Griffin Sharps
  committed
  1ce7b55
- Fix memory safety and rename paths in chain-of-custody script
  Griffin Sharps
  committed
  efa1921
- Sync lockfile with pyproject.toml
  Griffin Sharps
  committed
  2565ea1
- Fix sign convention docs and minor review findings
  Griffin Sharps
  committed
  097377e
- Remove redundant unique before n_unique in chain-of-custody
  Griffin Sharps
  committed
  55776d7
Commits on Mar 5, 2026
- Allow glob patterns in billing pipeline interval-pattern
  Griffin Sharps
  committed
  706214b
- Add chunked processing to compute_hourly_loads for OOM safety
  Griffin Sharps
  committed
  678b2ee
- Add sub-file row chunking to compute_hourly_loads
  Griffin Sharps
  committed
  65b933e
- Add multi-pass merge to re-aggregation step
  Griffin Sharps
  committed
  1de02c7
- Reduce batch size and merge fan-in for OOM safety
  Griffin Sharps
  committed
  0b91130
- Fix OOM by limiting Polars to single thread
  Griffin Sharps
  committed
  1ea44d2
- Use streaming sink_parquet for merge passes
  Griffin Sharps
  committed
  9969a4e
- Reduce batch size to 100k and fan-in to 8
  Griffin Sharps
  committed
  bff00e0
Commits on Mar 6, 2026
- Replace final merge pass with dict-based aggregation to avoid OOM
  Griffin Sharps
  committed
  b968643
- Shard final merge by account hash to fit in memory
  Griffin Sharps
  committed
  c514ec6
- fix: restore hourly aggregation in sharded merge and shard bill computation for memory safety
  Griffin Sharps
  committed
  36f07f1
- fix: correct STOU midday_peak prices, carry delivery_service_class through pipeline
  Griffin Sharps
  committed
  91ddb3d
- fix: preserve Categorical dtype for delivery_service_class in hourly loads merge pipeline
  Griffin Sharps
  committed
  7b2e7e5
- revert: remove delivery_service_class pipeline changes, keep corrected STOU prices
  Griffin Sharps
  committed
  fb32fc4
- Add output/ to .gitignore
  Griffin Sharps
  committed
  de325e1
- Remove dead files and clean up lint config
  Griffin Sharps
  committed
  7c9fae0
- Rewrite infra teardown scripts with state drift recovery
  Griffin Sharps
  committed
  7deba68
- Add pricing simulation report scaffold
  Griffin Sharps
  committed
  03aa091
- Fix mkdocs and deptry CI failures
  Griffin Sharps
  committed
  8b4bfd1
- Fix OOM in delivery class lookup with streaming PyArrow reader
  Griffin Sharps
  committed
  b36661f
Commits on Mar 7, 2026
- Add STOU delivery delta post-processing script
  Griffin Sharps
  committed
  a4245da
Commits on Mar 9, 2026
- Fix inverted sign convention in GeoJSON export and bill stats scripts
  Griffin Sharps
  committed
  302d2f0
- Fix hourly aggregation grain in sharded merge passes
  Griffin Sharps
  committed
  d4f73a5
- Reduce peak memory in delivery class lookup with incremental dedup
  Griffin Sharps
  committed
  7dfca22
- Cast delivery_service_class to Utf8 before dedup in lookup builder
  Griffin Sharps
  committed
  2675c17
- Standardize pricing pilot scripts against repo conventions
  Griffin Sharps
  committed
  0788c1b
- Fix OOM in delivery class lookup with set-based dedup
  Griffin Sharps
  committed
  67ddb2e
- Remove Polars lazy-scan path that OOMs before Python can catch
  Griffin Sharps
  committed
  10750a9
- Add DTOU results packaging script
  Griffin Sharps
  committed
  6d0e49e
Commits on Mar 10, 2026
- Add statewide account→BG map builder from combined Parquets
  Griffin Sharps
  committed
  c96c49f
- Add statewide BG-level pricing regression analysis script
  Griffin Sharps
  committed
  b38400f
- Fix log-income regression bug in statewide pricing analysis
  Griffin Sharps
  committed
  ecd89c9
- Add statewide GeoJSON export for pricing simulation maps
  Griffin Sharps
  committed
  b62e532
- Add phase 0 repo audit script
  Griffin Sharps
  committed
  b6a232e
Commits on Mar 11, 2026
- Update 2026 STOU rates and compute supply deltas inline
  Griffin Sharps
  committed
  6070e10
Commits on Mar 12, 2026
- Add income-vs-savings scatterplot generator from combined parquets
  Griffin Sharps
  committed
  9a42c96
- Add end-to-end pipeline audit with synthetic data verification
  Griffin Sharps
  committed
  25978b0
- Write per-class BG-level CSV in statewide analysis regression loop
  Griffin Sharps
  committed
  d878fd5
- Add 16 per-class scenarios to GeoJSON export
  Griffin Sharps
  committed
  a9fb692
- Add cost_change field to GeoJSON for Felt-friendly coloring
  Griffin Sharps
  committed
  47831ea
Commits on Mar 19, 2026
- Add income quintile analysis to statewide pipeline
  Griffin Sharps
  committed
  ba5cf72
- Draft 1 of memo in index.qmd
  Griffin Sharps
  committed
  a7b381e
Commits on Mar 20, 2026
- Updated index.qmd to full draft
  Griffin Sharps
  committed
  3f9a8d8
Commits on Mar 23, 2026
- Wire index.qmd setup block to pickle-backed report variables
  Griffin Sharps
  committed
  04b53cd
- devcontainer: switch from features to Dockerfile, raise fd limit
  Griffin Sharps
  committed
  1d6ee79
- devcontainer: fix stale yarn repo blocking chromium install
  Griffin Sharps
  committed
  5466d76
- devcontainer: set ulimit via runArgs
  Griffin Sharps
  committed
  2757c42
- devcontainer: create /workspaces dir
  Griffin Sharps
  committed
  8c4877f
- fix: devcontainer startup — move venv to /opt/venv, add WORKDIR, add plotnine
  Griffin Sharps
  committed
  7d63cea
- Replace analysis notebook with CUB TOU equity analysis
  Griffin Sharps
  committed
  ac096ef
- updated memo draft with correct python format to plug into notebooks/analysis.qmd
  Griffin Sharps
  committed
  d5a8c9f
- Fix deptry dependency issues in CI
  Griffin Sharps
  committed
  a24c5df
- Fix deptry dependency issues in CI
  
  Griffin Sharps
  and
  claude
  committed
  741aec4
Commits on Mar 24, 2026
- [smart-meter-analysis] Fix column names in analysis notebook
  Griffin Sharps
  committed
  473b47a
- [smart-meter-analysis] Add Quarto project config, references, and report targets
  Griffin Sharps
  committed
  55f2026
- Fix month filter comparisons to use integers in assign-regression-vars
  Griffin Sharps
  committed
  327b1f7
- Add Felt map iframes for DTOU and Rate BEST bill change maps
  Griffin Sharps
  committed
  be1f1dd
- add report_variables.pkl to cache, ignore .quarto build dir
  Griffin Sharps
  committed
  2e7baca
- WIP: proposed edits to report
  
  Max Shron (Switchbox)
  authored and
  Max Shron (Switchbox)
  committed
  d4268e8
- Merge branch '60-smart-meter-analysis-pricing-simulation' of https://github.com/switchbox-data/smart-meter-analysis into 60-smart-meter-analysis-pricing-simulation
  
  Max Shron (Switchbox)
  authored and
  Max Shron (Switchbox)
  committed
  997a7b6
- Add Switchbox shared lib (plotnine theme, quarto helpers) from reports2
  Griffin Sharps
  committed
  2105d4e
- Add build dependencies and uncommitted project files
  Griffin Sharps
  committed
  d2ad44b
- Merge branch '60-smart-meter-analysis-pricing-simulation' of https://github.com/switchbox-data/smart-meter-analysis into 60-smart-meter-analysis-pricing-simulation
  
  Max Shron (Switchbox)
  authored and
  Max Shron (Switchbox)
  committed
  ff89d5d
- Merge branch '60-smart-meter-analysis-pricing-simulation' of https://github.com/switchbox-data/smart-meter-analysis into 60-smart-meter-analysis-pricing-simulation
  
  Max Shron (Switchbox)
  authored and
  Max Shron (Switchbox)
  committed
  7cce8f9
Commits on Mar 26, 2026
- Memo draft 2 for review/comment
  Griffin Sharps
  committed
  fa9cae5
Commits on Mar 27, 2026
- Proposed edits to final draft
  Max Shron
  committed
  d04eddf
- Line edits for memo
  Griffin Sharps
  committed
  c163dea
- Updated visualization titles
  Griffin Sharps
  committed
  00ee6b0
Commits on Mar 30, 2026
- Add phase1 streaming audit script for production parquet outputs
  Griffin Sharps
  committed
  1d44756
- Vectorize inner loop: numpy comparisons + pyarrow group_by for acct-day counts
  Griffin Sharps
  committed
  859887f
Commits on Mar 31, 2026
- Split phase1 audit into --mode integrity and --mode acct-day
  Griffin Sharps
  committed
  6a994f7
- Optimize integrity audit to boundary-only checks (~300 comparisons vs ~20M)
  Griffin Sharps
  committed
  b5c2915
- Add phase4 S3 publish script for versioned parquet releases
  Griffin Sharps
  committed
  3b8183f
- Add compact_all_months wrapper to batch-compact production months
  Griffin Sharps
  committed
  829f7c5
- Fix needs_compaction path: use YYYY/MM not year=/month= Hive layout
  Griffin Sharps
  committed
  a5e4fbf
- Broaden parquet glob to *.parquet to pick up part-* files after compaction
  Griffin Sharps
  committed
  8a7239f
Commits on Apr 1, 2026
- Add BESTEC off-peak sensitivity script for CUB presentation
  Griffin Sharps
  committed
  68b3c29
Commits on Apr 8, 2026
- full draft after revenue neutrality adjustment
  Griffin Sharps
  committed
  a07ea81

Commits

Commits on Jan 26, 2026

Commits on Feb 4, 2026

Commits on Feb 5, 2026

Commits on Feb 6, 2026

Commits on Feb 9, 2026

Commits on Feb 10, 2026

Commits on Feb 11, 2026

Commits on Feb 16, 2026

Commits on Feb 17, 2026

Commits on Feb 18, 2026

Commits on Feb 19, 2026

Commits on Feb 20, 2026

Commits on Feb 21, 2026

Commits on Feb 23, 2026

Commits on Feb 24, 2026

Commits on Feb 26, 2026

Commits on Feb 27, 2026

Commits on Mar 2, 2026

Commits on Mar 3, 2026

Commits on Mar 5, 2026

Commits on Mar 6, 2026

Commits on Mar 7, 2026

Commits on Mar 9, 2026

Commits on Mar 10, 2026

Commits on Mar 11, 2026

Commits on Mar 12, 2026

Commits on Mar 19, 2026

Commits on Mar 20, 2026

Commits on Mar 23, 2026