diff --git a/context/README.md b/context/README.md index 79e71ac8..6b48c2b6 100644 --- a/context/README.md +++ b/context/README.md @@ -77,18 +77,18 @@ Cost-reflective TOU design: theory and window optimization. How we derive BAT marginal-cost inputs from regulatory studies. -| File | Purpose | -| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| ny_dist_mc_bat_methodology.md | Shared methodology for deriving sub-TX and distribution BAT marginal cost inputs from the seven NY MCOS workbooks: pipeline overview, key conceptual choices (incremental not cumulative, levelized 7-year window, no discounting, 2026 dollar year), per-utility deviations and harmonization, final values and sanity check, research appendices. | -| ny_mcos_studies_comparison.md | Cross-utility comparison of the seven NY 2025 MCOS studies: what MCOS numbers mean (undiluted/diluted, grid topology), methodology comparison, annualization mechanics (workbook-verified step-by-step), system-wide diluted/undiluted tables, component taxonomy mapping (physical hierarchy → cost-center labels), bulk vs local TX scrub, reassessment of bulk TX MC gap. | -| ny_bulk_transmission_marginal_cost.md | How to construct bulk TX marginal cost for NY BAT: the gap, data sources (OATT ATRR, CLCPA project costs, FERC Order 1920), five options with tradeoffs, recommended approach (OATT as upper-bound proxy), hourly PoP allocation. | -| ri_bulk_transmission_marginal_cost.md | How to construct bulk TX marginal cost for RI BAT: data sources (RNS rate, AESC 2024 avoided PTF cost, ISO-NE 2050 study), four options with tradeoffs, recommended approach (AESC primary, RNS upper bound), hourly PoP allocation, comparison with NY. | -| ct_bulk_transmission_marginal_cost.md | How to construct bulk TX marginal cost for CT BAT: same AESC 2024 $69/kW-year and top-100 exceedance as RI, plus `--allocation-load` flag to toggle between NE system aggregate and CT zone load for peak-hour identification; utilities `ct_eversource` and `ct_ui`; implementation files and Just commands. | -| capacity_market_comparison_nyiso_isone.md | Comparison of NYISO ICAP (Strip/Monthly/Spot) vs ISO-NE FCM (FCA/ARA/MRA) capacity market structures: structural analogy table, forward horizons, price volatility differences, implications for BAT marginal cost signal (Spot vs FCA vs MRA), current approach in NY vs RI, trade-offs. | -| dist_mc_definition_choice.md | Cross-cutting: which LRMC definition the `sub_tx_and_dist` BAT input uses, and why it differs by state. Taxonomy (Turvey / AIC-avoided / FLIC) and why AIC and FLIC differ ~10-50x, source-number table (NY FLIC $1-8, RI AESC $80, BGE Brattle $32, BGE E3 $203-258, OPC $25-34, BAT paper $54.46), what is implemented per state (NY=FLIC, RI/MD=avoided), the platform tension (NY-vs-RI/MD; methodology-doc §3-vs-§7), rationale, and BGE sensitivities. | -| md_bge_dist_mc_methodology.md | BGE (MD) sub-TX + distribution MC implementation: adopts Brattle avoided distribution cost $32/kW-yr (2022$), RI-style single-scalar config + CPI handling, why avoided cost is consistent with BGE's own peak-driven COS, the ~82% volumetric over-recovery intervention hook, FLIC/E3 sensitivities, and the Stage-B allocator MD-enablement dependency. | -| ct_eversource_dist_mc_methodology.md | eversource (CT) sub-TX + dist MC: how to derive CL&P/Eversource CT's `sub_tx_and_dist` MC from Exhibits CLP-MCOS-1/2. Stage A: system-wide diluted $20.17/kW-yr (Table 3) vs locational $86.58/kW-yr, why diluted mirrors NY's FLIC convention. Stage B: CL&P's own hourly probability-of-peak method (std-dev of 2022-2025 substation load, BTM solar/HP-adoption forecast adjustment) vs. what data we actually have (Table 2A TOU/season buckets only — Back-Up 36's raw PoP curve is listed in the TOC but not in the PDF); Tier 1 TOU-bucket mapping (implementable now) vs Tier 2 load-based PoP proxy (stretch); task list and open questions (UI gap, locational sensitivity, TOU-window confirmation). | -| md_bulk_transmission.md | How to construct bulk TX marginal cost for MD BAT: PJM NITS rates as OATT ATRR proxy (E3 ICC-VDER precedent), zone-to-utility mapping (BGE/DPL/PEPCO/APS covering all 13 MD utilities), calendar-year day-weighted blending, full-year top-150 PCAF allocation (following E3), reference data at `data/pjm/bulk_tx/nits/`, implementation in `bulk_tx_pjm.py`. | +| File | Purpose | +| ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| ny_dist_mc_bat_methodology.md | Shared methodology for deriving sub-TX and distribution BAT marginal cost inputs from the seven NY MCOS workbooks: pipeline overview, key conceptual choices (incremental not cumulative, levelized 7-year window, no discounting, 2026 dollar year), per-utility deviations and harmonization, final values and sanity check, research appendices. | +| ny_mcos_studies_comparison.md | Cross-utility comparison of the seven NY 2025 MCOS studies: what MCOS numbers mean (undiluted/diluted, grid topology), methodology comparison, annualization mechanics (workbook-verified step-by-step), system-wide diluted/undiluted tables, component taxonomy mapping (physical hierarchy → cost-center labels), bulk vs local TX scrub, reassessment of bulk TX MC gap. | +| ny_bulk_transmission_marginal_cost.md | How to construct bulk TX marginal cost for NY BAT: the gap, data sources (OATT ATRR, CLCPA project costs, FERC Order 1920), five options with tradeoffs, recommended approach (OATT as upper-bound proxy), hourly PoP allocation. | +| ri_bulk_transmission_marginal_cost.md | How to construct bulk TX marginal cost for RI BAT: data sources (RNS rate, AESC 2024 avoided PTF cost, ISO-NE 2050 study), four options with tradeoffs, recommended approach (AESC primary, RNS upper bound), hourly PoP allocation, comparison with NY. | +| ct_bulk_transmission_marginal_cost.md | How to construct bulk TX marginal cost for CT BAT: same AESC 2024 $69/kW-year and top-100 exceedance as RI, plus `--allocation-load` flag to toggle between NE system aggregate and CT zone load for peak-hour identification; utilities `ct_eversource` and `ct_ui`; implementation files and Just commands. | +| capacity_market_comparison_nyiso_isone.md | Comparison of NYISO ICAP (Strip/Monthly/Spot) vs ISO-NE FCM (FCA/ARA/MRA) capacity market structures: structural analogy table, forward horizons, price volatility differences, implications for BAT marginal cost signal (Spot vs FCA vs MRA), current approach in NY vs RI, trade-offs. | +| dist_mc_definition_choice.md | Cross-cutting: which LRMC definition the `sub_tx_and_dist` BAT input uses, and why it differs by state. Taxonomy (Turvey / AIC-avoided / FLIC) and why AIC and FLIC differ ~10-50x, source-number table (NY FLIC $1-8, RI AESC $80, BGE Brattle $32, BGE E3 $203-258, OPC $25-34, BAT paper $54.46), what is implemented per state (NY=FLIC, RI/MD=avoided), the platform tension (NY-vs-RI/MD; methodology-doc §3-vs-§7), rationale, and BGE sensitivities. | +| md_bge_dist_mc_methodology.md | BGE (MD) sub-TX + distribution MC implementation: adopts Brattle avoided distribution cost $32/kW-yr (2022$), RI-style single-scalar config + CPI handling, why avoided cost is consistent with BGE's own peak-driven COS, the ~82% volumetric over-recovery intervention hook, FLIC/E3 sensitivities, and the Stage-B allocator MD-enablement dependency. | +| ct_eversource_dist_mc_methodology.md | Eversource (CT) sub-TX + dist MC. Stage A: MCOS-2 Table 3 system-wide diluted $20.17/kW-yr vs locational $86.58/kW-yr. Default 8760 (§6): OCC-863 Attachment 2 secondary month-hour $/kWh tables expanded by weekday/weekend, with `include_holidays` (default true; federal holidays, Saturday observed Friday and Sunday observed Monday). One `--year` for the calendar, timestamps, and Hive partition, written to `dist_and_sub_tx_eversource_derived/`. Fallback (§3): PoP of the $20.17 figure to `dist_and_sub_tx/` when `use_eversource_marginal_cost=false`. | +| md_bulk_transmission.md | How to construct bulk TX marginal cost for MD BAT: PJM NITS rates as OATT ATRR proxy (E3 ICC-VDER precedent), zone-to-utility mapping (BGE/DPL/PEPCO/APS covering all 13 MD utilities), calendar-year day-weighted blending, full-year top-150 PCAF allocation (following E3), reference data at `data/pjm/bulk_tx/nits/`, implementation in `bulk_tx_pjm.py`. | ### methods/ (root) diff --git a/context/methods/marginal_costs/ct_eversource_dist_mc_methodology.md b/context/methods/marginal_costs/ct_eversource_dist_mc_methodology.md index 6501afec..313ae285 100644 --- a/context/methods/marginal_costs/ct_eversource_dist_mc_methodology.md +++ b/context/methods/marginal_costs/ct_eversource_dist_mc_methodology.md @@ -2,11 +2,13 @@ **Status: IMPLEMENTED.** This document lays out how the `sub_tx_and_dist` BAT marginal-cost input for Eversource CT (CL&P) is derived from the two MCOS exhibits, and how it is allocated to an 8760 -hourly signal. The config CSV, allocator wiring, and S3 output described below exist and have been -run; see §3.4 for the implementation notes (including two upstream data gaps — ISO-NE utility-level -load and 2026 CPI data — that had to be filled first). Only CT **bulk transmission** MC was -implemented before this -([ct_bulk_transmission_marginal_cost.md](ct_bulk_transmission_marginal_cost.md)). +hourly signal. Two 8760s exist. The default `create-dist-mc-data` recipe expands Eversource's +OCC-863 Attachment 2 month-hour `$/kWh` tables (§6). The older path PoP-allocates the MCOS-2 +Table 3 `$20.17/kW-yr` figure (§3) and is what `use_eversource_marginal_cost=false` still runs. +The config CSV, allocator wiring, and S3 output described below exist and have been run; see §3.4 +for the PoP implementation notes (including two upstream data gaps — ISO-NE utility-level load and +2026 CPI data — that had to be filled first). Only CT **bulk transmission** MC was implemented +before this ([ct_bulk_transmission_marginal_cost.md](ct_bulk_transmission_marginal_cost.md)). For the underlying LRMC framework and cross-state definition choice, see [dist_mc_definition_choice.md](dist_mc_definition_choice.md). For the BGE/RI implementation pattern @@ -100,6 +102,12 @@ the existing CPI-inflation logic in `generate_utility_tx_dx_mc.py` handle any ru ## 3. Stage B — hourly allocation via Probability of Peak (PoP) +This is the fallback, not the default. `create-dist-mc-data` runs it only when +`use_eversource_marginal_cost=false` (or `USE_EVERSOURCE_MARGINAL_COST=false` through +`just s ct`). It writes `dist_and_sub_tx/utility=ct_eversource/year=/data.parquet`. +The default recipe is the OCC-863 month-hour expansion in §6, which does not use the +`$20.17/kW-yr` figure or the load curve. + ### 3.1 Method: standard PoP allocation (same as NY, RI, MD) Use the same Probability of Peak (PoP) allocation method already implemented in @@ -269,7 +277,82 @@ allocation or as a direct replacement. This is a nice-to-have, not a blocker for `tests/test_ct_dist_mc.py`. 6. ~~Update this doc and [dist_mc_definition_choice.md](dist_mc_definition_choice.md) §2–3, adding CT to the source-number and per-state tables.~~ Done. -7. **Remaining**: wire `path_dist_and_sub_tx_mc` into a CT scenario config (`scenarios_ct_eversource.yaml` - or equivalent) once CT scenario YAMLs exist, so a CAIRO run actually consumes this MC output. No CT - scenario configs exist yet in `rate_design/hp_rates/ct/config/scenarios/` — that's a separate, - larger piece of CT onboarding beyond this MC-generation task. +7. **CAIRO consumption.** `pipeline_ct_eversource.yaml` points + `marginal_costs.dist_and_sub_tx` at + `s3://data.sb/switchbox/marginal_costs/ct/dist_and_sub_tx/utility=ct_eversource/year=2026/data.parquet`. + That file predates the §6 path split: its timestamps are the 2025 calendar and its `year` + column is 2026. Batch `ct_20261007_b` read it. New default runs of `create-dist-mc-data` write + §6's `dist_and_sub_tx_eversource_derived/` path instead and do not replace this file. + +--- + +## 6. OCC-863 Attachment 2 month-hour 8760 (current default) + +`create-dist-mc-data` defaults to `--use-eversource-marginal-cost true`. It does not +PoP-allocate §2's `$20.17/kW-yr`. It looks up each hour of one calendar year in Eversource's +OCC-863 Attachment 2 secondary month-hour `$/kWh` tables (Docket 26-05-10) and writes an 8760 +in the same schema as the PoP file (`timestamp`, `utility`, `year`, `mc_total_per_kwh`). + +The tables live at +`s3://data.sb/switchbox/sources/ct/eversource/docket-26-05-10/occ-863/parquet/`: + +- `month_hour_marginal_costs_weekday_secondary.parquet` +- `month_hour_marginal_costs_weekend_holidays_secondary.parquet` + +Each has 288 rows (`month` 1–12, `hour` 0–23, `num_days`, `mc_total_per_kwh`). The recipe uses +the **secondary** voltage pair. Implementation is `expand_month_hour_mc_to_8760` in +`utils/data_prep/marginal_costs/generate_utility_tx_dx_mc.py`. + +### 6.1 Day type + +Saturday and Sunday take the weekend-and-holiday table. Every other Monday–Friday takes the +weekday table, except federal holidays when holidays are included. + +`include_holidays` defaults to `true`. Those holidays are the eleven dates in 5 U.S.C. § 6103. +A fixed-date holiday that falls on Saturday is observed the Friday before, and one that falls +on Sunday is observed the Monday after. With `include_holidays=false`, a holiday on +Monday–Friday stays on the weekday table; Saturday and Sunday are unchanged. + +```bash +just -f ct/Justfile create-dist-mc-data 2025 --upload +just -f ct/Justfile include_holidays=false create-dist-mc-data 2025 --upload +INCLUDE_HOLIDAYS=false just s ct create-dist-mc-data 2025 --upload +``` + +### 6.2 One year for the calendar, the timestamps, and the partition + +`--year` (the recipe's `year_arg`) is the only year. It chooses the calendar used to classify +weekdays, weekends, and holidays, and it is stamped on `timestamp`, on the `year` column, and +on the Hive partition `year=YYYY`. There is no separate partition year. + +CAIRO does not join this file to the run on the timestamp year. When both series have 8,760 +hours, `utils/cairo.py`'s `_align_mc_to_index` copies the marginal-cost values by position onto +the run's hours. Hour 0 of the file lands on January 1 at midnight of the run. For weekends and +holidays to fall on the right days, `--year` has to be the run's calendar. The CT pipeline year +is 2025. A leap year still emits 8,760 rows (December 31 is dropped), but its positions diverge +from a non-leap run after February 28. + +### 6.3 Where it is written + +With the flag true, the script rewrites an output base that ends in `dist_and_sub_tx/` to +`dist_and_sub_tx_eversource_derived/`. The recipe still passes the `dist_and_sub_tx/` base; the +redirect is what keeps this file off the PoP path. + +```text +s3://data.sb/switchbox/marginal_costs/ct/dist_and_sub_tx_eversource_derived/utility=ct_eversource/year=YYYY/data.parquet +/ebs/data/switchbox/marginal_costs/ct/dist_and_sub_tx_eversource_derived/utility=ct_eversource/year=YYYY/data.parquet +``` + +`use_eversource_marginal_cost=false` leaves the base alone and writes the §3 PoP file: + +```text +s3://data.sb/switchbox/marginal_costs/ct/dist_and_sub_tx/utility=ct_eversource/year=YYYY/data.parquet +``` + +```bash +just -f ct/Justfile use_eversource_marginal_cost=false create-dist-mc-data 2025 --upload +USE_EVERSOURCE_MARGINAL_COST=false just s ct create-dist-mc-data 2025 --upload +``` + +Tests for the lookup, the observed-holiday rule, the holiday toggle, leap-year length, and the +output-base rewrite are in `tests/test_expand_month_hour_mc.py`. diff --git a/context/methods/marginal_costs/dist_mc_definition_choice.md b/context/methods/marginal_costs/dist_mc_definition_choice.md index 0ee632de..ee408803 100644 --- a/context/methods/marginal_costs/dist_mc_definition_choice.md +++ b/context/methods/marginal_costs/dist_mc_definition_choice.md @@ -88,10 +88,13 @@ locational figure (`$86.58`) is itself much larger than NY's undiluted MCOS figu | **MD (BGE)** | **AIC / avoided** (Brattle replacement-value ÷ peak) | 32 (2022$) | BGE's own COS classifies the plant 100% peak-driven; BGE's own marginal numbers use a peak denominator (see §5) | | CT (ct_eversource / CL&P) | **FLIC-like** (system-wide diluted from MCOS-2 Table 3) | 20.17 (2026$) | CL&P's own MCOS dilutes locational cost by expansion-area load share, structurally the same move as NY's diluted FLIC (see [ct_eversource_dist_mc_methodology.md](ct_eversource_dist_mc_methodology.md) §2) | -All four feed the _same_ PoP allocator +NY, RI, and MD feed the same PoP allocator ([generate_utility_tx_dx_mc.py](../../../utils/data_prep/marginal_costs/generate_utility_tx_dx_mc.py)) in the identical `sub_tx_and_dist_mc_kw_yr` slot, with CPI inflation via the optional `dollar_year` -column. +column. CT's `$20.17` figure is that allocator's input only when +`use_eversource_marginal_cost=false`. The default CT 8760 is the OCC-863 Attachment 2 +month-hour expansion, which does not use this scalar +([ct_eversource_dist_mc_methodology.md](ct_eversource_dist_mc_methodology.md) §6). --- diff --git a/rate_design/hp_rates/ct/Justfile b/rate_design/hp_rates/ct/Justfile index e9ad9ada..6ab6356b 100644 --- a/rate_design/hp_rates/ct/Justfile +++ b/rate_design/hp_rates/ct/Justfile @@ -14,6 +14,10 @@ set dotenv-path := "state.env" state := env_var_or_default('STATE', 'ct') path_s3_utility_loads := "s3://data.sb/isone/hourly_demand/utilities/" +path_s3_occ863_parquet := "s3://data.sb/switchbox/sources/ct/eversource/docket-26-05-10/occ-863/parquet" +path_local_dist_mc := "/ebs/data/switchbox/marginal_costs/ct/dist_and_sub_tx/" +use_eversource_marginal_cost := env_var_or_default('USE_EVERSOURCE_MARGINAL_COST', 'true') +include_holidays := env_var_or_default('INCLUDE_HOLIDAYS', 'true') import '../Justfile' @@ -126,8 +130,19 @@ create-bulk-tx-mc-data-all *extra_args: # Generate dist+sub-tx MC for ct_eversource and one year. # -# just -f ct/Justfile create-dist-mc-data 2025 -# just -f ct/Justfile create-dist-mc-data 2025 --upload +# By default the 8760 comes from Eversource's OCC-863 Attachment 2 secondary +# month-hour $/kWh tables: weekends take the weekend table, and federal +# holidays do too unless include_holidays=false. Other days take the weekday +# table, on year_arg's calendar. That same year is stamped on the timestamps +# and on the Hive partition. The file is written under +# dist_and_sub_tx_eversource_derived/, so it never overwrites the PoP 8760. +# Set use_eversource_marginal_cost=false for the PoP allocation of the +# $/kW-yr value, written under dist_and_sub_tx/year=year_arg. +# +# just -f ct/Justfile create-dist-mc-data 2025 --upload +# just -f ct/Justfile use_eversource_marginal_cost=false create-dist-mc-data 2025 --upload +# just -f ct/Justfile include_holidays=false create-dist-mc-data 2025 --upload +# USE_EVERSOURCE_MARGINAL_COST=false just s ct create-dist-mc-data 2025 --upload create-dist-mc-data year_arg *extra_args: cd {{ path_repo }} && uv run python {{ path_repo }}/utils/data_prep/marginal_costs/generate_utility_tx_dx_mc.py \ --state CT \ @@ -137,7 +152,12 @@ create-dist-mc-data year_arg *extra_args: --mc-table-path {{ path_config }}/marginal_costs/ct_marginal_costs_{{ year_arg }}.csv \ --utility-load-s3-base {{ path_s3_utility_loads }} \ --output-s3-base s3://data.sb/switchbox/marginal_costs/ct/dist_and_sub_tx/ \ + --path-local-output-base {{ path_local_dist_mc }} \ --n-hours {{ upstream_hours }} \ + --use-eversource-marginal-cost {{ use_eversource_marginal_cost }} \ + --path-weekday-mc {{ path_s3_occ863_parquet }}/month_hour_marginal_costs_weekday_secondary.parquet \ + --path-weekend-mc {{ path_s3_occ863_parquet }}/month_hour_marginal_costs_weekend_holidays_secondary.parquet \ + --include-holidays {{ include_holidays }} \ {{ extra_args }} # ============================================================================= diff --git a/rate_design/hp_rates/ct/config/scenarios/pipeline_ct_eversource.yaml b/rate_design/hp_rates/ct/config/scenarios/pipeline_ct_eversource.yaml index 5e20e4e2..0b845486 100644 --- a/rate_design/hp_rates/ct/config/scenarios/pipeline_ct_eversource.yaml +++ b/rate_design/hp_rates/ct/config/scenarios/pipeline_ct_eversource.yaml @@ -56,70 +56,70 @@ scenarios: # Fair seasonal rate: summer at base Rate 1, winter adjusted for annual constraint. quartet: single_rate_uncalibrated tariff_base: icos_fair_seasonal - # hp_seasonal_percustomer_passthrough: - # quartet: multi_rate_collapsed - # depends_on: default - # promote: hp - # residual_allocation: - # delivery: percustomer - # supply: passthrough - # subclass_config: - # group_col: has_hp - # subgroups: - # hp: - # values: ["true"] - # structure: seasonal - # non-hp: - # values: ["false"] - # structure: base - # hp_seasonal_epmc_passthrough: - # quartet: multi_rate_collapsed - # depends_on: default - # promote: hp - # residual_allocation: - # delivery: epmc - # supply: passthrough - # subclass_config: - # group_col: has_hp - # subgroups: - # hp: - # values: ["true"] - # structure: seasonal - # non-hp: - # values: ["false"] - # structure: base - # hp_flat_percustomer_passthrough: - # quartet: multi_rate_collapsed - # depends_on: default - # promote: hp - # residual_allocation: - # delivery: percustomer - # supply: passthrough - # subclass_config: - # group_col: has_hp - # subgroups: - # hp: - # values: ["true"] - # structure: flat - # non-hp: - # values: ["false"] - # structure: base - # hp_flat_epmc_passthrough: - # quartet: multi_rate_collapsed - # depends_on: default - # promote: hp - # residual_allocation: - # delivery: epmc - # supply: passthrough - # subclass_config: - # group_col: has_hp - # subgroups: - # hp: - # values: ["true"] - # structure: flat - # non-hp: - # values: ["false"] - # structure: base + # hp_seasonal_percustomer_passthrough: + # quartet: multi_rate_collapsed + # depends_on: default + # promote: hp + # residual_allocation: + # delivery: percustomer + # supply: passthrough + # subclass_config: + # group_col: has_hp + # subgroups: + # hp: + # values: ["true"] + # structure: seasonal + # non-hp: + # values: ["false"] + # structure: base + # hp_seasonal_epmc_passthrough: + # quartet: multi_rate_collapsed + # depends_on: default + # promote: hp + # residual_allocation: + # delivery: epmc + # supply: passthrough + # subclass_config: + # group_col: has_hp + # subgroups: + # hp: + # values: ["true"] + # structure: seasonal + # non-hp: + # values: ["false"] + # structure: base + # hp_flat_percustomer_passthrough: + # quartet: multi_rate_collapsed + # depends_on: default + # promote: hp + # residual_allocation: + # delivery: percustomer + # supply: passthrough + # subclass_config: + # group_col: has_hp + # subgroups: + # hp: + # values: ["true"] + # structure: flat + # non-hp: + # values: ["false"] + # structure: base + # hp_flat_epmc_passthrough: + # quartet: multi_rate_collapsed + # depends_on: default + # promote: hp + # residual_allocation: + # delivery: epmc + # supply: passthrough + # subclass_config: + # group_col: has_hp + # subgroups: + # hp: + # values: ["true"] + # structure: flat + # non-hp: + # values: ["false"] + # structure: base # The one stage every other run's bills are compared against, joined onto the # master tables as baseline_elec_* columns by post-processing. bill_change_baseline: diff --git a/rate_design/hp_rates/ct/config/tariffs/electric/ct_eversource_tariff_summary.yaml b/rate_design/hp_rates/ct/config/tariffs/electric/ct_eversource_tariff_summary.yaml index 0b9d6d49..61fd1493 100644 --- a/rate_design/hp_rates/ct/config/tariffs/electric/ct_eversource_tariff_summary.yaml +++ b/rate_design/hp_rates/ct/config/tariffs/electric/ct_eversource_tariff_summary.yaml @@ -5,40 +5,35 @@ # (filed tariff schedules) and Exhibit CLP-RATES-6 (Rate 6 worksheet). # # Provenance doc: context/code/data/ct_eversource_custom_rates.md - # ───────────────────────────────────────────────────────────────────────────── # Shared delivery riders — flat per-kWh, identical for Rate 1 and all Rate 6 # variants. These are pass-through adjustment clauses set outside the rate case. # ───────────────────────────────────────────────────────────────────────────── shared_riders: - transmission: 0.05050 # Exh CLP-RATES-3 p.243 / p.249 - competitive_transition_assessment: 0.00496 # Exh CLP-RATES-3 p.243 / p.249 - systems_benefits_charge: -0.00196 # Exh CLP-RATES-3 p.243 / p.249 (credit) - conservation_adjustment_mechanism: 0.00600 # Exh CLP-RATES-3 p.243 / p.249 - renewable_energy: 0.00100 # Exh CLP-RATES-3 p.243 / p.249 - fmcc_delivery: -0.01911 # Exh CLP-RATES-3 p.243 / p.249 (credit) - electric_system_improvements: 0.00000 # Rolled into distribution (ESI sunsets) - revenue_decoupling_mechanism: 0.00000 # Rolled into distribution - total: 0.04139 # = sum of above - + transmission: 0.05050 # Exh CLP-RATES-3 p.243 / p.249 + competitive_transition_assessment: 0.00496 # Exh CLP-RATES-3 p.243 / p.249 + systems_benefits_charge: -0.00196 # Exh CLP-RATES-3 p.243 / p.249 (credit) + conservation_adjustment_mechanism: 0.00600 # Exh CLP-RATES-3 p.243 / p.249 + renewable_energy: 0.00100 # Exh CLP-RATES-3 p.243 / p.249 + fmcc_delivery: -0.01911 # Exh CLP-RATES-3 p.243 / p.249 (credit) + electric_system_improvements: 0.00000 # Rolled into distribution (ESI sunsets) + revenue_decoupling_mechanism: 0.00000 # Rolled into distribution + total: 0.04139 # = sum of above # Supply commodity (Genability 2025 seasonal, same for all tariffs) supply_commodity: - jan_jun: 0.11190 # $/kWh - jul_dec: 0.09748 # $/kWh - + jan_jun: 0.11190 # $/kWh + jul_dec: 0.09748 # $/kWh # ───────────────────────────────────────────────────────────────────────────── # Rate 1 Proposed — ct_eversource_rate1_proposed # Flat volumetric, no seasons, no tiers. # Source: Exh CLP-RATES-3 p.243; Exh CLP-RATES-6 p.3 line 172 # ───────────────────────────────────────────────────────────────────────────── rate1_proposed: - fixed_charge_per_month: 12.36 # Exh CLP-RATES-2.4 R1 RY Proposed+GET Customer Charge - + fixed_charge_per_month: 12.36 # Exh CLP-RATES-2.4 R1 RY Proposed+GET Customer Charge all_months_all_kwh: - distribution: 0.12196 # incl. GET; ESI/RDM folded in - riders: 0.04139 # shared_riders.total - delivery_total: 0.16335 # = distribution + riders → JSON rate - + distribution: 0.12196 # incl. GET; ESI/RDM folded in + riders: 0.04139 # shared_riders.total + delivery_total: 0.16335 # = distribution + riders → JSON rate # ───────────────────────────────────────────────────────────────────────────── # Rate 6 Filed — ct_eversource_rate6 # Seasonal tiered: winter (Nov–Mar) Block 1 ≤700 kWh / Block 2 >700 kWh; @@ -46,27 +41,23 @@ rate1_proposed: # Source: Exh CLP-RATES-3 p.249; Exh CLP-RATES-6 p.2 # ───────────────────────────────────────────────────────────────────────────── rate6_filed: - fixed_charge_per_month: 30.95 # MCC $28.98 × 1.068 GET - - winter_block1_le_700kwh: # Nov–Mar, first 700 kWh - distribution: 0.09431 # Class EC RR $796,054,171 / 8,440,783,007 kWh - riders: 0.04139 - delivery_total: 0.13570 - - winter_block2_gt_700kwh: # Nov–Mar, above 700 kWh - distribution: 0.07517 # Block 1 $0.09431 − discount $0.01914 - riders: 0.04139 - delivery_total: 0.11656 + fixed_charge_per_month: 30.95 # MCC $28.98 × 1.068 GET + winter_block1_le_700kwh: # Nov–Mar, first 700 kWh + distribution: 0.09431 # Class EC RR $796,054,171 / 8,440,783,007 kWh + riders: 0.04139 + delivery_total: 0.13570 + winter_block2_gt_700kwh: # Nov–Mar, above 700 kWh + distribution: 0.07517 # Block 1 $0.09431 − discount $0.01914 + riders: 0.04139 + delivery_total: 0.11656 # Discount build-up (Exh CLP-RATES-6 p.2): # MC subtotal = $0.04152 (FC recovery w/GET $0.03894 + MC Energy w/GET $0.00258) # Reconciliation = $0.03365 (residual $0.05279 × coinc. factor 63.7%) # Block 2 = MC subtotal + reconciliation = $0.07517 - - non_heating_all_kwh: # Apr–Oct, all kWh - distribution: 0.09431 - riders: 0.04139 - delivery_total: 0.13570 - + non_heating_all_kwh: # Apr–Oct, all kWh + distribution: 0.09431 + riders: 0.04139 + delivery_total: 0.13570 # ───────────────────────────────────────────────────────────────────────────── # Rate 6 RATE=MC — ct_eversource_rate6_ratemc # Same as filed Rate 6, but coincidence-factor adjustment set to 0%, @@ -74,23 +65,19 @@ rate6_filed: # Source: Google Sheets tab "CLP-RATES-6, 2 of 3 (RATE=MC)" # ───────────────────────────────────────────────────────────────────────────── rate6_ratemc: - fixed_charge_per_month: 30.95 # unchanged - + fixed_charge_per_month: 30.95 # unchanged winter_block1_le_700kwh: - distribution: 0.09431 # unchanged - riders: 0.04139 - delivery_total: 0.13570 - + distribution: 0.09431 # unchanged + riders: 0.04139 + delivery_total: 0.13570 winter_block2_gt_700kwh: - distribution: 0.04152 # = MC subtotal only (FC recovery $0.03894 + MC Energy $0.00258) - riders: 0.04139 # reconciliation adj zeroed (coinc. factor = 0%) - delivery_total: 0.08291 - + distribution: 0.04152 # = MC subtotal only (FC recovery $0.03894 + MC Energy $0.00258) + riders: 0.04139 # reconciliation adj zeroed (coinc. factor = 0%) + delivery_total: 0.08291 non_heating_all_kwh: - distribution: 0.09431 - riders: 0.04139 - delivery_total: 0.13570 - + distribution: 0.09431 + riders: 0.04139 + delivery_total: 0.13570 # ───────────────────────────────────────────────────────────────────────────── # Rate 6 FIXED+RATE=MC — ct_eversource_rate6_fixed_ratemc # Marginal Facilities Cost ($24.51/mo) moved into fixed charge; @@ -98,22 +85,19 @@ rate6_ratemc: # Source: Google Sheets tab "CLP-RATES-6, 2 of 3 (FIXED + RATE=MC)" # ───────────────────────────────────────────────────────────────────────────── rate6_fixed_ratemc: - fixed_charge_per_month: 57.13 # ($28.98 MCC + $24.51 MFC) × 1.068 GET - + fixed_charge_per_month: 57.13 # ($28.98 MCC + $24.51 MFC) × 1.068 GET winter_block1_le_700kwh: - distribution: 0.05537 # EC RR $467,385,662 / 8,440,783,007 kWh - riders: 0.04139 - delivery_total: 0.09676 - + distribution: 0.05537 # EC RR $467,385,662 / 8,440,783,007 kWh + riders: 0.04139 + delivery_total: 0.09676 winter_block2_gt_700kwh: - distribution: 0.00258 # = MC Energy w/GET only (FC recovery in fixed charge, recon = 0) - riders: 0.04139 - delivery_total: 0.04397 - + distribution: 0.00258 # = MC Energy w/GET only (FC recovery in fixed charge, recon = 0) + riders: 0.04139 + delivery_total: 0.04397 non_heating_all_kwh: - distribution: 0.05537 - riders: 0.04139 - delivery_total: 0.09676 + distribution: 0.05537 + riders: 0.04139 + delivery_total: 0.09676 # ───────────────────────────────────────────────────────────────────────────── # Side-by-side summary (delivery only) diff --git a/rate_design/hp_rates/ct/config/tariffs/electric/custom_rates/ct_eversource_icos_fair_seasonal.yaml b/rate_design/hp_rates/ct/config/tariffs/electric/custom_rates/ct_eversource_icos_fair_seasonal.yaml index fd3771e1..8a8616d1 100644 --- a/rate_design/hp_rates/ct/config/tariffs/electric/custom_rates/ct_eversource_icos_fair_seasonal.yaml +++ b/rate_design/hp_rates/ct/config/tariffs/electric/custom_rates/ct_eversource_icos_fair_seasonal.yaml @@ -8,24 +8,24 @@ fixed_charge: 12.36 seasons: winter: months: - - 12 - - 1 - - 2 - - 3 - - 11 + - 12 + - 1 + - 2 + - 3 + - 11 delivery: distribution: 0.03351 transmission: 0.01387 other_riders: -0.00911 summer: months: - - 4 - - 5 - - 6 - - 7 - - 8 - - 9 - - 10 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 delivery: distribution: 0.12196 transmission: 0.0505 diff --git a/tests/test_expand_month_hour_mc.py b/tests/test_expand_month_hour_mc.py new file mode 100644 index 00000000..36733ceb --- /dev/null +++ b/tests/test_expand_month_hour_mc.py @@ -0,0 +1,137 @@ +"""Month-hour marginal costs expanded onto a Cairo 8760.""" + +from __future__ import annotations + +from datetime import datetime + +import polars as pl +import pytest + +from utils.data_prep.marginal_costs.generate_utility_tx_dx_mc import ( + eversource_derived_output_base, + expand_month_hour_mc_to_8760, +) + + +def test_eversource_derived_output_base() -> None: + assert ( + eversource_derived_output_base( + "s3://data.sb/switchbox/marginal_costs/ct/dist_and_sub_tx/" + ) + == "s3://data.sb/switchbox/marginal_costs/ct/dist_and_sub_tx_eversource_derived/" + ) + assert ( + eversource_derived_output_base( + "/ebs/data/switchbox/marginal_costs/ct/dist_and_sub_tx" + ) + == "/ebs/data/switchbox/marginal_costs/ct/dist_and_sub_tx_eversource_derived/" + ) + already = ( + "s3://data.sb/switchbox/marginal_costs/ct/dist_and_sub_tx_eversource_derived/" + ) + assert eversource_derived_output_base(already) == already + with pytest.raises(ValueError, match="dist_and_sub_tx/"): + eversource_derived_output_base( + "s3://data.sb/switchbox/marginal_costs/ct/bulk_tx/" + ) + + +def _table(value_for_month_hour: float, *, weekend: bool) -> pl.DataFrame: + sign = -1.0 if weekend else 1.0 + rows = [ + { + "month": month, + "hour": hour, + "num_days": 1, + "mc_total_per_kwh": sign * (month + hour / 100 + value_for_month_hour), + } + for month in range(1, 13) + for hour in range(24) + ] + return pl.DataFrame(rows) + + +def test_weekday_weekend_and_holiday_lookup() -> None: + expanded = expand_month_hour_mc_to_8760( + _table(0.0, weekend=False), + _table(0.0, weekend=True), + 2025, + "ct_eversource", + ) + assert expanded.columns == ["timestamp", "utility", "year", "mc_total_per_kwh"] + assert expanded.height == 8760 + assert expanded.schema["timestamp"] == pl.Datetime("us") + assert expanded.schema["year"] == pl.Int32 + + def value_at(stamp: str) -> float: + return expanded.filter(pl.col("timestamp") == datetime.fromisoformat(stamp))[ + "mc_total_per_kwh" + ].item() + + # Wednesday that is not a holiday. + assert value_at("2025-01-08 00:00:00") == pytest.approx(1.0) + # New Year's Day and July 4 use the weekend-and-holiday table. + assert value_at("2025-01-01 00:00:00") == pytest.approx(-1.0) + assert value_at("2025-07-04 16:00:00") == pytest.approx(-7.16) + # The Thursday before July 4 stays on the weekday table. + assert value_at("2025-07-03 16:00:00") == pytest.approx(7.16) + # Saturday and Sunday: weekend table, including the spring-forward 02:00 slot. + assert value_at("2025-01-04 16:00:00") == pytest.approx(-1.16) + assert value_at("2025-03-09 02:00:00") == pytest.approx(-3.02) + + +def test_saturday_holiday_is_observed_on_friday() -> None: + expanded = expand_month_hour_mc_to_8760( + _table(0.0, weekend=False), + _table(0.0, weekend=True), + 2026, + "ct_eversource", + ) + + def value_at(stamp: str) -> float: + return expanded.filter(pl.col("timestamp") == datetime.fromisoformat(stamp))[ + "mc_total_per_kwh" + ].item() + + # July 4, 2026 is a Saturday, so Friday July 3 is the observed holiday. + assert value_at("2026-07-03 16:00:00") == pytest.approx(-7.16) + assert value_at("2026-07-06 16:00:00") == pytest.approx(7.16) + + +def test_weekday_holidays_stay_on_weekday_table_when_holidays_excluded() -> None: + expanded = expand_month_hour_mc_to_8760( + _table(0.0, weekend=False), + _table(0.0, weekend=True), + 2025, + "ct_eversource", + include_holidays=False, + ) + + def value_at(stamp: str) -> float: + return expanded.filter(pl.col("timestamp") == datetime.fromisoformat(stamp))[ + "mc_total_per_kwh" + ].item() + + # Wednesday New Year's and Friday July 4 stay on the weekday table. + assert value_at("2025-01-01 00:00:00") == pytest.approx(1.0) + assert value_at("2025-07-04 16:00:00") == pytest.approx(7.16) + # Saturday still uses the weekend table. + assert value_at("2025-01-04 16:00:00") == pytest.approx(-1.16) + + +def test_leap_year_drops_december_31() -> None: + expanded = expand_month_hour_mc_to_8760( + _table(0.0, weekend=False), + _table(0.0, weekend=True), + 2024, + "ct_eversource", + ) + assert expanded.height == 8760 + assert expanded.filter(pl.col("timestamp").dt.month() == 2).height == 29 * 24 + assert ( + expanded.filter( + (pl.col("timestamp").dt.month() == 12) + & (pl.col("timestamp").dt.day() == 31) + ).height + == 0 + ) diff --git a/utils/data_prep/marginal_costs/generate_utility_tx_dx_mc.py b/utils/data_prep/marginal_costs/generate_utility_tx_dx_mc.py index 147e8a02..9580c052 100644 --- a/utils/data_prep/marginal_costs/generate_utility_tx_dx_mc.py +++ b/utils/data_prep/marginal_costs/generate_utility_tx_dx_mc.py @@ -23,6 +23,9 @@ - RI default base: s3://data.sb/switchbox/marginal_costs/ri/dist_and_sub_tx/ - MD default base: s3://data.sb/switchbox/marginal_costs/md/dist_and_sub_tx/ - CT default base: s3://data.sb/switchbox/marginal_costs/ct/dist_and_sub_tx/ + - CT Eversource month-hour (--use-eversource-marginal-cost true): + s3://data.sb/switchbox/marginal_costs/ct/dist_and_sub_tx_eversource_derived/ + The partition year is --year, the same calendar stamped on the timestamps. - Default partition: utility=X/year=YYYY/data.parquet - Alternate-load-year base: _loadYYYY/ Example: dist_and_sub_tx_load2018/utility=X/year=2025/data.parquet @@ -59,7 +62,7 @@ import argparse import io -from datetime import datetime, timedelta +from datetime import date, datetime, timedelta from pathlib import Path import polars as pl @@ -67,6 +70,7 @@ from dotenv import load_dotenv from utils.data_prep.marginal_costs.supply_utils import ( + build_cairo_8760_timestamps, output_base_for_load_year, remap_year_if_needed, warn_if_multiple_partition_parquets, @@ -253,6 +257,34 @@ def load_marginal_cost_table(mc_table_path: str) -> pl.DataFrame: return df +def eversource_derived_output_base(base: str) -> str: + """Point a dist-and-sub-tx output root at the Eversource-derived dataset. + + ``.../dist_and_sub_tx/`` becomes ``.../dist_and_sub_tx_eversource_derived/``. + A base that already names that dataset is returned unchanged. + """ + trimmed = base.rstrip("/") + parent, leaf = trimmed.rsplit("/", 1) + if leaf == "dist_and_sub_tx_eversource_derived": + return f"{trimmed}/" + if leaf != "dist_and_sub_tx": + raise ValueError( + "--use-eversource-marginal-cost writes under " + "dist_and_sub_tx_eversource_derived/, so the output base must end " + f"in dist_and_sub_tx/ (got {base!r})" + ) + return f"{parent}/dist_and_sub_tx_eversource_derived/" + + +def _parse_bool(value: str) -> bool: + normalized = value.strip().lower() + if normalized in {"true", "1", "yes"}: + return True + if normalized in {"false", "0", "no"}: + return False + raise argparse.ArgumentTypeError(f"expected true or false, got {value!r}") + + def validate_mc_table_path(mc_table_path: str) -> None: """Fail early if marginal cost table path does not exist.""" if mc_table_path.startswith("s3://"): @@ -455,6 +487,130 @@ def validate_allocation( return validation_results +def expand_month_hour_mc_to_8760( + weekday: pl.DataFrame, + weekend: pl.DataFrame, + year: int, + utility: str, + *, + include_holidays: bool = True, +) -> pl.DataFrame: + """Expand month-hour marginal costs onto one calendar year's Cairo 8760. + + ``weekday`` and ``weekend`` are the OCC-863 Attachment 2 matrices, each with + ``month`` (1–12), ``hour`` (0–23), and ``mc_total_per_kwh``. Saturday and + Sunday take the weekend value. When ``include_holidays`` is true, US + federal holidays do too: a holiday that falls on Saturday is observed on + Friday, and one that falls on Sunday is observed on Monday. When it is + false, a holiday on Monday through Friday takes the weekday value. + Any other Monday through Friday takes the weekday value. + + ``year`` is both the calendar used to classify each day and the year + stamped on ``timestamp`` and the ``year`` column. Passing 2018 classifies + 2018's weekdays, weekends, and holidays and writes 2018 timestamps. + + Timestamps match ``build_cairo_8760_timestamps``: naive hour-beginning + local time, 8,760 rows, with December 31 dropped in a leap year. + """ + clock = build_cairo_8760_timestamps(year).with_columns( + pl.col("timestamp").cast(pl.Datetime("us")) + ) + holidays = _us_federal_holidays(year) if include_holidays else set() + clock = clock.with_columns( + pl.col("timestamp").dt.month().alias("month"), + pl.col("timestamp").dt.hour().alias("hour"), + # Polars weekday is ISO: Monday = 1, Sunday = 7. + (pl.col("timestamp").dt.weekday() >= 6).alias("is_weekend"), + pl.col("timestamp").dt.date().is_in(sorted(holidays)).alias("is_holiday"), + ) + weekday_rates = _month_hour_rates(weekday, "weekday").rename( + {"mc_total_per_kwh": "weekday_mc"} + ) + weekend_rates = _month_hour_rates(weekend, "weekend").rename( + {"mc_total_per_kwh": "weekend_mc"} + ) + joined = clock.join(weekday_rates, on=["month", "hour"], how="left").join( + weekend_rates, on=["month", "hour"], how="left" + ) + expanded = joined.with_columns( + pl.when(pl.col("is_weekend") | pl.col("is_holiday")) + .then(pl.col("weekend_mc")) + .otherwise(pl.col("weekday_mc")) + .alias("mc_total_per_kwh") + ) + if expanded["mc_total_per_kwh"].null_count() != 0: + raise ValueError(f"month-hour lookup missed hours in {year}") + return expanded.select( + "timestamp", + pl.lit(utility).alias("utility"), + pl.lit(year).cast(pl.Int32).alias("year"), + "mc_total_per_kwh", + ) + + +def _us_federal_holidays(year: int) -> set[date]: + """US federal holidays observed during ``year``. + + The eleven holidays in 5 U.S.C. § 6103. A fixed-date holiday on Saturday is + observed the Friday before, and one on Sunday the Monday after. That + observed set is what Attachment 2's weekend-and-holiday day counts match + for 2025. + """ + fixed = [ + date(year, 1, 1), + date(year, 6, 19), + date(year, 7, 4), + date(year, 11, 11), + date(year, 12, 25), + date(year + 1, 1, 1), + ] + floating = [ + _nth_weekday(year, 1, 0, 3), + _nth_weekday(year, 2, 0, 3), + _nth_weekday(year, 5, 0, -1), + _nth_weekday(year, 9, 0, 1), + _nth_weekday(year, 10, 0, 2), + _nth_weekday(year, 11, 3, 4), + ] + observed = {_observe_holiday(day) for day in fixed + floating} + return {day for day in observed if day.year == year} + + +def _nth_weekday(year: int, month: int, weekday: int, n: int) -> date: + """The nth ``weekday`` of ``month`` (Monday = 0). ``n`` of -1 is the last.""" + if n > 0: + first = date(year, month, 1) + shift = (weekday - first.weekday()) % 7 + return first + timedelta(days=shift + 7 * (n - 1)) + if month == 12: + last = date(year, 12, 31) + else: + last = date(year, month + 1, 1) - timedelta(days=1) + return last - timedelta(days=(last.weekday() - weekday) % 7) + + +def _observe_holiday(day: date) -> date: + if day.weekday() == 5: + return day - timedelta(days=1) + if day.weekday() == 6: + return day + timedelta(days=1) + return day + + +def _month_hour_rates(frame: pl.DataFrame, label: str) -> pl.DataFrame: + required = {"month", "hour", "mc_total_per_kwh"} + missing = required - set(frame.columns) + if missing: + raise ValueError(f"{label} month-hour table is missing {sorted(missing)}") + rates = frame.select("month", "hour", "mc_total_per_kwh") + n_unique = rates.select("month", "hour").n_unique() + if rates.height != 12 * 24 or n_unique != rates.height: + raise ValueError( + f"{label} month-hour table must have 288 unique month-hour rows" + ) + return rates + + def save_allocated_costs( df: pl.DataFrame, utility: str, @@ -462,6 +618,7 @@ def save_allocated_costs( s3_base: str, validation_results: dict, storage_options: dict[str, str], + path_local_base: str | None = None, ): """Save allocated marginal costs to S3 with Hive-style partitioning. @@ -471,13 +628,15 @@ def save_allocated_costs( Args: df: DataFrame with allocated costs utility: Utility name - year: Output year for the Hive partition and timestamps. This is also - the default target dollar year of the allocated ``$/kW-yr`` (CPI - inflation to this year happens before save). Distinct from the + year: Output year for the Hive partition and the ``year`` column. For + the PoP allocation this is also the timestamp year and the default + target dollar year of the allocated ``$/kW-yr``. Distinct from the PoP allocation load year. s3_base: Base S3 path for marginal costs validation_results: Validation results (unused, kept for interface consistency) storage_options: Polars S3 storage options with AWS bucket region + path_local_base: Optional local mirror of ``s3_base``; the same + ``utility=X/year=YYYY/data.parquet`` is written there too. """ output_df = df.select( [ @@ -518,6 +677,16 @@ def save_allocated_costs( ) print(f"\n✓ Saved allocated costs to {output_path}") + if path_local_base: + path_local = ( + Path(path_local_base) + / f"utility={utility}" + / f"year={year}" + / "data.parquet" + ) + path_local.parent.mkdir(parents=True, exist_ok=True) + output_df.write_parquet(path_local) + print(f"✓ Saved local copy to {path_local}") print(f" Rows: {len(output_df):,}") print(f" Columns: {', '.join(output_df.columns)}") @@ -605,12 +774,60 @@ def main(): action="store_true", help="Upload results to S3 (default: False, for data inspection only)", ) + parser.add_argument( + "--use-eversource-marginal-cost", + type=_parse_bool, + default=False, + metavar="{true,false}", + help=( + "true: build the 8760 from Eversource's OCC-863 Attachment 2 " + "month-hour $/kWh tables (weekday vs. weekend-and-holiday). " + "false: PoP allocation of the $/kW-yr value. CT only; default false." + ), + ) + parser.add_argument( + "--path-weekday-mc", + type=str, + default=None, + help="Month-hour weekday $/kWh parquet (with --use-eversource-marginal-cost)", + ) + parser.add_argument( + "--path-weekend-mc", + type=str, + default=None, + help=( + "Month-hour weekend-and-holiday $/kWh parquet " + "(with --use-eversource-marginal-cost)" + ), + ) + parser.add_argument( + "--include-holidays", + type=_parse_bool, + default=True, + metavar="{true,false}", + help=( + "true: federal holidays take the weekend-and-holiday table. " + "false: only Saturday and Sunday do; a weekday holiday stays on " + "the weekday table. Used with --use-eversource-marginal-cost." + ), + ) + parser.add_argument( + "--path-local-output-base", + type=str, + default=None, + help="Local mirror of --output-s3-base; with --upload, also write there", + ) args = parser.parse_args() - validate_mc_table_path(args.mc_table_path) load_dotenv() storage_options = get_aws_storage_options() + if args.use_eversource_marginal_cost: + _run_eversource_month_hour(args, storage_options) + return + + validate_mc_table_path(args.mc_table_path) + output_year = args.year load_year = args.load_year if args.load_year else output_year target_dollar_year = ( @@ -705,6 +922,7 @@ def main(): output_s3_base, validation_results, storage_options, + path_local_base=args.path_local_output_base, ) print("\n" + "=" * 60) print("✓ Marginal cost allocation completed and uploaded") @@ -716,5 +934,77 @@ def main(): print("=" * 60) +def _run_eversource_month_hour( + args: argparse.Namespace, storage_options: dict[str, str] +) -> None: + """Build and optionally save the 8760 from OCC-863 month-hour tables.""" + if args.state != "CT": + raise ValueError("--use-eversource-marginal-cost is only defined for CT") + if not (args.path_weekday_mc and args.path_weekend_mc): + raise ValueError( + "--use-eversource-marginal-cost needs --path-weekday-mc and " + "--path-weekend-mc" + ) + year = args.year + output_s3_base = eversource_derived_output_base(args.output_s3_base) + path_local_base = ( + eversource_derived_output_base(args.path_local_output_base) + if args.path_local_output_base + else None + ) + + print("=" * 60) + print("EVERSOURCE MONTH-HOUR MARGINAL COST → 8760") + print("=" * 60) + print(f"Utility: {args.utility}") + print(f"Year (calendar, timestamps, and partition): {year}") + print(f"Include federal holidays: {args.include_holidays}") + print(f"Weekday table: {args.path_weekday_mc}") + print(f"Weekend-and-holiday table: {args.path_weekend_mc}") + print(f"Output S3 base: {output_s3_base}") + if path_local_base: + print(f"Local output base: {path_local_base}") + print(f"Upload to S3: {'Yes' if args.upload else 'No (inspection only)'}") + print("=" * 60) + + weekday = _read_parquet_any(args.path_weekday_mc, storage_options) + weekend = _read_parquet_any(args.path_weekend_mc, storage_options) + expanded = expand_month_hour_mc_to_8760( + weekday, + weekend, + year, + args.utility, + include_holidays=args.include_holidays, + ) + + annual = float(expanded["mc_total_per_kwh"].sum()) + nonzero = expanded.filter(pl.col("mc_total_per_kwh") > 0).height + print(f"\n Rows: {expanded.height:,}") + print(f" Hours with nonzero MC: {nonzero:,}") + print(f" Sum over 8760 (flat 1 kW, $/kW-yr): ${annual:.4f}") + print("\nTop 10 hours by marginal cost:") + print(expanded.sort("mc_total_per_kwh", descending=True).head(10)) + + if args.upload: + save_allocated_costs( + expanded, + args.utility, + year, + output_s3_base, + {}, + storage_options, + path_local_base=path_local_base, + ) + print("\n✓ Eversource month-hour 8760 completed and uploaded") + else: + print("\n⚠️ No data uploaded to S3 (use --upload flag to enable)") + + +def _read_parquet_any(path: str, storage_options: dict[str, str]) -> pl.DataFrame: + if path.startswith("s3://"): + return pl.read_parquet(path, storage_options=storage_options) + return pl.read_parquet(path) + + if __name__ == "__main__": main()