Conversation
…ance for multistage
…d Integer_Build column
`write_costs` built the `Total` column of `cFix` from the co-located VRE+storage inverter, solar and wind terms but omitted `eTotalCFixElec`, while the per-zone columns did include `eCFixElec`. Zones therefore did not sum to the total for any case with an electrolyzer component. Verified on example_systems/8_three_zones_w_colocated_VRE_storage_electrolyzers, the only case in the repo with VS_ELEC resources. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`write_costs` built the Total column from the aggregate eTotal* expressions and
the zone columns from per-resource ones, in two independent passes. Nothing tied
the two together, so they could and did disagree.
Split into two functions:
cost_breakdown attributes every cost term to a resource id, returning
vectors indexed by resource (plus per-zone NSE and the
system-wide terms that have no resource attribution)
assemble_costs sums those vectors for the Total column and over each zone's
resources for the zone columns, then applies ParameterScale
once to every numeric cell
Because both columns now come from the same attribution, the zone columns sum to
the Total column by construction rather than by coincidence. That is the property
the multi-stage discounting work needs, since it has to scale investment and
operating costs by different factors without the two halves drifting apart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cTotal exceeded the sum of its rows whenever these were active: capacity-reserve virtual charge/discharge -> cVar hourly matching slack -> cUnmetPolicyPenalty retrofit lifetime slack -> cUnmetPolicyPenalty LDES slacks (eObjSlack, eObjSlackVreStor) -> cUnmetPolicyPenalty Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds stage_opex_multiplier, stage_discount_factor, payment_years_remaining and overnight_capital_cost_factor, and routes compute_overnight_capital_cost, initialize_cost_to_go and write_multi_stage_costs through them. The formulas are unchanged; the discount factor was previously computed by an inline loop in two separate places. Unit tests assert each helper reproduces the expression it replaced, under both equal and uneven stage lengths. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
run_myopic_multistage never updated CurStage, so it held whatever model generation left behind, the final stage, for every stage it wrote. The post-solve loop in run_genx_case_multistage! does set it, but that loop is skipped precisely when Myopic and WriteIntermittentOutputs are both on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tech_wacc defaults to zero when the WACC column is absent, which left investment annuities undiscounted rather than discounted at the social rate. Network.csv was worse: a missing WACC column raised an error, so the column was effectively mandatory. annuity_discount_rate now substitutes the general discount rate from multi_stage_settings.yml wherever a WACC is missing, blank or non-positive, and overnight_capital_cost_factor discounts at that rate. An explicit zero is treated as absent; a genuinely undiscounted annuity needs a zero general rate. No behaviour change for the oracle or test cases: all supply positive WACCs (resources 0.017-0.039, lines 0.062), so obj_true is unchanged. Both column descriptions in multi_stage_input.md now mark WACC optional and state the fallback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
OPEXMULT placed each year's operating cost at the start of the year (1/(1+DR)^(j-1)) while investment annuities are discounted from the end of year one (1/(1+WACC)^j), so the two were inconsistent by a factor of (1+DR). OPEXMULT now uses 1/(1+DR)^j. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
get_retirement_stage measured a technology's life from the end of the current stage, so capacity had to be gone by the time that stage finished. It now measures to the start of the current stage, matching the documented convention that a unit reaching end of life within a stage stays available for that whole stage and retires at the beginning of the next. The return value is the latest build-stage whose capacity must already have retired, used as the bound of `for p in 1:get_retirement_stage(...)`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds MultiStageCostFactors and ComponentAnnuity, built during input configuration and stored under inputs["MULTISTAGE_COST_FACTORS"]: the stage discount factor, the operating multiplier, the stage length, and per-component annuity factors with their payment counts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each stage's costs.csv becomes the present value at the start of the horizon, what the objective weighs, with a companion costs_undiscounted.csv giving the cash flow over the stage. The summaries stack both without further scaling, and the cTotal row returns. A fixed-cost expression is investment plus fixed O&M and the two scale differently, so the writer separates them: O&M is a rate times capacity, and investment is the remainder. Both are recovered to annual terms and re-scaled. Under myopic the investment field holds one year's annuity, so multiplying by the annuity factor adds back the payments the objective never charged. That is what makes the two foresight modes comparable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
model_output.md gains sections for costs_undiscounted.csv and the two multi-stage summaries, plus notes on which costs.csv rows carry no zonal attribution and why myopic reported totals exceed the stage objective. multi_stage_input.md gains a discounting section with the formulas actually used: the stage discount factor, the operating multiplier, the annuity factor and its truncation at the horizon, the age-based retirement rule, and the myopic add-back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Empty cells read back as missing while the column stays numeric, and are distinct from a genuine zero. Breaking for anything testing the literal "-". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The check rejecting a zero Capital_Recovery_Period alongside a positive investment cost lived inside compute_overnight_capital_cost, which only runs under perfect foresight. Myopic runs skip that path, so they never validated it. That became harmful once reporting started using annuity factors: a myopic resource with a zero recovery period gets an annuity factor of zero, so its investment vanished from both the discounted and undiscounted views while the objective charged it in full. Self-consistent and wrong, which is the worst combination. Extracted the check into validate_capital_recovery_period and call it for every component as the reporting factors are built, so both foresight modes validate the same way. Each component is checked against the investment field it prices. Found in review by a second pair of eyes, not by the tests or the oracle. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two silent-skip paths of the same shape as the recovery-period bug, both found in review: - _accumulate_fixed! skipped the investment/O&M split whenever a component's annuity was absent. For a single-stage run that is correct, but in multi-stage it would drop the component from the discounted and undiscounted views while leaving it in the model view, so the file would look self-consistent and understate costs. The lookup now errors in multi-stage instead. - write_multi_stage_costs wrote nothing when a stage file was missing. That is right when cost output is switched off entirely, but not when only some stages have the file; it now warns and names the stages. The new guard immediately caught a real inconsistency in the caller: the transmission component was looked up unconditionally even though it is legitimately absent without network expansion, which would have tripped on every single-zone multi-stage case. Now looked up only when the model has it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fixed-cost and transmission paths each carried the same three lines turning a model-held investment cost into its discounted and undiscounted views. Pulled into _investment_views, so there is one place to read and one place to test. That matters because the transmission path had no direct coverage: the multi-stage test fixture is single-zone with no network, so only example 6 exercises it, and only as a whole-file diff. The arithmetic is now tested on its own, including the case where both modes must agree on the reported value, and the zero-payment-years guard that would otherwise divide by zero. Also corrects two docstrings flagged in review: get_retirement_stage listed a fourth argument that does not exist, and initialize_cost_to_go claimed investment costs are scaled by 1/WACC when the code divides by 1/OPEXMULT. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Before, the overnight capital cost summed each resource's annuities as its own tech WACC, falling back to the dr ony when none was give. Co-Authored-By: Claude <noreply@anthropic.com>
Every existing cost test compares the implementation against itself: the objective identity, the zone sums, the ratios between views. All would pass under a uniformly wrong discount factor. This one compares against values derived by hand from the input CSVs and the documented conventions. The case forces its own solution, so no optimisation is needed to know the answer: one zone, one thermal generator with no existing capacity, flat demand stepping up each stage, and nothing else that could substitute for it. Capacity equals demand, generation equals capacity times 8760, new build is 100/50/50. Stage lengths are uneven (10, 5, 8) so annuity truncation differs by stage: with a 20-year recovery period the payments inside the horizon are 20, 13 and 8. Equal stages would have left that untested. Checks each component separately so a failure localises, not just the totals. Verified to have teeth: flipping the operating-cost convention to start-of-year fails 12 of 37 assertions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The setting held the planner's time value of money, but shared its name with the per-resource WACC column of Resource_multistage_data.csv, which is a technology's cost of capital. Two different quantities, one name, in two input files a user reads side by side. That confusion cost real time this week. WACC is still read and translated, with a warning naming the file, so existing cases keep working. Supplying both keys is an error rather than a guess. Local variables and docstrings follow the new name; the per-resource and per-line WACC columns keep theirs, since those genuinely are costs of capital. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three rows without per-resource attribution were already documented, but not the consequence: cTotal is the one row whose zone columns do not sum to its Total, by exactly those rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three autodoc'd docstrings still pointed users at the WACC field of multi_stage_settings.yml. The code reads DiscountRate; only the prose was stale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rtol=1e-8 was a default, not a derived bound. Measured error is ~1e-16 for checks that only re-sum the writer's own output, so those move to 1e-12. Comparisons against objective_value cross the solver boundary and the two investment checks lose digits to cancellation; those move to 1e-10. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
For a co-located resource eCFix is entirely the grid-connection cost, every other component having its own expression. The memo read eCGrid, which matches eCFix except that eCFix scales the investment term by Cap_Size for a resource in CAP_SIZE_SCALED. A co-located resource can reach that set through Discrete_Build, in which case the memo understated the cFix it quotes. cTotal excludes the memo, so totals and the objective were never affected. Reading eCFix makes the two agree by construction and leaves the :grid annuity component unused, so it is dropped; it duplicated :discharge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The single-stage branch no longer quotes the objective because rows are missing; since every objective term has a row, the two agree and the objective serves as a check on the rows. The multi-stage branch sums rows because eObj is not in the rescaled units, not because no objective exists: the sum equals objective_value - vALPHA. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Single-stage cTotal is written from the objective rather than from the rows, so comparing the two tests that every objective term reached a row. Nothing asserted this before; the multi-stage path already did. Covers both the continuous and the DiscreteInvestments run. Tolerances measured on this case: cTotal against the row sum crosses the objective-evaluation boundary at 7e-15, zone sums are a pure re-sum at 2e-16. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DEMAND stood for three quantities at once: demand, total installed capacity and the generation level. They coincide only because the generator builds exactly peak demand and nothing retires. FOM is charged on capacity, so derive it as cumsum(NEWCAP) and assert that it equals DEMAND rather than leaving the reader to reconstruct it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cTotal and objective_value - vALPHA come from the same solution, so solver error cancels; measured 3.8e-16. The remaining rtol=1e-6 checks compare against hand-computed values and stay as they are: the fixture sets Feasib_Tol 1e-5, so a conforming solver may return capacity off by 1e-7 relative, and the convention errors this test exists to catch are percent-scale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The old blanket warning that multi-stage costs were approximate is gone, since they no longer are. One caveat it covered still stands: run_ddp returns whatever models it has at its iteration limit, and the costs are then reported exactly as for a converged run. Check the final bounds against ConvergenceTolerance and say so when the gap is still open. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: created as a draft because it's built on top of #915.
Description
This PR fixes multi-stage cost discounting and reporting, which was internally inconsistent in ways that made
costs.csvunstable for perfect-foresight runs and not comparable between myopic and perfect foresight.Two symptoms motivated the work:
Totalcolumn.cTotaldid not equal the sum of its own rows.In addition to these fixes, this PR changes the convention adopted for when capacity is retired and comes online (see section below).
Important conventions adopted
Operating costs are now discounted end-of-year, matching how investment annuities were already discounted:
OPEXMULTchanges fromΣ 1/(1+DR)^(j-1)toΣ 1/(1+DR)^j. Previously an operating dollar and an annuity dollar in the same year differed in weight by a factor of(1+DR), which biased the model slightly toward capital-intensive builds.Investment annuities are now discounted at the general discount rate rather than at each technology's or line's own
WACC.Each stage now reports on two explicit bases:
costs.csvis the present value at the start of the horizon (what the objective weighs), and a newcosts_undiscounted.csvis the cash flow over the stage with no time value.costs_multi_stage.csvand the newcosts_undiscounted_multi_stage.csvstack the per-stage columns with no further scaling, so summing a row across stages gives the horizon cost.Myopic reports add back the investment annuities falling beyond the stage being solved, since the myopic objective charges only one year of each annuity; this makes myopic and perfect-foresight output comparable.
Age-based retirement now measures a technology's life from the start of the stage it was built in to the start of the current stage. Capacity reaching end of life part way through a stage stays available for that whole stage. This only differs when stage lengths are unequal; equal-length runs are unchanged.
The standing warning that "the cost calculation of the multi-stage GenX is approximate currently" is removed, since the approximation it described is what this PR fixes.
Breaking changes
multi_stage_settings.yml:WACCis renamedDiscountRate. The old name collided with the per-resourceWACCcolumn, which is a different quantity.WACCis still accepted and translated with a deprecation warning; setting both keys is an error."-". The string placeholder forced the entire zone column to be read back as text, so arithmetic on it failed. Scripts testing for the literal"-"need updating.OPEXMULT, and for any resource whoseWACCdiffered from the discount rate. Myopic objectives are unchanged (OPEXMULT = 1there).Capital_Recovery_Period = 0, as perfect-foresight runs already did.A note on
cTotaland zone columnsThree rows are system-wide and have no zone (
cUnmetRsv,cNetworkExp,cUnmetPolicyPenalty).cTotalis therefore the one row whose zone columns deliberately do not sum to itsTotal; the difference is exactly the sum of those three rows. This is documented inmodel_output.md. Every other row sums exactly, by construction.What type of PR is this? (check all applicable)
Related Tickets & Documents
Fixes #862, #838, #660
Addresses #841
Checklist
How this can be tested
Automated. The full suite passes (unit tests plus end-to-end consistency checks on both the DDP and myopic paths)
Manual. Run
example_systems/6_three_zones_w_multistagewithMyopic: 1andMyopic: 0This branch was verified against the pre-branch commit on
1_three_zones,4_three_zones_w_policies_slack,6_three_zones_w_multistage(myopic and DDP),7_three_zones_w_colocated_VRE_storage,8_three_zones_w_colocated_VRE_storage_electrolyzersand11_three_zones_w_allam_cycle_lox, under Gurobi, with time-domain-reduction clustering held fixed across runs so the comparison isolates code changes.Post-approval checklist for GenX core developers
After the PR is approved