[FBumann]

Prompt: Lets update specsolve to the latest mathspec

Note

The following content was generated by AI.

specsolve now requires mathspec 0.3.0. The engine and the linopy lane build 0.3.0's program and apply each declaration's missing:. Every spec is in the 0.3.0 language. The PyPSA parity gate runs the 0.3.0 corpus: 68 rungs match, and rung 63 differs as a known failure of that corpus.

What changed in the engine

• Relations. A sum through a relation is now a Sum over the axes of a Join, and a lookup is a bare Join. A where reads a JoinedPredicate or a NamedMask. relations.py maps JoinColumns onto the old consumed/produced/joined roles: dropped, added, kept.
• missing: at attach (attaching.py):
  • A parameter or relation short of a row is refused by default. The message names the first missing coordinate and the number of others.
  • A declared value fills the table, so nothing downstream sees the gap. A bare numeric name in a where still asks whether the data gave the row, through the given frames, as 0.3.0 says.
  • A dimensionless parameter may arrive with no row when it declares missing:.
• missing: absent in the build.
  • The parameter's piece carries a presence frame, so absence follows mathspec's rules: a constant or divisor removes the row, a summand is one fewer, a bound leaves its side open.
  • A constant piece's presence now restricts the rows (assembly.py).
• missing: neutral. It keeps 0.2.1's reading. The engine refuses a missing row only where it stands as the whole constant side, as a divisor or as a bound. Under a sum, a missing row is one summand fewer: mathspec's own PyPSA file needs this for sum(Transformer_phase_shift_weight, over=transformer).
• given: masks:, new in 0.3.0, is refused like every other given: name. The refusal named no mask before (this spec reads under given:). a-mask-another-file-declares in test_composition.py fails without the fix.
• A latent bug, fixed (pieces.py). A product carried a presence whose key was only implied onto a piece with more dims, which crashed. A masked variable under a filled shift could reach it before. The absent breakpoint table of a ragged curve now reaches it, and test_a_ragged_curve_down_to_one_point_is_refused covers it.
• The archive's INPUTS_LAYOUT rises to 2, because spec.yaml is now written in the 0.3.0 language.

How the specs were migrated

• The call syntax was rewritten mechanically: sum(x, by=l, over=a, into=b) → sum(x, over=a, by=l[b]); at(x, by=l, over=a, into=b) → at(x, by=l[a]); by=l, within=c → within=l[c]; rel.col → rel[col] in a where; absence: zero/undefined → missing: neutral/absent.
• ordered: and missing: were decided by what each declaration means, not by the errors. Time axes, periods, breakpoints and SOS sizes are ordered; sets of things are not. A parameter keeps the default refused unless its description, or the intent of its test, says the table is sparse.
• Errors after that pass were treated as findings:
  • OSeMOSYS build_cost and holding_cost are given only for technologies that have a cost, so they are neutral.
  • The tests of the engine's own refusals (sparse divisor, sparse constant side, NULL bound) now declare missing: neutral, the only reading that still reaches those guards. tests/test_missing.py covers the default refusal and each reading on both lanes.
  • A test that shifted along generator now shifts along snapshot, because 0.3.0 refuses a shift along a set.
• Generated pages were regenerated or re-synced: the gallery fences, gallery math, constructs, the walkthrough golden file and the ladder pages.

PyPSA parity: the corpus moves to v0.3.0

The v0.2.1 corpus does not load under 0.3.0, so pypsa-parity.yml now checks out v0.3.0. That corpus pins PyPSA master at 51986084, not a release, and the workflow pins the same commit. prep.py prepares the 9 parameters 0.3.0 adds. Rungs that patch the base file are now read through their PATCH. The projections keep the masks: a rung needs. untested.txt lists the gaps as the gate measures them; the gate reports 0 unlisted.

The linopy lane now builds rungs 56, 66 and 68. Each has a dimension with no labels, and the lane's operator_at raised IndexError there. It now stands a made-up label in, which the mask removes again. test_a_lookup_into_a_dimension_with_no_labels_reads_nothing_on_both_lanes fails without the fix. On rung 66 the lane keeps rows of zero coefficients, as linopy does, so the gate's row reader now skips a label with no nonzero coefficient, as linopy's LP writer and row count do. All three rungs are structurally equal to PyPSA.

A dual that differs is checked against the range PyPSA's model admits. Where an optimum is degenerate, a row has a range of optimal duals, and the two solvers may return different ones. For each differing row, the gate moves the row's right-hand side by ±0.001 in PyPSA's own model, re-solves on HiGHS over a copy of its LP file, and measures the slope of the objective on each side (differential/pypsa/slopes.py). Ours counts as equal if it lies between the two slopes. PyPSA's own dual must lie there too, or the run stops; that self-check caught a sign error in the first version. The ladder pages show the count as inside PyPSA's range.

• Rung 55: 8 differing rows, all inside.
• Rung 67: 6 differing rows, all inside. Process-p-ramp_limit_down is written negated, as Link-p-ramp_limit_down is, and deviations.yaml now says so. Rungs 17 and 45 check that record too.
• Rung 57: the duals: reason for Generator-p_nom_set (two identical pins that split one dual) is gone, because the range check now accepts that split and the gate refuses a reason no rung needs.

Rung 63 is a known failure of the 0.3.0 corpus. pypsa.yaml builds p_nom and its floor row for an extendable unit that is not active, and PyPSA builds neither (energy-models/mathspec#883). The fix is energy-models/mathspec#886. It adds a <Component>_active_asset parameter for each of seven components, so this gate's prep.py must feed them when the corpus moves past 0.3.0. differential/pypsa/known_failing.yaml lists the rung with that issue. The gate and the ladder-page test expect it to differ, and fail once it matches, so the line goes with the corpus move that carries the fix.

Mutation table of the dual range check (taken by hand)

Each mutation was made in differential/pypsa/slopes.py, the file restored with git checkout --, and __pycache__ dropped on both sides. tests/test_slope_ranges.py ran each time.

guard	result
a dual outside the range is refused (inside)	caught
a side that makes the model infeasible has an infinite slope	caught
the range takes both sides, not one	caught
the tolerance at an end of the range	still green, then caught by the probe past-an-end-by-less-than-the-tolerance
PyPSA's own dual lies in the range (parity.py)	not reachable from the suite; the gate's own self-check

Mutation table of the engine (taken by hand)

Each guard is a condition inside a line, which tools/mutate.py cannot express as a line deletion. Each mutation was taken in a clean worktree at c801bf1. The file was restored with git checkout --, __pycache__ was dropped on both sides, and the full suite ran without the two parity-dependent files. The tree came back clean.

guard	result
a relation short of a key is refused (attaching.py)	caught
a parameter short of a row is refused (attaching.py)	caught
a declared value fills a missing row (attaching.py)	caught
a bare numeric name asks whether the data gave the row (predicates.py)	caught
an absent parameter carries its presence (compiler.py)	caught
an absent bound leaves its side open (compiler.py)	caught
a constant piece restricts the rows (assembly.py)	caught
an absent parameter is not owed its coordinates (assembly.py)	still green, then caught by the purpose-built probe absent-summand-on-a-constant-side-is-not-owed
a product spells out the keys it carries (pieces.py)	caught

The assembly.py filter this last probe tested was removed later, in e154a77, when summed gaps stopped being refused. The probe still checks the behaviour.

Gates run, what was not run, and defaults departed from

• Run:
  • Full suite, at 86324dd: 5759 passed.
  • The PyPSA parity gate, in full at b913a56: 68 match, 1 differs (rung 63). Its exit at 86324dd was checked on the committed stamps: 0 with rung 63 listed as a known failure, and 1 when a rung that matches is listed.
  • ruff check ., ruff format --check . and pyrefly check pass.
• Not run: pixi run check (no pixi in this container; the same gates ran through uv), docs-build, and the benchmarks.
• One PR, not three. The pin forces the engine port, the spec migration and the corpus move together, because none of them loads without the others. The commits keep them apart for review.
• Not done:
  • The refused check where the model reads the value, which feat(language)!: a missing row is refused only where the model reads it, a file names its value instead of neutral, and the program says where each parameter is read energy-models/mathspec#880 introduces, is left for the release that carries it. This PR follows 0.3.0 as released.

🤖 Generated with Claude Code

https://claude.ai/code/session_01A481aCHLAzU6coTE2VB1zP

[read-the-docs-community]

Documentation build overview

📚 specsolve | 🛠️ Build #35171321 | 📁 Comparing 86324dd against latest (15f298d)

  🔍 Preview build  

122 files changed · + 9 added · ± 112 modified · - 1 deleted

+ Added

• examples/pypsa_ladder/rung_61_voltage_angle_limits/index.html
• examples/pypsa_ladder/rung_62_startup_cost_per_snapshot/index.html
• examples/pypsa_ladder/rung_63_modular_branches_and_storage/index.html
• examples/pypsa_ladder/rung_64_commitment_edges/index.html
• examples/pypsa_ladder/rung_66_cycles_per_period/index.html
• examples/pypsa_ladder/rung_67_linearized_link_process/index.html
• examples/pypsa_ladder/rung_68_outage_factors_per_period/index.html
• examples/pypsa_ladder/rung_69_storage_retires_before_close/index.html
• examples/pypsa_ladder/rung_70_risk_preference_zero/index.html

± Modified

• index.html
• run/index.html
• sweep/index.html
• about/changelog/index.html
• about/linopy/index.html
• development/architecture/index.html
• development/verification/index.html
• development/walkthrough/index.html
• examples/dispatch/index.html
• examples/facility_location/index.html
• and 102 more...

- Deleted

• examples/pypsa_ladder/rung_54_scenario_delay/index.html

[codspeed]

Merging this PR will not alter performance

✅ 104 untouched benchmarks
⏩ 307 skipped benchmarks¹

---

_{Comparing claude/ecstatic-einstein-yxxrme (86324dd) with main (15f298d)}

Footnotes

1. 307 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[FBumann]

Note

The following content was generated by AI.

Red checks after 32c2e17, and why they stay red for now:

• floors failed because tests/test_missing.py imported pandas, which that environment does not hold. Fixed in 32c2e17: it now imports pandas through tests.oracle and skips there, like every other lane test.
• suite and floors: test_every_structure_difference_has_a_reason_and_is_on_the_index and test_every_negated_dual_is_checked_rather_than_excused fail on rungs 55 and 67.
• parity fails on the same two rungs and on rung 63.

None of these is a difference in the model.

Rung 63: filed as energy-models/mathspec#883.

Rungs 55 and 67: the duals there are not unique, and both solves report a valid one. I checked this by moving each differing row's right-hand side by ±0.001 in PyPSA's own model and measuring the slope of the objective on each side. The dual is any value between the two slopes.

• Rung 55, bus b at 03:00, where the load of 45 equals the transformers' transfer limit: the slopes are 30 from below and 150 from above, which is hydro's 10 and diesel's 50 times the weight. PyPSA reports 90 and we report 18, scaled by scenario weight.

• Rung 67, at 03:00: every differing dual lies inside PyPSA's interval.

  row	interval	PyPSA	ours
  Link-p-ramp_limit_down	[384.79, 769.58]	384.79	769.58
  Link-com-partly-shut-down	[−384.79, 0]	−384.79	0
  Process-p-ramp_limit_down	[320.46, 799.05]	320.46	774.08
  Process-com-partly-shut-down	[−420.33, 0]	−420.33	0
  Process-com-p-before	[−167.33, −109.07]	−167.33	−134.04
  Process-com-p-lower	[0, 33.29]	33.29	0

An earlier version of this comment said rung 67's rows are parallel. They are not: several rows bind at the same point of the solution, so their duals form a range rather than one value.

A duals: reason in deviations.yaml would excuse the name on every rung, Bus-nodal_balance included, so I recorded none. How the gate should accept a non-unique dual is with the maintainer.

---

Generated by Claude Code