[FBumann]

Prompt: Lets do it — And we need to document the now more less ambigouos behaviour and cotract!

Prompt: Why should we not do one PR per topic? Keep the enum one, combine the "missing" ones, and combine the ones making the spec fully sorted and predictable?

Note

The following content was generated by AI.

What this changes

missing: neutral is gone. A file writes the value that a missing row reads as, such as 0 or false, and a variable writes the number it stands as outside its mask. refused still needs a row at every coordinate. A divisor that the file itself makes 0, through a missing: 0 row or an edge=0 shift, is refused at load.

This PR supersedes #889, which stays open as a draft. #942 and #912 were squash-merged into this branch earlier, and each kept its own review.

How to review

Part	Files	How to read it
src/	8	Closely: the closed missing: vocabulary, the divisor check at load in _divisors.py, and a variable's number
docs/reference/	10	Closely: the table in absence.md, "A table with gaps on purpose", "Zero divisors in the data", and the edge=0 rule in expressions.md
tests/, hand-written	8	The new cases, the edge=0 cases from #889, and the neutral cases that became 0 and false
examples/, docs/examples/	43	Skim: neutral becomes 0 or false, and #912 picks a reading for each PyPSA parameter
schema, golden output, tests/expand/	9	Generated. Regenerated, not edited

What a missing row reads as

• A parameter takes refused, absent or a value. missing: 'neutral' gets the error for a closed vocabulary, which names the valid readings.
• absence.md has a table with one row for each position (coefficient, summand, divisor, bound, offset=/window= amount, where comparison, bare flag, bare number or label), and one column each for absent and for a value. Under refused, the table has a row at every combination of labels, so no position reads a missing row.
• A divisor that a missing row of a missing: 0 parameter can make 0 is refused at load. The check follows a factor, the numerator of a quotient, the base of a power, and the operand of a negation, a sum, a window, a shift, a lookup or a named expression. It also follows each region of a case. An addition can be 0 only where every term can, so c + 1 loads. A guard that is false at every missing row spares the divisor (fix(language): a divisor that a where, a case or a masked variable guards loads under missing: 0, and so does a variable number outside its bounds #942): a conjunct of the constraint's where, of a case's when, or of the mask of a missing: absent variable that multiplies the quotient.
• An engine must refuse a divisor that is 0 in a row it builds, when the data is attached. absence.md states it under "Zero divisors in the data", and reading.md links it. The load check reads only the shape of a divisor, so a zero from numbers that cancel loads: missing: 1 with x / (1 - c), x / ((c + 1) - 1) under missing: 0, a case that holds 1 - 1, or a present row that is 0. A reported quotient stays absent where its divisor is 0. specsolve already refuses each of these, but names the wrong parameter for a case (a divisor that a case makes 0 is refused as a parameter with missing rows, though the parameter has every row fluxopt/specsolve#2064).
• A variable's missing: takes absent or a finite number. The variable needs a where:. The number must fit the domain of the variable: 0 or 1 for binary, an integer for integer. The bounds hold only inside the mask, so the number may lie outside them (fix(language): a divisor that a where, a case or a masked variable guards loads under missing: 0, and so does a variable number outside its bounds #942). A curve gate takes 0 for no curve, or 1 for an ungated curve, which is also what absent gives.

A divisor that an edge=0 shift makes 0 (from #889)

p / shift(c, along=t, offset=1, edge=0) loaded and divided by 0 at the edge, whatever the data held. It is now refused at load, with the rewrite named:

Constraint 'cap': the divisor is 0 where shift(along=g, edge=0) vacates, which no number divides by. Write edge='wrap', or an edge= value other than 0.

#889 added a second walk, _zero_in, in _expression_resolver.py. Here the rule is one more case in this PR's walk: _zero_where_missing in _divisors.py is now _zero_in, and it returns the cause of the zero, either a missing: 0 parameter or an edge=0 shift. So the shapes it follows are the same for both causes: a factor, a numerator, a base, the operand of a negation, a sum, a window, a shift, a lookup or a named expression, a region of a case, and a sum only where every term can be 0. No guard spares an edge=0 zero. expressions.md states the rule beside the other rules for /, and links to absence.md for the missing: 0 case.

Reproduced first. #889's 15 tests were copied onto this branch before the fix. The 12 refusal cases failed with DID NOT RAISE LanguageError, and the 3 cases that load passed.

Mutation. With the edge=0 case deleted from _zero_in, the full suite gives 12 failures, all in test_a_divisor_a_shift_fills_with_zero_is_refused. The other branches of the walk are shared with the missing: 0 check and are in the table below. The run used the same precautions as below.

A table with gaps on purpose

absence.md has a new section for a table that has rows only where the model needs them. The file declares it absent, and an assumption says where it must have a row:

parameters:
  capacity: { dims: [g] }
  p_max: { dims: [g], missing: absent }
variables:
  dispatch:
    dims: [g]
    where: "capacity > 0"
    bounds: { lower: 0, upper: p_max }
assumptions:
  p_max_where_dispatch_exists:
    holds: "p_max"
    where: "capacity > 0"

The tool that attaches the data checks the assumption, so a missing p_max[gas] is an error, and a missing p_max[old] with capacity[old] = 0 is not.

Migration

The migration writes 0 for a float or int parameter and false for a bool parameter. It covers examples/, tests/expand/ and the golden fixture. The seven PyPSA *_nom_mod parameters are declared missing: absent instead: "no value means the build is continuous". Generator_p_nom_mod is a divisor in *_maint_modstatus_*, and before #942 missing: 0 was refused there. Those rows are guarded by *_p_nom_mod > 0, so missing: 0 loads now, and #912 decides which reading the files take. The two load refusals that need a row at every coordinate stay, and their rewrites no longer name neutral:

• a bare int or str name in a where, on a refused parameter;
• a refused values table of a curve with points:.

The schema, the golden output, and the gallery and notation pages are regenerated.

#912 judges each PyPSA missing: parameter. It declares the *_set parameters and the global-constraint weights missing: absent, as asked in review, and more besides. #942 fixes two load checks above that refused valid files.

Breaking changes, for the release notes

• Files. missing: neutral is refused, on a parameter and on a variable. Write 0 or false on a parameter, and 0 on a variable.
• A divisor that an edge=0 shift can make 0 is refused at load. Write edge='wrap', or another edge= value.
• A missing row in a bound or on the constant side. Under neutral, a missing row there was refused when the data was attached. Under missing: 0 it reads as 0, with no refusal: x <= cap becomes x <= 0. To keep a refusal, do one of these:
  • guard the constraint with the bare name, where: cap, which has no row where the data has none. Every such use in examples/pypsa.yaml already does this;
  • state where the table must have a row, assumptions: {cap_given: {holds: "cap", where: <where it is read>}}, and the tool that attaches the data refuses a missing row there.
• A where comparison. Under neutral, a comparison at a missing row was false. A value compares as itself, so cost < 1 is true at a missing row under missing: 0. Every comparison on a migrated parameter in examples/ is > 0, which is false both ways. To keep the old reading, write where: cost AND cost < 1.
• Imports. Each type of missing: is named after the entry that declares it: RelationMissing is Literal['refused', 'absent'], ParameterMissing is RelationMissing | bool | float, and VariableMissing is Literal['absent'] | float. MissingReading and Missing are gone. A given parameter is a GivenParameter (dims, dtype, description) under program.given.parameters, so Parameter.missing is never None.

Why

neutral read a missing row as 0 or false. That contributes nothing only in a sum. As a factor it removed the whole term, as an exponent it gave 1, and as an offset= it read the coordinate itself. A value says which number the file means.

Verification

Gates on the head, 216660d, which adds the edge=0 case from #889:

• pixi run lint passes.
• pixi run test passes: 3122 passed, 1 skipped.
• pixi run docs-build passes.
• Every generator re-run gives no diff.
• compile-tex did not run, because the container cannot download the TeX bundle. CI runs it.

Gates on 021e433, which merged main (0.4.0, #931, #946, #947) into this branch: lint and docs-build passed, and test gave 3107 passed, 1 skipped. The merge had two conflicts. In CHANGELOG.md, the lines that 0.4.0 released left "Upcoming version", and the #942, #912 and #894 lines stay. In reading.md, this branch's paragraph on zero divisors and #947's paragraph on relation comparisons both stay.

Earlier gates. On 9fb7dea, which merged main with #898, #899, #900, #901, #904, #907, #909 and #911: lint and docs-build passed, and test gave 3077 passed, 1 skipped. #942 holds its own gates and mutation table. On 547b034: test gave 3097 passed, 1 skipped.

Mutation table on 2121ec7. b1869aa merges the two checks on a variable's number into one validator; its where: and domain branches were deleted again on b1869aa and are caught by the same tests. A script deleted each guard and ran the full suite. It restored the file with git checkout --, dropped __pycache__ before and after each run, and the tree came back clean. #942 later removed the bounds rows of the last line.

Guard deleted	Caught by
the divisor refusal is called	12 cases of test_a_divisor_that_reads_a_missing_row_as_zero_is_refused
a bare missing: 0 parameter can make a divisor 0	the same 12
through a negation, a sum, a window or a shift	[negated], [summed], [shifted]
through a lookup or a named expression	[through-a-named-expression], [in-a-case], [looked-up-in-every-addend]
through a factor / a numerator / a base / a region of a case	[a-factor] / [a-numerator] / [a-base] / [in-a-case]
an addition only where every term can be 0	examples/pypsa.yaml no longer loads: CVaR_def divides by 1 - CVaR_alpha, and the mutated check refuses that divisor
a variable number needs a where:	test_a_rule_decided_without_data[zero-without-a-mask]
a variable number is finite and not a flag	test_a_variable_missing_that_reads_data_is_refused[a-flag], [an-infinity]
binary / integer / below lower / above upper	test_a_variable_missing_outside_what_the_variable_takes_is_refused[binary] / [integer] / [below-lower] / [above-upper]
the edge=0 case of _zero_in (on 216660d)	12 cases of test_a_divisor_a_shift_fills_with_zero_is_refused

Coverage moved. The neutral cases became 0 and false cases. an-int-under-neutral, neutral-on-a-label and the neutral reading of the exclusivity tests are removed, and their value counterparts stay.

Not done

• specsolve must drop neutral, and accept a number as the missing: of a variable. It already reads a value on a parameter. That is a separate PR in specsolve.
• The load check does not evaluate a divisor, so it does not catch numbers that cancel. That is left to the engine, by the rule above.
• sum_back stops short at the edge and sums fewer terms. That is not a 0 the file puts in, so it is not refused.
• missing: <parameter> on a variable is not added. A named expression with cases: already gives the variable inside the mask and the data outside it.
• The 0.3.0 release notes still name neutral. They are history.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7L7U7Av5v8wnKV6eDpXfi

[read-the-docs-community]

Documentation build overview

📚 mathspec | 🛠️ Build #35355387 | 📁 Comparing 216660d against latest (7cc9515)

  🔍 Preview build  

45 files changed · + 1 added · ± 43 modified · - 1 deleted

+ Added

• reference/changelog/index.html

± Modified

• several-files/index.html
• about/limits/index.html
• examples/pypsa_linearized_uc/index.html
• howto/compose/index.html
• reference/api/index.html
• reference/glossary/index.html
• reference/language/index.html
• reference/notation/index.html
• reference/program/index.html
• reference/reading/index.html
• and 33 more...

- Deleted

• changelog/index.html

[brynpickering]

@FBumann how does this PR link to the one I previously commented on (#869)? It was closed in preference for #880 which was closed in preference for this PR.

These trails of superseded PRs doesn't feel particularly productive. Would be better to hash out these in single issues first, then open a single PR following that discussion.

[FBumann]

I was trying to improve the missing: feature and was not satisfied with the solution I came up with. Your comments on it and me not being satisfied lead to me closing them and rethinking the concept of missing data.

This new PR now removes the missing: neutral feature and instead relies on default values carrying the meaning of "neutral", which is much more predictable for users. Im convinced that we wont be missing the "neutral" option.

Sorry for the churn. Im not trying to merge this quickly if you or @FabianHofmann want to review it and have the time, I would appreciate it.

[brynpickering]

Having the ability to define a value does seem neater, and nice to force the user to think about what is being filled in on absence rather than doing it automatically for them.

Interesting though that all missing are 0 in the examples. I would have expected 1 for efficiency / p_max_pu.

FYI: In Calliope, we were using this value option only and it caused us some issues. Namely, when applying a variant switching from capacity expansion to dispatch (see calliope-project/calliope#862 (comment)). I think this is a non-issue in mathspec as we can use missing: absent rather than a value.

Comments are about:

1. switching some pypsa entries from having 0 missing to absent
2. adding missing information elsewhere (e.g. p_max_pu).

Second point might not be worthwhile as pypsa always pre-fills them with defaults (I guess that's why most params don't have a missing at present).

[brynpickering]

should these _set options always be absent? I.e., they only get applied if they are present so if they are missing that should be considered an absence rather than being filled in.

[FBumann]

Yes. But this result in a sparser model, which is probably not what pypsa does today.
Your pointing at what the role of these files is: Replicating pypsa 100%, or showing how pypsa would be implemented in mathspec. We settled on replicating 100% for now, so this should probably be 0, although absent is what we ultimately want.

[FBumann]

I dived in a bit and I think we should just change it to the correct one (absent here). Not sure what pypsa does exactly.

[brynpickering]

OK, agreed that replicating 100% is our goal. I leave @FabianHofmann to review these entries based on what happens in pypsa currently.

[FBumann]

@brynpickering I move those pypsa changes into a stacked PR #912. We can fold them in here of you prefer.

[brynpickering]

This will include all generators in the global expression constraint, with many possibly included with a weight of 0. If missing: absent then they just wouldn't be included in the sum, right? Same result, but perhaps the latter is quicker to parse and add to the solver?

[FBumann]

Yes. But this results in a sparser model, which is probably not what pypsa does today.
Your pointing at what the role of these files is: Replicating pypsa 100%, or showing how pypsa would be implemented in mathspec. We settled on replicating 100% for now, so this should probably be 0, although absent is what we ultimately want.

[FBumann]

I dived in a bit and I think we should just change it to the correct one (absent here). Not sure what pypsa does exactly.