[FBumann]

I wanted to leave off things a bit cleaner and combined the 3 topic related PRs into this big one. Hope this is better to work with. The other ones are drafts now and lost their label.

Prompt: Why should we not do one PR per topic? Keep the enum one, combine the "missing" ones, and combine the ones making the spec fully sorted and predictable?

Prompt: i think the sorting one should replace the others with the clear goal of making everything sorted and pedictable, saying the others are superseeded and making them drafts. Not closing them

Note

The following content was generated by AI.

A loaded spec is sorted by name and cannot be changed. to_yaml() writes back the fields its file wrote. Two files that differ only in order, defaults or the spelling of a condition write one canonical text.

This PR supersedes #918, #923 and #927, which stay open as drafts and keep their review history. It also still holds #914, #916 and #921, which were squashed into it earlier.

Part	From	What changes
Sorted	this PR	Entries, dims: lists, relation columns, cases: regions, a macro's kwargs: and the lists of names in a call are sorted by name
Cannot be changed	#923	A field cannot be set. A section is a read-only Frozen mapping, and a list is a tuple. override is how you change a spec
Writes back what the file wrote	#918	to_yaml() writes the fields the file set, plus version:. The canonical form writes no defaults. A patch may change only description, ordered and missing of a dimension or a relation that its base declares
One text per condition	#927	The canonical form prints each where:, when:, condition: and holds: from its parsed tree. AND/OR operands are sorted, keywords are upper case, and spacing and brackets are made the same

Breaking changes, for the release notes

Spec API (#923).

• A spec field cannot be set, and a section cannot gain or lose an entry. Use override(spec, [patch]).
• A list in a spec is a tuple: spec.variables['p'].dims == ('g',). to_dict() and to_yaml() still write lists.
• A section is a Frozen[str, X] mapping, not a dict. Code that tests isinstance(section, dict) must test Mapping.
• Sealed in _sealed.py is now Frozen in _frozen.py. The same class holds the program's groups.

Order (this PR).

• Mask.dims is a tuple sorted by name, not a frozenset. A tool that compares it with a set must build a set first.
• Relation.columns lists the key's columns first, and each side is sorted by role. Relation.dims lists each dimension once, sorted by name.
• Every dims and frame in the program is sorted by name. specsolve's result layouts change.
• A spec's sections iterate in name order, and to_yaml() writes them in that order.

Patches (#918). A patch entry over a dimension or a relation that its base declares may write only description, ordered (only to true) and missing. Otherwise it must declare the whole entry, and a field it leaves out reads as its default.

Written output (#918). to_yaml() keeps ordered: false and missing: refused where the file wrote them, and no longer adds dtype: str, dtype: float or domain: continuous where the file did not write them. It always writes version:.

How sorting and freezing work together

#923 and the sorting part conflicted in spec.py, _lowering.py and four test files. The rule is: sort first, then freeze.

• Dims = Annotated[Listed[str], AfterValidator(_sorted)]. Listed is feat(spec)!: a loaded spec cannot be changed, so the file it writes back out is always the one its program comes from #923's tuple that is written back out as a list. _sorted returns a sorted tuple. A macro's kwargs: uses the same validator.
• _by_name, which sorts a section's entries and the regions under cases:, now returns a Frozen mapping. RelationSpec._sorted_by_role returns a sorted tuple, or a sorted Frozen mapping.
• _frame_of stays deleted, as the sorting part deleted it.
• The tests from feat(spec)!: a loaded spec cannot be changed, so the file it writes back out is always the one its program comes from #923 that compared a field with a tuple now compare it with the sorted tuple.

New probe: test_a_loaded_spec_cannot_be_written_to[a-relation-column-removed]. Before it, a relation side written as a mapping could come back as a plain dict after sorting, and no test failed (row 2 below).

Gates and mutation table

Gates on 188f13e, which also merges main (0.4.0, #946, #947):

• pixi run lint: passed.
• pixi run test: 3591 passed, 1 skipped (3590 before the new probe).
• pixi run docs-build: passed.
• Every generator re-run (schema, golden output, gallery, notation, spec_math, home_math, expansion_math): no diff.
• compile-tex: did not run. Tectonic cannot download its bundle in this sandbox. CI runs it.

Mutation table for the sort-then-freeze rule, on 76f3ff2 and on 188f13e for the probe. Each change was made by a script. __pycache__ was dropped before and after each run, the full suite ran, the file was restored with git checkout --, and the tree came back clean each time.

Change	Caught by
_by_name returns a dict	3 failures: test_a_loaded_spec_cannot_be_written_to[an-entry-added], [an-entry-removed], [a-case-removed]
_sorted_by_role returns a dict for a mapping	nothing on 76f3ff2. On 188f13e: [a-relation-column-removed]
_sorted returns a list	130 failures, among them test_a_patch_names_only_the_field_it_changes

The mutation tables of each part are in their own PRs, and those rows were not re-run here: sorting in this PR's earlier body (below), the write-back and patch rules in #918, the frozen spec in #923, and the canonical conditions in #927. The merges changed no line those rows mutate, except the three rows above.

Sorting: method and tests (from the earlier body of this PR)

Six validators on the spec models do the sorting, and nothing after the load sorts again. RelationSpec._sorted_by_role sorts key and values. field_validator('*') on Spec and on GivenSpec, and field_validator('cases') on ExpressionSpec, sort a mapping by key through _by_name. The canonical form's normalised sorts a NameListNode and the columns of a ColumnsNode. The program reads the spec's order through sorted_dims.

Piecewise links and a macro's args: keep the file's order, because their position carries meaning.

Change	Caught by
a dims: list kept as written	447 failures
a relation side written as a list kept as written	16 failures
a relation side written as a mapping kept as written	31 failures
GivenSpec keeps the file's order	121 failures
cases: keeps the file's order	5 failures
Spec keeps the file's order	346 failures
the form prints a list of names as written	1 failure
the form prints a relation's columns as written	1 failure
a macro's kwargs: kept as written	1 failure

Those rows ran on cc2d044 and 887723f.

Not done

• A declared order for dims or entries. It can come later as one order for the whole spec.
• The sides of a comparison are not turned to one direction (0 < p_max to p_max > 0), in a condition or in an expression.
• to_yaml() writes an expression or a condition string as the file wrote it. Only the canonical form reprints them.
• Comments and line layout are not kept by to_yaml(). That would need ruamel.yaml, a new runtime dependency.
• A looser patch rule, where a field that a patch leaves out keeps the base's value. It can come later without breaking a patch that loads now.
• one_of: stays in feat(language): a dimension may list the labels it allows, and a condition that names another label fails at load #915, and missing: stays in feat(language)!: a missing row reads as the value the file names, and missing: neutral is gone #894. Each is its own topic.
• specsolve follows after a release, as a separate PR there.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7L7U7Av5v8wnKV6eDpXfi

[read-the-docs-community]

Documentation build overview

📚 mathspec | 🛠️ Build #35355323 | 📁 Comparing 188f13e against latest (0f892c0)

  🔍 Preview build  

53 files changed · ± 53 modified

± Modified

• index.html
• first-spec/index.html
• several-files/index.html
• examples/commitment/index.html
• examples/dispatch/index.html
• examples/operators/index.html
• examples/pypsa/index.html
• examples/pypsa_linearized_uc/index.html
• howto/compose/index.html
• howto/see-an-expansion/index.html
• and 43 more...

[brynpickering]

I'm fine with imposing ordering strictly everywhere we can. It might come up against opposition later when people see their beautifully crafted specs reordered in a way they didn't expect, but let's see if people complain about that before we make concessions on this. Maximising the value of a spec diff should be our priority and ordering enables that.

[FBumann]

I'm fine with imposing ordering strictly everywhere we can. It might come up against opposition later when people see their beautifully crafted specs reordered in a way they didn't expect, but let's see if people complain about that before we make concessions on this. Maximising the value of a spec diff should be our priority and ordering enables that.

My approach would be to do this now, uniformly and strictly, and soften it if users demand it.

If users always had a sorted spec, they will not miss custom sorting that much I think. And if they do, they probably have good reasons to.

Im not entirely sure if you agree to do it like this. Could you clarify?