feat(language): a file may declare sets of labels and the properties of a dimension or a set

CHANGELOG.md

@@ -12,6 +12,7 @@ it releases that version ([RELEASING.md](https://github.com/energy-models/mathsp
 
 ## Upcoming version
 
+- feat(language): a file may declare sets of labels and the properties of a dimension or a set ([#1](https://github.com/fluxopt/mathspec/pull/1))
 - feat(language): a parameter may declare the coordinates it has with `where:`, and outside them it is absent ([#945](https://github.com/energy-models/mathspec/pull/945))
 - fix(language): a divisor that a where, a case or a masked variable guards loads under missing: 0, and so does a variable number outside its bounds ([#942](https://github.com/energy-models/mathspec/pull/942))
 - docs(pypsa): a PyPSA schedule, weight or limit that the data leaves out is absent rather than zero, and a committable extendable unit is assumed to have its big-M ([#912](https://github.com/energy-models/mathspec/pull/912))

docs/reference/glossary.md

@@ -46,7 +46,7 @@ A consumer is responsible for checking that the host provides each given name
 ([what a program does not build](reading.md#what-a-program-does-not-build)).
 
 **Section**
-: One of the thirteen top-level keys of a file, such as `parameters:` or
+: One of the fifteen top-level keys of a file, such as `parameters:` or
 `given:`, and the entries under it ([file shape](language/file.md)).
 
 **Entry**

docs/reference/language/file.md

@@ -5,14 +5,16 @@ SPDX-License-Identifier: CC-BY-4.0
 
 # File shape
 
-A spec file is a YAML mapping with **thirteen section keys**, plus
-`version` and `description`, and any subset of the thirteen is accepted.
+A spec file is a YAML mapping with **fifteen section keys**, plus
+`version` and `description`, and any subset of the fifteen is accepted.
 
 | Key           |                                                                                                           |
 | ------------- | --------------------------------------------------------------------------------------------------------- |
 | `given`       | what this file reads but does not build ([given](parameters-variables-constraints.md#given))              |
 | `dimensions`  | the dimensions ([dimensions](dimensions.md))                                                              |
 | `relations`   | named relations between dimensions ([relations](relations.md))                                            |
+| `sets`        | which combinations of labels exist ([sets](sets.md#sets))                                                 |
+| `properties`  | a label for each row of a dimension or a set ([properties](sets.md#properties))                           |
 | `parameters`  | the data the spec expects ([parameters](parameters-variables-constraints.md#parameters))                  |
 | `conditions`  | named conditions, read by name in a `where:`, a `when:` or a `holds:` ([conditions](named.md#conditions)) |
 | `variables`   | what the solver decides                                                                                   |

docs/reference/language/index.md

@@ -47,7 +47,7 @@ names the fix.
 
 | #   | Rule                                                                                                                                                                               |                                                                 |
 | --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
-| 1   | A file has thirteen section keys, plus `version` and `description`. An unknown key is refused, with the nearest valid key named.                                                   | [File shape](file.md)                                           |
+| 1   | A file has fifteen section keys, plus `version` and `description`. An unknown key is refused, with the nearest valid key named.                                                    | [File shape](file.md)                                           |
 | 2   | `to_spec` checks everything it can check without data when the file loads.                                                                                                         | [Errors](errors.md)                                             |
 | 3   | Every name is declared once. A parameter and a dimension both called `snapshot` is refused.                                                                                        | [Names](expressions.md#name-resolution)                         |
 | 4   | The kind of a name decides where it may stand. A dimension follows `over=` or `along=`, and is never multiplied.                                                                   | [Names](expressions.md#name-resolution)                         |

docs/reference/language/sets.md

@@ -0,0 +1,144 @@
+<!--
+SPDX-FileCopyrightText: mathspec contributors
+SPDX-License-Identifier: CC-BY-4.0
+-->
+
+# Sets and properties
+
+You declare which combinations of labels exist as a **set**, and a fact about
+each row of a dimension or a set as a **property**. The data supplies the rows
+of both.
+
+```yaml
+dimensions:
+  generator: { dtype: str }
+  bus: { dtype: str }
+  line: { dtype: str }
+  period: { dtype: int }
+  zone: { dtype: str }
+sets:
+  connection: { dims: [generator, bus], missing: absent } # the pairs that are wired
+  generator_period: { dims: [generator, period] }
+properties:
+  generator:
+    bus: bus # each generator is on one bus
+  line:
+    bus0: bus
+    bus1: { into: bus, missing: absent } # a line may have an open end
+  generator_period:
+    zone: zone # each generator is in one zone in each period
+```
+
+## `sets`
+
+A set is a table whose rows are combinations of labels. `connection` holds the
+pairs of a generator and a bus that are wired, and every other pair is not in
+the set.
+
+| Field         |                                                                     |                   |
+| ------------- | ------------------------------------------------------------------- | ----------------- |
+| `dims`        | required. The dimensions a row combines, as a list or as a mapping  |                   |
+| `missing`     | `refused` or `absent`: whether every combination of `dims` is a row | default `refused` |
+| `description` | free text                                                           | default `null`    |
+
+A set has at least two dimensions. For a subset of one dimension, declare a
+`bool` parameter. Where two members of a set are over one dimension, the
+mapping form names them, and each name is a column of the set's table:
+
+```yaml
+dimensions:
+  generator: { dtype: str }
+  snapshot: { dtype: int, ordered: true }
+sets:
+  cover: { dims: { generator: generator, start: snapshot, covered: snapshot }, missing: absent }
+```
+
+A member is named after its own dimension or after no dimension, so
+`dims: {bus: generator, zone: zone}` is refused. A set names dimensions only,
+and a set in the `dims:` of another set is refused.
+
+### Entries over a set
+
+No parameter, variable, constraint or expression is over a set. An entry is
+over the dimensions of the set, and its `where:` names the set:
+
+```yaml
+dimensions:
+  generator: { dtype: str }
+  bus: { dtype: str }
+  snapshot: { dtype: int }
+sets:
+  connection: { dims: [generator, bus], missing: absent }
+parameters:
+  load: { dims: [bus, snapshot] }
+variables:
+  flow:
+    dims: [generator, bus, snapshot]
+    where: connection
+    bounds: { lower: 0 }
+constraints:
+  balance:
+    dims: [bus, snapshot]
+    expression: sum(flow, over=generator) == load
+```
+
+$$\mathit{flow}_{g,b,t} \ge 0 \qquad \forall\, g \in \mathcal{G},\ b \in \mathcal{B},\ t \in \mathcal{T} \,:\, \left( g,\ b \right) \in \mathrm{connection}$$
+
+`flow` is [absent](absence.md) at every pair that `connection` does not list.
+A parameter's `where:` reads a set in the same way
+([a parameter's mask](parameters-variables-constraints.md#a-parameters-mask)).
+An entry written over the set is refused, and the message names the rewrite:
+
+```text
+Variable 'flow' is over set 'connection', and no entry is over a set. Write dims: [generator, bus, snapshot] with where: connection.
+```
+
+## `properties`
+
+A property maps each row of its **subject** to one label of a dimension. The
+subject is a dimension or a set, and `properties:` lists the properties of each
+subject under the subject's name. `generator: {bus: bus}` gives each generator
+one bus.
+
+| Field         |                                                                |                   |
+| ------------- | -------------------------------------------------------------- | ----------------- |
+| `into`        | required. The dimension the value is a label of                |                   |
+| `missing`     | `refused` or `absent`: whether every row of the subject has it | default `refused` |
+| `description` | free text                                                      | default `null`    |
+
+`bus: bus` is the short form of `bus: {into: bus}`. Two properties of one
+subject may map into one dimension, as `bus0` and `bus1` do. A property of a
+set maps each row of the set, so `generator_period: {zone: zone}` gives each
+generator one zone in each period.
+
+`to_spec` refuses a property in these cases:
+
+- The subject is not a declared dimension or set.
+- `into:` is not a declared dimension. A property maps into one dimension, so
+  for a set, declare one property for each dimension of the set.
+- The property has the name of a member of its subject. `connection: {bus: zone}`
+  is refused, because `bus` is a member of `connection`.
+- The property is named after a dimension it does not map into. `period: zone`
+  is refused. Name the property after what it holds.
+- The subject is a set with two members over one dimension, such as `cover`.
+  A subject has one member over each dimension.
+
+A property name is not declared in the namespace of the file, so two subjects
+can both have a property `bus`.
+
+## A missing row
+
+`missing:` checks a key at two levels, and never a value. A value that is not a
+label of its dimension is refused when the data is attached, so no value fills
+a gap.
+
+| On a     | `missing:` asks                                  | `absent` means                              |
+| -------- | ------------------------------------------------ | ------------------------------------------- |
+| set      | is every combination of `dims` a row?            | a combination with no row is not in the set |
+| property | does every row of the subject have the property? | a row of the subject with no value lacks it |
+
+The default `refused` refuses data that leaves a row out. For
+`generator_period` above, every pair of a generator and a period is a row, and
+every one of those rows has a `zone`. For `connection`, only the listed pairs
+are rows, and a property of `connection` with `missing: refused` needs a value
+at each listed pair, not at every pair of a generator and a bus.

mkdocs.yml

@@ -52,6 +52,7 @@ nav:
           - Parameters, variables, constraints and the objective: reference/language/parameters-variables-constraints.md
           - Dimensions: reference/language/dimensions.md
           - Relations: reference/language/relations.md
+          - Sets and properties: reference/language/sets.md
           - Expressions: reference/language/expressions.md
           - Named expressions, conditions and macros: reference/language/named.md
           - Operators: reference/language/operators.md

schema/mathspec.schema.json

@@ -808,6 +808,49 @@
       "title": "PiecewiseSpec",
       "type": "object"
     },
+    "PropertySpec": {
+      "anyOf": [
+        {
+          "additionalProperties": false,
+          "description": "A fact about each row of a subject, which is a dimension or a set: one label of the dimension it maps into.\n\nDeclared under ``properties:``, keyed by the subject and then by the\nproperty's name. ``bus: bus`` is the short form of ``bus: {into: bus}``::\n\n    properties:\n      generator:\n        bus: bus\n      line:\n        bus0: bus\n        bus1: {into: bus, missing: absent}\n      generator_period:\n        zone: zone\n\n``missing:`` asks whether every row of the subject has the property. The\nvalue is never missing: a row that has the property has one label of\n``into``. The table is data, and arrives under ``subject.property``, with\nthe subject's key columns and one column named after the property.",
+          "properties": {
+            "description": {
+              "anyOf": [
+                {
+                  "type": "string"
+                },
+                {
+                  "type": "null"
+                }
+              ],
+              "default": null,
+              "title": "Description"
+            },
+            "into": {
+              "title": "Into",
+              "type": "string"
+            },
+            "missing": {
+              "default": "refused",
+              "enum": [
+                "refused",
+                "absent"
+              ],
+              "title": "Missing",
+              "type": "string"
+            }
+          },
+          "required": [
+            "into"
+          ],
+          "title": "PropertySpec",
+          "type": "object"
+        },
+        {
+          "type": "string"
+        }
+      ]
+    },
     "RelationSpec": {
       "additionalProperties": false,
       "description": "A named relation between dimensions: the columns a row is keyed by, and the columns that key determines.\n\nEach side is a dimension, a list of them, or a mapping of column name to\ndimension where two columns share one. ``key:`` is the claim the language\nchecks when the data is attached: one row per key tuple, so every ``values:`` column is a\nfunction of it. A relation with no ``values:`` is **bare** \u2014 every column is\nin its key, a row is its own identity, and nothing reads it::\n\n    relations:\n      gen_bus: {key: generator, values: bus}\n      gen_bt: {key: [generator], values: [bus, technology]}\n      zone_of: {key: [generator, period], values: zone}\n      ends: {key: line, values: {bus0: bus, bus1: bus}}\n      connection: {key: [generator, bus]}\n\nA sum joins the table on the columns ``over=`` names and every other key\ncolumn, and groups by the columns ``by=relation[...]`` names. A lookup\njoins on the columns it names. The entry fixes no direction. A\ncolumn is named after its own dimension or after none, so a name in a\ncall reads the same as a column and as a dimension. The map itself is\ndata, and arrives with the rest of it, under the relation's name, one\ncolumn per role.",
@@ -884,6 +927,55 @@
       "title": "RelationSpec",
       "type": "object"
     },
+    "SetSpec": {
+      "additionalProperties": false,
+      "description": "A named set: which combinations of the labels of its dimensions exist.\n\n``dims:`` lists the dimensions a row combines, or maps each member's name to\nits dimension where two members share one. No entry is over a set. An entry\nis over the member dimensions, and ``where:`` names the set::\n\n    sets:\n      connection: {dims: [entity, port], missing: absent}\n      cover: {dims: {generator: generator, start: snapshot, covered: snapshot}, missing: absent}\n\nThe rows are data, and arrive under the set's name, one column per member.\nA set is also the subject of the [`PropertySpec`][] entries declared under\nits name.",
+      "properties": {
+        "description": {
+          "anyOf": [
+            {
+              "type": "string"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Description"
+        },
+        "dims": {
+          "anyOf": [
+            {
+              "items": {
+                "type": "string"
+              },
+              "type": "array"
+            },
+            {
+              "additionalProperties": {
+                "type": "string"
+              },
+              "type": "object"
+            }
+          ],
+          "title": "Dims"
+        },
+        "missing": {
+          "default": "refused",
+          "enum": [
+            "refused",
+            "absent"
+          ],
+          "title": "Missing",
+          "type": "string"
+        }
+      },
+      "required": [
+        "dims"
+      ],
+      "title": "SetSpec",
+      "type": "object"
+    },
     "SosSpec": {
       "additionalProperties": false,
       "description": "A special-ordered set over one dimension of one variable.\n\nOne set per coordinate of the variable's ``dims`` minus ``along``; the\nmembers are the variable's *existing* coordinates along ``along``, in that\ndimension's declared order.\n\n``type: 1`` admits at most one nonzero member, ``type: 2`` at most two,\nand those two consecutive. A consumer with the concept takes the set as\none; [`Spec.expand`][] states it as binaries instead, and the rows it\nwrites multiply by the member's own ``bounds``, which is why a member\nneeds both.",
@@ -1001,7 +1093,7 @@
   },
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "additionalProperties": false,
-  "description": "The declared math \u2014 one YAML file, or one dict, validated. Nothing here has seen data.\n\nA ``Spec`` that exists has passed the whole language: constructing one by\nany route \u2014 ``to_spec``, [`model_validate`][], the constructor \u2014 runs\nevery load-time check, expression pass included, and raises\n[`LanguageError`][mathspec.errors.LanguageError] on a spec the language refuses.\nHolding one is the proof, so nothing downstream checks it again.\n\nThe API is the thirteen sections plus ``version`` and\n``description``, three ways back out \u2014 [`to_dict`][] for the spec as\ndata, [`to_yaml`][] for the file a reviewer reads, [`expand`][] for the\nspec with its formulations written out as plain rows \u2014 and [`program`][], the\nspec typed, which every reader after load walks. Everything else on this\nclass is pydantic's, not a contract this package keeps.",
+  "description": "The declared math \u2014 one YAML file, or one dict, validated. Nothing here has seen data.\n\nA ``Spec`` that exists has passed the whole language: constructing one by\nany route \u2014 ``to_spec``, [`model_validate`][], the constructor \u2014 runs\nevery load-time check, expression pass included, and raises\n[`LanguageError`][mathspec.errors.LanguageError] on a spec the language refuses.\nHolding one is the proof, so nothing downstream checks it again.\n\nThe API is the fifteen sections plus ``version`` and\n``description``, three ways back out \u2014 [`to_dict`][] for the spec as\ndata, [`to_yaml`][] for the file a reviewer reads, [`expand`][] for the\nspec with its formulations written out as plain rows \u2014 and [`program`][], the\nspec typed, which every reader after load walks. Everything else on this\nclass is pydantic's, not a contract this package keeps.",
   "properties": {
     "assumptions": {
       "additionalProperties": {
@@ -1100,6 +1192,17 @@
       "title": "Piecewise",
       "type": "object"
     },
+    "properties": {
+      "additionalProperties": {
+        "additionalProperties": {
+          "$ref": "#/$defs/PropertySpec"
+        },
+        "type": "object"
+      },
+      "default": {},
+      "title": "Properties",
+      "type": "object"
+    },
     "relations": {
       "additionalProperties": {
         "$ref": "#/$defs/RelationSpec"
@@ -1108,6 +1211,14 @@
       "title": "Relations",
       "type": "object"
     },
+    "sets": {
+      "additionalProperties": {
+        "$ref": "#/$defs/SetSpec"
+      },
+      "default": {},
+      "title": "Sets",
+      "type": "object"
+    },
     "sos": {
       "additionalProperties": {
         "$ref": "#/$defs/SosSpec"