Bind combined runs as an ordered composition of reform identities

changelog.d/578.added.md

@@ -0,0 +1 @@
+Added `composed_reform_identity` and the `policyengine-uk.scenario_composition/v1` representation, so an execution receipt can bind several reforms applied in one run, in order.

docs/receipts.md

@@ -107,6 +107,21 @@ A measures file edited after import, or a constant, default, import or function
 
 Some modifiers are refused outright because their behaviour depends on values the digest cannot bind: closures, lambdas, nested functions, methods, `functools.partial` and builtins. Anything that is not a `Scenario` is refused too, rather than read as current law.
 
+### Several reforms in one run
+
+A combined run, such as a whole Budget package, applies several measures in one simulation. Bind each measure on its own, then bind the package with `composed_reform_identity(members, applied_before_data_load=...)`, passing the members' identities in application order and the phase the combined `Scenario` runs in. Its representation is `policyengine-uk.scenario_composition/v1`, and its digest covers the phase and each member's representation, digest and binding, in that order.
+
+The representation means one simulation, in the stated phase, that applies every member's parameter changes and then runs each member's simulation modifier in member order, whatever the policyengine-uk version. That order is part of the payload (`"application": "parameter_changes_then_modifiers"`). The composer must run exactly that, for example as a single simulation modifier. It is not `Scenario.__add__`: that merge depends on how members are grouped, and it leaves the order to the engine (policyengine-uk 2.125.1 runs a Scenario's modifier before its parameter changes). The rules that follow:
+- The phase has no default. The composition's phase is the one that runs, whatever phase a member scenario had on its own.
+- No two members may change the same parameter, in any form, not even to the same value. With disjoint changes the result does not depend on grouping or order. The composer must check this; `composed_reform_identity` sees only identities, so it cannot.
+- Order is part of the digest: the same members in another order give a different digest. A nested composition's members are applied as if listed in its place.
+- A baseline shared by several members (one that applies parameter changes only) appears once, at the position of the first member that shares it; being parameter changes, its position does not affect the run.
+- Each member is revalidated, so an identity edited past its rules is refused rather than dropped or hashed.
+- A member with a live re-check handle (from `scenario_reform_identity`, or a composition) is recomputed before composing and refused if it changed. The package's identity keeps a handle that recomposes from those members, so both receipt builders re-check every member, nested compositions included, after the run. An identity loaded from JSON has no handle, as for any identity.
+- `current_law` members are dropped, and a composition with no remaining member is `current_law`. The binding is `code_revision` if any member's is.
+
+A reader who holds the members' identities can recompute the composition's digest and compare.
+
 To check a certificate, call `verify_certificate(receipt, bundle)` with the policyengine manifest for `certificate.bundle_version`. It rebuilds the certificate from that manifest's certification record, the same way the builder does. Then it compares every identity member: the record digest, the basis, the certified model, core and dataset digest, the certifier and the bundle version. It also checks that the receipt's whole `data` identity is the certified dataset with the certified bytes. An empty list means the bundle certifies exactly what the receipt claims, and through the receipt's own rules that pins `model.version`, `core.version` and `wrapper.version` too. `certificate_source`, `qualification_receipt_sha256`, `tro_composition_fingerprint` and wheel hashes are not determined by the bundle, so it does not check them.
 
 Pass the release artifact that was loaded as `dataset_path`. A derived per-year file such as `enhanced_frs_2024_25_year_2026.h5` has different bytes from the artifact the bundle pins, so a receipt for it is refused.
@@ -140,10 +155,10 @@ The pre-promotion replay interface with policyengine-scorecard follows a differe
 | `execution_identity_sha256` | `schema_version`, `country_id`, `wrapper`, `model`, `core`, `data`, `certification_status`, and the certificate's content: bundle version, record digest, basis, certified versions, dataset digest and certifier. Results with the same execution identity recorded the same package identities (versions, install origins and wheel evidence) and the same data under the same certificate, however the receipt was built. For a certified receipt those identities name the code that ran. An uncertified one does not bind its packages' contents: two runs from an editable checkout edited in between share an execution identity. |
 | `receipt_sha256` | The whole receipt except `runtime` and the reform and baseline `name` labels. Renaming a measure therefore does not change it. |
 | `certificate.certificate_sha256` | The bundle manifest's `data_releases.uk` record. |
-| `reform.reform_sha256` | `{"representation": ..., "payload": ...}`, with the payload exactly as given, nulls kept (strict JSON conversion, then RFC 8785, with no null removal). |
+| `reform.reform_sha256` | `{"representation": ..., "payload": ...}`, with the payload exactly as given, nulls kept (strict JSON conversion, then RFC 8785, with no null removal). For a composition the payload is `{"applied_before_data_load": ..., "application": "parameter_changes_then_modifiers", "members": [{"representation", "reform_sha256", "binding"}, ...]}`, with members in application order. |
 | `outputs_sha256` | The output payload. |
 
-Reform digests are never normalized across representations. The same keys can legitimately mean different things in different representations: for example, a bare year in a policyengine-uk country dictionary changes that one year only. So each digest is tied to the representation that was submitted. The v1 representations are `policyengine.parameter_values/v1`, `policyengine.reform_dict/v1`, `policyengine-uk.reform_dict/v1`, `policyengine-uk.scenario/v1` and `policyengine-sim-api.policy/v1`.
+Reform digests are never normalized across representations. The same keys can legitimately mean different things in different representations: for example, a bare year in a policyengine-uk country dictionary changes that one year only. So each digest is tied to the representation that was submitted. The v1 representations are `policyengine.parameter_values/v1`, `policyengine.reform_dict/v1`, `policyengine-uk.reform_dict/v1`, `policyengine-uk.scenario/v1`, `policyengine-uk.scenario_composition/v1` and `policyengine-sim-api.policy/v1`.
 
 ## Rules JSON Schema cannot express
 

src/policyengine/data/schemas/execution_receipt.v1.schema.json

@@ -438,7 +438,8 @@
                 "policyengine.reform_dict/v1",
                 "policyengine-uk.reform_dict/v1",
                 "policyengine-sim-api.policy/v1",
-                "policyengine-uk.scenario/v1"
+                "policyengine-uk.scenario/v1",
+                "policyengine-uk.scenario_composition/v1"
               ],
               "type": "string"
             },
@@ -700,7 +701,7 @@
     "execution_identity_sha256": "{schema_version, country_id, wrapper, model, core, data, certification_status, certificate?} where certificate keeps only bundle_version, country_id, certificate_sha256, compatibility_basis, certified_for_model_version, certified_for_core_version, certified_dataset_sha256 and certified_by",
     "outputs_sha256": "the canonical output payload the receipt binds",
     "receipt_sha256": "the whole receipt without runtime and without reform.name and baseline.name",
-    "reform.reform_sha256": "RFC 8785 over {representation, payload} where payload is the reform exactly as given in that representation, nulls kept (a null in a reform can mean something, unlike an absent receipt field)"
+    "reform.reform_sha256": "RFC 8785 over {representation, payload} where payload is the reform exactly as given in that representation, nulls kept (a null in a reform can mean something, unlike an absent receipt field); for policyengine-uk.scenario_composition/v1 the payload is {applied_before_data_load, application: parameter_changes_then_modifiers, members: [{representation, reform_sha256, binding}, ...]} with members in application order"
   },
   "x-policyengine-rules": [
     "years is sorted ascending with no duplicates, and each year is written as an integer (2027, not 2027.0).",

src/policyengine/provenance/__init__.py

@@ -99,6 +99,9 @@
 from .receipt import (
     build_execution_receipt as build_execution_receipt,
 )
+from .receipt import (
+    composed_reform_identity as composed_reform_identity,
+)
 from .receipt import (
     reform_identity as reform_identity,
 )

src/policyengine/provenance/receipt.py

@@ -51,6 +51,8 @@
   caller's code revision,
   and says so (``binding="code_revision"``). The digest is the same wherever
   the code is checked out and whichever Python runs it.
+- Several reforms applied in one run are bound as an ordered composition of
+  their identities (:func:`composed_reform_identity`).
 - A certificate is trusted only after :func:`verify_certificate` has checked
   it, and the receipt's data identity, against the published bundle manifest
   it names. ``certificate_source`` is the producer's statement, not evidence.
@@ -141,6 +143,7 @@
     "build_execution_receipt",
     "build_execution_receipt_for_microsimulation",
     "certificate_sha256",
+    "composed_reform_identity",
     "content_sha256",
     "fiscal_year_label",
     "installed_package_identity",
@@ -183,8 +186,16 @@
     # A policyengine-uk ``Scenario`` built from parameter changes and, at
     # most, a module-level modifier: see scenario_reform_identity().
     "policyengine-uk.scenario/v1",
+    # Several reforms applied in one policyengine-uk ``Scenario``, in a stated
+    # order: see composed_reform_identity().
+    "policyengine-uk.scenario_composition/v1",
 ]
 
+#: Representation of a composition of member reforms.
+COMPOSITION_REPRESENTATION: Literal["policyengine-uk.scenario_composition/v1"] = (
+    "policyengine-uk.scenario_composition/v1"
+)
+
 #: v1 rules that JSON Schema cannot express. Published in the schema file.
 CROSS_FIELD_RULES: tuple[str, ...] = (
     "years is sorted ascending with no duplicates, and each year is written as an integer (2027, not 2027.0).",
@@ -207,7 +218,7 @@
     "execution_identity_sha256": "{schema_version, country_id, wrapper, model, core, data, certification_status, certificate?} where certificate keeps only bundle_version, country_id, certificate_sha256, compatibility_basis, certified_for_model_version, certified_for_core_version, certified_dataset_sha256 and certified_by",
     "receipt_sha256": "the whole receipt without runtime and without reform.name and baseline.name",
     "certificate.certificate_sha256": "the bundle manifest's data_releases.<country_id> record",
-    "reform.reform_sha256": "RFC 8785 over {representation, payload} where payload is the reform exactly as given in that representation, nulls kept (a null in a reform can mean something, unlike an absent receipt field)",
+    "reform.reform_sha256": "RFC 8785 over {representation, payload} where payload is the reform exactly as given in that representation, nulls kept (a null in a reform can mean something, unlike an absent receipt field); for policyengine-uk.scenario_composition/v1 the payload is {applied_before_data_load, application: parameter_changes_then_modifiers, members: [{representation, reform_sha256, binding}, ...]} with members in application order",
     "outputs_sha256": "the canonical output payload the receipt binds",
 }
 
@@ -806,6 +817,8 @@ def reform_identity(
         raise ReceiptError("A non-empty reform needs its representation.")
     if representation == "policyengine-uk.scenario/v1":
         raise ReceiptError("Bind a Scenario with scenario_reform_identity().")
+    if representation == COMPOSITION_REPRESENTATION:
+        raise ReceiptError("Bind a composition with composed_reform_identity().")
     digest = _payload_sha256({"representation": representation, "payload": payload})
     return ReformIdentity(
         kind="reform", representation=representation, reform_sha256=digest, name=name
@@ -2705,6 +2718,127 @@ def checked(identity: ReformIdentity) -> ReformIdentity:
     )
 
 
+#: The order a ``policyengine-uk.scenario_composition/v1`` run applies its
+#: members in. It is part of the payload, so the digest states it.
+COMPOSITION_APPLICATION = "parameter_changes_then_modifiers"
+
+
+def composed_reform_identity(
+    members: Sequence[ReformIdentity],
+    *,
+    applied_before_data_load: bool,
+    name: Optional[str] = None,
+) -> ReformIdentity:
+    """Bind several reforms applied together in one run, in a stated order.
+
+    ``members`` are complete identities from :func:`reform_identity`,
+    :func:`scenario_reform_identity` or this function, in application order.
+    The digest covers ``applied_before_data_load``, the application order and
+    each member's representation, digest and binding, in member order, and
+    nothing else: member labels are not part of it.
+
+    What ``policyengine-uk.scenario_composition/v1`` means, whatever the
+    policyengine-uk version: one simulation, in the phase given by
+    ``applied_before_data_load``, that applies every member's parameter
+    changes (a reform dictionary member's entries, a scenario member's
+    ``parameter_changes``) and then runs each member's simulation modifier in
+    member order. This is what the composer must run, for example as a
+    single simulation modifier; it is not ``Scenario.__add__``, whose merge
+    depends on how members are grouped and which leaves the order to the
+    engine (policyengine-uk 2.125.1 runs a Scenario's modifier before its
+    parameter changes). The composer must also make sure no two members
+    change the same parameter, in any form: with disjoint changes the merged
+    result does not depend on grouping or order. This function sees only
+    identities and cannot check either.
+
+    A member's own phase is overridden by the composition's, and a nested
+    composition's members are applied as if listed in its place.
+
+    - Each member is revalidated, so an identity edited past its model's
+      rules (``model_copy(update=...)``) is refused rather than dropped.
+    - A member with a re-check handle (from :func:`scenario_reform_identity`,
+      or a composition) is recomputed first and refused if it changed, and
+      the result keeps a handle that recomposes from those members, so the
+      receipt builders re-check every member again.
+    - ``current_law`` members change nothing and are dropped; if none
+      remains the result is ``current_law``.
+    - A single remaining member still gets the composition representation,
+      so its digest differs from the member's own, as across any two
+      representations.
+    - The binding is ``code_revision`` if any member's is, else ``content``.
+    """
+
+    if type(applied_before_data_load) is not bool:
+        raise ReceiptError("applied_before_data_load must be True or False.")
+    if isinstance(members, (str, bytes, Mapping)) or not isinstance(members, Sequence):
+        raise ReceiptError("A composition is a sequence of reform identities.")
+    # The members as given, handles included; a later edit to the caller's
+    # list does not change what the handle recomposes.
+    members = tuple(members)
+
+    def checked(identity: ReformIdentity) -> ReformIdentity:
+        object.__setattr__(
+            identity,
+            "_recompute_identity",
+            lambda: composed_reform_identity(
+                members,
+                applied_before_data_load=applied_before_data_load,
+                name=name,
+            ),
+        )
+        return identity
+
+    bound: list[dict[str, Any]] = []
+    binding: Literal["content", "code_revision"] = "content"
+    for index, member in enumerate(members):
+        if not isinstance(member, ReformIdentity):
+            raise ReceiptError(
+                f"Member {index} is {type(member).__name__}, not a ReformIdentity; "
+                "bind it with reform_identity() or scenario_reform_identity() first."
+            )
+        try:
+            member._recheck()
+            member = ReformIdentity.model_validate(member.model_dump())
+        except ReceiptError as exc:
+            raise ReceiptError(f"Member {index}: {exc}") from exc
+        except ValueError as exc:  # pydantic's ValidationError included
+            raise ReceiptError(
+                f"Member {index} is not a valid identity: {exc}"
+            ) from exc
+        if member.kind == "current_law":
+            continue
+        bound.append(
+            {
+                "representation": member.representation,
+                "reform_sha256": member.reform_sha256,
+                "binding": member.binding,
+            }
+        )
+        if member.binding == "code_revision":
+            binding = "code_revision"
+    if not bound:
+        return checked(ReformIdentity(kind="current_law", name=name))
+    digest = _payload_sha256(
+        {
+            "representation": COMPOSITION_REPRESENTATION,
+            "payload": {
+                "applied_before_data_load": applied_before_data_load,
+                "application": COMPOSITION_APPLICATION,
+                "members": bound,
+            },
+        }
+    )
+    return checked(
+        ReformIdentity(
+            kind="reform",
+            representation=COMPOSITION_REPRESENTATION,
+            reform_sha256=digest,
+            binding=binding,
+            name=name,
+        )
+    )
+
+
 def _distribution_metadata_directory(name: str) -> Path:
     """Locate dist-info through the distribution's public metadata API."""