Record the capital gains realisation response each UK run applies

changelog.d/602-cgt-provenance.added.md

@@ -0,0 +1 @@
+Added a record of the capital gains realisation response to UK runs (`policyengine.tax_benefit_models.uk.capital_gains_responses`). Each population run records the policyengine-uk release and where it was installed from when that is not a package index, the caller's explicit `gov.simulation.capital_gains_responses` settings in the order they apply and, where the installed policyengine-uk reports it, the response it resolved after every reform and modifier. The record is kept with the output dataset (`metadata`), in the release bundle, the run record and the saved output file; household results carry it under `provenance`, and managed microsimulations under `policyengine_bundle["capital_gains_responses"]` for each year. UK cache and saved-output identity now includes a digest of the explicit settings, and a saved UK output is reused only under the same settings and the same policyengine-uk release from the same source (never from a local directory install). Each saved UK output is therefore calculated once more after upgrading, so a job that relies on a warm UK cache takes as long as a cold run the first time. Added `StaleOutputError`, a `ValueError` that `load()` raises for a saved output it refuses (UK and US alike); `Simulation.ensure()` logs the reason at INFO and runs the simulation again, instead of logging a warning with a traceback.

docs/dev.md

@@ -106,6 +106,6 @@ src/policyengine/
 
 **Country-specific model classes**: `PolicyEngineUSLatest` and `PolicyEngineUKLatest` inherit from a shared `MicrosimulationModelVersion` base (variable/parameter loading, manifest certification, `save`/`load`). Each subclass only implements `run()` and a handful of country hooks (`_load_system`, `_load_region_registry`, `_dataset_class`, `_get_runtime_data_build_metadata`). The US `run` applies reforms as a dict at `Microsimulation(reform=...)` construction time; the UK `run` wraps inputs as `UKSingleYearDataset` and applies reforms via a modifier after construction.
 
-**LRU cache + file caching**: `Simulation.ensure()` checks an in-process LRU cache (max 100 entries), then tries loading from disk, then falls back to `run()` + `save()`.
+**LRU cache + file caching**: `Simulation.ensure()` checks an in-process LRU cache (max 100 entries), then tries loading from disk, then falls back to `run()` + `save()`. When `load()` refuses an out-of-date saved output it raises `StaleOutputError` (a `ValueError`), which `ensure()` logs at INFO as an expected rerun. Any other load error is logged as a warning.
 
 **Output pattern**: All output types inherit from `Output`, implement `.run()`, and populate result fields. Convenience functions (e.g., `calculate_us_poverty_rates()`) create, run, and return collections of output objects.

docs/households.md

@@ -165,6 +165,8 @@ For UK households, a `YYYY-MM-DD` key is effective from that date until the next
 
 UK households reject bare years such as `"2027"` or `2027`, because the underlying packages assign different meanings to them. Use `"2027-04-06.2028-04-05"` for one UK fiscal year or `"2027-01-01"` for an effective date. See [Reforms](reforms.md) for the shared `Simulation(policy=...)` and `Simulation(dynamic=...)` date contract. The US household route retains its country package's period-key handling.
 
+UK household results record the capital gains realisation response the calculation applied, under `result["provenance"]["capital_gains_responses"]`: the policyengine-uk release, the reform's explicit `gov.simulation.capital_gains_responses` settings, and the assumption policyengine-uk resolved (see [The UK capital gains realisation response](microsim.md#the-uk-capital-gains-realisation-response)). `result.write(...)` includes it.
+
 Structural reforms (new variables, formula swaps) require the `Simulation` path — see [Reforms](reforms.md).
 
 ## Year

docs/microsim.md

@@ -30,7 +30,7 @@ total_snap.run()
 total_snap.result
 ```
 
-`Simulation.ensure()` loads a cached result if one exists, or runs and caches on miss. Call `Simulation.run()` explicitly if you want to bypass the cache.
+`Simulation.ensure()` loads a cached result if one exists, or runs and caches on miss. Call `Simulation.run()` explicitly if you want to bypass the cache. A UK simulation's explicit capital gains response settings are part of its cache and saved-output identity; see [The UK capital gains realisation response](#the-uk-capital-gains-realisation-response).
 
 ### Canonical US SPM settings and provenance
 
@@ -288,6 +288,51 @@ checked. A saved UK simulation output records the data year its run anchored
 on; `Simulation.load()` refuses an output saved without one, and
 `Simulation.ensure()` runs it again.
 
+### The UK capital gains realisation response
+
+policyengine-uk scales realised capital gains in response to a reform, at an
+elasticity set under `gov.simulation.capital_gains_responses`. Its default is
+the model's, so it can change between policyengine-uk releases. Each UK run
+records the response it applied, read after every reform and modifier.
+
+- **Where it is kept.** It is in `simulation.release_bundle["capital_gains_responses"]`,
+  the run record's `results.json` and the saved output file.
+- **What it holds.**
+  - `model`: the policyengine-uk release. Its `source` is `null` for a
+    release from a package index; otherwise it says where the install came
+    from: a version control commit, an archive's SHA-256, or a local
+    directory and whether it is editable.
+  - `request`: the caller's explicit settings under that path, in the order
+    they apply (policy, then dynamic), and their digest.
+  - `resolved`: the assumption the release reports. That is the convention,
+    the elasticities for main-rate gains and for gains qualifying for Business
+    Asset Disposal Relief, the MTR elasticity, the separate relief switch, and
+    whether each is the default, an override or set to zero for an MTR
+    elasticity. It is `null` with a release that doesn't report it.
+- **Modifiers.** Changes a simulation modifier makes aren't listed under
+  `request`, as run records don't describe modifiers. `resolved` reflects them.
+  Neither the `storage_id` nor `Simulation.load()` sees them, so two
+  simulations with the same `id` whose modifiers set the response differently
+  share a saved output. Give such simulations different ids.
+- **Identity.** A simulation with explicit settings adds `-cgr-<digest>` to its
+  `storage_id`, so its cache entry and saved output are kept apart from runs
+  with other settings. The digest covers each setting as given, including
+  whether it came from the policy or the dynamic and its dates. So the same
+  value set another way, or with an explicit end date instead of none, is
+  saved separately.
+- **Reuse.** `Simulation.load()` reuses a saved UK output only when its record
+  shows the same policyengine-uk release, installed from the same source, and
+  the same settings. A branch installed from git or a local directory can
+  carry the version number of a published release with another default.
+  With policyengine-uk installed from a local directory, editable or not, no
+  saved UK output is reused, as its code can change without its version
+  changing.
+- **Reruns.** Otherwise `load()` raises `StaleOutputError`, and
+  `Simulation.ensure()` runs the simulation again and logs the reason at INFO.
+  Outputs saved before outputs recorded the response are calculated once
+  more, so a job that relies on a warm UK cache takes as long as a cold run
+  the first time it runs after the upgrade.
+
 ## Simulations
 
 A `Simulation` needs a dataset, a tax-benefit model version, and optionally a policy (reform):
@@ -379,6 +424,14 @@ Pass `allow_unmanaged=True` with a custom `dataset=` to opt out of the release
 bundle. Explicit local paths and Hugging Face URIs remain supported in this
 mode. GCS dataset URIs are not supported.
 
+A UK managed microsimulation also records, in
+`sim.policyengine_bundle["capital_gains_responses"]`, the capital gains
+realisation response it applies in each year of its dataset (see
+[The UK capital gains realisation response](#the-uk-capital-gains-realisation-response)).
+Its `reform=` goes to policyengine-uk unchanged, so no request is listed.
+Settings that conflict are recorded as an error for that year, and raise when
+the response is calculated.
+
 For managed simulations, `sim.policyengine_bundle` records the actual source
 package, repository type, revision, verified SHA-256, and local path.
 

docs/run-records.md

@@ -46,7 +46,7 @@ The directory contains:
 | `bundle.trace.tro.jsonld` | The certified bundle TRO (model + data pins) |
 | `reform.json` | The reform as parameter values with effective dates |
 | `input.json` | Input dataset hash, dynamics, scoping, extra variables |
-| `results.json` | Output dataset hash and per-entity table summaries; for US runs, the SPM receipt and the renamed stored inputs mapped onto live ones (see [Renamed inputs in stored data](microsim.md#renamed-inputs-in-stored-data)) |
+| `results.json` | Output dataset hash and per-entity table summaries; for US runs, the SPM receipt and the renamed stored inputs mapped onto live ones (see [Renamed inputs in stored data](microsim.md#renamed-inputs-in-stored-data)); for UK runs, the capital gains realisation response the run applied (see [The UK capital gains realisation response](microsim.md#the-uk-capital-gains-realisation-response)) |
 
 All payload files are written with the same canonical JSON used for
 hashing, so the record verifies offline exactly as written.

src/policyengine/core/__init__.py

@@ -27,6 +27,7 @@
 )
 from .simulation import Simulation as Simulation
 from .tax_benefit_model import TaxBenefitModel as TaxBenefitModel
+from .tax_benefit_model_version import StaleOutputError as StaleOutputError
 from .tax_benefit_model_version import (
     TaxBenefitModelVersion as TaxBenefitModelVersion,
 )

src/policyengine/core/run_record.py

@@ -20,6 +20,7 @@
 
 from __future__ import annotations
 
+import copy
 from dataclasses import dataclass, field
 from pathlib import Path
 from typing import TYPE_CHECKING, Any, Mapping, Optional, Union
@@ -202,6 +203,29 @@ def build_simulation_run_record_payloads(
     if legacy_spm_declarations is not None:
         results_payload["legacy_spm_declarations"] = dict(legacy_spm_declarations)
 
+    # The capital gains realisation response a UK run applied: the model
+    # resolves its default, so the record binds the assumption itself. See
+    # ``policyengine.tax_benefit_models.uk.capital_gains_responses``.
+    capital_gains_responses = (
+        getattr(simulation.output_dataset, "metadata", None) or {}
+    ).get("capital_gains_responses")
+    if capital_gains_responses is not None:
+        from policyengine.tax_benefit_models.uk.capital_gains_responses import (
+            simulation_request,
+        )
+
+        if (
+            capital_gains_responses["request"]["digest"]
+            != simulation_request(simulation)["digest"]
+        ):
+            raise ValueError(
+                "Capital gains response settings changed since this output was "
+                "calculated; run again before writing a run record"
+            )
+        results_payload["capital_gains_responses"] = copy.deepcopy(
+            capital_gains_responses
+        )
+
     return {"reform": reform, "input": input_payload, "results": results_payload}
 
 

src/policyengine/core/simulation.py

@@ -1,3 +1,4 @@
+import copy
 import hashlib
 import json
 import logging
@@ -13,7 +14,7 @@
 from .policy import Policy
 from .scoping_strategy import ScopingStrategy
 from .spm import SPMProvenance, SPMSelection
-from .tax_benefit_model_version import TaxBenefitModelVersion
+from .tax_benefit_model_version import StaleOutputError, TaxBenefitModelVersion
 
 logger = logging.getLogger(__name__)
 
@@ -124,12 +125,22 @@ def spm_provenance(self) -> Optional[dict[str, Any]]:
 
     @property
     def storage_id(self) -> str:
-        """Include resolved SPM settings in cache and saved-result identity."""
+        """Include resolved SPM settings, and a UK run's explicit capital
+        gains response settings, in cache and saved-result identity."""
+        storage_id = self.id
         config = self.spm_config
-        if config is None:
-            return self.id
-        encoded = json.dumps(config, sort_keys=True, separators=(",", ":")).encode()
-        return f"{self.id}-spm-{hashlib.sha256(encoded).hexdigest()}"
+        if config is not None:
+            encoded = json.dumps(config, sort_keys=True, separators=(",", ":"))
+            storage_id += f"-spm-{hashlib.sha256(encoded.encode()).hexdigest()}"
+        if getattr(self.tax_benefit_model_version, "country_code", None) == "uk":
+            from policyengine.tax_benefit_models.uk.capital_gains_responses import (
+                simulation_request,
+            )
+
+            digest = simulation_request(self)["digest"]
+            if digest is not None:
+                storage_id += f"-cgr-{digest}"
+        return storage_id
 
     @model_validator(mode="after")
     def _compile_dict_reforms(self) -> "Simulation":
@@ -192,6 +203,12 @@ def ensure(self):
         except FileNotFoundError:
             self.run()
             self.save()
+        except StaleOutputError as error:
+            # An expected rerun, such as the first after an upgrade that
+            # changes what saved outputs record.
+            logger.info("Running simulation %s again: %s", self.id, error)
+            self.run()
+            self.save()
         except Exception:
             logger.warning(
                 "Unexpected error loading simulation %s; falling back to run()",
@@ -266,4 +283,11 @@ def release_bundle(self) -> dict[str, Any]:
         ).get("legacy_spm_declarations")
         if legacy_spm_declarations is not None:
             result["legacy_spm_declarations"] = dict(legacy_spm_declarations)
+        # The capital gains realisation response a UK run applied. See
+        # ``policyengine.tax_benefit_models.uk.capital_gains_responses``.
+        capital_gains_responses = (
+            getattr(self.output_dataset, "metadata", None) or {}
+        ).get("capital_gains_responses")
+        if capital_gains_responses is not None:
+            result["capital_gains_responses"] = copy.deepcopy(capital_gains_responses)
         return result

src/policyengine/core/tax_benefit_model_version.py

@@ -24,6 +24,16 @@
     from .variable import Variable
 
 
+class StaleOutputError(ValueError):
+    """A saved output that ``load()`` refuses to reuse, and why.
+
+    The output was calculated under something that has since changed (a
+    model release, settings, or how outputs are recorded).
+    ``Simulation.ensure()`` runs the simulation again and logs the reason at
+    INFO, as an expected rerun rather than an error.
+    """
+
+
 class TaxBenefitModelVersion(BaseModel):
     model_config = {"arbitrary_types_allowed": True}
 
@@ -77,6 +87,8 @@ def save(self, simulation: "Simulation"):
         )
 
     def load(self, simulation: "Simulation"):
+        """Load a saved output: ``FileNotFoundError`` when there is none,
+        ``StaleOutputError`` when it must not be reused."""
         raise NotImplementedError(
             "The TaxBenefitModel class must define a method to load simulations."
         )