Fix disable_simulated_benefits and read one claimant-or-partner award

changelog.d/disable-simulated-benefits-scoped-awards.fixed.md

@@ -0,0 +1,4 @@
+- Fix the `disable_simulated_benefits` reform (`gov.contrib.policyengine.disable_simulated_benefits`), which raised an error before setting any benefit. The simulation now links itself to its tax-benefit system before applying structural reforms, so a reform that sets inputs can reach it (the public service budget adjustments failed on the same missing link); a clone or branch that applies a reform links the system to itself while it does, so the reform never sets its parent's inputs. A reform applied after the simulation is built (one passed as `reform=`, or applied to a built simulation, clone or branch) now keeps through `reset_calculations` and `derivative` every input it stores and leaves in place, however it stores it, including one that replaces an input set since the build; an input set since the build that the reform did not store is still reset, as before. The reform reads the dataset year whether it is stored as text or a number. It no longer lists Attendance Allowance, DLA and PIP, which have no reported amount: the data gives their receipt as a rate category and the model pays that category's rate.
+- Under `disable_simulated_benefits`, set the claimant's and partner's income-related ESA and income-based JSA (`claimant_or_partner_esa_income`, `claimant_or_partner_jsa_income`) for each year from the dataset-year reports of whoever is the claimant or partner that year, and set Income Support from their reports alone. The reform holds each benefit at its dataset-year amount while the reported amounts are uprated, so in a later year the stored award matched neither reading of the reports and another member's award was taken to be the claimant's. Another member's own Income Support report is left out of household income, as it is with the reform off, where Income Support is calculated only for the claimant's family.
+- Neutralizing `esa_income` or `jsa_income` also neutralizes the claimant's and partner's award, so an award entered for them, or set by the reform, no longer outlives the benefit it is part of.
+- Read the claimant's and partner's income-related ESA and income-based JSA in `income_support_eligible` from `claimant_or_partner_esa_income` and `claimant_or_partner_jsa_income`, as the means tests and passports do, so an award entered for them directly, or set by the reform, is read the same way everywhere. The two variables share one formula, which now treats an award within half a penny of zero as no award, as `income_support_eligible` did, so the float residual left by reports that exactly match the tariff income is no longer the claimant's or partner's award, and no passport or status that reads their award counts it. A member outside the couple is still on their own award as before: it is read from their own report, without that rule. When `esa_income` or `jsa_income` holds the plain total of the reported awards, Income Support now reads the plain total of the claimant's and partner's reports, as the two variables already did; it used to read their award after the capital test.

policyengine_uk/reforms/policyengine/disable_simulated_benefits.py

@@ -1,58 +1,131 @@
 from policyengine_core.model_api import *
 
+# Benefits whose simulated amount the reform replaces with the amount reported
+# in the dataset year. Each needs a <name>_reported variable. Attendance
+# Allowance, DLA and PIP are not listed: the data records their receipt as a
+# rate category (aa_category, dla_m_category, dla_sc_category, pip_dl_category,
+# pip_m_category), and the model pays that category's rate. The model has no
+# reported amount for them to copy.
+BENEFITS = [
+    "afcs",
+    "bsp",
+    "carers_allowance",
+    "child_benefit",
+    "child_tax_credit",
+    "council_tax_benefit",
+    "esa_contrib",
+    "esa_income",
+    "housing_benefit",
+    "iidb",
+    "incapacity_benefit",
+    "income_support",
+    "jsa_contrib",
+    "jsa_income",
+    "pension_credit",
+    "sda",
+    "ssmg",
+    "state_pension",
+    "universal_credit",
+    "winter_fuel_allowance",
+    "working_tax_credit",
+]
+
+# Tax credits are not paid below a minimum award, and each has a variable
+# before that minimum, which takes the same reported amount.
+PRE_MINIMUM = {
+    "child_tax_credit": "child_tax_credit_pre_minimum",
+    "working_tax_credit": "working_tax_credit_pre_minimum",
+}
+
+# Benefit-unit awards that cover only the claimant and partner, set from their
+# own reports. The reported amounts of anyone else in the benefit unit (such
+# as a non-dependent adult, who claims in their own right) stay in esa_income
+# and jsa_income, which count in household income. They never enter the
+# claimant's or partner's award.
+CLAIMANT_OR_PARTNER_AWARDS = {
+    "esa_income": "claimant_or_partner_esa_income",
+    "jsa_income": "claimant_or_partner_jsa_income",
+}
+
+# Income Support is the claimant's family's award, so it is set from the
+# claimant's and partner's reports only. That is all the model ever pays as
+# income_support: with the reform off, would_claim_IS and
+# income_support_eligible read only their reports too. Another member's own
+# Income Support report (a non-dependent adult claiming in their own right) is
+# in household income neither with the reform nor without it. ESA and JSA
+# differ: esa_income and jsa_income count every member's award either way.
+CLAIMANT_OR_PARTNER_ONLY = ["income_support"]
+
+YEARS_IN_FUTURE = 10
+
 
 def disable_simulated_benefits(parameters, period):
     if parameters(period).gov.contrib.policyengine.disable_simulated_benefits:
 
         class DisableSimulatedBenefits(Reform):
+            """Use the dataset year's reported benefit amounts in every year.
+
+            Each listed benefit is set, in the dataset year and the nine years
+            after it, to its amount reported in the dataset year. The amount
+            is not uprated, while most of the *_reported variables are.
+
+            The claimant-or-partner awards are set the same way, from the
+            dataset-year reports of whoever is the claimant or partner in
+            each year (is_claimant_or_partner); Income Support is set from
+            those reports alone. So every amount is the same in every year,
+            except that these three change when who is the claimant or
+            partner changes. Readers of those awards, including the
+            Income Support gate, cannot tell an award set here from one
+            entered directly. In a later year, esa_income and jsa_income match
+            neither the award on that year's uprated reports nor their total.
+            Without the claimant-or-partner awards set here, the readers would
+            take the whole of esa_income or jsa_income, including another
+            member's award, to be the claimant's or partner's.
+
+            As the amounts are inputs, a reform that changes a benefit's
+            parameters does not change them. A reform that neutralizes a
+            benefit removes it, and neutralizing esa_income or jsa_income
+            also neutralizes its claimant-or-partner award
+            (CountryTaxBenefitSystem.variable_parts). A neutralized variable
+            ignores what the reform sets for it.
+            """
+
             def apply(self):
                 simulation = self.simulation
+                system = simulation.tax_benefit_system
+                time_period = int(str(simulation.dataset.time_period)[:4])
+
+                def reported(variable, members=None):
+                    """A benefit's reported amount in the dataset year, for
+                    the given members, summed over the benefit's entity."""
+                    amounts = simulation.calculate(variable + "_reported", time_period)
+                    if members is not None:
+                        amounts = amounts * members
+                    entity = system.variables[variable].entity.key
+                    population = simulation.populations[entity]
+                    if population.entity.is_person:
+                        return amounts
+                    return population.sum(amounts)
 
-                BENEFITS = [
-                    "afcs",
-                    "attendance_allowance",
-                    "bsp",
-                    "carers_allowance",
-                    "child_benefit",
-                    "child_tax_credit",
-                    "council_tax_benefit",
-                    "dla_m",
-                    "dla_sc",
-                    "esa_contrib",
-                    "esa_income",
-                    "housing_benefit",
-                    "iidb",
-                    "incapacity_benefit",
-                    "income_support",
-                    "jsa_contrib",
-                    "jsa_income",
-                    "pension_credit",
-                    "pip_dl",
-                    "pip_m",
-                    "sda",
-                    "ssmg",
-                    "state_pension",
-                    "universal_credit",
-                    "winter_fuel_allowance",
-                    "working_tax_credit",
-                ]
-                time_period = simulation.dataset.time_period
-                YEARS_IN_FUTURE = 10
-                for variable in BENEFITS:
-                    entity = simulation.tax_benefit_system.variables[
-                        variable
-                    ].entity.key
-                    reported_value = simulation.calculate(
-                        variable + "_reported", time_period, map_to=entity
+                for year in range(time_period, time_period + YEARS_IN_FUTURE):
+                    claimant_or_partner = simulation.calculate(
+                        "is_claimant_or_partner", year
                     )
-                    for year in range(time_period, time_period + YEARS_IN_FUTURE):
-                        simulation.set_input(variable, year, reported_value)
-
-                    if variable in ["child_tax_credit", "working_tax_credit"]:
-                        # CTC and WTC have their own pre_minimum variables because tax credits aren't paid if
-                        # below a threshold.
-                        variable = variable + "_pre_minimum"
-                        for year in range(time_period, time_period + YEARS_IN_FUTURE):
-                            simulation.set_input(variable, year, reported_value)
+                    values = {}
+                    for variable in BENEFITS:
+                        members = (
+                            claimant_or_partner
+                            if variable in CLAIMANT_OR_PARTNER_ONLY
+                            else None
+                        )
+                        values[variable] = reported(variable, members)
+                        if variable in PRE_MINIMUM:
+                            values[PRE_MINIMUM[variable]] = values[variable]
+                        if variable in CLAIMANT_OR_PARTNER_AWARDS:
+                            values[CLAIMANT_OR_PARTNER_AWARDS[variable]] = reported(
+                                variable, claimant_or_partner
+                            )
+                    for target, value in values.items():
+                        simulation.set_input(target, year, value)
 
         return DisableSimulatedBenefits

policyengine_uk/simulation.py

@@ -50,6 +50,32 @@
 
 from microdf import MicroDataFrame
 
+
+class _InputKeyLog(set):
+    """A simulation's record of its inputs while Simulation.apply_reform runs.
+
+    policyengine-core records each input in the simulation's
+    ``_user_input_keys`` as (variable, branch, period). ``Holder._set`` adds
+    the key once it has stored the value, again when it replaces one, and on
+    every route: ``Simulation.set_input``, ``Holder.set_input``, a
+    ``set_input`` helper, another thread. A value core does not store adds
+    no key: one ``Simulation.set_input`` gets for a period after the
+    variable's end, one for a neutralized variable, or one for a sub-period
+    that a ``set_input`` helper skips because it already holds an input.
+    This set also adds each key to every set in ``logs``: one for each
+    reform being applied to the simulation, the innermost last.
+    """
+
+    def __init__(self, keys, logs: list):
+        super().__init__(keys)
+        self.logs = logs
+
+    def add(self, key) -> None:
+        super().add(key)
+        for log in self.logs:
+            log.add(key)
+
+
 # Cache for fully-loaded, uprated, enum-pre-encoded multi-year datasets keyed
 # by URL. Avoids repeating HDF5 reading (0.84s), uprating (0.69s) and enum
 # encoding (0.67s) on every Simulation() call after the first.
@@ -201,7 +227,12 @@ def __init__(
 
         self.tax_benefit_system.reset_parameter_caches()
 
-        # Apply structural reforms based on parameters
+        # Apply structural reforms based on parameters. Core applies a reform
+        # to the tax-benefit system, so reforms that set inputs (such as
+        # disable_simulated_benefits and adjust_budgets) reach the simulation
+        # through it. Core's own constructor sets this link; this one does not
+        # call it, so set it here (apply_reform also sets it for each reform).
+        self.tax_benefit_system.simulation = self
         structural_reform = create_structural_reforms_from_parameters(
             self.tax_benefit_system.parameters,
             period_(self.default_input_period),
@@ -239,6 +270,53 @@ def __init__(
             if scenario.parameter_changes is not None:
                 self.apply_parameter_changes(scenario.parameter_changes)
 
+    def apply_reform(self, reform) -> None:
+        # Reforms that set inputs (disable_simulated_benefits, adjust_budgets)
+        # reach the simulation through its tax-benefit system's link. A clone
+        # or branch may share its parent's system, so link the system to this
+        # simulation while the reform is applied, then restore the link.
+        system = self.tax_benefit_system
+        linked = getattr(system, "simulation", None)
+        system.simulation = self
+        # Log the inputs the reform stores (see _InputKeyLog). Within another
+        # reform applied to this simulation (a tuple's parts, or a reform
+        # that applies one), log them for that one too.
+        keys = getattr(self, "_user_input_keys", None)
+        enclosing = keys.logs if isinstance(keys, _InputKeyLog) else []
+        stored = set()
+        self._user_input_keys = _InputKeyLog(keys or (), enclosing + [stored])
+        try:
+            super().apply_reform(reform)
+        finally:
+            # Core rebuilds the record as a plain set once the reform is
+            # applied, but not if it raised. Resume the enclosing reform's log.
+            keys = set(self._user_input_keys)
+            self._user_input_keys = _InputKeyLog(keys, enclosing) if enclosing else keys
+            if linked is not None:
+                system.simulation = linked
+        # input_variables is taken when the simulation is built.
+        # reset_calculations and derivative delete the values of every
+        # variable it does not name, and the marginal rate formulas those of
+        # every such variable that has a formula. A reform applied afterwards
+        # (a Reform class passed as reform=, or one applied to a clone or
+        # branch) can store inputs: new ones, or ones that replace an input
+        # set since the build. Name every variable with an input the reform
+        # stored that this simulation still holds, under a branch name it
+        # reads and a reset deletes, or a reset would delete what the reform
+        # stored and recalculate the formula. Assign a new list: a clone or
+        # branch shares its parent's.
+        input_variables = getattr(self, "input_variables", None)
+        if input_variables is not None:
+            branches = set(self._get_visible_branch_names())
+            supplied = {
+                name
+                for name, branch, _ in stored & self._user_input_keys
+                if branch in branches
+            }
+            added = sorted(supplied - set(input_variables))
+            if added:
+                self.input_variables = list(input_variables) + added
+
     def delete_arrays(self, variable: str, period: Period = None) -> None:
         super().delete_arrays(variable, period)
         # From policyengine-core 3.32.27 (policyengine-core#561) the deletion

policyengine_uk/tax_benefit_system.py

@@ -1,7 +1,7 @@
 # Standard library imports
 import copy
 from pathlib import Path
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Tuple
 
 # PolicyEngine core imports
 from policyengine_core.parameters.operations.propagate_parameter_metadata import (
@@ -76,6 +76,20 @@ class CountryTaxBenefitSystem(TaxBenefitSystem):
 
     variables: Dict[str, Variable]
 
+    # Variables that hold part of another variable's value. Neutralizing a
+    # variable neutralizes its parts, so a value entered or set for a part
+    # (as disable_simulated_benefits sets the claimant's and partner's awards
+    # for each year) cannot outlive the award it is part of.
+    variable_parts: Dict[str, Tuple[str, ...]] = {
+        "esa_income": ("claimant_or_partner_esa_income",),
+        "jsa_income": ("claimant_or_partner_jsa_income",),
+    }
+
+    def neutralize_variable(self, variable_name: str) -> None:
+        super().neutralize_variable(variable_name)
+        for part in self.variable_parts.get(variable_name, ()):
+            super().neutralize_variable(part)
+
     def reset_parameter_caches(self):
         """Reset all caches in the tax-benefit system."""
         self._parameters_at_instant_cache = {}