Skip to content

to_input_dataframe() drops dataset inputs for variables with formulas, such as state_pension_reported #2297

Open

Description

@MaxGhenis

What happens

Simulation.to_input_dataframe() leaves out values a dataset supplies for a variable that has a formula. A simulation built from the exported frame then recalculates that variable instead of using the data, so it does not give the original's results. state_pension_reported is one such variable: the reloaded simulation pays no State Pension.

Reproduction

Run on #2270's branch with policyengine-core 3.32.27 and with 3.33.2; both print the same. The frame has one person, so it also loads on main:

import pandas as pd
from policyengine_uk import Simulation
from policyengine_uk.data import UKSingleYearDataset

data = UKSingleYearDataset(
    person=pd.DataFrame(
        {
            "person_id": [1],
            "person_benunit_id": [1],
            "person_household_id": [1],
            "age": [71],
            "state_pension_reported": [9_500.0],
        }
    ),
    benunit=pd.DataFrame({"benunit_id": [1]}),
    household=pd.DataFrame(
        {
            "household_id": [1],
            "household_weight": [1.0],
            "region": ["LONDON"],
            "council_tax": [0.0],
            "rent": [0.0],
            "tenure_type": ["OWNED_OUTRIGHT"],
        }
    ),
    fiscal_year=2025,
)
original = Simulation(dataset=data)
frame = original.to_input_dataframe()
print([c for c in frame.columns if "state_pension" in c])  # []
print(original.calculate("state_pension", 2025))                  # [9500.]
print(Simulation(dataset=frame).calculate("state_pension", 2025)) # [0.]

to_input_dataframe(include_computed_variables=True), taken before the simulation calculates anything, keeps the column, and the reloaded simulation then matches on every variable (tested in #2270).

Cause, read in the code

policyengine-core's to_input_dataframe exports a variable only if Variable.is_input_variable() is true: no formulas, and no adds or subtracts (_is_exportable_input_variable in policyengine_core/simulations/simulation.py). state_pension_reported has formula_2022, which returns the previous year's value, so the value the dataset supplied is not exported. Any dataset column for a variable with a formula or adds is dropped the same way, such as a supplied consumption.

Options

  1. Give the UK Simulation its own to_input_dataframe that exports every value the simulation was given as an input and still stores, whatever the variable's formula. policyengine_uk.utils.supplied_inputs already tracks those (and Give UK simulations a working subsample #2265 adds supplied_input_arrays).
  2. Change the rule in policyengine-core to export supplied inputs of formula variables.

Either needs a test that a dataset-built simulation's default export reloads to the same results when the dataset supplies a formula variable.

Found while fixing #2266 (#2270 documents the limit in its tests and does not change the export).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions