Repository navigation
build_from_populations keeps _user_input_keys for the holders it replaces #605
Description
Simulation.build_from_populations replaces every population, and so every holder, but keeps Simulation._user_input_keys. After a rebuild, the record of set_input values still lists (variable, branch, period) entries for values the new holders never received. This is the same drift as #559, through a path #561 did not cover. #561 keeps the record in step for Holder.delete_arrays and clone, and subsample resets it before rebuilding.
Reproduction (core only, 3.32.27; same code path in 3.33.0)
from policyengine_core import periods
from policyengine_core.country_template import CountryTaxBenefitSystem
from policyengine_core.simulations import SimulationBuilder
FEB = periods.period("2018-02")
tbs = CountryTaxBenefitSystem()
households = {"h": {"parents": ["a"]}}
sim = SimulationBuilder().build_from_entities(
tbs,
{
"persons": {"a": {"salary": {"2018-02": 1000}, "disposable_income": {"2018-02": 5000}}},
"households": households,
},
)
# Rebuild the same simulation with new populations and no disposable_income input.
sim.build_from_populations(tbs.instantiate_entities())
SimulationBuilder().build_from_entities(
tbs,
{"persons": {"a": {"salary": {"2018-01": 1000, "2018-02": 1000}}}, "households": households},
sim,
)
print(sim.get_holder("disposable_income")._get_array_from_storage(FEB)) # None
print(("disposable_income", "default", FEB) in sim._user_input_keys) # True
sim.calculate("disposable_income", "2018-02") # the formula value is cached under that recordOutput on 3.32.27:
after rebuild: stored 2018-02: None
after rebuild: record has disposable_income 2018-02: True
calculated disposable_income 2018-02: [1430.0]
to_input_dataframe exports disposable_income: False ['salary__2018-01', 'salary__2018-02']
to_input_dataframe does not export the stale entry. Code that reads the record directly does see it, though. PolicyEngine-UK's supplied-input helpers (policyengine_uk/utils/supplied_inputs.py) treat a recorded (variable, branch, period) with a stored value as supplied. After a rebuild, a value carried over or calculated into that period passes as a supplied input. PE-UK reproduced this for a Class 4 trading loss: profits came out at £20,000 and NIC at £445.80, where they should be £30,000 and £1,045.80. PE-UK keeps a sweep in Simulation.calculate for this case (PolicyEngine/policyengine-uk#2260); it can go once core fixes the rebuild.
PE-UK's Simulation.build_from_situation calls build_from_populations(tax_benefit_system.instantiate_entities()) and then SimulationBuilder.build_from_dict(..., self). Its property tests rebuild cloned template simulations this way (test_state_pension_age.py, test_state_pension_credit_qualifying_age.py).
Suggested fix
In build_from_populations, reset _user_input_keys to an empty set, as subsample does. Every holder is new at that point, so none of the old entries can name a stored value. Also reset _user_input_contexts if no set_input can be running across a rebuild. Add a regression test: set an input, rebuild without it, calculate that period, and check that the record no longer names it.
Activity
MaxGhenis commentedon Oct 10, 2026
The rebuild drift also reaches core's own input export, not only code that reads
_user_input_keysdirectly. My first reproduction used a formula variable (disposable_income), whose calculated valueto_input_dataframeskipped. A value the engine carries over into an input variable's dropped period is exported as an input.PolicyEngine-UK, with
hours_workedas an input variable that carries over:Output on core 3.32.27 and on 3.33.0 (the same on both):
The 2026 entry is not an input. It is 2025's value carried over, and the record of the dropped 2026 input passes it off as one. A
Simulation.delete_arrays("hours_worked", 2026)after the rebuild does not help on core alone, because the deletion forgets only the records of values it removes and there is no stored 2026 value to remove. PE-UK keeps adelete_arraysoverride that drops records with no stored value for this reason (PolicyEngine/policyengine-uk#2262).Resetting the record in
build_from_populations, as suggested above, would fix both cases.