Uprate only from inputs, so uprated values don't depend on calculation order

[MaxGhenis]

#562 (input-only carry-over) has merged, as 58e23f56. This PR is five commits on master bd459bdf: the uprating fix, three fixes that its review on master asked for, and a changelog-only fifth commit (45f2bfc3, the head). The code is the fourth commit's, c5c7bf4c.

Problem

Simulation._calculate uprates a variable that has uprating and no formula result for a period. It takes the latest earlier period stored in the variable's own unit and scales that value by the index ratio. That period could be one the simulation had calculated itself: an earlier uprated, masked or truncated value. So an uprated value depended on which periods had been calculated first.

Executed on #562 (index +3.7% a year, input for 2012; 2015 asked alone vs after 2013 and 2014):

Variable	2015 asked alone	2015 after 2013, 2014
int [1001, 77]	[1116, 85]	[1115, 83]: truncation compounds
float [1001.3, 77.7]	1116.6072998046875	1116.607177734375: float32 rounding compounds
float, defined_for False for person 0 in 2013 only	[1115.16, 1115.16]	[0.0, 1115.16]: the 2013 mask carries into 2015

The same drift happens when one of these, cached at an intermediate period, becomes the value uprated:

• a default;
• a monthly input carried into a calculated year;
• the result of a formula that a reform gives an uprated variable up to an end.

#551's path-independence test held only to rel=1e-5, which hid the bitwise drift.

Fix

Uprate only from inputs (first commit)

Uprate only from an input this branch reads. #562 marks every value the simulation caches as derived. Holder.get_input_periods(branch_name), also from #562, lists the periods whose value get_array(period, branch_name) actually reads is an input. It ranks stored keys in get_array's lookup order (the branch, then its ancestors, then default; memory before disk) and leaves out periods stored only under branches this one cannot read. For variables with uprating, the uprating path keeps only the earlier same-unit periods in that set. With no earlier input, control falls through to #562's carry-over or the default, exactly as when the period is asked alone.

It also fixes a crash that was already on master: a period stored only under a branch this simulation cannot read, for example through Holder.set_input(..., "sibling"), used to be picked as the uprating source and read back as None, which raised TypeError. A regression test covers it.

This commit's production diff is one filtered list in simulation.py, plus a comment in #562's no-input branch. It rebased onto master without conflicts. git range-diff shows it is patch-identical to the commit reviewed before #562 merged (57c0d36b).

A yearly input given through a set_input helper replaces months already calculated (second commit)

A monthly variable stores a yearly input month by month through its set_input helper: set_input_dispatch_by_period copies it to each month for a stock, and set_input_divide_by_period divides it between them for a flow. Both skipped any month that already held a value, including one the simulation had calculated:

• dispatch kept the calculated month and copied it into the later months as an input;
• divide took it out of the year's total and divided the rest among the other months.

The month read back as calculated. Since uprating now starts only from inputs, the next January was uprated from an earlier month.

An executed probe compared 108 cases with a fresh simulation given the same input. The cases covered both helpers, carry-over on and off, root, branch and nested branch, and three calculate-first patterns. Mismatches:

• master: 84;
• the first commit alone: 96;
• this PR: 0.

Master matched only where the calculated value happened to equal the input.

The helpers now count only a stored input as present (get_stored_input), so a calculated month is replaced, as an input set for that month alone would replace it. Simulation.set_input also drops the fast-cache entries calculate made for the months a helper writes. Otherwise a month read back from the fast cache would still give the calculated value. This holds for any input a helper spreads over sub-periods that were calculated, such as a two-year input, not only a yearly one.

A simulation given all its inputs before it calculates anything behaves as before: the helpers meet no calculated value. A calculated month restored from a dump (which keeps #562's marks) is replaced the same way.

Clones never write over each other's files on disk (third commit)

OnDiskStorage.clone shares the directory and every file stored so far, and put wrote each value to its key's path. So a write through one storage changed what the other read:

• On disk, a helper input given to a Simulation.clone() (which keeps the branch name) wrote over the source's file. The review of the second commit found this.
• An own-unit set_input on a clone did the same, already on master.
• So did the source writing a period again after cloning. The first review found this.

A storage now writes over a file only if it wrote it and has not shared it since. If any storage in its clone family wrote that file, or held it when a clone was made, the value goes to a new file of its own under replaced/, which restore() never reads. Storages not cloned from one another, such as restore() and a separate writer over the same directory, still write over each other's files as before. Pickling counts as cloning, except for files the storage only read back with restore() (see "Not in scope").

Each clone records its own inputs; a storage keeps its own files across restore() (fourth commit)

The review of the third commit found two blocking defects:

• Simulation.clone() shared _user_input_keys with its source. Already on master, an input set on a clone for a period the source had calculated made the source's apply_reform keep its calculated value there as an input, and the source's input export listed it. Executed: input 2012 = [1001, 77], the source calculates 2013, a clone sets its own 2013 input, the source applies a no-op reform; the source's 2015 is [1116, 84] where a fresh simulation gives [1116, 85]. The second commit added another route: a yearly helper input on the clone replaces a month the source had calculated and records that month's key in the shared set, so the source's apply_reform kept its calculated month as an input and uprated from it. Each clone now gets a copy of the record. apply_reform and the input export look up the recorded keys in the simulation's own holders, so nothing needs the record shared.
• restore() forgot which files a storage had written (third commit). A storage that was never cloned then moved its next write to replaced/, where a separate reader of the directory did not see it. A storage now remembers, for each key, the file it last wrote and has not shared since (_own_paths). restore(), delete and apply_reform's wipe keep that record; clone() and pickling clear it. Writing a deleted key again therefore reuses its file. Before, each delete-and-write cycle after a clone left one more file (100 cycles left 101 files).

Invariants

The domain: uprated variables of each value type (float, int, and bool masks via defined_for), inputs in each variable's own unit (or any unit for a variable with no set_input helper), and earlier requests that are plain calculations in either unit, ADD or DIVIDE. Over that domain:

1. Order independence, byte for byte (dtype, shape and bytes, so -0.0 differs from 0.0). What a simulation returns for a period equals what a fresh simulation with the same inputs returns for that period alone. This holds when reading from the simulation, from a branch, or from a nested branch forked after those requests, including a branch that sets its own input. It holds with auto-carry-over on and off.
2. Reference rule. The result is the latest earlier own-unit input × index(period) / index(input period), cast to the variable's type and masked by its own defined_for. With no such input, it is Carry over only inputs, the latest at or before the requested period #562's carry-over input or the default.
3. Inputs still count. An input at an intermediate period is still uprated from. That covers an input set over a period already calculated, one set on a branch over a period the parent calculated, and one stored only under a branch while the parent calculates the same period.

For yearly inputs given through a set_input helper to a monthly stock or flow, stored in memory or on disk, after calculations inside that year:

4. Helper inputs. Every month of the year then holds an input. Every month of the year, and any other month, is byte for byte what a fresh simulation returns when given the same inputs in the same order. This holds from the simulation, a branch or a nested branch, with auto-carry-over on and off.
5. Clone isolation. The input changes nothing stored by a clone of the simulation taken just before it, and the clone's apply_reform still keeps only the clone's own inputs. At the storage level, a write through one storage never changes what a storage clone()d from it, or one it was cloned from, reads, unless one of them has since called restore(). For copies made by pickling or deepcopy, it holds only for files the storage or its clone family wrote before the copy (see "Not in scope").

Tests

• tests/core/test_uprating_order.py: 97 regressions with no Hypothesis dependency, so the smoke job runs them too.
  • 33 cover uprating: int truncation, float32 and monthly rounding, defined_for masks and all-false defaults, cached non-zero defaults, cross-unit carried inputs, a reform formula with end, branches and nested branches, branch-only inputs, and over-correction guards.
  • 56 cover helper inputs: both helpers × carry-over on and off × root, branch and nested branch × four calculate-first patterns. They also check uprating from December and keeping an input already set in the year.
  • 8 cover a helper input set on a clone and the source's apply_reform: both helpers × carry-over on and off × memory and disk.
• tests/core/test_uprating_order_property.py: two derandomized Hypothesis properties. Each docstring states its domain.
  • The first runs 500 examples plus 9 @example rows and covers invariants 1 and 2 byte for byte. Half the requests target the variable checked, and branches may set their own input.
  • The second runs 300 examples plus 4 rows and covers invariant 4. It covers invariant 5 for a clone taken before the input, in memory and on disk, including the clone's apply_reform afterwards.
• tests/core/test_on_disk_storage_clones.py: 17 tests.
  • The review's sequence, for both helpers and carry-over on and off.
  • An own-unit set_input on a clone, and the source rewriting a period.
  • Storage-level cases: clones and clones of clones; writes over a storage's own files; independent storages over one directory; restore() reading only usual file names; a restored storage's clone; a never-cloned storage written after restore(); deleting and writing a key again after a clone; deepcopy and pickle; and pickles from before this change.
• tests/fixtures/uprating_order.py: the variables, the reference rule (which rejects inputs it doesn't model) and assert_bitwise_equal. It now also has uprated_monthly_stock, a stock with a month-by-month index, so which month a value is uprated from shows. It reuses Carry over only inputs, the latest at or before the requested period #562's simulation helpers.
• Fix variable uprating when the uprating parameter is undefined, and pick the latest earlier period #551's test_result_does_not_depend_on_intermediate_years_computed: now asserts exact equality instead of rel=1e-5.

Each commit's production code, run with this head's test_uprating_order.py, test_uprating_order_property.py, test_on_disk_storage_clones.py and variables/test_variable_uprating.py (349 tests):

Code	Failures
master bd459bdf (with #562)	112: 23 of 33 uprating regressions, 56 of 56 helper regressions, 8 of 8 clone regressions, both properties, 8 of 8 tightened #551 cases, 15 of 17 disk tests
first commit 14785411	80: 56 of 56 helper regressions, 8 of 8 clone regressions, the helper property, 15 of 17 disk tests
second commit 168a710a	24: 8 of 8 clone regressions, the helper property (on disk, with a clone), 15 of 17 disk tests
third commit 1ee63a80	12: 8 of 8 clone regressions, the helper property (the clone's inputs after a reform), 3 of 17 disk tests (writing after restore(), reusing a file after delete, and the old-pickle test, whose setup reads the renamed _own_paths field)
fourth commit c5c7bf4c (this PR's code)	0

The tests that pass on earlier code check behavior it already had: the over-correction guards (inputs still count), an uprated value cached as derived (#562 marks it), the cross-unit case with carry-over off, storages not cloned from one another still writing over each other's files, and, before the third commit, a never-cloned storage still writing its own file after restore().

Mutation check and ablations.

• The uprating filter (mutants.py): all 6 mutants are killed, and the property alone kills all 6. The mutants are: no filter, a filter that ignores the branch, an inverted filter, max over the unfiltered list, dropping the uprating guard, and uprating from the earliest input.

• The second commit's helper regressions:

  • with neither part: 56 of 56 fail;
  • with the helper change alone: 52 fail (the four that pass don't read December back);
  • with the fast-cache change alone: 56 fail.

• The helper property, generated examples only (explicit rows removed). It fails:

  • without the second commit;
  • with either half of the second commit alone;
  • on the second commit: the source's input changes a month its clone stored on disk;
  • on the third commit: the clone's inputs change after a no-op reform.

  On the fourth commit it passes all 300 generated examples.

Local runs on c5c7bf4c (this PR's code):

• Full suite: 1485 passed, 4 skipped, 1 xfailed.
• Country-template YAML: 39 passed.
• ruff format --check: clean.
• The repro behind the table above: all three variables bitwise equal.
• The helper probe (108 cases: three index and default setups × stock and flow × carry-over on and off × root, branch and nested branch × three calculate-first patterns): 0 mismatches with a fresh simulation; 84 on master.

CI: 18 of 18 checks pass on c5c7bf4c and on the head, 45f2bfc3, Windows included.

Independent reviews.

• Before Carry over only inputs, the latest at or before the requested period #562 merged: a hard-tier review (GPT-6.1 Sol) on an earlier head and a final-commit review (Opus, on 2120f78f) each returned APPROVE WITH NITS and found no defect introduced by the PR. Their documentation and test nits are fixed in the first commit, except an optional one: get_input_periods runs twice for an uprated variable with no earlier input when carry-over is on. The pre-existing defects they reported are fixed here or listed under "Not in scope".
• On master: four hard-tier rounds (GPT-6.1 Sol), each on the head at the time:
  1. On the first commit: REQUEST CHANGES. A yearly helper input over a calculated month was mishandled, and this PR made it worse (fixed in the second commit). Its four other findings are pre-existing. One, a source rewriting a period on disk after cloning, which changed what its clone read, is fixed in the third commit; the other three are under "Not in scope".
  2. On the second commit: REQUEST CHANGES. A helper input on an on-disk clone wrote over the source's file (fixed in the third commit). Its two other findings, a stale yearly ADD and a day-unit input, are under "Not in scope".
  3. On the third commit: REQUEST CHANGES. The shared input record and restore() forgetting a storage's own files were blocking; replacement files piled up over delete-and-write cycles (all three fixed in the fourth commit). Its pre-existing findings are under "Not in scope".
  4. On the fourth commit, c5c7bf4c: APPROVE. Every blocking round-3 repro (findings 1, 2 and 4) is resolved, and its pre-existing finding-3 cases reproduce the same on master and on this code. It also ran 16 input-record scenarios (branches, nested branches, clones, a subclass clone, both input exports, reforms on sources, branches and clones) and 48 file-ownership cases across clone, deepcopy, pickle, delete and restore(). It rebuilt its round-3 clone-family state machine and ran it: 400 examples and 280,298 value and provenance checks. The seven related test files passed (237 tests).

Measurements

The confirmation pass and the final-head check ran the first commit's filter (get_input_periods), before the rebase onto master. The first pass (1e88ee16) ran an earlier form, not holder.is_derived(p, branch), which differs only for a period stored only under a branch this one cannot read, where it raised TypeError. The full-eCPS 2025-2026 row comes from the first pass only.

The other three commits change results only in the cases they fix:

• an input given through a set_input helper after a period it covers was calculated (or restored from a dump as calculated): a yearly or multi-year input, or a day input to a monthly variable;
• a write to an on-disk key whose file a clone, or a copy made by pickling, also reads;
• apply_reform, or an export of inputs, on a simulation after an input was set on its clone, or on the clone after an input was set on the source.

The PE-UK and PE-US runs were not repeated for them.

These are real PE-UK and PE-US microsimulations, with no scaling. Cores compared: master b78b0ba9, #562 and this change.

• First pass: the full-eCPS and multi-year runs used Carry over only inputs, the latest at or before the requested period #562 27d1bc35, with this change at 1e88ee16.

• Confirmation pass: on Carry over only inputs, the latest at or before the requested period #562 35e20d9f with this change at a2782333. It reran PE-UK single and multi, the PE-US subsample and full-eCPS 2026. Every comparison matched the first pass.

• Final heads: Carry over only inputs, the latest at or before the requested period #562 912a4ac5 and this change at 2120f78f, whose production code the first commit keeps. They changed only Carry over only inputs, the latest at or before the requested period #562's carry-over tie-break for equal-extent inputs in another unit. Checked against the confirmation outputs, they matched byte for byte:

  • UK 2031-2033: 41/41 arrays each year;
  • full eCPS 2026: 30/30;
  • the 2025-2026 subsample: 30/30 each year.

  This held for both Carry over only inputs, the latest at or before the requested period #562 and this change.

• Shadow logger: every run on this change was repeated with a shadow logger, which left outputs bitwise unchanged. It records each uprating decision where master's rule picks a different source period.

• PE-UK: policyengine-uk 7b9fc379, Enhanced FRS 2024-25, full sample.

• PE-US: policyengine-us 6c5170fd, Enhanced CPS (inputs at 2024), a 3,000-household subsample (seed core-cow-bench) and the full sample.

Single-year simulations (what policyengine.py and the APIs build): no output changes.

Run	master vs #562 vs this change
PE-UK 2026	41/41 arrays bitwise identical. No uprating decision has an earlier stored period: PE-UK loads every year through 2030 as inputs.
PE-UK 2032	41/41 identical. 97 uprating decisions; master and this change pick the same source in all of them (the 2030 input).
PE-US 2026, 3,000 households, with MTR branches	31/31 identical. 197 decisions pick a different source, and none changes a value: they are monthly variables with no input, where master chains a cached default.
PE-US 2026, full eCPS	#562 → this change: 30/30 identical. 99 of 198 decisions pick a different source; none changes a value. (Master → #562 differs in spm_unit_net_income for 14 SPM units: #562's state_fips/FPG fix, see #562.)

Multi-year in one simulation: this change removes uprated-input drift.

Run	#562 → this change	Single-year vs multi-year (same year)
PE-UK 2025-2030	41/41 identical every year. 130 decisions pick a different source, 0 change a value.	—
PE-UK 2031-2033	2032: 24 of 41 arrays change, by 1e-7 to 1e-9 relative. Examples: household net income +£41k on £2.03tn, income tax +£14k on £427bn, weights 5e-8. 2033 is similar. These are uprated inputs: weights, employment, self-employment, pension, savings and dividend income. They were chained from cached 2031 values; they now come from the 2030 input.	2032 inputs now match single-year 2032 exactly. Master/#562 differ in 8 input arrays.
PE-US 2025-2026, 3,000 households	2026: 28 of 30 arrays change, by 1e-7 to 1e-8 relative. Example: income tax +$258k on $1,962bn. The cause is the same: inputs uprated through cached 2025 values.	26/30 arrays match single-year 2026 exactly; income tax is identical. Master matches 0/30 and #562 1/30.
PE-US 2025-2026, full eCPS	2026: 28 of 30 arrays change, by 1e-7 to 1e-8 relative. Examples: income tax +$242k on $2,141bn; household net income +$0.73m on $14.46tn; weights 1.6e-8.	Asked alone vs after 2025: income tax differs for 28 tax units (#562: 30,957), household net income for 30 households (#562: 23,034), state income tax for 3 (#562: 24,131). All inputs and weights match exactly. The rest is 24 tax units whose itemization flips in PE-US's persistent branches (the same 24 on #562).

The multi-year differences that remain are not uprating:

• PE-UK 2032+ (12 arrays, incl. child benefit −2.3%, UC, MTR, poverty). PE-UK's current_education formula copies last year's value only if last year happens to be cached, and otherwise imputes from age. That's a PE-UK order dependence. Separate task.
• PE-US 2026 (itemizing outputs: one tax unit's itemization flips, ctc_limiting_tax_liability and tax_liability_if_itemizing for 63-120 units). These are computed in PE-US's persistent itemizing/not_itemizing branches, created in 2025 and reused in 2026. policyengine-us#9738 (get_branch_for_period) re-forks them.
• PE-US 2027 after 2025-2026 raises ParameterNotFoundError (md...informal.rates.UNKNOWN) on Carry over only inputs, the latest at or before the requested period #562 and this change. It doesn't raise on master, but master's 2026 is wrong there (income tax $33bn, the Carry over only values stored at the variable's definition period #557 age/12 bug). The cause is in PE-US: its county formula reads the default branch's latest period from inside a reused branch. The carry-over session is filing the PE-US fix.
• spm_unit_is_in_spm_poverty raises SPMInputError on every core, master included. This is the known eCPS SPM composition defect, us: name the SPM composition defect in seconds, and refuse it by name in the release microcosm#948. It isn't measured here.

Not in scope

These are pre-existing on master. The reviews found most of them. None is made worse here, except the day-unit case noted below.

• Stale cached targets after a later set_input: Make set_input on a branch drop values calculated from the input it replaces #560 covers branches. For the root simulation, apart from the months a helper writes (second commit), values calculated from an input stay cached when it changes, as on master: later periods, dependent variables and a yearly ADD sum. This includes:
  • a yearly ADD sum cached before a helper input to that year, which stays cached at the year (the months, and ADD itself, are recomputed);
  • a month calculated before a day-unit input to a monthly helper variable, which stays in the fast cache. Day inputs are outside the helpers' documented domain (periods larger than the variable's). On master, dispatch ignored such an input after the month was calculated, and divide raised Inconsistent input. Now both store it, as a fresh simulation would, but calculate for that month still returns the cached value.
• apply_reform's replay of user inputs:
  • it can promote a value recalculated over a deleted input to an input;
  • after a dump is restored, it drops the restored inputs.
• Dumps of a branch:
  • they write the default branch's values;
  • a period stored only under an unreadable sibling dumps as None, which fails to restore.
• Subsample promotes calculated values to inputs, and STOCK/DIVIDE caches: task_857ef4c2.
• On-disk copies and restores (the third review):
  • two copies made by pickling or deepcopy that each store a key neither had before the copy write the same file;
  • a clone that calls restore() reads files its source still writes over;
  • subsampling a simulation after cloning it, on disk, can change the arrays its clone reads;
  • a key that a storage read back with restore() but never wrote: after a pickle or deepcopy, either copy writing it changes what the other reads.
• Equal-start inputs: for a variable with no set_input helper given inputs for year:2012:2 and 2012, the uprating source is whichever was set first, as on master: task_5fbde569.
• Other on-disk storage defects and underscore branch names: Give each disk-backed holder storage its own directory #558, Keep branch period reads, dumps, and disk deletion consistent #552, task_a9863c4b. Give each disk-backed holder storage its own directory #558 gives each disk-backed storage its own directory. It overlaps this PR's third commit, so it needs a rebase after this merges.
• Other holder issues: restore_simulation drops the input registry, Holder.set_input has a stale fast cache, and Enum uprating raises. These are a new task.
• Old dumps: dumps written before Carry over only inputs, the latest at or before the requested period #562 carry no provenance marks, so a restored calculated period is treated as an input.

axiom: n/a: engine fix to uprating source selection and input storage in policyengine-core; no policy rule changes

🤖 Generated with Claude Code