Carry over only values stored at the variable's definition period by MaxGhenis · Pull Request #557 · PolicyEngine/policyengine-core

MaxGhenis · 2026-10-01T15:21:19Z

Problem

Since 3.24.0 (#457, "bug H1"), auto-carry-over in Simulation._calculate takes the latest-starting known period of any unit: max(known_periods, key=lambda p: p.start). A YEAR flow variable requested for a month caches a twelfth of the year's value at that month (calculate_divide). So once any month after January has been requested, a YEAR input known only for that year carries a twelfth of its value into every later year.

Before #457, sorted(known_periods)[-1] happened to pick the latest year for a YEAR variable, because "month" < "year".

Where it bites

policyengine-us monthly_age calls person("age", month). Illinois CCAP and others reach it while computing taxes and benefits. In a policyengine-us Microsimulation on a single-year (variable-centric) dataset such as the Enhanced CPS, calculating 2024 and then 2025 gives every person a twelfth of their age in 2025. That flips dependency, filing status, AGI and everything downstream.

Measured on a 3,000-household Enhanced CPS subsample, with policyengine-us main and core master 7950c01:

2025 result	after calculating 2024 first	2025-only simulation
Federal income tax	$31.6bn	$1,846.9bn
Itemizing tax units	3.0m	16.5m
State income tax	−$19.9bn	$471.9bn

On the full Enhanced CPS, 2025 income tax was $3.9bn after calculating 2024 first. With this fix and policyengine-us#9738, a 2024-then-2025 run gives $2,013.7bn for 2025, bitwise identical in all 24 benchmark arrays to a 2025-only run.

The trigger needs all of these: auto_carry_over_input_variables = True (US, UK, Canada); a FLOW YEAR input (float by default; an int declared FLOW also qualifies) with no formula and no uprating; and a month request in an earlier year within the same simulation or a branch cloned from it. int, bool and enum variables default to STOCK, and the automatic STOCK conversions return the year's or the month's value without caching it, so in practice those variables were unaffected. (An explicit calculate_add on a STOCK variable does cache a sum.)

Fix

For a variable with a (truthy) set_input helper, carry-over prefers known periods at the variable's definition unit. It falls back to every known period only when there are none at that unit. The change is in the carry-over branch of Simulation._calculate only; the rest of master's code, including the uprating path, is unchanged. Variables without a helper keep master's rule, the latest period of any unit; these are DAY variables and any variable with a falsy set_input, such as set_input = None.

Why prefer the own unit. The built-in helpers store inputs at the definition period. So beside own-unit values, a value at another unit is ordinarily a cache derived from them: calculate_divide's twelfth at a month, or calculate_add's sum at a larger period. Carrying either forward rescales the variable.

Why fall back. A value at another unit can still be a genuine input, and there it is the variable's only data. This happens when an input was stored before a reform changed the variable's definition period or gave it a helper, after an automatic reform baseline, after a restore into a system with other definitions, or with a custom helper. Master carries those inputs, and so does this.

Known limits, pinned in the tests.

A helper variable that holds both an own-unit value and a later genuine input at another unit carries the own-unit value. No built-in path stores such a pair.
A cached value at another unit with nothing at the own unit still carries over, as on master.
A variable without a helper still carries a cached twelfth, as on master.

No non-ETERNITY variable in policyengine-us, policyengine-uk or policyengine-canada lacks a helper. ETERNITY variables have none, but they never carry across periods.

Five designs on this branch were reviewed and replaced:

Commit	Design	Why it was replaced
`9e5bc21e`	Filter every variable by unit	Dropped real inputs of helper-less variables
`99b0ca09`	Match recorded input keys	Leaked through branch-name collisions, `clone()` and similar paths
`504e983e`	Per-period derived marks	Went stale on `apply_reform` replay, dump/restore, branch-scoped deletion and direct writes
`bab4325a`	Prefer own-unit periods for every variable	Made helper-less variables order-dependent
`7af22b16`	Drop other units for helper variables, no fallback	Lost genuine inputs after definition-changing reforms, automatic baselines, cross-system restores and custom helpers

On the fifth review's 32-case storage-premise harness, this head differs from master in one case only: after a same-system restore, master carries the derived twelfth (10) and this carries the year input (120).

Invariants

For variables with a helper:

Order independence for YEAR inputs and MONTH flow inputs. What a simulation calculates for a later period doesn't depend on which earlier periods it calculated first, including from a branch forked after those requests. (A MONTH stock input asked for a later year, then for an earlier month of that year, still gets the default for the month, as on master.)
Carry-over of a yearly input is the input itself.
A month input carries its latest month, not the year sum.
No backward carry-over from a later input.
A cached value at another unit is not carried when an own-unit value exists. This holds for a same-named branch elsewhere, a clone, an input set after branching, derivative(), dump/restore, an inherited twelfth, an explicit divide on a stock, a dispatch-helper flow, and an anchored-year request.
Genuine inputs at another unit survive reforms that change a definition period or install a helper, and cross-system restores.

For variables without a helper:

Unchanged from master. Inputs at any unit carry over directly, from branches, after apply_reform, and after a branch's own cache is deleted. The latest of several wins, and a later input beats an earlier cached default or sum.

Everywhere:

Unchanged where the bug didn't fire. Single-year results are bitwise identical to master: all 24 benchmark arrays match for 2024 and 2025 on the Enhanced CPS subsample above. policyengine.py's multi-year create_datasets loop is also bitwise identical (179 input arrays per year, 2024-2026).

Tests

tests/core/test_carry_over_definition_period.py has 244 cases. Failures by version: master 51, bab4325a 4, 7af22b16 6, this head 0.

Mutants of the rule all fail at least one case: is not None gating instead of truthiness, preference for FLOW only, for the divide helper only, the early return on unfiltered periods, no fallback, no gating, and min for max.

The MONTH order cases are controls. A month input's latest month always outranks its year sum, so they pass on master as well.

Run locally: the test file passes 244 cases, and ruff format --check and ruff check are clean. CI runs the full suite.

Keep branch period reads, dumps, and disk deletion consistent #552 edits the same block. It scopes known periods to the branches the current branch can read; it doesn't address units. A reviewer applied Keep branch period reads, dumps, and disk deletion consistent #552 on this branch with 532 tests passing and found no semantic conflict. Keep both: holder.get_known_periods(self.branch_name), then this preference. A reviewer ran the combination with 866/866 targeted cases passing on the previous head.
policyengine-us follow-up: a regression test that calculates two years in one simulation against single-year simulations, a core pin bump, and per-period refresh of the itemization, SALT and state refundability branches. Those branches were also kept across periods, which left a smaller residual: on the same subsample, 124 tax units' ctc_limiting_tax_liability and one tax unit's income tax still differed in 2025.

axiom: n/a: engine fix to carry-over in policyengine-core; no policy rule changes

🤖 Generated with Claude Code

Since 3.24.0 (#457, "bug H1") auto-carry-over took the latest-starting known period of any unit. A yearly variable also caches month values: calculate_divide stores a twelfth of the year's value. So once any month after January had been requested, a yearly input known only for that year carried a twelfth of its value into every later year. policyengine-us requests age by month through monthly_age, so a Microsimulation that calculated 2024 and then 2025 on a single-year dataset aged everyone to a twelfth of their age in 2025. On a 3,000-household Enhanced CPS subsample, 2025 federal income tax came to $31.6bn, against $1,846.9bn in a simulation that calculated only 2025. Filter known periods to the definition period's unit before carrying over, as the uprating path beside it already does. A MONTH variable's year-sum cache (calculate_add) is excluded the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Review of the first version found that filtering known periods to the definition period's unit also dropped authoritative inputs. A variable with no set_input helper, and every DAY variable, stores an input at whatever period it is given (a YEAR stock set for 2024-12, a MONTH stock set for 2024, a DAY stock set for 2024-12), and master carried those forward. The first version returned the default instead. The values that must not carry over are only the derived caches: calculate_divide's twelfth of a YEAR flow at a month, and calculate_add's sum of a MONTH flow at a year. STOCK paths cache nothing in another unit. So restore master's known-period handling and, in the carry-over branch, accept a period in another unit only when it was set as an input on a branch this simulation reads (_user_input_keys, filtered to visible branches, now shared with the input-export code through _get_user_input_periods). Tests: the four off-unit input cases, directly and through a branch and a nested branch. On master the 41 derived-cache cases fail; on the first version the 8 off-unit input cases fail; all 242 pass here. The docstring no longer claims the STOCK path caches month values, and names the control families. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

MaxGhenis · 2026-10-01T16:59:28Z

Independent review (Codex lane) on 9e5bc21: REQUEST_CHANGES. Responses, addressed in 99b0ca0:

Inputs stored in another unit were dropped. This was a real regression in the first commit. A variable with no set_input helper, and every DAY variable, stores an input at whatever period it is given, and master carried it forward. Fixed: a period in another unit now counts for carry-over if it was set as an input (_user_input_keys on a visible branch). Only the derived caches are excluded: calculate_divide's twelfth of a YEAR flow at a month, and calculate_add's MONTH-flow year sum. The uprating path is back to master's code. The four repro cases (DAY stock month input, MONTH stock year input without a helper, YEAR stock month input without a helper, YEAR flow month input without a helper) are now tests, directly and through a branch and a nested branch. They fail on 9e5bc21 and pass here.
The STOCK caching claim was wrong. The STOCK path returns the year's or month's value without caching. Corrected in the code comment, the test docstring and the PR description.
Control families. The int, bool, STOCK and MONTH cases pass on master. The docstring and description now name them as controls.
Pre-existing limits, not changed here. These are: size-two inputs carried into elementary periods, and on-disk branch names containing _. The latter is Keep branch period reads, dumps, and disk deletion consistent #552's subject.
Keep branch period reads, dumps, and disk deletion consistent #552 overlap. Agreed: a textual rebase only. The combined form keeps Keep branch period reads, dumps, and disk deletion consistent #552's holder.get_known_periods(self.branch_name) and applies this input filter in the carry-over branch.
Benchmark totals not reproduced by the reviewer. The arrays, scripts and logs are in ~/reviews/pe-us-second-year-2026-10-01 on the investigation host (us_bench.py, cmp.py, run4.sh, exp7.py).

Re-review requested on 99b0ca0.

The partial re-review of 99b0ca0 found cases where a derived month cache shared its period with a recorded input and so still carried over: a same-named branch in another subtree, Simulation.clone() (which shares _user_input_keys), a parent input set after branching, an input deleted and then derived again, and derivative(). The input-key scan also costs O(input keys) per carry-over (about 19 ms at 22,000 keys). Mark the values core derives by rescaling instead: calculate_divide's twelfth of a yearly flow at a month, and calculate_add's sum over more than one sub-period. Holder.put_in_cache(..., derived=True) records the period; any later write to it (Holder._set) clears the mark; Holder.clone copies the marks. Carry-over skips marked periods and nothing else, so inputs at any unit, and off-unit values cached without the mark, carry over as on master. The uprating path is unchanged. Tests add the five leak scenarios and two that pin inputs winning over a derived cache. On master 46 of 249 cases fail, on 9e5bc21 12, on 99b0ca0 6; all pass here. Full suite 1344 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

MaxGhenis · 2026-10-01T18:43:07Z

Re-review of 99b0ca0 (an Opus lane) was cut off by the lane's usage limit before it wrote a verdict. Its saved probes confirmed the STOCK repro fixed and the test matrix (242 / 41 / 8), and found no fresh-path difference from master in 150 random trials.

It also found leak cases where a derived month cache sat at a period recorded as an input, so it still carried over:

a same-named branch in another subtree;
Simulation.clone(), which shares _user_input_keys;
a parent input set after branching;
a deleted input;
derivative().

The recorded-input scan also cost about 19 ms per carry-over at 22,000 input keys.

504e983 replaces the recorded-input match with derived-cache marks:

calculate_divide and multi-sub-period calculate_add call put_in_cache(..., derived=True);
any later write clears the mark, and Holder.clone copies the marks;
carry-over skips marked periods only.

The five leak scenarios are now tests, plus an input written over a derived cache and an unmarked off-unit put_in_cache. Failures by version: 46 on master, 12 on 9e5bc21, 6 on 99b0ca0, 0 here (249 cases). Full suite: 1344 passed.

On the PE-US bench (3,000 eCPS households), year 2 equals the 2025-only run bitwise, and single-year 2024 and 2025 are bitwise identical to released core. The policyengine.py create_datasets loop is bitwise identical for 2024-2026 (179 arrays per year). A fresh re-review is queued.

The third review of 504e983 found that per-period derived marks go stale or get lost on paths that write storage without Holder._set or drop entries per branch: apply_reform's input replay, dump/restore, branch-scoped Holder.delete_arrays exposing an ancestor entry, and direct storage puts. Making provenance follow every storage entry would spread across both storages, the dumper and #556's shared arrays. Use a stateless rule instead: carry over the latest period at the variable's own definition-period unit, and fall back to periods in other units only when there is none at its own unit. Derived caches (calculate_divide's twelfth at a month, calculate_add's sum at a year) always coexist with the own-unit value they came from, so they are never picked. Inputs that exist only at another unit (a variable with no set_input helper, every DAY variable) carry over as on master. Where a helper-less variable holds both an own-unit value and an input at another unit, the own-unit value now wins, as before 3.24.0; no country package has such a variable. holder.py is back to master. Tests: the lifecycle reproductions (reform replay, dump/restore, branch-scoped deletion both ways) join the leak scenarios. 253 cases; failures on master 49, 9e5bc21 11, 99b0ca0 8, 504e983 7, here 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…finition-period

MaxGhenis · 2026-10-02T05:09:55Z

Third review (on 504e983, an Opus lane): REQUEST_CHANGES. Per-period derived marks go stale or get lost on paths that bypass Holder._set or drop entries per branch:

apply_reform input replay (0 instead of master's 600);
dump/restore (the restored copy carried the twelfth);
branch-scoped Holder.delete_arrays exposing an ancestor entry, in both directions;
direct storage puts.

The rest of that review confirmed:

all 11 differences from master in its 42 scenarios were intended;
300/300 fresh-path trials match master;
the full suite passes;
Keep branch period reads, dumps, and disk deletion consistent #552 composes with 532 tests passing.

bab4325 drops the marks for a stateless rule: carry over the latest own-unit period, and fall back to other units only when there is none.

Derived caches always coexist with their own-unit source, so they're never picked.
Inputs that exist only at another unit carry over as on master.
Nothing is tracked, so none of those lifecycle paths can go stale. holder.py is back to master.
One intended change: a helper-less variable with both an own-unit value and an off-unit input carries the own-unit value, as before 3.24.0. No country package has a helper-less or DAY variable.

The lifecycle reproductions are now tests. Failures on the 253 cases: master 49, 9e5bc21 11, 99b0ca0 8, 504e983 7, here 0. Merged with master (#556); CI runs the full suite. I'll rerun the PE-US bench on this version later today, when the shared host is free.

MaxGhenis · 2026-10-02T06:11:57Z

Overlap: #562 (draft, from another session) changes the same carry-over block. It carries over only inputs, the latest at or before the requested period, using derived marks held in the storages, and it keeps this PR's own-unit preference among inputs. All 253 tests in this PR's file pass on #562's head a80a3b9. Max decides whether this PR lands first or is closed in favour of #562, so I'm holding this merge until that ruling. I'll keep the branch green and reviewed in the meantime.

The fourth review of bab4325 found that preferring own-unit periods for every variable made variables without a set_input helper (DAY variables, set_input = None) depend on request order where master did not: a default cached for 2025, or a calculate_add sum stored for 2024, now beat a later input at another unit. A set_input helper stores every input at the definition period (Holder.set_input routes other periods to it; both helpers store at the definition unit or raise), so for a helper variable a value at another unit is never an input: it is calculate_divide's twelfth, calculate_add's sum, or an explicit put_in_cache. Drop those from the known periods for helper variables, at the start of the no-formula branch; master's carry-over code is otherwise unchanged. Variables without a helper keep master's rule (latest period of any unit). No variable in policyengine-us, -uk or -canada lacks a helper. Tests rewritten around that split (235 cases): helper variables pin order independence, the leak and lifecycle scenarios, no backward carry-over, and never carrying another unit; helper-less variables pin master's behaviour, including the review's four order cases and the known limitation that they still carry a cached twelfth. The 17 tautological empty-request cases are gone. Master fails 48 (all helper-variable fix cases) and passes every helper-less case; bab4325 fails 4; mutants (no early return, min for max, filter all, filter none, inverted) are killed. Docstring, code comment and changelog no longer overstate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

MaxGhenis · 2026-10-02T06:35:46Z

Fourth review (bab4325, Opus lane): REQUEST_CHANGES, narrower than before.

For helper variables (every US, UK and Canada variable) the fix only differs from master where master carried a derived cache, so it is never worse.
For helper-less variables, the own-unit preference added order dependence master didn't have. Examples: a default cached for 2025 beat a later 2025-09 input, and a stored 2024 sum beat a 2025-03 input.
Claims in the changelog, PR body, docstring and code comment overstated.
Tests: 17 cases were tautological, and some mutants survived.

7af22b1 adopts the review's tested suggestion. For a variable with a set_input helper, carry-over ignores periods at other units, which can never be inputs because the helper stores every input at the definition unit. Helper-less variables keep master's rule exactly.

The test file is rewritten around that split (235 cases):

Master fails 48, every helper-variable fix case, and passes every helper-less case.
bab4325 fails the review's 4 order cases.
Mutants are killed: no early return, min for max, filter-all, filter-none, inverted. The two survivors are equivalent in practice.

Description, changelog and comments are corrected. A fifth review is queued.

MaxGhenis · 2026-10-02T06:46:57Z

Like-for-like PE-US benchmark: core #557 vs #562. Setup: Enhanced CPS 2024, 3,000-household subsample (fixed seed), policyengine-us main 20ccd5ac with and without policyengine-us#9738. Each cell compares all 24 benchmark arrays bitwise (income tax, itemization, both branch liabilities, CTC, EITC, DE/ID/VA and state income tax, household net income, SNAP and more).

Comparison	core master `b78b0ba` (with #556)	#557 at `7af22b1`	#562 at `a80a3b9`
2024 single-year vs master	n/a	identical	identical
2025 single-year vs master	n/a	identical	identical
2025 from a 2024-then-2025 run vs 2025-only, PE-US main	20 of 24 differ (age at 1/12: 1,114 tax units' income tax change)	8 differ (formula-branch residual: 124 tax units' `ctc_limiting_tax_liability`, 1 tax unit's income tax)	8 differ (same residual)
Same, with PE-US #9738	n/a	identical	identical
#557 vs #562, 2025 from the two-year run with #9738	n/a	identical	identical

So on this model the two core PRs are interchangeable. Both remove the age/12 error, both leave single-year results bitwise unchanged, and both need PE-US #9738 for the second year to match exactly. They differ in scope:

Carry over only values stored at the variable's definition period #557 is an 18-line stateless filter for helper variables.
Carry over only inputs, the latest at or before the requested period #562 reworks carry-over to take only inputs, with derived marks held in the storages and dumper. That also covers helper-less variables, and it changes the "latest at or before" selection.

Scripts and arrays: ~/reviews/pe-us-second-year-2026-10-01/cmp_bench.sh, rerun_a2.sh, cmp/ on the investigation host.

The fifth review of 7af22b1 found that dropping other-unit periods for helper variables loses genuine inputs on supported paths: a reform that changes a variable's definition period (YEAR<->MONTH) or installs a helper after an off-unit input, an automatic reform baseline, a restore into a system with other definitions, and custom helpers. In each the off-unit input is the variable's only data, and master carries it. Prefer own-unit periods for a variable with a (truthy) set_input helper and fall back to every period when there are none, in the carry-over branch only; master's code is otherwise unchanged. The fifth review's 32-case premise harness now differs from master in one case, restore_builtin_same_system, where master carries the derived twelfth (10) and this carries the year input (120). Tests (244): the reform, helper-installation and cross-system restore paths keep their input; a helper variable with nothing at its own unit uses another unit, as before; set_input = False counts as no helper; mutant killers for FLOW-only, divide-helper-only, early return on unfiltered periods, no fallback and no gating. The known limit (a helper variable with both units carries its own unit) is pinned. Failures: master 51, bab4325 4, 7af22b1 6, here 0; every listed mutant fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

MaxGhenis · 2026-10-02T09:16:35Z

Fifth review (7af22b1, Opus lane): REQUEST_CHANGES, blocking on genuine-input loss. A helper variable can hold a real input at another unit, and there the strict filter dropped it where master keeps it. Paths:

a reform changing a definition period (YEAR to MONTH, or MONTH to YEAR);
a reform installing a helper after an off-unit input;
an automatic reform baseline;
a restore into a system with other definitions;
custom helpers.

The review also flagged overstated claims and six surviving mutants. Its other findings: earlier scenarios clean, every difference intended, and #552 composes (866/866).

97d4582 uses the fourth review's original suggestion: a preference with fallback for helper variables (truthy set_input), in the carry-over branch only. On the review's own 32-case premise harness it differs from master in one case, where master carries the derived twelfth after a same-system restore and this carries the year input.

Tests now have 244 cases: master 51 / bab4325 4 / 7af22b1 6 / here 0 failures, and every listed mutant is killed. Claims are rewritten, and the remaining limits are stated and pinned. A sixth review is queued.

MaxGhenis · 2026-10-02T19:29:59Z

Closing as superseded by #562, per Max's ruling on decision d820 (2026-10-02).

#562 carries over only inputs, with derived marks held in the storages. It covers everything this PR did:

it passes 242 of this PR's final 244 tests;
the 2 differences are helper-less-variable cases where Carry over only inputs, the latest at or before the requested period #562 is equal or better;
on the same PE-US data the two are bitwise identical.

It lands with or after PolicyEngine/policyengine-us#9738. This PR's investigation and benchmarks stay in its comments for reference.

…562) * Carry over only inputs, the latest at or before the requested period Auto-carry-over took the latest-starting stored period of any kind and returned the default if it started after the requested period. Any later stored period therefore hid an earlier input: a later input (inputs for 2012 and 2014 gave the default for 2013), or a later period the simulation had already calculated (2014 calculated first made 2013 the default). And values the simulation calculated carried like inputs: a twelfth cached at a month by calculate_divide (the policyengine-us monthly_age bug #557 fixes), a value masked by defined_for, a default cached where defined_for was false everywhere, or a formula result from before the formula's end. Results depended on which periods were calculated first. The rule now: a period takes the input stored for the latest-starting period that starts no later than it (on a tie, the one ending last), preferring the variable's own definition-period unit; with none, the default. Values the simulation calculates are stored with derived=True: the mark lives in the storage with the value (InMemoryStorage and OnDiskStorage keep it per key, every put sets or clears it, it counts only while the key is stored, and clones copy it), so deletions, direct writes and #556's shared arrays cannot leave it stale. Holder.is_derived(period, branch_name) reports the mark of the value get_array reads, so a value one branch calculated never hides another branch's input. A derived value never replaces an input the branch reads (calculate_add/calculate_divide used to overwrite inputs stored in another unit), calculate_add caches only sums over several sub-periods, and dump/restore keeps the marks. Storages gain has(), which neither reads nor copies an array. Supersedes #557's carry-over change: its 253 tests pass here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep uprating, input helpers, dumps and deletions consistent with input-only carry-over Round-2 review findings: - A default cached for a period with no earlier input became the base the uprating path uprated later periods from; the default is not cached for variables with uprating (as before). - A value calculated inside a set_input helper was written to the input's branch and recorded as a user input, bypassing the guard that keeps inputs; derived writes now stay on the branch they were calculated on. - dump_simulation reads each value's mark from the same branch and period as the value it dumps, so #552's branch dumps keep matching marks. - Carry-over checks candidates latest first and stops at the first input, instead of resolving the storing branch for every known period. - Storages drop the marks of deleted keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test that deleting a whole branch drops its derived marks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cache the default only where master cached a carried value With no input at or before the period but a later period stored, master returned the default without caching it; caching it changed what formulas that test whether a value is stored see (policyengine-uk's maintenance loan and current_education formulas check get_array(period.last_year)), and gave the uprating path a calculated base. Return it uncached there, as master did; cache it (derived) otherwise, as master cached the value it carried. This replaces the uprating-only special case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Read input periods in one pass over readable keys; clear marks with the caches Round-3 review findings: - A period stored only under a branch this one cannot read was taken for an input (own-unit preference ranked it first) and read back as NaN. Holder.get_input_periods(branch_name) now lists, in one pass over the stored keys, the periods whose readable value is an input, taking for each period the key get_array reads first; carry-over picks from those. This also replaces the per-period walk up the branch chain. - apply_reform's cache wipe kept the storages' marks, and rebuilding a disk index brought a stale mark back onto a new input; the wipe now clears the marks and OnDiskStorage.restore starts without any. - Storages pickled before the marks existed failed to clone, put or delete; __setstate__ now defaults them (and #556's shared-key set). - The public carry-over rule now states the own-unit preference and that an input for the period itself is read back as stored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test memory-over-disk precedence, in-memory mark clearing and index rebuilds Closes three mutants the suite did not kill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Break equal-extent carry-over ties by unit, then period Two inputs covering the same days in units other than the variable's (month:2013-01:2 and day:2013-01-01:59) resolved to whichever was stored first. Prefer the larger unit, then the period's string form, so the choice depends only on the inputs (round-4 review). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Record a storage's inputs, not its derived values, and share them with clones Measured on a policyengine-us household (policyengine-us 2.18.3, one household_net_income calculation, 6,185 storages, 30,925 holder clones across 5 branches), master with #578 retains 20.94 MB per simulation. With derived marks as a set per storage that holds a derived value, the merge retained 24.35 MB: 1,514 sets in the simulation and a copy of its source's marks in every holder clone. That is 2.6 times the 1.3 MB per simulation that ran policyengine-us's CI out of memory (#577). 5,201 of the 5,204 stored values are derived, so record the inputs instead: is_derived is "stored and not an input". The 3 storages holding inputs have a set (648 bytes); the rest read _NO_INPUTS. A storage and its clones share one frozenset of inputs until one of them changes its own. Retained memory is now 21.25 MB per simulation, +0.31 MB on master: one more 8-byte attribute slot in each of the 37,110 storage objects. A pickle records that inputs were recorded; a state from before (no marker) counts every stored value as an input, as it did then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Carry over an input that ends after the last representable date The carry-over sort key evaluated period.stop for every candidate input. A period that ends after 9999-12-31 (day:9999-12-30:3, or day:2012-01-01:4000000) has no stop: computing it raises OverflowError. Master carried such inputs ([10, 20]); this branch raised. Sort by _end_order, which puts such a period after every period that has a stop, so it still ends last. Found by the review of #582, which uses the same key for uprating. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Order period ends by day number, so ends after 9999-12-31 compare too 6ddbaf9 sorted every period whose stop cannot be computed after those that have one, but equal to each other, so of two inputs ending after 9999-12-31 the string form decided: day:9999-12-30:9 was carried over day:9999-12-30:10, and 24 months over 800 days from the same day (delta review, P2). Month and year periods past 9999 also raised ValueError, or returned a stop no date can hold, rather than OverflowError. _end_order is now the number of the day after the period's last day, counted as date.toordinal counts, by integer arithmetic on the same calendar. It equals stop + 1 day wherever stop has a value and orders every other period by its true end. Tests: the review's two cases in both set orders; a property that _end_order agrees with Period.stop where it has a value and with numpy's calendar everywhere (1,000 examples, sizes to 5,000,000); a longer period from the same day ends later. Also asserts that a pickled state without a set of shared keys gets none (a surviving mutant). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

MaxGhenis mentioned this pull request Oct 1, 2026

Calculate each year's formula branches fresh so multi-year simulations match single-year ones PolicyEngine/policyengine-us#9738

Merged

6 tasks done

MaxGhenis mentioned this pull request Oct 2, 2026

Make set_input on a branch drop values calculated from the input it replaces #560

Merged

MaxGhenis and others added 2 commits October 2, 2026 01:09

Merge remote-tracking branch 'upstream/master' into fix-carry-over-de…

bab4325

…finition-period

This was referenced Oct 2, 2026

Keep the record of set_input values in step with storage #561

Merged

Carry over only inputs, the latest at or before the requested period #562

Merged

MaxGhenis mentioned this pull request Oct 2, 2026

Uprate only from inputs, so uprated values don't depend on calculation order #563

Merged

MaxGhenis mentioned this pull request Oct 2, 2026

Delete the periods within a deleted period from disk storage #565

Draft

MaxGhenis closed this Oct 2, 2026

MaxGhenis deleted the fix-carry-over-definition-period branch October 3, 2026 14:12

Navigation Menu

Repository navigation

Carry over only values stored at the variable's definition period - #557

Carry over only values stored at the variable's definition period#557
MaxGhenis wants to merge 7 commits into
masterfrom
fix-carry-over-definition-period

MaxGhenis commented Oct 1, 2026 •

edited

Loading

Uh oh!

MaxGhenis commented Oct 1, 2026

Uh oh!

MaxGhenis commented Oct 1, 2026

Uh oh!

MaxGhenis commented Oct 2, 2026

Uh oh!

MaxGhenis commented Oct 2, 2026

Uh oh!

MaxGhenis commented Oct 2, 2026

Uh oh!

MaxGhenis commented Oct 2, 2026

Uh oh!

MaxGhenis commented Oct 2, 2026

Uh oh!

MaxGhenis commented Oct 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

MaxGhenis commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Where it bites

Fix

Invariants

Tests

Related

Uh oh!

MaxGhenis commented Oct 1, 2026

Uh oh!

MaxGhenis commented Oct 1, 2026

Uh oh!

MaxGhenis commented Oct 2, 2026

Uh oh!

MaxGhenis commented Oct 2, 2026

Uh oh!

MaxGhenis commented Oct 2, 2026

Uh oh!

MaxGhenis commented Oct 2, 2026

Uh oh!

MaxGhenis commented Oct 2, 2026

Uh oh!

MaxGhenis commented Oct 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

MaxGhenis commented Oct 1, 2026 •

edited

Loading