Measure a branch's behavioural responses against baseline policy, with its own cache

[MaxGhenis]

Summary

Measure a reform simulation's branches against baseline policy applied to their own inputs. Give each simulation its own behavioral-response measurement dictionary, clear it on direct cloning and cache invalidation, and let new perturbation branches inherit an independent copy of their household's measurements.

Previously an early branch could measure reform policy against itself and report zero response. A calculated clone could reuse another household's measurements, and measurements for another period could leak into its parent.

Round-1 findings

1. No reform crashed. The baseline measurement now falls back to the simulation's own policy when baseline is None. Regressions cover both a no-reform household and a reform simulation's own baseline, including all five elasticity/change helpers returning zero.
2. An alleged household output change was refuted for the cited examples. The saved round-1 executions compare TX wages plus long-term gains and CA wages plus weekly hours: all eight outputs and six measurement arrays match. In those executions, MTR first calculated household net income, caching raw gains/hours, whose original input records preserved them in perturbation branches. The review's claim that both arms lose those values overlooked these cached raw values. This establishes equality for those examples, not every household. Evidence is preserved in the job's untracked hub-evidence/main_r1.json and fix_r1.json. Those historical executions used the prior household input bookkeeping; Round 3 separately fixes records left behind when the original arrays are deleted.
3. Capital-gains MTR depended on calculation order. Calculate household net income before creating either capital-gains perturbation branch, so they inherit measured responses even when capital-gains MTR is requested first. The regression compares MTR-first with income-first calculations and their resulting household net income.

Main merge and Core integration

Plain-merged main at a844fc1717e8632bc92a9b0d7c9d41dda5359fdc. spm.py merged cleanly: retain #9994 (09d7d9d0b8)'s rebuilt-baseline reset, incompatible-holder pruning, policy rebinding, tracing isolation and county receipts alongside this PR's _invalidate_all_caches, clone and get_branch overrides. These serve complementary purposes: policy ownership and holder suitability versus measurement ownership and lifetime.

The textual conflict in capital_gains_responses.py retains main's user_set_variables handling and this PR's income-before-branch ordering. Baseline measurement branches also prune holders whose entity or ETERNITY storage cannot serve baseline policy before rebinding; structural-reform regressions cover both differences.

Current fetched main's uv.lock pins Core 3.32.8, with a >=3.32.8 project requirement. Round 3 leaves that lock unchanged. The complete behavioral-response copies regression file passes on both 3.32.8 and 3.33.1, including the fully-calculated-clone regression, perturbed measured-branch property, and input-provenance regression. The remaining measurements and SPM checks are tracked below. #10069's SPM property remains intact: after subsampling, recalculate reform-only variables because 3.33.1 rebuilds from recorded inputs rather than exporting calculated values as inputs.

Round-3 findings and fixes

Core 3.32.8 deletes a holder's arrays without forgetting their records in _user_input_keys. After this package moves the household's original behavioral inputs to their pre-response slots, those stale original-name records can make a fully calculated clone treat inherited post-response earnings as input. In the regression, calculate household net income at $50,000, clone, clear inherited formula values, and set employment_income_before_lsr to $100,000: obsolete earnings enter both measurement arms, so the clone disagrees with a fresh $100,000 household.

Fix the provenance at migration in both Simulation and Microsimulation: when dropping the original holder's arrays, remove records for the exact storage keys that were dropped. Detach the input-record set before pruning so the already-created baseline retains its original raw inputs. Keep the migrated pre-response records and later explicit overrides, including original-name overrides for another period. This handles Core 3.32.8 and 3.33.1 without relying on the newer Core deletion behavior or bumping uv.lock.

The saved fully-calculated-clone regression is included. A provenance regression covers all five migrated inputs—employment income, self-employment income, SSTB self-employment income, weekly hours, and long-term capital gains—and checks baseline preservation plus a clone's deliberate later-period employment-income override surviving formula-value cleanup without reaching its source.

The Hypothesis copy property now exercises an explicit branch created after measurement and then changed from $50,000 to $100,000, rather than replacing its perturbed wages with the source wages. Under d1021, each cached measurement array must equal its original household snapshot, and the response must scale with the branch's new wages using those fixed measurements and this reform's uniform elasticities. Direct clones and branches created before measurement still use the fresh-household oracle.

d1021 and landing order

d1021 was RULED on 2026-10-08: keep reuse. MTR and perturbation branches hold response measurements at the household's measured values. New branches copy the dictionary; later measurements stay private. Direct clones measure afresh. The ruling states “#9698 and #9913 match.” #9698 is hub-owned and unchanged here; whichever of #9698 and #9913 lands second must rebase the overlapping simulation/behavioral-response files and rerun the checks.

Invariants and Python coverage

• A direct clone, or a branch created before measurement, agrees with a fresh reform simulation using its inputs. Hypothesis varies copy kind, timing and wages, with explicit regression examples.
• Measurements for another period cannot enter the parent dictionary. Clones discard inherited measurements; branches created after measurement inherit a separate dictionary.
• Applying a reform and actual subsampling remove stale measurements, including baseline/child arms. The SPM subsample property exercises repeated draws, changed parameters and structural reforms.
• The baseline measurement uses its own private baseline policy with no further baseline, preserves the measured simulation's inputs, drops computed reform values and unsuitable holders, and isolates parameter tracing.
• Capital-gains MTR is independent of whether household income was calculated first.

Python coverage is required because YAML cannot exercise clone/branch APIs, cache ownership, calculation order or storage rebinding. The new copies file belongs to the existing core Rest Python group, measurements to heavy-b, and the SPM tests to their existing separate process on the Microsimulation runner. No jobs or concurrent heavy processes are added. These local macOS resource measurements include process setup and are not Linux CI measurements.

Tests

All runs below completed. Local macOS elapsed times include process setup; peak memory is reported per process and is not a Linux CI measurement.

Check	Result	Elapsed including setup	Peak memory
Fully-calculated-clone regression before the fix at 7bb51930ad5a0510f303d2a88b0561aa7bab0958, Core 3.32.8	1 failed as expected, 14 deselected	9,204.14 s	4,948.56 MiB (5,188,943,872 bytes)
Fully-calculated-clone regression after the fix at 04be263a411c77a032b79eabfb8ee285bc13b93e, Core 3.32.8	1 passed, 15 deselected; 14 existing RuntimeWarnings	111.40794012503466 s	5,131.75 MiB (5,381,029,888 bytes)
Full test_behavioral_response_measurement_copies.py at 04be263a411c77a032b79eabfb8ee285bc13b93e, Core 3.32.8, including the fully-calculated-clone regression	16 passed; 117 existing RuntimeWarnings	1,753.2014067500131 s	5,790.203125 MiB (6,071,468,032 bytes)
Full test_behavioral_response_measurement_copies.py at 04be263a411c77a032b79eabfb8ee285bc13b93e, Core 3.33.1, including the fully-calculated-clone regression	16 passed; 350 existing RuntimeWarnings	1,392.481941416976 s	5,879.562500 MiB (6,165,168,128 bytes)
test_behavioral_response_measurements.py, Core 3.32.8	15 passed	139.60109945799923 s	3,735.312500 MiB (3,916,759,040 bytes)
test_spm_system.py, Core 3.32.8	76 passed	398.5 s	3,679.11 MiB (3,857,825,792 bytes)

The unfixed clone returned a labor-supply response of 631.1710815429688, while a fresh $100,000 household returned -52.118892669677734. This reproduces the stale-earnings failure on the locked Core version. The selected saved regression now passes on Core 3.32.8 at the fixed head; its resource record is preserved in hub-evidence/regression-after-core-3.32.8.json.

The completed Core 3.32.8 full copies run validates all 16 cases, including the saved clone regression, input-provenance regression, and perturbed measured-branch property. Its resource record is preserved in hub-evidence/copies-after-core-3.32.8-standard.json. The Core 3.33.1 full copies run also passes all 16 cases at the same fixed head; its resource record is hub-evidence/copies-after-core-3.33.1.json. The Core 3.32.8 measurements file passes all 15 cases; its resource record is hub-evidence/measurements-core-3.32.8.json. The Core 3.32.8 SPM file passes all 76 cases. All completed after-fix checks use the same fixed head.

Two earlier full copies attempts had the optional faulthandler_timeout=300 diagnostic setting enabled and were interrupted with incomplete diagnostic dumps, after two passes and no completed tests respectively. Those partial attempts are distinct from the completed standard faulthandler_timeout=0 run above.

Tests run directly in the foreground, one file at a time, with OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, VECLIB_MAXIMUM_THREADS, and NUMEXPR_NUM_THREADS capped at 4 and OMP_WAIT_POLICY=PASSIVE. Hypothesis remains enabled; no tests or strategies are disabled. No YAML files are changed relative to merged main. make format and its linter passed before each Round-3 commit. Failing-regression evidence stays outside Git in hub-evidence/; earlier interrupted attempts and mutation-run records are historical evidence, rather than completed validation of this head.

axiom: n/a: simulation machinery for behavioral-response measurement, cloning and cache isolation; no policy rule or legal provision changes.

Hub microsimulation impact (current head)

A real microsimulation on the default dataset, main at the merge base a844fc1717 against this PR's head 04be263a41, for 2026. The runs went one at a time under the shared heavy-job lock. Sums are weighted.

Output	2026 change	2026 records changed (weighted)
employment_income	+0	0 (0)
self_employment_income	+0	0 (0)
long_term_capital_gains	+0	0 (0)
weekly_hours_worked	+0	0 (0)
labor_supply_behavioral_response	+0	0 (0)
state_income_tax	+0	0 (0)
income_tax	+0	0 (0)
household_net_income	+0	0 (0)
household_benefits	+0	0 (0)

All measured outputs are unchanged: behavioural responses are off by default, and this PR changes how measurements are kept and cloned, not any default calculation.

Comparison files are in fixes/9913-impact/ on the hub host.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.61%. Comparing base (909176a) to head (04be263).
⚠️ Report is 476 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##              main    #9913      +/-   ##
===========================================
- Coverage   100.00%   98.61%   -1.39%     
===========================================
  Files            4        2       -2     
  Lines           76      144      +68     
  Branches         2        7       +5     
===========================================
+ Hits            76      142      +66     
- Misses           0        2       +2     

Flag	Coverage Δ
unittests	98.61% <100.00%> (-1.39%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[MaxGhenis]

Context from the session that opened this PR, for the US + core hub, which owns it now. I won't push to the branch. My uncommitted round-1 edits match 23e7661 byte for byte, so nothing in my worktree is lost.

Everything below is in ~/reviews/pe-us-lsr-branch-baseline/.

Round-1 review. review-r1-prompt.md and review-r1-out.md (Opus via Subfleet, REQUEST_CHANGES at 57b74a7). What I executed against its findings, in review-r1-checks/ (check.py, main_r1.json, main_helpers.json, fix_r1.json):

• Finding 1, no-reform crash: confirmed.
  • On main, after labor_supply_behavioral_response, a no-reform household and a reform simulation's baseline both return 0 for income_elasticity_lsr, substitution_elasticity_lsr, relative_wage_change, relative_income_change and relative_capital_gains_mtr_change.
  • At 57b74a7 every one raised AttributeError.
  • These variables can't be requested before labor_supply_behavioral_response (requires_computation_after), so any check must request it first.
• Finding 2, household output change: refuted by execution.
  • Households: TX with $60k wages and $200k long-term gains; CA with $15k wages and a 25-hour weekly_hours_worked input.
  • Reform: rate cut, capital-gains rate 0.25, income elasticity −0.05, substitution elasticity 0.25, capital-gains elasticity −0.62.
  • All 8 outputs and all 6 stored measurements match between main and 57b74a7.
  • Why (mtr_input_check.py): a perturbation branch made after household_net_income keeps the cached post-response values. The household input_variables lists those names, so the deletion loop keeps them and deletes only the _before_* inputs. A probe branch kept long_term_capital_gains at 200,000 while long_term_capital_gains_before_response read 0.
• Finding 3, capital-gains MTR ordering: fixed in 23e7661. test_capital_gains_marginal_tax_rate_does_not_depend_on_calculation_order covers it, but I never got to run it on main.

Nit responses, never sent to the reviewer:

• _no_change passed as a class: kept. pe-us passes reform classes to apply_reform (see create_structural_reforms_from_parameters).
• No slow marker: RUN_HEAVY_TESTS gates default-dataset microsims that CI skips entirely, so gating would drop the regression coverage. The file sits in the core group, not heavy-b.
• mark_tree_shared on the measured simulation: performance only. Marginal-tax-rate branches already mark the tree.

Round-2 brief. review-r2-prompt.md is drafted for the 57b74a7 → fix delta. Replace HEAD_SHA with the current head before dispatching.

Tests on main. onmain/on_main.log: the 4 behavioural tests in the new file fail on upstream/main at the expected assertions. That run predates the 3 round-1 tests.

Local timings on a host at load 200–450, an upper bound: property test 496 s, cross-period test 201 s, cheap tests 25–50 s each. Peak memory was 6.1 GB.

Impact harnesses (not yet run to completion).

• Microsim: microsim/run_lsr_microsim.py, with lockrun_lsr.sh taking the same heavy lock, host gate, nice check and watchdog as lockrun.sh (that run.py builds only no-reform microsims).
  • jobs_ab.txt holds main vs fix runs (LSR, and LSR with capital gains) on a 5,000-household subsample. Each saves the reform, baseline and an early-branch arm record by record.
  • It also holds a paired 1,000-household arm comparing reuse with re-measurement (LSR_REMEASURE=1), which sizes d1021's alternative.
  • compare.py compares runs record by record.
  • It never got the heavy lock, so there are no results.
• Household: household/us_household_ab.py (core#587's harness, plus copies given doubled earnings), run_rest.sh, compare.py.
  • The only finished run is main-core3328.json (main b79705b4 on core 3.32.8).
  • The fix run was cut by a reboot.
• Before reuse: repoint the checkout paths to your heads. I'm removing my A/B worktrees (pe-us-lsr-ab-main, pe-us-lsr-ab-fix, core-master-lsr-ab, core-587-lsr-ab).
• Operational: Bash background tasks die at 2 hours. A lock wait that outlives that cap kills the microsim mid-run, so I ran the lock waiters in a terminal tab.

Repro evidence. repro.py with main_core3328.json, main_core3328_p2.json and fix_core3328.json (main vs 3d174a7).

[MaxGhenis]

US + core hub merge audit, us#9913 at 04be263a411c77a032b79eabfb8ee285bc13b93e (squash): Measure a branch's behavioural responses against baseline policy, with its own cache.

• Review: independent GPT-6.1 Sol review r3 APPROVE at this head: round 2's P2 (stale earnings in fully calculated clones on core 3.32.8) is fixed by forgetting exactly the dropped input records, and the regression fails without the fix; the branch-reuse property follows d1021 (keep reuse).
• Gates checked live at merge:
  • gh pr checks exits 0;
  • the latest Pull request run at the head concluded success;
  • MERGEABLE;
  • not a draft;
  • no reviewer's latest review requests changes;
  • the semantic-overlap guard;
  • head pinned.
• Impact (hub full-file microsimulation, merge base a844fc1717 against this head, in the description): 2026: no record changes in the measured outputs.
• Axiom: n/a: simulation machinery for behavioral-response measurement, cloning and cache isolation; no policy rule or legal provision changes.