Repository navigation
Honor dataset-supplied tax-unit roles - #9617
Conversation
Populace's tax-unit constructor (microunit) writes each person's role (tax_unit_role_input: HEAD, SPOUSE, DEPENDENT) and each unit's filing status (filing_status_input) into the certified build, but no variable read them, so the loader dropped both and the role variables fell back to age ordering. Add the two input variables under the producer's exact column and member names, with an UNSPECIFIED default, and make is_tax_unit_head, is_tax_unit_spouse and filing_status defer to them when supplied. Malformed supplied values raise instead of falling back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review found that neutralizing filing_status_input in remove_head_of_household re-derived every supplied status, moving 158 non-head-of-household units on the certified build, and that the gov.abolitions parameters for head_of_household_eligible and surviving_spouse_eligible did nothing to supplied statuses. filing_status now re-derives a supplied HEAD_OF_HOUSEHOLD or SURVIVING_SPOUSE only when that rule is neutralized or abolished, using core's own switch-off test, and the reform is back to its original single neutralization. Also: check a supplied JOINT status against the supplied roles so abolishing a role variable cannot raise; keep an explicit head out of the supplied spouse role; document precedence, the fallback's separation gate, and that supplied adult dependents' income counts toward no return. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9617 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 4 6 +2
Lines 76 120 +44
Branches 2 3 +1
=========================================
+ Hits 76 120 +44
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
State seven properties of supplied tax-unit roles and filing statuses (partition, supplied roles hold, age-ordering fallback, supplied statuses hold, equivalence with the explicit-pin workaround, switch-off locality, error contract) and check each over generated households. Adds hypothesis to the dev extra; uv.lock regenerated with CI's uv 0.12.13 so it changes only by that package. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Independent review of d8b02c1 found the switch-off-locality property could generate no unit supplied with the abolished status, so it never exercised the re-derivation it claims to check. Each example now holds one forced target unit beside other supplied units. Also narrow the docs and changelog: a dependent's income is left out of the claiming return's gross income, but the deductible share of a dependent's self-employment tax still lowers the filer's AGI through self_employment_tax_ald. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts: - pyproject.toml, uv.lock: main already added hypothesis>=6.100.0 to the dev group, this branch's only new dependency, so both files are main's; `uv lock` leaves uv.lock unchanged. - is_tax_unit_head.py, is_tax_unit_spouse.py: #9630 narrowed the age-ordered fallback to head_or_spouse_candidates (skipping input dependents); this branch wraps that fallback in where(tax_unit_roles_supplied, <supplied role>, <fallback>). Kept both: supplied roles still decide when every member has one, and otherwise the fallback uses main's candidates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Filing status is a policy calculation, so the model computes it from the household's facts, as on main. Supplied data may set who is head, spouse or dependent, but not the filing status. - filing_status.py returns to main's formula; no supplied status is read. - Remove filing_status_input and its YAML tests. The Populace build's filing_status_input column now matches no variable, so the loader skips it. - Revert the romney remove_head_of_household cases, which tested only how the repeal re-derived a supplied status. - Keep tax_unit_role_input, tax_unit_roles_supplied and the head/spouse formulas composed with #9630's inference. - Unit tests: drop the supplied-status tests; add a dataset test showing a filing_status_input column that disagrees with the rules is not read, in the data year and an extended year. - Property tests: roles only. Filing status is joint exactly when there is a spouse; the role input matches pinning the role flags (neither side pins filing_status); a new property composes supplied roles with input dependents, against an oracle that skips them as #9630 does. - Microsim test: check the build's roles only. - Docs: roles only, the fallback as it stands after #9630, and dependents' above-the-line deductions after #9801. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The age-ordered fallback skips adults input as tax unit dependents unless every adult in the unit is one, in which case every adult stays a candidate. Say so in tax_unit_role_input's documentation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per Max's ruling d318, the dataset supplies tax-unit roles but not filing status, so the fragment no longer says filing_status_input sets filing_status. It drops the 2024 aggregates measured with supplied filing status, which no longer describe this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The loader declares every tax unit in a dataset's tax_unit_id column, members or not, and all() is True over no members. So tax_unit_roles_supplied counted a memberless unit as supplied with no HEAD and raised, where main gives the unit no head. It now needs at least one supplied member. The certified default build has no memberless units, so its results do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
US + core hub merge audit, us#9617 at
|
…uses The #9617 cross-check found one Mississippi pure head/spouse swap that still changed state tax: the microdata supply the health savings account deduction at the tax-unit level, and person_share_of_tax_unit_amount gave a deduction with no per-person amounts to the head's column. It now takes an opt-in split_between_spouses flag, which the Mississippi health savings account, self-employed health insurance and self-employed retirement adjustments use: the head and spouse each take half. Medicaid AGI keeps the head fallback. The property test now draws tax-unit deductions with no per-person split, and fails on the previous head for this case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
The certified Populace US build ships the tax units its constructor built, with each person's role. policyengine-us has had no variable to read that column, so the loader drops it and re-derives roles by age: the oldest adult heads the unit and the next-oldest is the spouse. This PR adds the role input under the producer's exact name and makes the role variables use it when supplied. Age ordering, with #9630's rule for input dependents, remains the fallback.
Roles only: filing status is computed by the model, not supplied. Max ruled on 2026-10-06 (decision d318) for roles only: filing status is a policy calculation, so the model computes it from the household's facts. Supplied data may set the tax-unit roles (who is head, spouse or dependent) but not the filing status. Earlier versions of this PR also honored a supplied
filing_status_input; that part is reverted.filing_status.pyis byte-identical to main, andfiling_status_inputno longer exists. The build still writes afiling_status_inputcolumn. No variable has that name, so the loader skips it, as it did before this PR.tax_unit_role_input(person:HEAD/SPOUSE/DEPENDENT). It has no formula and defaults toUNSPECIFIED.is_tax_unit_headandis_tax_unit_spouseuse the supplied roles when every member of the unit has one (tax_unit_roles_supplied).is_tax_unit_dependentfollows as before, andfiling_statusis computed from the resulting roles by main's formula.This round (hub fix, 2026-10-06)
filing_status.pyis identical to main (6440b3e). Thefiling_status_inputvariable and its test filefiling_status_input.yamlare removed rather than left as a dead input.remove_head_of_household.yamltest is back to main. Each of the three cases this PR had added setfiling_status_input, so they tested a supplied status, not roles. With no supplied status there is nothing left for them to test.tax_unit_role_input,tax_unit_roles_supplied,is_tax_unit_headandis_tax_unit_spouse. Supplied roles decide when every member of the unit has one; otherwise Exclude input dependents from tax unit head and spouse inference #9630's inference runs, which skips input dependents.filing_status_input. They also check that a dataset carrying that column gets the computed status, in the data year and in an extended year.is_tax_unit_dependentinputs.docs/usage/microsimulation.mdstates the roles-only contract. It describes the fallback as it stands after Exclude input dependents from tax unit head and spouse inference #9630, which skips adults input as dependents unless every adult is one.tax_unit_idcolumn, members or not, and core'sall()is True over no members. Sotax_unit_roles_suppliedcounted a memberless unit as supplied with noHEADand raised, where main gives the unit no head. The unit now needs at least one supplied member. The default build has no memberless units (all 79,729 tax-unit rows have members), so its results do not change. Regression test:test_a_tax_unit_without_members_falls_back_rather_than_raising.The problem, with evidence
This section is carried over from earlier rounds of this PR, minus its filing-status parts. This round re-checked only the role counts in the cached H5 and the core loader code.
The build is
populace-us-2024-spm-20260915(populace_us_2024.h5, sha2566496cc43…aee84). It is the default in policyengine.py bundle 6.1.1. The same bytes, tagged 20260909, are policyengine-us'sDEFAULT_DATASET. In the cached copy of that tag, the person table'stax_unit_role_inputhas 79,729HEAD, 35,592SPOUSEand 51,000DEPENDENT, oneHEADper tax unit (79,729 units).policyengine-core sets a dataset column only if a variable has its name. In
policyengine_core/simulations/simulation.py(3.32.15, the version this round tested with), the loader collects any other column asunknown_columnsand only logs that they "do not match any variable in the tax-benefit system and were ignored".Re-running
microunit==0.1.0construct_tax_units(mode="policyengine")on the H5's own raw ASEC columns reproduces every role in the ASEC half. The PUF-clone half copies each person's role from its source person. The legacy CPS pipeline never shipped this column. policyengine-us-data #824 (merged 2026-04-25) says so:What the age rule does to the certified build in 2024, measured earlier on this PR (main 2.11.2):
What #9467 has to do with it
The release manifest's compatibility claim reads:
The "role variable" there is the SPM independence role
is_spm_independent_minor_role, not a tax-unit role. No tax-unit role mechanism was ever intended or shipped. This PR follows #9467's pattern, a source-owned role with a fallback, but uses a formula-free input rather thanDATASET_SOURCE_INPUTS. Two reasons:is_tax_unit_headitself would revert to the age rule after the extension's last year.Design
Which side owns the contract. The producer already emits the role column with the right name and values, so no rebuild is needed. policyengine-us is the consumer that dropped it, so the fix lives here, and the certified H5 bytes flow through unchanged.
Per-unit fallback. A unit uses supplied roles only when every member has one. Otherwise
is_tax_unit_headandis_tax_unit_spouserun main's inference: the oldest adult heads the unit, skipping adults input as tax-unit dependents unless every adult is one (#9630), and the next such adult is the spouse unless any member is separated (#9620). Datasets without the column, and household situations that don't set it, get main's roles: no role is supplied, sotax_unit_roles_suppliedis false and the role variables evaluate main's expression. That includes a tax-unit row no person belongs to (see "Memberless tax units no longer raise" above).Composition with #9630. When a unit's roles are supplied, they decide; #9630's input-dependent skipping applies only to units without them.
tax_unit_roles_suppliedis all-or-none per unit, sowhereevaluating both branches cannot change either side's result.Filing status. The model computes it with main's formula from the resulting roles. Because a supplied role can make an adult a dependent rather than a spouse, the computed status moves for those units (for example, a parent whose adult student was age-ordered as their spouse files as head of household rather than jointly).
Invariants
tests/core/test_tax_unit_role_input_properties.pychecks each of these with Hypothesis (derandomized). Invariants 1–4 run over generated households of up to four tax units with up to four members each, the differential (5) over up to three supplied units, and the error contract (6) samples 40 examples from the 340 possible ordered role lists for a single unit of one to four members:is_tax_unit_dependentinputs.tax_unit_role_inputgives the same filing status and income tax as pinningis_tax_unit_head/spouse/dependentdirectly. Neither side pinsfiling_status.HEADand at most oneSPOUSE.Microsimulation impact
The hub compared merge base
6440b3e604with roles-only branchf99b4e04daonpopulace_us_2024.h5@populace-us-2024-spm-20260909, for 2025 and 2026. Source was checked at current PR headf0cb844fe3. Its only subsequent model change is the memberless-unit fallback; all 79,729 units in these arrays have members, so that fallback does not affect this dataset. Roles are supplied; filing status remains computed. No microsimulation was rerun for this writeup.Weighted aggregate changes, in $ millions (PR minus baseline):
Role shifts: swaps versus reclassifications
The arrays contain 166,321 people, 79,729 tax units and 57,240 households. Tax-unit IDs, person-to-tax-unit membership and unit sizes are unchanged. The role transitions are identical in both years; annual weights differ. Counts below are person records and weighted people, not tax units.
In particular, 1,165 people become dependents (3,335,143 weighted in 2025; 3,358,960 in 2026), while 308 cease being dependents (335,767; 338,165). The 1,473 dependent-flag changes count both directions. Spouse status changes for 27,109 people: 14,109 lose it and 13,000 gain it (63,445,425 weighted people changing in 2025; 63,898,495 in 2026).
Any role changes for 27,382 people (63,647,168 weighted in 2025; 64,101,678 in 2026), across 14,412 units (33,565,319; 33,805,012 weighted units). 12,944 units simply exchange head and spouse, keeping every dependent unchanged. These represent 29,934,642 weighted units in 2025 and 30,148,408 in 2026. Their aggregate AGI changes by only −$356 / +$604 and federal tax by −$95 / +$125, consistent with numerical precision. Standard deduction, CTC and EITC are unchanged. State tax has a material exception below.
The other 1,468 units change who is a dependent. These include 1,160 straightforward spouse-to-dependent reclassifications, 250 headless singleton dependents becoming heads, and two headless pairs becoming head/spouse. Three units retain the same head/spouse/dependent counts while changing dependent identities: two
(H,S,D) → (D,H,S)cycles and one spouse/dependent exchange. Grouping by role counts alone would mistakenly treat these as harmless swaps. Every person has exactly one role on both sides; the PR removes all 252 formerly headless units and leaves exactly one head and at most one spouse per unit.The fallback orders eligible adults by age; the supplied roles replace that ordering. These arrays contain no ages, relationships or person-level income components, so they establish which roles change, without establishing whether each newly recognized dependent satisfies dependency requirements.
Tax-unit reconciliation
These disjoint groups reconcile the 2025 totals. Weighted counts here are tax units. Dollar totals include all signed differences; displayed near-zero swap values round to zero.
The first group includes the 1,160 straightforward spouse-to-dependent units and two units whose old head becomes dependent while the old spouse becomes head. Conversely, 303 units gain a filer from a previous dependent: the 51 spouse gains, 250 singleton heads and two formerly headless pairs. Together they restore $11.059bn AGI and $1.789bn federal tax in 2025 ($11.686bn / $1.920bn in 2026). The two three-person cycles alone remove $2.901bn AGI and $256.199M federal tax in 2025. All seven saved tax-unit money variables are exactly unchanged in the 65,317 units whose roles are unchanged.
The code's gross-income consumer excludes dependents' income from the claiming return; above-the-line deductions also exclude dependents' own amounts. Thus changing who is a dependent changes whose income and deductions enter AGI. EITC earned income likewise sums nondependents. Changing only which of the same two adults is head generally preserves those federal sums.
AGI is not taxable income. Computed filing status determines basic standard deduction and tax brackets. Losing a spouse ends joint status, with the replacement status computed from eligibility. The standard-deduction decline goes in the opposite direction: AGI minus standard deduction falls $10.925bn. This difference is not measured taxable income: actual deductions, exemptions and taxable-income arrays were not exported.
The small federal net change is substantial cancellation: $5.301bn in decreases versus $5.170bn in increases in 2025; $5.608bn versus $5.487bn in 2026. In 2025, 239 units have lower AGI yet higher federal tax: AGI −$3.527bn, standard deduction −$4.844bn, refundable CTC −$88.336M, EITC −$333.598M and federal tax +$861.049M. The arrays establish these concurrent changes, rather than separately assigning each dollar to a consumer.
CTC here includes the other-dependent component: its maximum can add $500 for a newly recognized dependent, while phaseout depends on AGI and computed filing status.
ctcis entitlement after phaseout, not necessarily realized tax relief; nonrefundable credits are capped at liability and refundable CTC has its own earnings and refund limits. The spouse-loss group adds $1.560bn CTC but loses $78.163M refundable CTC; the spouse-gain group loses $123.307M / $46.211M respectively. The latter also loses $48.676M EITC, offsetting the former's $87.196M gain. Federalincome_taxalready includes refundable credits, so these credit changes must not be added again to the tax reduction.Anonymized record examples
2025 dollar amounts are rounded to the nearest dollar. H = head, S = spouse, D = dependent; member order is held fixed within each example. Labels identify examples only, not dataset IDs.
(S,H,D,D) → (H,S,D,D)(H,S,D) → (H,D,D)(H,S) → (H,D)(D) → (H)(H,S,D) → (D,H,S)(H,S) → (H,D)B illustrates tax rising despite lower AGI: standard deduction falls $31,500 → $23,625 and CTC rises $500 → $1,000, but tax rises $4,430. E preserves role counts while replacing a taxable adult. F loses $1.144M AGI and has a weight of 7,169.889 units, making a single record a large aggregate contributor; the saved arrays cannot validate that dependency claim.
Household net income reconciliation
Federal and state income-tax reductions sum to +$734.813M in 2025 and +$747.597M in 2026. Observed household net income rises $520.227M / $578.368M, leaving an aggregate residual of −$214.586M / −$169.229M.
Household net income includes market income, benefits and refundable credits, less taxes before refunds and health costs. The saved arrays lack those remaining components and household membership IDs, so they cannot allocate this residual to programs, households, other taxes or aggregation weights. No particular benefit program is credited with it. The tax-unit reconciliation above is complete for the saved tax-unit outcomes; the household reconciliation remains component-incomplete.
Possible defects found by the impact
No possible defect was fixed as part of this writeup.
Verification
Each test file was run on its own, in the foreground, under the hub's shared heavy lock. The environment was Python 3.13, policyengine-core 3.32.15 and hypothesis 6.168.3.
tests/core/test_tax_unit_role_input.py9d233a2: 1 failed, 11 passed
tests/core/test_tax_unit_role_input_properties.pytests/policy/baseline/household/demographic/tax_unit/tax_unit_role_input.yamltests/policy/contrib/congress/romney/family_security_act/remove_head_of_household.yaml(main's version)The property tests and role YAML also pass on the old head. They test role behaviour that the old head already had, and none of their cases supply a filing status. This round's regressions are the unit tests below.
Regression: a dataset's filing status is no longer read. These three unit tests fail on b77b56f and pass on this head:
test_a_supplied_filing_status_column_is_not_read[2024]and[2026]. The dataset'sfiling_status_inputcolumn saysSURVIVING_SPOUSEfor a lone 16-year-old andSINGLEfor the parent of a 21-year-old full-time student. On the old head the model returns['JOINT', 'SURVIVING_SPOUSE', 'SINGLE']. On this head it computes['JOINT', 'SINGLE', 'HEAD_OF_HOUSEHOLD'](26 USC 152(c)(3)(A)(ii), 2(b)(1)(A)(i)). The loader logs thatfiling_status_inputmatches no variable and was ignored.test_supplied_input_carries_the_producers_column_name_and_values. On the old head,filing_status_inputis still a variable.Regression: a tax unit with no members. The same dataset as above, plus a fourth tax-unit row that no person belongs to:
test_a_tax_unit_without_members_falls_back_rather_than_raisingfails withValueError: tax_unit_role_input gives 1 tax units something other than exactly one HEAD and at most one SPOUSE.tax_unit_roles_suppliedis[True, True, True, False], and the supplied roles hold.ruff format --checkandruff checkpass on the seven changed Python files. The new case intests/microsimulation/test_microsim.pywas not run locally. It loads the full default dataset, and the host's run rule leaves microsimulations to the hub and CI.Recertification
This section is carried over from earlier rounds; this round did not re-check the policyengine.py or microcosm references.
policyengine.py bundle 6.1.1,
data_releases.us, has to be recertified before it can ship a policyengine-us with this change. Itsbundle_idisus-6.1.1, its build ispopulace-us-2024-spm-20260915, and it is certified for policyengine-us 2.2.1.>=2.0.0,<2.3, basispublisher_claim) rests on how those bytes are read, so it must not be widened to cover this change.scripts/bundle.py certify-data.Also affected:
DEFAULT_DATASET, tag 20260909, same sha256). Its standalone results change on release; its hash pin stays valid.tax_unit_role_inputenters its pinned input registry and should be registered in_POOL_SIMULATION_PRESERVED_ENGINE_INPUTSalongsideis_related_to_head_or_spouse. Register tax_unit_role_input and filing_status_input as preserved engine inputs when moving past pe-us 2.2.1 microcosm#1027 still namesfiling_status_inputtoo; with roles only, that input no longer exists, so onlytax_unit_role_inputneeds registering.Follow-ups (not in this PR)
ar_agi_indiv.py:17, Delawarede_agi_indiv.py:18and Mississippims_agi.py:21assign dependents' income to the head before nonlinear per-person tax. Iowaia_net_income.py:22and Montanamt_agi_indiv.py:32use the same pattern in their separate-tax paths, which are disabled for 2025–2026. Fix these existing consumers in follow-up work with swap-invariance regressions. Pure head/spouse swaps move aggregate state tax by about −$8.9M in 2025 (−$8.931M); the saved arrays lack state/component data to attribute that amount to particular states.tax_unit_role_input(and no longerfiling_status_input) as a preserved engine input when microcosm moves past pe-us 2.2.1.axiom: n/a: data-contract change; no provision's rule changes
🤖 Generated with Claude Code