Bind state × AGI-band SOI targets and re-pin the US feed for TY2023 state bands (#940)

[MaxGhenis]

Summary

Binds IRS SOI Historic Table 2 state × AGI-band targets, return counts and AGI amounts, for every state's bands from $100k up, with $500k–$1M and $1M+ split. It re-pins the US Chronicle feed to add the TY2023 state file that publishes those split classes.

Fixes #940.

Why

A state's total return count and AGI calibrate tightly, while its income above $1M, which progressive state rate schedules tax, is unconstrained.

• State top-tail income mass is uncalibrated: CO M+ AGI at 24% of SOI in buildp-acs-local (fails progressive-bracket scoring) #940 measured Build P's ACS-local release within 0.12% of Colorado's SOI returns and AGI. That release held 24% of Colorado's SOI AGI above $1M ($7.8B against $31.7B).
• After Stop the US PUF imputation from predicting income from itself (#982) #1033 (US PUF imputation predicts each income item from the recipient's own survey value, so the PUF clone half reproduces ASEC income #982) the PUF clones reach the national tail. One pre-calibration Colorado record ($125.4M proxy AGI, weight 867 in the ×2 view) carries 84% of Colorado income above $1M.
• National Table 1.1 targets (Bind the SOI Table 1.1 size-of-AGI distribution as calibration targets #960) do not say which state that income lands in.

Before this PR, the state return-count and AGI bands already in the feed (TY2022, 9 bands with a summed 500k_plus) compiled to nothing: _is_untransformed_cross_period_agi_slice refuses every bounded AGI slice from another tax year, and only the EITC and taxable-interest families were rescued.

What changes

The rule (us_runtime/fiscal_targets.py)

• _soi_state_agi_band_status and _is_soi_state_agi_band_fact flag a fact only when all of these hold:
  • its measure is return_count or adjusted_gross_income;
  • it comes from a per-state HT2 AGI record set (irs_soi.<ty>.historic_table_2.state_agi.<st>) at state geography;
  • it is for all filing statuses;
  • its band's lower edge is at or above US_SOI_STATE_AGI_BAND_MINIMUM_LOWER_BOUND ($100,000). The national size-of-AGI floor uses the same edge for the same reason: the SOI slice materializer counts every tax unit in a band, filer or not.
• Every other state HT2 band of those measures is refused at every period, so a same-year vintage can never bind raw sub-floor levels. HT2 us rows never qualify; the national shape belongs to Table 1.1.
• _rebase_soi_state_agi_bands runs after the capital-gains rebase. For each state and measure it binds the newest vintage whose published partition tiles the AGI line. Each band's value is control × band / partition:
  • partition sums the state's published bands of that vintage (sub-floor bands and negative AGI under $1 included).
  • control is the state's latest HT2 state_broad all-returns total not after the build period, the row that already binds the state total. Congressional-district <st>_total rows never anchor a state.
• The value lands at the control's period (uprating_to_period). Target aging then ages AGI on the CBO AGI series with the state total; counts never age. Each rebased row records state_agi_band_share, uprating_factor and the control's record id.
• Vintage handling:
  • Binding bands keep their period in the latest-selection key, so every vintage reaches the pass, which selects one.
  • A gapped newest vintage falls back to the last complete one.
  • Overlapping bands raise, and so does an id re-emitted with different values. An identical re-emission collapses.

Other code

• target_aging.find_period_contract_violations reads uprating_to_period for rebased rows, the period _age_spec ages them from.
• New release requirements irs_state_agi_top_tail and irs_state_agi_top_tail_returns: a $1M+ AGI row and a $1M+ return-count row for all 51 states. The release builder already enforces US_FISCAL_TARGET_COVERAGE_REQUIREMENTS.
• The alias soi-historic-table-2-state-agi-2023 is registered in source_coverage.py and DIRECT_ACTIVE_ALIASES.

Feed re-pin

• The pin moves from c5e5bf8 to 51aa40f: Chronicle c5e5bf8 plus only the TY2023 state AGI package from Add IRS SOI Historic Table 2 TY2023 state AGI-band facts chronicle#291. That commit lives on branch feed/us-c5e5bf8-state-agi-2023 with tag microcosm-us-feed-940-v1.
• Why not Chronicle main. At main, the artifact-year restamp guard (chronicle#292) makes ten scoped pairs produce no row: the CD file, W-2 items, IRA tables and state_2022 US rows no longer restamp to ty2023. chronicle#304 also renames the HT2 and CD capital-gains concepts. Adopting both is Read restamped Chronicle facts at their data year before aging #1030's re-pin and needs its own decisions. I ran the main-based build; it failed on exactly those pairs.
• What moved (experiments/940-state-agi-bands/feed-repin/compare.json):
  • All 39,158 c5e5bf8 cells are present with equal values and no changed field.
  • 1,020 cells were added, all irs_soi.ty2023.historic_table_2.state_agi.
  • Compiled at 2024 as the release does (CD crosswalk, aging, Medicaid substitution), no target other than the state bands is added, removed, or changed in value or metadata. The state bands go from 306 (TY2022, what the rule binds on the old feed) to 408 (TY2023).
• Pins
  • Scope: 586 → 637 pairs, 62 → 63 runs. Feed: 40,178 rows, 965a29ac….
  • Parity regenerated: still 32 compiled / 52 reviewed; irs_soi.historic_table_2 5,654 → 6,674.
  • Feed-gated pins, re-measured:
    • registry: 32,842 → 33,250 targets;
    • national_state: 5,694 → 6,102 at f96be9f41781;
    • ACS-local state: irs_soi 3,819 → 4,227;
    • ACS-local state_cd: SOI 25,864 → 26,272.
  • Every change is exactly +408.
  • tools/spec_engine_coverage.py --check is unchanged (42,184/42,184).

Invariants (property-tested in tests/engine_free/us/test_us_soi_state_agi_bands.py)

1. Share identity: each bound band = control × band / partition (Hypothesis over generated partitions, controls and up to three states).
2. Control homogeneity: scaling one state's total scales exactly that state's bands.
3. Feed-order invariance: shuffled facts compile to the same bands.
4. Partition after aging: each aged AGI band divided by the aged state total equals its published share; counts are unaged (synthetic and feed-gated, for all 408 rows).
5. One vintage per state and measure: the bound bands tile; a collapsed 500k_plus never binds beside split rows.
6. Means inside edges: every state band's aged AGI per return lies inside the band (feed-gated, all 204 AGI rows).
7. Differential against the national owner: summed over the 51 states, the aged $500k–$1M and $1M+ rows are 97–100% of Table 1.1's TY2023 classes aged the same way. They measure 98–99%; Table 1.1 also counts returns filed from other areas and Puerto Rico.

There are also example tests for: the floor at every period, a gap falling back to an older complete vintage, CD totals never anchoring a state, control period eligibility, overlap and duplicate handling, the period contract, and both coverage requirements. The Colorado fixture is the published TY2023 cells, with a test that they sum to the file's stub 0.

Review

• Round 1 (adversarial workflow, math-and-selection lens) found five issues, all fixed in 8042842 and 7a90473:

  • the Colorado fixture's stubs 1–7 were wrong;
  • the floor held only across periods;
  • the period contract read the wrong period;
  • count rows could drop with no gate;
  • malformed vintages that could never bind crashed the compile.

  Fixing the vintage fallback exposed a sixth issue: per-band latest selection shadowed older vintages. It is fixed by the selection-key change.

• An independent review of this head is linked in a comment.

Launch notes (not done here)

• No build, calibration or release was run, and no numbers are published (decision d228 is about ballot revenue scores).
• Support is thin in places. experiments/940-state-agi-bands/README.md measures pre-calibration support per state and band on the US PUF imputation predicts each income item from the recipient's own survey value, so the PUF clone half reproduces ASEC income #982 outputs, all proxy AGI:
  • no state × band cell is empty, and the count cap never binds;
  • 9 states (DE, IA, ID, ME, MI, MT, RI, WI, WY) have a $1M+ SOI mean above every pre-calibration record, so their $1M+ count and amount cannot both be reached by reweighting alone;
  • 4 states have one record above half their $1M+ amount.
• These rows enlarge the target surface. The next exact-k launch must re-freeze --frozen-target-surface-sha256 and re-score the incumbent.
• Bind the SOI Table 1.1 size-of-AGI distribution as calibration targets #960 (national Table 1.1 classes, d416) is compatible: the differential above measures the agreement. Whichever PR merges second resolves the adjacent edits in fiscal_targets.py and docs/us-fact-to-target.md.

axiom: n/a: calibration-target compilation and data feed (no tax-benefit rule changes)

🤖 Generated with Claude Code