Choose the capital-gains rebase control by concept and period, not feed order

[MaxGhenis]

Fixes #1035.

Summary

On the pinned feed, the route-A national_state surface calibrates 51 Historic Table 2 (HT2) state capital-gains return counts that sum to 29,599,604. The same surface calibrates the national Table 1.4 target of the same model quantity at 12,392,020, so the states sum to 2.39x the nation. This PR fixes the two causes:

1. Feed order picked the rebase control. The control chooser now takes the latest concept-matched fact, and a tie refuses the compile.
2. Two different IRS counts were treated as one concept. HT2 rows now enter only as declared shares of the Table 1.4 level.

The 51 targets return to 12,289,836, which is exactly the Build P release's value. This moves published target values, so it waits on Max's ruling before merge.

What each IRS count measures

The four feed-backed files (22in14ar.xls, 23in14ar.xls, 22in55cmcsv.csv, 22incd.csv) are sha256-identical to the feed's source.source_sha256. The supplementary workbooks (Tables 1.3 and 1.4A, TY2020/21 Table 1.4, TY2023 HT2) are not in the feed and were fetched from irs.gov. Evidence is in ~/PolicyEngine/_reviews/cg-returns-control-20260925/ (irs-concepts.md).

• Table 1.4, col 37/38 ("Sales of capital assets reported on Form 1040, Schedule D: Taxable net gain"): Schedule D returns that net to a gain.
  • Distributions-only returns (col 35) and Schedule D loss returns (col 39) are separate columns.
  • Table 1.3 row 18, "Sales of capital assets net gain", equals col 35 + col 37 exactly for TY2022 and TY2023.
• HT2 and CD N01000/A01000: both doc guides define these as "Number of returns with net capital gain (less loss)" and its amount, at Form 1040 line 7. That is gain or loss: a Schedule D gain, a loss capped at $3,000, or distributions only.
• The identity: HT2 US N01000 = Table 1.4 col 35 + 37 + 39 within 0.25% in each of TY2020–TY2023. In TY2022, col 37 is 42.4% of N01000.
• No state-level gain-only count exists. HT2, CD, ZIP and county files carry only N01000/A01000 for capital gains.

Which one capital_gains_gross / indicator_sum should target

Table 1.4 col 37.

• capital_gains_gross maps to PE-US capital_gains, which is short-term plus long-term, i.e. Schedule D before the loss limit (capital_gains.py adds).
• The SOI materializer counts a tax unit when its summed value is > 0 (tools/build_us_fiscal_refresh_release.py _signed_component / _soi_component_row).
• Distributions reported without Schedule D are the separate non_sch_d_capital_gains, which has its own col-35 target.
• Build P met col 37 at 12,391,446 against the 12,392,020 target.
• PE-US allows negative gains, but the Build P release data has no tax unit with a net loss, so it could not reach an N01000 count at any level.

How the control drifted

• The chooser kept the first of two equal-period facts.
• The July pin listed Table 1.4 ty2023 first, so Build P used it (factor 0.406751165649407).
• The September re-pin sorts rows by aggregate_fact_key, a content hash. The congressional-district US row, which is TY2022 data stamped ty2023 (CD record sets stamp 22incd.csv as ty2023 — a tax-year vintage error; add Table 1.4 taxable-interest facts (populace#489) chronicle#117), now sorts first and won (factor 0.97964).
• The amount control was just as fragile: reversing the feed hands it to the CD row (0.92455 instead of 0.77190).

What changes (fiscal_targets.py)

• _SOI_CAPITAL_GAINS_FAMILY_CONCEPTS: a reviewed register that maps each SOI record-set family to what its net_capital_gains_* columns count.
  • table_1_4 → schedule_d_taxable_net_gain
  • historic_table_2 and congressional_district (any vintage) → form_1040_line_7_net_gain_or_loss
  • _SOI_CAPITAL_GAINS_MODEL_CONCEPT records what capital_gains_gross measures.
• _soi_capital_gains_active_totals:
  • Only a family registered with the model's concept can be a control.
  • The latest national all-AGI fact not after the build period wins.
  • Different records at the winning period raise AmbiguousSoiCapitalGainsControlError. There is no feed-order tie-break.
  • Unregistered families are never controls.
• _rebase_stale_soi_capital_gains_distributions:
  • Rebased HT2 rows carry soi_source_concept / soi_control_concept, so the bridge is declared.
  • A line-7 row with no control at or after its own period is dropped. It used to ship as a line-7 level on a Schedule D variable. This does not happen on the pinned feed.
• _soi_record_set_family: strips the period token and the CD data vintage.

Effect on the pinned feed

Release compile: target_period=2024, age_targets=True, packaged CD crosswalk, Medicaid substitution.

	before	after
compiled targets	32,842	32,842
national_state targets	5,694	5,694
national_state registry	d315c75804ef	65e4dde11c83
full registry	0fc096598516	e02123644d42
51 HT2 state return counts, sum	29,599,604	12,289,836
CA / TX / FL / NY / IL	3,774,052 / 2,108,705 / 2,093,099 / 1,974,435 / 1,235,812	1,566,997 / 875,540 / 869,060 / 819,791 / 513,113
national Table 1.4 returns target	12,392,020	12,392,020
51 HT2 state amounts	unchanged values (control was already Table 1.4, factor 0.771900044145164)	same values, plus the two concept keys

• Exactly 120 specs differ:
  • 60 return counts move by the same factor, 0.415202720927. These are the 51 national_state rows plus 9 CD-classified HT2 proxies on full.
  • 60 amount specs gain only the concept metadata.
• No spec enters or leaves either surface.
• CA's 1,566,996.6631293728 equals the July scorecard (experiments/replacement_scorecard/incumbent_48b9d479.json).
• The ACS-local release's state SOI mode ships the same 51 rows, so it gets the same fix.

The share assumption. HT2 shares of line-7 returns stand in for shares of Schedule D gain returns, because IRS publishes no state gain-only count. AGI composition is the one measurable source of difference. Weighting each state's HT2 N01000 by AGI class with Table 1.4's per-class gain share would move TY2022 state targets by −2.9% (WV) to +4.3% (DC), with a median absolute change of 1.6%. That refinement is not applied here.

Invariants

• Concept-matched. A control always comes from a family registered with the model's concept. HT2, CD and unregistered families are never chosen. (Hypothesis property)
• Order-free and deterministic. The chosen controls are identical under every permutation of the facts, and a tie at the winning period raises instead of resolving. (Hypothesis property, plus example tests in both orders)
• Latest eligible. The control is the latest matched fact not after the build period, and there is no control when none exists. (Hypothesis property)
• Shares conserved. Rebased state rows sum to control × (Σ HT2 states / HT2 US). When the HT2 states sum to at most the HT2 US total, they sum to at most the control. (Hypothesis property over compiles)
• States never exceed the nation. On national_state, no SOI state family sums past the largest national target of the same quantity and filters (63 families compared, pinned feed). Before this PR the capital-gains returns family violated this at 2.39x, and it was the only one.
• Values move only where intended. Pinned diff: 60 value moves, all by the same factor. Everything else is unchanged apart from the two metadata keys on the 60 amount rows.

Tests

• New unit tests:
  • controls in either feed order;
  • no-control and older-control rows dropped;
  • tie refusal, including one record id carrying two values and an equivalent period label (one identical record twice is not a tie; a tie at a superseded period does not matter);
  • _soi_record_set_family shapes, including a future congressional_district_2024.
• New properties: two Hypothesis properties (300 and 25 examples).
• New pinned-feed tests: the 51 rows' control, factor, concept keys and sum; the states-never-exceed-the-nation invariant.
• Registry pin: moved to 65e4dde11c83.
• Mutation check: each of these fails at least one new test (the property alone falsifies the first):
  • reinstating the feed-order chooser;
  • removing the tie refusal;
  • restoring keep-as-level.
• Local runs on the pinned feed, peak RSS 3.7 GB:
  • test_us_fiscal_targets.py: 192 passed, 0 skipped.
  • test_release_target_parity.py, test_us_chronicle_feed.py, test_us_acs_local_release_tool.py (with MICROCOSM_US_CHRONICLE_FACTS set), test_us_spine_blindness.py and test_us_source_coverage.py: 622 passed.
• ruff check . is clean; ruff format --check is clean on the touched files (59 untouched files on main would be reformatted by the locked ruff, unrelated to this PR).

Review

Independent adversarial review (Subfleet review/standard, served by GPT-6 Astra, report at ~/PolicyEngine/_reviews/cg-returns-control-20260925/review-1036.md): APPROVE. It re-read the IRS headers and doc guides, reproduced the identities, a fresh compile (65e4dde11c83 / e02123644d42, 60 value moves by 0.415202720927, 0 added or removed, 51-state sum 12,289,835.97, all 51 equal to Build P's diagnostic targets), 189 fiscal-target tests, and the three mutation failures. Its P3 notes are addressed in 6ebfcf7: the chooser now also refuses one record id with two values at the winning period (the public compile already rejected duplicate ids earlier, in ledger_targets.py), tests cover superseded ties and equivalent period labels, and the no-loss-returns and AGI-mix median wording is qualified.

Interaction with #1030

#1030 (open) adds test_pinned_feed_capital_gains_returns_control_reads_its_data_year. That test pins the CD US row as the control of 51 specs, and its doc bullet predicts a 58.5% fall once the stamp is truthful. With this PR the CD row is never a control, so:

• Read restamped Chronicle facts at their data year before aging #1030's rebased-control correction stops firing for capital gains, because Table 1.4 is not restamped.
• Whichever PR merges second must drop that test and bullet and re-pin the registry.

PolicyEngine/chronicle#292, which restamps the CD package to ty2022, would also have flipped the control by period alone. With this PR the control does not depend on that stamp.

Not changed here (chips filed)

• Chronicle's HT2 N01000/A01000 concept id (irs_soi.returns_with_taxable_net_capital_gains) is Table 1.4's. The TY2023 HT2 package generator in progress will clone it.
• CD-file capital-gains targets on full. 488 return counts at the line-7 level on the Schedule D indicator; the US row is 2.41x the Table 1.4 row on that surface.
• _is_soi_congressional_district_record_set matches only congressional_district_2022.
• 14 other quantities carry both a stale HT2 US row and a newer Table 1.x national row on national_state, 0.3% to 12.2% apart. Taxable Social Security is the largest gap.

🤖 Generated with Claude Code