Fill IRS SOI uprating series gaps with Publication 1304 actuals

[MaxGhenis]

Summary

The national-total uprating series under calibration/gov/irs/soi/ had no values between 2015 and their next year (2020, 2021, 2022 or 2023), so parameter(instant) held the 2015 value through the gap. Each series' per-capita sibling (total / population, built by tools/per_capita_uprating.py) therefore drifted down with population across the gap and then jumped, and any dollar input dated in or before the gap was uprated past it by the wrong factor. Employment income per capita, for example:

	2015	2016	2017	2018	2019	2020
before	21,811	21,620	21,473	21,324	21,210	24,989
after	21,811	21,940	22,878	23,709	24,672	24,989

This PR fills every gap with IRS SOI actuals, fixes three existing values that were keyed to the wrong tax year or misread, and adds a guard so a gap can't come back. gov.ssa.uprating (the Social Security COLA index that veterans' benefits follow) had the same problem for 2016-2021 and is filled with third-quarter CPI-W.

Sources and method

Every value comes from IRS SOI Publication 1304, all returns, "All returns, total" row: Table 1.4 (YYin14ar.xls) for income items, Table 1.4A (YYin14acg.xls) for capital gains. Columns were located by their header labels in each year's file, never by position: the layouts change across years (TY2018 combined IRA distributions with pensions; TY2021 split partnerships from S corporations; TY2022 renamed wages). Each file now cites one reference per tax-year file, per the repo's references rule, and has a header comment naming the column.

Each series uses the column its existing values came from. I checked every existing value against the tables first (the CBO AGI series' 2015 value turned out to be tax-year 2013 data, #10114):

Series	Variables	Column(s), all returns	Gap filled	Existing values
employment_income	17	salaries and wages	2016-2019	2015, 2020, 2021 match
long_term_capital_gains	14	1.4A net long-term capital gain	2016-2019, 2021-2022	2015, 2020, 2023 match
taxable_pension_income	13	pensions and annuities, taxable	2016-2019	2015, 2020 match
tax_exempt_pension_income	6	pensions and annuities, total less taxable	2016-2019	2015, 2020 match
self_employment_income	5	business or profession net income	2016-2019	2015, 2020 match
social_security	4	social security benefits, total	2016-2019	2015 matches; 2021 was the TY2020 total
qualified_dividend_income	3	qualified dividends	2016-2019	2015, 2020 match
non_qualified_dividend_income	1	ordinary less qualified dividends	2016-2019	2015, 2020 match
alimony_income	2	alimony received	2016-2020	2021 matches; 2015 was $2,000 high (typo)
farm_income	2	farm net income less net loss	2016-2020	2015, 2021 match
farm_rent_income	1	farm rental net income less net loss	2016-2020	2015 matches; 2021 subtracted the number of loss returns, not the loss
partnership_s_corp_income	2	partnership and S corporation net income less net loss	2016-2021	2015 rounded; 2022 was the TY2021 total
rental_income	1	rent and royalty net income less net loss	2016-2020	2015 and 2021 rounded (47B, 46B)
short_term_capital_gains	1	1.4A net short-term capital gain	2016-2021	not SOI: a PUF figure and a backcast (below)
tax_exempt_interest_income	1	tax-exempt interest	2016-2019	2015, 2020 match
taxable_interest_income	1	taxable interest	2016-2019	2015, 2020 match
unemployment_compensation	1	unemployment compensation	2016-2019, 2022	2015, 2020 match; 2021 rounded (208B)

Corrections to existing values

• Partnership and S corporation, 2022: 976B was the TY2021 total (partnership 469.8B - 168.2B + S corporation 766.7B - 92.7B = 975.66B). It moves to 2021 and 2022 gets TY2022: 1,031.2B. It was added in Calibration improvements #3985 (Feb 2024), when TY2021 was the latest year published.
• Social Security, 2021: 774.0B was the TY2020 total. It moves to 2020 and 2021 gets TY2021: 791.2B.
• Farm rental, 2021: 6.114B is 6,186,997 - 72,972 (thousands), where 72,972 is the number of returns with a farm rental loss (the column beside the loss amount). TY2021 net is 6,186,997 - 631,042 = 5.556B.
• Rounded totals (partnership 2015, rental 2015 and 2021, unemployment 2021) and alimony 2015 are restated exactly.

Judgment calls (reviewers: please check these)

1. Short-term capital gains. The file said "Taken from the IRS PUF in the absence of a SOI target": -72B keyed to 2022, and 2015 = -72B x 63% (the old long-term gains 2015/2020 ratio). But Table 1.4A publishes short-term gains, so the series now uses its "Net short-term capital gain" column for 2015-2022 (38.2B in 2015, 77.7B in 2022), replacing the two old values. That is the counterpart of the long-term series, which uses "Net long-term capital gain": returns with a gain, net losses left out. A first draft used gain less loss (Schedule D line 7 summed over all returns). The independent review rejected that, and I agree: as an uprating index it is a large negative total that moves against gains, multiplying a 2021 input by 4.1 into 2022, the year gains fell 63%. From 2022 on the series still grows with CBO net capital gains, so projected ratios are unchanged. Projected levels flip sign (-72B to +77.7B at 2022), which nothing reads.
2. 2018 pensions. The 2018 Form 1040 combined IRA distributions with pensions and annuities, so SOI published no separate figure (TY2019 Table 1.4, note 3). Taxable pensions 2018 = TY2018 combined taxable (1,087.2B) x the mean pension share of that total in 2017 and 2019 (0.7179, 0.7071) = 774.7B. Tax-exempt pensions 2018 = geometric mean of 2017 and 2019, since the combined total includes nontaxable IRA distributions that no other year reports. Each file says how. This only affects inputs dated exactly 2018.
3. gov.ssa.uprating 2015-2021. On main it starts at 2022 and backdate_parameters copies 2022's value back to 2015, so a veterans' benefit dated 2021 got no uprating through 2022 (the 2022 COLA was 5.9%). I checked whether that was intentional: the file never had pre-2022 values on main, and the one attempt to add them (Extend economic uprating parameters to 2100 using SSA Trustees data #6744, which rebased the index) was closed unmerged. The new values are third-quarter CPI-W averages from SSA's published monthly CPI-W, the same method as the existing values. 2016 keeps the 2014Q3 level: 2015Q3 (233.278) fell below 2014Q3 (234.242), so no COLA was payable for 2016 (42 U.S.C. 415(i)). 2016Q3 / 2014Q3 - 1 = 0.35% reproduces the 0.3% COLA for 2017, where comparing with 2015Q3 would give 0.8%.
4. Unemployment compensation start_instant. It was there because CBO's outlays start in 2022, a year after the old last SOI value. With TY2022 in the file it is the default behaviour, so it is removed.

Effect

Full parameter tree (every leaf after system init, before vs after; dump_params.py/diff_params.py from #10114): 35 parameters change, the 17 SOI totals, their 17 per-capita siblings and gov.ssa.uprating. Nothing else moves, including the SSI amounts and other parameters that uprate by gov.ssa.uprating, and the variable-to-series map is unchanged. Core parameter uprating grows a series from its last authored value, so:

• Interior fills change only the filled years (employment, long-term gains, pensions, self-employment, dividends, interest, alimony, farm, gov.ssa.uprating).
• The six series whose last authored value changed (partnership, Social Security, farm rental, rental, short-term gains, unemployment) shift every projected year by one constant factor: x1.0566, x1.0221, x0.9087, x1.0126, x-1.0796 (a sign flip) and x0.1454. Ratios between any two years at or after the last authored year are unchanged. A new test pins this for every SOI series.
• Microsimulation: no change. Datasets are 2024-based and extend forward by ratios between years from 2024 on; every series' last authored year is 2023 or earlier.
• Other readers of these series, found by GitHub code search across the PolicyEngine org: Microcosm's static aging (microcosm-frame uprating_series, base year 2024) uses each total's growth from the base year, so it is unaffected. The SPM threshold paper's uprating analysis covers 2024 to 2025 only. ACA-Calc reads employment income at 2026, which is unchanged. In this repo, the Trustees long-run extension grows each series from its 2036 level, so ratios carry through.

Household inputs dated 2015-2022 projected later are what change. Factor to 2025 for an input dated in year Y, old to new (selected; full table in the session notes):

Series	2016	2018	2019	2020	2021
employment_income	1.503 to 1.481	1.524 to 1.371	1.532 to 1.317
long_term_capital_gains	2.081 to 2.404	2.110 to 1.626	2.121 to 1.831		1.474 to 0.816
taxable_pension_income	1.820 to 1.811	1.846 to 1.644	1.856 to 1.632
social_security	1.672 to 1.669	1.695 to 1.541	1.704 to 1.467	1.712 to 1.368
unemployment_compensation	8.433 to 1.313	8.550 to 1.539	8.597 to 1.591	0.580 to 0.084	1.134 to 0.164
gov.ssa.uprating	1.150 to 1.318	1.150 to 1.288	1.150 to 1.253	1.150 to 1.234	1.150 to 1.218

(Unemployment compensation was the worst: the pandemic-year totals were carried forward, so a 2019 input was multiplied by 8.6 by 2025.)

Blast radius

Static scan of all 4,998 YAML files (34,302 cases): an input of a variable that follows a changed series, dated year Y, with an output or case period in a later year T where index(T)/index(Y) changes. No formula requests a later period than the one it runs in (only last_year and negative offsets), so dated inputs are the only way these factors reach a test. Two cases match, and neither asserts a value that changes:

• medicare_cost.yaml "unit test 5 - 2021 data, premium-free": tax_exempt_interest_income 2019 to 2021 (factor 0.770 to 0.773), but the input is 0.
• medicare_irmaa_magi_two_years_prior.yaml Case 5: tax_exempt_interest_income dated 2015, period 2017, but the variable is read at 2015 (two years prior), so it is never uprated.

Instrumented runs (a sitecustomize hook that logs every runtime uprating on a changed series) of the HHS, SSA, IRS and household shards and the uprating Python tests are queued under heavy_run; results to follow.

Tests and invariants

Invariants, each enforced exhaustively over the finite set of series and years (not sampled):

• Every series an input follows has a January 1 value for every year from its first to its last (test_series_has_a_january_value_for_every_year, parametrized over _series_that_inputs_follow()). On the parent commit it fails for all 17 SOI series, gov.ssa.uprating and CBO AGI. It subsumes Fill CBO total AGI for 2015-2018 with IRS SOI actuals #10114's CBO-only test.
• From its last authored year, each SOI series grows exactly with the series in its uprating metadata, so filling earlier years cannot move projected growth (test_soi_series_follows_its_projection_from_its_last_authored_year).
• Each SOI value cites its own tax year's Table 1.4/1.4A file, except the documented 2018 tax-exempt pension estimate (test_soi_series_cite_the_table_for_each_tax_year). The long-term gains test now requires one Table 1.4A file per value year.
• The corrected values are pinned to their table totals, with the arithmetic in comments (test_soi_corrected_values_hold_their_table_1_4_totals).
• Differential checks: the independent reviewer re-derived every value from the table rows by hand (below), and the session building the stacked TY2023 extension (Extend IRS SOI uprating series to tax year 2023 #10132) recomputed every authored value at c96ccb7 from its own extraction of the IRS files with zero mismatches (its scripts/series_new.py; the 2018 pension estimates are not table values and were skipped).

The new guard fails for calibration.gov.cbo.income_by_source.adjusted_gross_income until #10114 merges; this PR rebases after it.

Review

Independent review by GPT-6.1 Sol (subfleet run --task review --tier hard), two rounds:

• Round 1 at 0383877: request changes, one finding. It re-derived all 120 SOI values from the table rows with its own column choices and arithmetic, and every one matches. It confirmed the three corrections, that partnership plus S corporation net income less loss is the same quantity before and after the 2021 table split, the 2018 estimates, the COLA index and the 2016 no-COLA treatment, and every mechanism claim against core's code (uprate_parameter appends only after the last authored value; variable uprating divides the index at the target by its value at the latest earlier known period). The finding was the short-term capital gains concept (judgment call 1).
• Round 2 at c96ccb7: approve. All eight new short-term values match Table 1.4A column 6, the finding is resolved, and both 2018 estimates are now listed in the test. It asked for two checks it could not run. git diff --stat 03838777 c96ccb7 shows exactly the three intended files. policyengine-us-data, which it could not see, is archived and has no reference to calibration.gov.irs.soi in its tracked Python, YAML or TOML.

Remaining condition from the review: merge after #10114.

axiom: n/a: calibration and uprating data, no policy rule

Follow-ups filed separately: CPI-W monthly series switches from not seasonally adjusted to seasonally adjusted values in January 2024; extend the SOI series to tax year 2023 (moves projections, so kept out of this gap fill).

🤖 Generated with Claude Code