Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/940-state-agi-band-targets.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
State x AGI-band calibration targets from IRS SOI Historic Table 2 (#940): return counts and AGI for each state's bands from $100k up, with $500k-$1M and $1M+ split, bound as shares of the state's published partition rebased onto the state total that binds. The US Chronicle feed is re-pinned to add the TY2023 state AGI-band facts (40,178 rows), and two release coverage requirements demand the $1M+ AGI and return-count row in every state.
25 changes: 21 additions & 4 deletions docs/us-acs-local-soi-target-surface.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,21 @@ Chronicle `b571381` consumer artifact gives the same counts.
| `irs_soi` | 3,819 | 607 | 30,913 |
| **Admin specs** | **3,972** | **760** | **31,066** |

**Re-measured 2026-10-04 on `consumer_facts_us_51aa40f.jsonl`** (sha256
`965a29ac…`, the #940 re-pin: the c5e5bf8 feed plus the TY2023 Historic
Table 2 state AGI-band facts). The 408 state AGI-band rows (51 states x the
four bands from $100k up x returns and AGI; `docs/us-fact-to-target.md`,
"State AGI bands bind as shares of the state total") join `state` and `full`;
their role is `soi_fiscal_distribution`, so `totals` is unchanged:

| Family | `state` | `totals` | `full` |
|---|---:|---:|---:|
| `irs_soi` | 4,227 | 607 | 31,321 |
| **Admin specs** | **4,380** | **760** | **31,474** |

With the 487 population marginals `state` now calibrates to 4,867 targets.
The rest of this section describes the 2026-09-22 measurement.

The 487 population marginals (51 states and 436 congressional districts) are
added on top in every mode, so `state` calibrates to 4,459 targets: Build P's
set, and Build O's 4,461 minus the Vermont under-$1 taxable-interest pair the
Expand All @@ -80,7 +95,7 @@ Historic Table 2 state tables:
| Record set spec | Specs | Content |
|---|---:|---|
| `irs_soi.historic_table_2.state_broad_totals.v1` | 2,397 | 47 all-income-range measures x 51 states, including AGI, income tax, and ACA premium tax credit returns and amounts |
| `irs_soi.historic_table_2.state_agi_counts_and_amounts.v1` | 912 | taxable interest by AGI band |
| `irs_soi.historic_table_2.state_agi_counts_and_amounts.v1` | 912 | taxable interest by AGI band (1,320 on the #940 feed: plus 408 TY2023 return-count and AGI bands) |
| `irs_soi.historic_table_2.state_eitc.v1` | 510 | EITC returns and amounts by number of qualifying children |

It holds no congressional-district SOI row and no row from the TY2023
Expand Down Expand Up @@ -307,15 +322,17 @@ change forces the same review.
|---|---:|
| `usda_snap` | 102 |
| `cms_medicaid` (enrollment) | 51 |
| `irs_soi` state, Historic Table 2 | 3,819 |
| `irs_soi` state, Historic Table 2 | 4,227 |
| `irs_soi` state, district file (district-file-only measures) | 302 |
| `irs_soi` district (427 districts x 51 measures, less 34 banded) | 21,743 |
| **Admin specs** | **26,017** |
| **Admin specs** | **26,425** |

Of the 21,743 district rows, 19,181 are rebased to a Historic Table 2 parent
and 2,562 keep a district-file parent. The 2,189 (state, concept) district
blocks each sum to their parent within 1e-9. Adding the 487 population
marginals gives 26,504 targets, before the holdout. The feed-gated test
marginals gives 26,912 targets, before the holdout. The Historic Table 2
state count includes the 408 state AGI-band rows of #940 (3,819 and 26,017
admin specs before the 2026-10-04 re-pin). The feed-gated test
`test_pinned_feed_state_cd_surface_matches_its_contract` pins these counts
and the reconciliation.

Expand Down
43 changes: 37 additions & 6 deletions docs/us-chronicle-feed-repin.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,33 @@ any other feed, and `test_us_chronicle_feed.py` fails if the parity resources,
the generator and the pin disagree, or if the scope file changes without a
new pin.

## Why the feed moved
## The 2026-10-04 re-pin: TY2023 state AGI bands (microcosm#940)

The state x AGI-band targets of #940 need the TY2023 Historic Table 2 state
file, whose $500k-$1M and $1M+ classes are separate facts
(PolicyEngine/chronicle#291). The pin moved from `c5e5bf8` to `51aa40f`, a
commit that is `c5e5bf8` plus only that package (Chronicle branch
`feed/us-c5e5bf8-state-agi-2023`, tag `microcosm-us-feed-940-v1`), not to
Chronicle main:

- At Chronicle main the artifact-year restamp guard (chronicle#292) stops
building the TY2022 and TY2020 files at `--year 2023`, so ten scoped pairs
(the congressional-district file, the W-2 tips and 401(k) items, the IRA
tables, the `state_2022` US rows) produce no row, and chronicle#304 renames
the Historic Table 2 and district-file capital-gains concepts. Adopting
those is microcosm#1030's re-pin and needs its own decisions.
- Built at `51aa40f`, the feed keeps all 39,158 `c5e5bf8` cells with equal
values and no changed field, and adds exactly the 1,020 TY2023 state AGI
cells (`irs_soi.ty2023.historic_table_2.state_agi.<st>`, 51 new scope
pairs). Compiled at 2024 with the packaged CD crosswalk, aging and the
Medicaid substitutions, every target other than the state AGI bands is
identical on both feeds, name, value and metadata; the bands go from 306
TY2022 rows to 408 TY2023 rows. The comparison script and its report are in
`experiments/940-state-agi-bands/feed-repin/`.
- A later re-pin to Chronicle main carries the 51 pairs over: chronicle#291
adds the same package to main.

## Why the feed moved (2026-09-18)

`_validate_chronicle_hierarchy_labels`
(`packages/microcosm-build/src/microcosm/build/ledger_targets.py`) requires
Expand All @@ -28,15 +54,20 @@ the first labelled US export.

| Field | Value |
|---|---|
| Chronicle commit | `c5e5bf8aa84960c1a200ee47303b19c953092d0f` |
| Feed file | `consumer_facts_us_c5e5bf8.jsonl`, 39,158 rows, 164,603,204 bytes |
| `facts_sha256` | `b85437390021777e746f507c5890305496baf5fc7f2c78ba08ddb090f4839801` |
| Chronicle commit | `51aa40fd28e2ca81937752515e773d123fecfb71` (`c5e5bf8` plus the TY2023 state AGI package; tag `microcosm-us-feed-940-v1`) |
| Feed file | `consumer_facts_us_51aa40f.jsonl`, 40,178 rows, 169,038,757 bytes |
| `facts_sha256` | `965a29ac9458edb58a05184917ee4a19772fd3dc7b4588db6eb6da5803641aac` |
| Consumer fact schema | `chronicle.consumer_fact.v3`, schema file sha256 `bdb51e2a…` (unchanged from the UK pin) |
| Scope | 586 (record set, period) pairs; 62 package runs over build years 2020 to 2029 |
| Scope | 637 (record set, period) pairs; 63 package runs over build years 2020 to 2029 |
| Consumer artifact | none: refused at this commit, see below |

The previous pin was `c5e5bf8aa84960c1a200ee47303b19c953092d0f`
(`consumer_facts_us_c5e5bf8.jsonl`, 39,158 rows, `facts_sha256`
`b8543739…`, 586 pairs, 62 runs); the rest of this document describes its
2026-09-18 rebuild unless it says otherwise.

The feed is too large for the repository. Its home on the build machine is
`~/PolicyEngine/_buildh-runtime/inputs/consumer_facts_us_c5e5bf8.jsonl`,
`~/PolicyEngine/_buildh-runtime/inputs/consumer_facts_us_51aa40f.jsonl`,
beside the previous pins; `tools/build_us_target_parity_manifest.py` reads it
there by default. A holder of the Chronicle commit regenerates it byte for
byte with the commands below.
Expand Down
61 changes: 61 additions & 0 deletions docs/us-fact-to-target.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,67 @@ exclusion register instead.
wiring is safe and normal (the keogh ALD facts rode the feed unmapped for
weeks).

### State AGI bands bind as shares of the state total (microcosm#940)

A state's total return count and AGI leave its top tail free, and the top
tail is what progressive state rate schedules tax: #940 measured Build P's
ACS-local release within 0.12% of Colorado's SOI returns and AGI while
holding 24% of Colorado's SOI AGI above $1M. IRS SOI Historic Table 2
publishes each state's returns and AGI in ten AGI classes, including
$500k–$1M and $1M+ separately. `_soi_reference_from_fact` rescues those
bands from the cross-period refusal and `_rebase_soi_state_agi_bands`
binds them:

- **Which rows.** `return_count` and `adjusted_gross_income` from the
per-state HT2 AGI record sets (`irs_soi.<ty>.historic_table_2.state_agi.<st>`),
all filing statuses, bands from
`US_SOI_STATE_AGI_BAND_MINIMUM_LOWER_BOUND` ($100,000) up. The floor is the
national size-of-AGI floor for the same reason: the SOI slice materializer
counts every tax unit in a band, filer or not. The HT2 `us` rows never
qualify; the national AGI shape belongs to Table 1.1.
- **The value.** `control x band / partition`. The partition is the sum of
the state's published bands of the same vintage, measure and record set
over the whole AGI line, negative AGI under $1 included; HT2 publishes
amounts that add exactly and counts rounded to tens. The control is the
state's latest HT2 all-returns total not after the build period
(`irs_soi.<ty>.historic_table_2.state_broad.<st>.all.<measure>`), the row
that already binds the state's total. Congressional-district `<st>_total`
rows never anchor a state.
- **Periods.** The value lands at the control's period and ages with the
state total: AGI on the CBO AGI series, counts never. The bands therefore
stay the same share of the state total they bind beside, at every stage.
The control may be older than the bands (TY2023 bands on the TY2022 state
total): the newest published shares scale onto the level the state total
binds at, rather than binding a second, differently aged level.
- **One vintage.** Every vintage of a state band reaches the pass; per state
and measure the pass binds the newest vintage whose published partition is
complete and drops the rest, so the TY2022 package's summed `500k_plus` row
never binds beside TY2023's split rows, and a gapped newest vintage falls
back to the last complete one. Overlapping bands, or one source record id
re-emitted with different values, raise.
- **The floor holds at every period.** A state band below $100k, or for a
single filing status, is refused even in its own tax year, so a same-year
vintage can never bind raw sub-floor levels.
- **Receipts.** Every rebased row carries `state_agi_band_share`,
`uprating_factor` and the control's record id.
- **Release gate.** `irs_state_agi_top_tail` and
`irs_state_agi_top_tail_returns` in `US_FISCAL_TARGET_COVERAGE_REQUIREMENTS`
require a $1M+ AGI row and a $1M+ return-count row for all 51 states, so a
feed without the split bands, or a state whose partition or total is
missing for either measure, fails the release rather than shipping without
the constraint. Together the two rows pin each state's AGI above $1M.
- **Period contract.** Without aging, a band rebased onto an older state
total holds that total's period's dollars; `find_period_contract_violations`
reads `uprating_to_period` for rebased rows, the same period target aging
ages them from.
- **Agreement with Table 1.1.** On the pinned feed the states' $500k–$1M and
$1M+ rows sum to 98–99% of Table 1.1's TY2023 classes aged the same way;
Table 1.1 also counts returns filed from other areas and Puerto Rico
(`test_pinned_feed_state_top_tail_agrees_with_table_1_1`).
- **Support.** The rows can bind only where the pool has records in the
band. `experiments/940-state-agi-bands/` measures pre-calibration support
per state and band on the #982 fix's offline outputs.

## 4. Keep the exclusion register honest

`US_FISCAL_TARGET_SUPPORT_EXCLUSIONS` is keyed by `source_record_id` and its
Expand Down
Loading
Loading