Age donor dollar values to the base year per record, not through weights (SCF, SIPP, PUF) #981

Activity

1. MaxGhenis commented on Sep 22, 2026

   

   MaxGhenis

   on Sep 22, 2026

   ContributorAuthor

   More actions

   PUF mechanics, now read rather than inferred (replaces the "I have not read the current ETL" line above).

   Microcosm does not age the PUF. The puf_tax_detail stage in us/spec/sources.yaml declares kind: uprate from 2015 with to_year_from_build_config: true, but no runtime executes that kind: uprate appears only in the schema, source_manifest.py's kind list and source_stages.json. The build consumes release://policyengine/irs-soi-puf/1.8.0/puf_2024.h5 as pinned.

   That file was aged by the archived policyengine-us-data builder (datasets/puf/puf.py PUF.generate, lines 1378-1398, and datasets/puf/uprate_puf.py), in two legs:

   1. 2015 to 2021, uprate_puf. Each SOI line (wages, interest, dividends, pensions, UI, Social Security and others) is scaled by SOI aggregate growth divided by total return-count growth, so per-return. Business income, capital gains and partnership income age their positive and negative values separately. Five itemized deductions use a flat ITMDED_GROW_RATE = 0.02 a year. About 45 remaining PUF fields follow AGI per return. S006 grows by return-count growth.
   2. 2021 to 2024. Each variable is multiplied by create_policyengine_uprating_factors_table(): the policyengine-us uprating parameter's growth divided by population growth, skipping the divide for paths marked per capita or index. The mapping is whatever policyengine-us had when 1.8.0 was built, which predates Point dollar inputs at closer uprating series policyengine-us#9527 (rent, UC, alimony, capital gains components and others).

   Consequences for this issue:

   • The PUF is frozen at 2024. A 2025 base inherits 2024-dollar PUF values unless the artifact is rebuilt or Microcosm runs the declared step.
   • Every record in a column gets one factor, so each column keeps its 2015 distribution scaled. That is the mechanism behind PUF donor uprating overstates mean-reverting distribution series: non_sch_d at 2.39x the SOI concept #487's capital gain distributions at 2.39 times the SOI concept.
   • The right fix is the same per-record donor ager proposed above, run in Microcosm at build time from 2015 to the base year, with the corrected series and per-series SOI chains where they exist. Implementing the declared uprate kind with that ager would close the gap between the spec and what runs.
2. MaxGhenis commented on Oct 12, 2026

   

   MaxGhenis

   on Oct 12, 2026

   ContributorAuthor

   More actions

   Corrections to the 2026-09-22 comment above, from #1248.

   • Its second leg did not happen. The generator's 2021-to-2024 loop iterates the year columns of a pivoted table and its body never runs, so the donor is frozen at TY2021, with two columns at TY2015. On the build donor, 34 directly mapped dollar columns equal the 2015-to-2021 value on 100% of their nonzero source records.
   • The build does not consume the 1.8.0 release asset. The pinned donor is a later, larger file of the same lineage (316.9 MB against 241.0 MB); the spec's locator names the release path.
   • The published 1.8.0 puf_2021.h5 and puf_2024.h5 are identical in all 74 arrays.

   Two consequences for the ager proposed here. Running the declared uprate from 2015 on this donor would count 2015-to-2021 growth twice, and adding a handler for it would not run it, because the PUF stage does not go through the source runtime. And the donor has no single price year: #1248 proposes verifying the factor each column carries against the TY2015 source and dividing it out.