Read restamped Chronicle facts at their data year before aging

changelog.d/us-restamped-source-vintage-aging.fixed.md

@@ -0,0 +1 @@
+Read restamped Chronicle facts at their data year. The pinned US feed stamps five packages' pinned files as ty2023: 26,893 facts from the TY2020 W-2 table and the TY2022 congressional-district (PolicyEngine/chronicle#117), state and IRA tables. The compile detects a restamp either by its registered file digest or by a `raw_r2_key` year earlier than its period. It refuses the compile unless a reviewed `US_RESTAMPED_SOURCE_PACKAGES` entry matches. If one matches, aging starts from the data year. It also refuses where a restamp would outrank newer truthful data in latest-vintage selection, or would sit in an aging growth index. On the pinned feed the `national_state` W-2 Box 7 tips amount target moves from $28.28B, aged from 2023, to $34.29B, chained from TY2020. The surface keeps 5,694 targets, and its registry moves from `d315c75804ef` to `884bc45ef335`. On the full surface, 13,176 congressional-district-file dollar targets age from TY2022 under the current series policy: the measures on the AGI series move +3.05% and net capital gains -23.9%.

docs/us-chronicle-feed-repin.md

@@ -181,6 +181,126 @@ and target id, and the register entry carries a reviewed `state_name`
 (`_geography_fallback_label` is UK-only, so the label cannot be derived from
 the FIPS code).
 
+## Restamped packages
+
+Five Chronicle packages in this feed restamp their data: 26,893 observation
+facts carry a later period than their publisher file describes. A Chronicle
+package can pin one file with `artifact.artifact_year` and still render
+`{year}` into its period, record ids and vintage from the build year. The
+scope builds these five at 2023, so each emits its pinned file as `ty2023`.
+PolicyEngine/chronicle#117 reported the congressional-district case in July.
+microcosm#1030 found the other four, and PolicyEngine/chronicle#292 fixes the
+stamp at source.
+
+| Package | File | Data year | Facts stamped ty2023 |
+|---|---|---|---|
+| `soi-congressional-district-2022` | `22incd.csv` | TY2022 | 26,880 |
+| `soi-w2-statistics-2020` | `20in04w2all.xlsx` | TY2020 | 5 |
+| `soi-state-2022` | `22in54us.xlsx` | TY2022 | 4 |
+| `soi-ira-roth-contributions-2022` | `22in06ira.xlsx` | TY2022 | 2 |
+| `soi-ira-traditional-contributions-2022` | `22in05ira.xlsx` | TY2022 | 2 |
+
+The data years come from the files themselves: the title cells ("Tax Year
+2020", "Tax Year 2022"), cell-for-cell equality with the ty2020 W-2 twins,
+and the TY2022 county file for the congressional-district data.
+
+**When it started.** The previous pin already carried 26,891 of these facts:
+- every congressional-district, state and IRA row;
+- the ty2023 W-2 taxpayer count, 401(k) and Roth 401(k) rows.
+
+The 2026-09-18 re-pin added the other two: the ty2023 tips amount and return
+count. These are the "W-2 Social Security tips for 2023 (2)" cells listed
+under [What moved and what did not](#what-moved-and-what-did-not). Before
+them, the only tips amount was the ty2020 row, which would have aged on the
+chained SOI wages bridge from 2020. The previous pin cannot compile on main,
+so that is read from the code, not observed. The restamped row won
+latest-vintage selection and aged on the direct CBO ratio from 2023,
+reaching $28.28B at 2024 instead of the $34.29B it gets aged from its data
+year.
+
+**What the compile does** (`us_runtime/source_vintage.py`). Two rules detect a
+restamp:
+- the fact's `source.source_sha256` is a registered file and its period is
+  after that file's data year;
+- its `source.raw_r2_key` names an earlier year than its period.
+
+Each detected fact must match a reviewed `US_RESTAMPED_SOURCE_PACKAGES` entry,
+or the compile refuses. For matching specs, the readings that aging and the
+period contract use move to the data year:
+- `source_period`;
+- `uprating_to_period`, for specs rebased onto a restamped control.
+
+The stamp is kept in `source_vintage_stamped_period`. Values and target names
+do not change. Two readings still see the stamp, because moving them would
+change which fact wins:
+- **Latest-vintage selection.** The compile refuses where a restamp would
+  outrank a truthful fact dated after its data year.
+- **The choice of a rebase control.**
+
+A restamped fact inside an aging growth index is refused outright.
+
+The key-year rule is exact for single-year IRS files. A multi-year release
+keyed by its publication year can carry later columns: BEA's 2024-keyed
+`SAINC.zip` has a 2025 column. A truthful build of such a column needs a
+`US_LATER_PERIOD_OBSERVATION_EXEMPTIONS` entry.
+
+**On this feed**, compiled as the release does:
+
+- **`national_state`** keeps 5,694 targets, and its registry moves from
+  `d315c75804ef` to `884bc45ef335`.
+  - One value moves: the W-2 Box 7 tips amount, from $28,280,884,269 to
+    $34,287,530,779 (+21.2%).
+  - 53 specs change metadata only: 51 Historic Table 2 net-capital-gains
+    return counts rebased onto the congressional-district US row, and the two
+    `state_2022` counts. Counts do not age.
+  - The 51 rebased counts now read `uprating_from_period` and
+    `uprating_to_period` 2022: a same-year rescale of TY2022 shares onto the
+    TY2022 CD file's total.
+- **Full surface.** 13,176 congressional-district-file dollar targets age from
+  TY2022 instead of 2023.
+  - The 23 measures that age on the CBO AGI series (AGI, income tax, the EITC
+    amounts, taxable interest, dividends, pensions, SALT and the rest) move
+    +3.05%.
+  - Net capital gains move -23.9%, because the SOI Table 1.4 chain records
+    their TY2022 to TY2023 fall.
+  - Qualified dividends (+8.4%) and Schedule C and partnership income (-0.5%)
+    move mostly because their series changes: with no SOI bridge for them, the
+    aging model falls back to the AGI chain from 2022.
+  - These are changes relative to aging from the data year under the current
+    series policy, not realized growth. Taxable interest rose ×2.35 from
+    TY2022 to TY2023 in Table 1.4, far above the AGI link (#117).
+- **Metadata only.**
+  - 13,614 count specs change `source_period`: 13,612 CD-file counts plus the
+    two `state_2022` counts above.
+  - 9 more rebased counts are CD-classified and full-surface only.
+- **No target.** The 401(k), Roth 401(k), IRA, `state_2022` AGI and EITC
+  amount rows compile none.
+
+`test_pinned_feed_restamp_register_matches_the_feed` checks the register
+against this feed in both directions. It runs where the feed is present and
+skips in CI. A re-pin on a Chronicle commit that stamps truthfully leaves
+entries that match nothing, and the test fails until they are deleted. At that
+re-pin the scope moves these pairs to their data years (`ty2022`, and `ty2020`
+for W-2). IRS has also published TY2023 Historic Table 2 and IRA tables, which
+would give true `ty2023` facts.
+
+Two things need a decision before that re-pin:
+
+- **The Historic Table 2 capital-gains returns control.** The CD US row wins a
+  same-period tie on feed order. Stamped truthfully, it would lose to Table
+  1.4 ty2023, which counts returns with a taxable net gain (a different
+  population), and the 51 `national_state` counts would fall 58.5%. Today the
+  51 state counts sum to 2.39× the national Table 1.4 target of the same
+  compiled variable.
+- **`SOI_CONGRESSIONAL_DISTRICT_RECORD_SET_ID`**, which names the `ty2023`
+  record set.
+
+One mis-stamp runs the other way and is out of reach of both rules.
+`bea-regional-state-personal-income-components-2024` reads the 2024 column
+under a `cy2023` stamp (416 facts). No target compiles from it, but
+`bea_regional.state_wages_salaries` is a deferred parity family. Activating it
+before chronicle#292 would calibrate 2024 wages as cy2023.
+
 ## The consumer artifact is refused at this commit
 
 `chronicle build-consumer-artifact` validates every row against

packages/microcosm-build/src/microcosm/build/us_runtime/fiscal_targets.py

@@ -28,6 +28,13 @@
 from microcosm.build.us_runtime.congressional_district_vintage import (
     translate_congressional_district_facts_to_current_vintage,
 )
+from microcosm.build.us_runtime.source_vintage import (
+    RestampShadowsNewerVintageError,
+    SourceVintageCorrection,
+    apply_source_vintage_corrections,
+    check_restamps_stay_out_of_aging_indexes,
+    source_vintage_corrections,
+)
 from microcosm.build.us_runtime.target_aging import (
     age_us_dollar_targets,
     enforce_period_contract,
@@ -1015,10 +1022,17 @@ def compile_us_fiscal_target_registry(
             materialized_facts,
             congressional_district_vintage_crosswalk,
         )
+    # Restamped Chronicle facts (a pinned artifact labelled with a later
+    # build year, PolicyEngine/chronicle#117). Refuses a restamp that is not
+    # in the reviewed register, or one an aging index would read at its stamp.
+    restamps = source_vintage_corrections(materialized_facts)
+    if age_targets:
+        check_restamps_stay_out_of_aging_indexes(materialized_facts, restamps)
     references = (
         *_dynamic_us_fiscal_target_references(
             materialized_facts,
             target_period=target_period,
+            restamps=restamps,
         ),
         *_references_for_target_period(
             US_JCT_TAX_EXPENDITURE_TARGET_REFERENCES,
@@ -1051,6 +1065,10 @@ def compile_us_fiscal_target_registry(
             "target_period": target_period,
         },
     )
+    # From here on restamped facts are read at their data year: aging starts
+    # from it and the period contract checks it. Selection and the rebase
+    # controls above still saw the stamp.
+    registry = apply_source_vintage_corrections(registry, restamps)
     if age_targets:
         # Final nominal transform: age dollar amounts from their source period
         # to the build period on the fully within-surface-aligned registry
@@ -2290,8 +2308,13 @@ def _dynamic_us_fiscal_target_references(
     facts: tuple[object, ...],
     *,
     target_period: int | str,
+    restamps: Mapping[str, SourceVintageCorrection] | None = None,
 ) -> tuple[LedgerTargetReference, ...]:
-    selected = _latest_dynamic_target_references(facts, target_period=target_period)
+    selected = _latest_dynamic_target_references(
+        facts,
+        target_period=target_period,
+        restamps=restamps,
+    )
     _check_exclusion_vintage_scope(source_record_id for source_record_id, _ in selected)
     return tuple(reference for _, reference in selected)
 
@@ -2300,11 +2323,17 @@ def _latest_dynamic_target_references(
     facts: Iterable[object],
     *,
     target_period: int | str,
+    restamps: Mapping[str, SourceVintageCorrection] | None = None,
 ) -> tuple[tuple[str, LedgerTargetReference], ...]:
     """Select one fact per model target shape: the latest eligible period.
 
     Returns ``(source_record_id, reference)`` pairs so the exclusion vintage
     guard and receipt can see which fact won each key.
+
+    Selection compares stamped periods. With ``restamps`` (the compile's
+    :func:`source_vintage_corrections`), it refuses a key where a restamped
+    fact competes with a truthful fact dated after the restamp's data year:
+    the stamp would pick the older data (microcosm#1030 review).
     """
 
     candidates: list[
@@ -2331,6 +2360,12 @@ def _latest_dynamic_target_references(
                     reference,
                 )
             )
+    if restamps:
+        _refuse_restamps_that_shadow_newer_vintages(
+            candidates,
+            restamps,
+            target_period_key=_period_key_from_value(target_period),
+        )
     keys_with_positive_observations = {
         key for key, _, value, _, _ in candidates if value > 0
     }
@@ -2364,6 +2399,51 @@ def _latest_dynamic_target_references(
     )
 
 
+def _refuse_restamps_that_shadow_newer_vintages(
+    candidates: Iterable[
+        tuple[tuple[str, ...], tuple[int, int, str], float, str, object]
+    ],
+    restamps: Mapping[str, SourceVintageCorrection],
+    *,
+    target_period_key: tuple[int, int, str],
+) -> None:
+    """Refuse a restamped candidate that would outrank newer truthful data.
+
+    A TY2020 cell stamped 2023 beats a truthful TY2021 fact of the same
+    target shape on the stamp alone, and the newer data would drop silently.
+    A truthful fact dated from the year after the restamp's data year up to
+    its stamp is shadowed; only candidates eligible at the target period
+    count. None exists on the pinned feed: the ty2020 W-2 twins sit at the
+    data year, not after it.
+    """
+
+    eligible = [
+        candidate
+        for candidate in candidates
+        if _not_after_target_period(candidate[1], target_period_key)
+    ]
+    restamped_keys: dict[tuple[str, ...], list[SourceVintageCorrection]] = {}
+    for key, _, _, source_record_id, _ in eligible:
+        restamp = restamps.get(source_record_id)
+        if restamp is not None:
+            restamped_keys.setdefault(key, []).append(restamp)
+    if not restamped_keys:
+        return
+    conflicts: list[str] = []
+    for key, period_key, _, source_record_id, _ in eligible:
+        if source_record_id in restamps or not period_key[0]:
+            continue
+        year = period_key[1] // 100
+        for restamp in restamped_keys.get(key, ()):
+            if restamp.data_year < year <= restamp.stamped_year:
+                conflicts.append(
+                    f"{restamp.source_record_id} ({restamp.label}) would outrank "
+                    f"{source_record_id} ({year})"
+                )
+    if conflicts:
+        raise RestampShadowsNewerVintageError(tuple(sorted(conflicts)))
+
+
 def _exclusion_vintage_bypasses(
     source_record_ids: Iterable[str],
 ) -> dict[str, tuple[str, ...]]:
@@ -3410,9 +3490,11 @@ def _references_for_target_period(
 def _soi_target_role(fact: object, measure_id: str) -> str:
     # W-2 item facts (generic "amount" measure id, layout-routed via the
     # form_w2_item override) get a named role so target aging can pin them
-    # to the wages series: tips are a W-2 wage component, and the feed's
-    # TY2020 vintage needs the SOI wages actuals as its chain bridge into
-    # the CBO projection years (microcosm#451 item 3).
+    # to the wages series: tips are a W-2 wage component, and the data are
+    # TY2020 (the newest IRS W-2 table), so the SOI wages actuals are the
+    # chain bridge into the CBO projection years (microcosm#451 item 3).
+    # The feed's ty2023 rows restamp the same TY2020 cells; source_vintage
+    # reads them at 2020 before aging.
     if (
         measure_id == "amount"
         and _str_at(fact, "layout", "groupby_dimension")