Refuse pinned-artifact builds that only relabel another year (#117)

[MaxGhenis]

Summary

artifact_year pins which file a source package reads, but every {year} label still rendered from --year. So a pinned package built at another year relabelled one year's publisher data as that year. The default bundle and the microcosm US feed both did this:

• soi-congressional-district-2022 stamped 22incd.csv (TY2022) as ty2023: 26,880 facts. This is item 1 of CD record sets stamp 22incd.csv as ty2023 — a tax-year vintage error; add Table 1.4 taxable-interest facts (populace#489) #117.
• soi-state-2022, both IRA 2022 packages and soi-w2-statistics-2020 stamped TY2022 and TY2020 workbooks as ty2023. The W-2 case is the one microcosm#1029's reviewer traced.
• bea-regional-state-personal-income-components-2024 read the 2024 column (AI) at every --year, so its "cy2023" facts were CY2024 values. For example, US personal income at cy2023 was 24,897,613,000 thousand, which is the 2024 cell. The file's 2023 cell is 23,577,208,000 thousand.
• historic_table_2 and its two state packages stamped legal_vintage: tax_year_2023 on TY2022 facts. ici-fact-book-table-30 stamped source.vintage: calendar_year_2023 on its 2026 edition.

This PR does three things:

1. Adds a guard in the harness. A pinned build at year Y that selects exactly the cells the artifact-year build selects, while any rendered label differs, is refused.
2. Fixes the ten packages the guard flags, so each one keeps building at every year the default bundle or the microcosm scope uses.
3. Adds a year-header guard to BEA regional (third commit, which can be dropped on its own; see "Commit 3" below).

Fixes item 1 of #117 (the congressional-district stamp). Items 2 and 3 of #117, the Table 1.4 interest facts and HT2 TY2023, stay open. Context: microcosm#1029.

What changed

Harness: chronicle/source_package.py, chronicle/bundle.py (commit 2)

• artifact_year_restamp_issues(package, year) -> list[SourcePackageIssue] compares the build at Y with the build at the artifact year A, one (row, measure) cell at a time. A cell is read alike when all of these render the same at Y and A:

  • the artifact's parser, archive member, delimiter and header row, and its sheet when the parser is a spreadsheet;
  • the cell as the compiler resolves it: row.column or measure.column, the column-header row and header (the row's wins), both value scales, the divisor, rounding and cell type;
  • its guard cells and the selected_rows entry it reads.

  The artifact sheet_name counts as selection only for xlsx_table_full_rows. For delimited and JSON parsers it is a virtual sheet name, so it counts as a label. The used-range spreadsheet parsers never read it; their record set's sheet_name does the selecting.

  If any cell of a record set is read alike and anything else in that record set differs, or an artifact label (vintage, source_table, a delimited file's virtual sheet name) differs, the record set gets an artifact_year_restamp issue.

  • "Anything else" is right because every fact carries layout.record_set_spec_hash, a hash of the whole compiled record set. A fixed cell beside a year-selected one is relabelled by its neighbour's move.
  • The message names the package, A, Y and the fields that moved.
  • The check compiles record-set specs only; it never parses the artifact.
  • A package whose declarations contain no {year}, {filing_year} or *_by_year returns early.
  • Verdicts are memoised per package instance and year.

• ArtifactYearRestampError(ValueError) is raised by SourcePackage.build_source_rows, build_source_cells and build_source_record_set_specs. build_source_regions, build_source_record_specs, build_source_records and build_facts all go through build_source_record_set_specs, so they raise too.

• validate_source_package reports the issues as errors with code artifact_year_restamp. It still fills in the counts.

• bundle._source_unavailable_for_year treats artifact_year_restamp as "no data for this year".

  • A default bundle skips the package and records the reason in skipped_sources: "source package pins another year's artifact and would only relabel it as Y (artifact_year_restamp)".
  • An explicit --source still reaches build_source_suite, which raises. The bundle then records source_suite_build_failed, the report is invalid, and build-bundle exits 1.

• The rule is structural and names no package. Year-selecting pinned packages keep building off-year: selected_rows with a {year} Period (BEA NIPA, Census projections), column_by_year (CBO, CMS NHE, Z.1, JCT), sheet_name_by_year, and header guards that follow the year.

Package data (commit 1)

• irs_soi/w2_statistics_2020: {year} becomes 2020 in period, record_set_id, source_record_id_prefix, artifact.vintage and legal_vintage.
• irs_soi/congressional_district_2022, state_2022, ira_roth_contributions_2022 and ira_traditional_contributions_2022: the same fields get 2022.
• irs_soi/historic_table_2, historic_table_2_state_agi_2022 and historic_table_2_state_broad_2022: legal_vintage: tax_year_2022.
• ici/fact_book_table_30: vintage: calendar_year_2026, the value its artifact-year build emits today.
• bea/regional_personal_income_state is now year-selecting.
  • Each of the 8 measures uses column_by_year (one YAML anchor) for every year column that SAINC5N__ALL_AREAS_1998_2025.csv has: 1998 = I through 2025 = AJ, so 2023 = AH and 2024 = AI. All 416 selected rows are numeric in all 28 year columns.
  • source_column_id: '2024' becomes '{year}' on the 6 measures that had it. The two value ones are unchanged.
  • A year the file lacks (1997, 2026) fails with "No source artifact for year Y".
  • The package id and label still say 2024. Renaming them would move the alias and the microcosm scope, so that is left to the package owner.

Commit 3: BEA regional year-header guard (chronicle/sources/specs.py)

column_by_year binds each year to a column letter. Commit 3 checks that binding at build time: every BEA measure now expects row 1 of its year's column to name the year (expected_column_header_row: 1, expected_column_header_by_year, one anchor). If a refreshed file has shifted columns, the build then fails instead of relabelling one year's values as another's.

For this to work, _resolve_guard_cell needed one change. A CSV header row keeps the text '2024', but package YAML renders every digit-only string to the integer 2024 (_render_value), and the guard compared with !=. So no year-header guard could ever match a delimited file. _guard_value_matches now accepts a text cell that is exactly the expected integer's digits, and nothing looser: in the new text branch '02024', '2024.0' and booleans do not match. The existing raw == expected comparison is unchanged, so a raw True still equals an expected 1, as before. Only builds this used to reject can change; a build that passed before resolves the same cells.

Trade-off: commit 3 breaks byte-identity for BEA regional at 2024. Each fact's source_cell_keys gains exactly one key, the row-1 header cell of its column (AI1 at 2024), and layout.record_set_spec_hash changes, because the spec hash covers expected_column_header*. At 2023, 2024 and 2025 these fields do not change: value, period, record ids, source_row_keys, aggregate_fact_key and semantic_fact_key. Every year from 1998 to 2025 builds with the guard passing. If byte-identity at 2024 matters more than the build-time binding check, drop commit 3 (aa3e4de). Commits 1 and 2 stand on their own; with them alone BEA at 2024 is byte-identical to origin/main.

Docs

docs/agent-source-package-harness.md has a new "Pinned artifacts and year labels" subsection under the declarative authoring contract. It says that artifact_year pins the file, so labels must be literal unless --year also changes what is read. It also describes the guard, the validate code and the bundle behaviour. Chronicle has no changelog (changelog.d/ and CHANGELOG* are absent).

Why

Chronicle records each fact's publisher reference period (AGENTS.md). docs/pe-uk-source-checklist.md says that "artifact_year identifies the publication vintage; it does not redefine the fact period." Aligning a fact to another period is the consumer's job; microcosm already ages dollar targets from the fact period. With {year} labels on a pinned file, though, the period itself was wrong.

In the microcosm US feed pinned at c5e5bf8, 27,309 of 39,158 rows were restamps: CD 26,880, BEA regional 416, W-2 5, state 4, IRA 2 + 2. For example, the W-2 tips rows labelled ty2023 carry the TY2020 file's sha256, cells and values. A dollar target aged from ty2023 therefore skips three years of wage growth. The design notes and the adversarial audit behind this PR are in the review folder that goes with microcosm#1029.

Invariants

These are checked over the whole enumerated synthetic space, the review's seven counterexample patterns and the real tree (219 pinned packages at every year the default bundle or the microcosm scope uses). They are not proved for every possible YAML:

1. No restamp. The build at Y either raises, or emits no fact whose (value, source_cell_keys, source_row_keys) equals a fact of the A build while any other field differs.
2. The guard agrees with the built facts (differential). The guard fires for (package, Y) exactly when the unguarded build at Y would emit such a fact. It is exact on synthetic packages over the whole enumerated space, and on real bytes for origin/main's IRA and BEA regional YAML.
3. The guard is not over-strict. A build it allows emits exactly what the unguarded build emits. Year-selecting packages (column_by_year, selected_rows on {year}, sheet_name_by_year) are never refused, and a year their file lacks fails with its own error.
4. The artifact year is untouched. artifact_year_restamp_issues(package, A) == [], and unpinned packages are never checked.
5. Determinism. A package that templates nothing builds byte-identical facts at every year.

The selected cells must match together with source_row_keys: a year selected through selected_rows lands on the same virtual cell at every year, and only its source_row_keys differ.

Before/after evidence

All numbers below come from scripts in the review folder (scratch-E/), run on this branch and on a detached origin/main worktree.

• Guard vs prototype. On origin/main's package YAML, the implemented guard flags 22 (package, year) pairs in exactly the 10 packages above, out of 585 pairs over 219 pinned packages (A±1, 2023 and every microcosm scope year). That set is identical to the read-only prototype's.
• Guard after the data fix. On this branch the same scan flags 0 of 585 pairs.
• Artifact-year builds are unchanged. For all 10 packages, the facts and consumer rows built at the artifact year are byte-identical before and after commits 1-2 (cmp, 20 files). With literal labels, every IRS and ICI build at 2023 is byte-identical to its own artifact-year build.
• BEA regional at 2023. US personal income goes from 24,897,613,000,000 (the 2024 column) to 23,577,208,000,000 (the CSV's 2023 cell). 415 of 416 values change; the unchanged one is a residence adjustment that is 0 in both years. Record ids and source_row_keys are unchanged. Every year 1998-2025 builds, and US personal income equals the CSV row for each year.
• Commit 3 on BEA regional (2023, 2024 and 2025): only layout.record_set_spec_hash and source_cell_keys change, the latter by one added header-cell key per fact.

Tests

New module tests/test_chronicle_artifact_year_restamp.py (53 tests, about 8 s locally):

• Refuse. A pinned package with a {year} period raises ArtifactYearRestampError from all seven build entry points. The message names the package, both years and each moved label.
  • Each single templated label (legal_vintage, artifact vintage, source_table, a filter plus its constraint) is enough to refuse it.
  • validate_source_package reports artifact_year_restamp with unchanged counts, and the harness validate-package command exits 1.
• Allow. Off-year builds via column_by_year, selected_rows on {year} and sheet_name_by_year read that year's value (140 rather than 110), with different lineage. A year the file lacks fails with its own ValueError.
  • Literal labels build byte-identical facts at every year.
  • Unpinned packages, the artifact year itself and non-integer build keys (chronicle#79) are not checked.
  • The verdict is memoised per package and year.
• Per cell. The review's counterexamples, each checked against the unguarded build. Two more were added in the final pass: a fixed row column over a column_by_year measure, and an unused {year} artifact sheet name on a used-range spreadsheet.
  • A fixed selected_rows entry beside a {year} one, and a fixed-column measure beside a column_by_year one, are refused.
  • The same cells split into separate record sets are allowed.
  • A {year} virtual sheet name on a delimited file is refused.
• Bundles. A default bundle skips the restamping package, with artifact_year_restamp in the reason and the validation errors, and still builds a literal-label package. An explicit --source gives source_suite_build_failed, an invalid report and build-bundle exit 1.
• Property test. Chronicle does not depend on Hypothesis (it is absent from pyproject.toml and uv.lock), so this test enumerates the whole space instead of sampling it: 2^6 templated-label subsets x 4 selection mechanisms x 5 build years = 1,280 cases. It asserts invariants 1-3 for each case against the unguarded build.
• Real tree.
  • The guard passes at A-1, A+1 and 2023 for every pinned package whose YAML renders {year}/{filing_year} or has a *_by_year key (13 packages after this change).
  • Built-facts invariant plus guard/built-facts agreement at A-1 and A+1 for 11 fast representative pinned packages: W-2, both IRA, Historic Table 2, ICI, BEA regional, both CBO, CMS NHE Table 24, Z.1 and Census projections. The slow builds are left out of the default suite: soi-state-2022 takes 50-90 s a build, the BEA NIPA packages about 6 s, and CD has 26,880 facts. The guard-only test covers them, and the before/after evidence above covers their facts. The repo has no slow marker convention, so none was added.
  • Real-bytes differential: copies of ira_roth_contributions_2022 and bea/regional_personal_income_state edited back to origin/main's labels or fixed column. Without the guard, every artifact-year fact comes back relabelled; with it, the build is refused.
• tests/test_chronicle_source_package.py: test_bea_regional_state_personal_income_components_read_the_requested_year checks 2023 values against the CSV's AH cells, the 2025 US value, and that 1997 and 2026 fail.
• tests/test_chronicle_source_cells.py (commit 3): a guard expecting 2024 matches the header text '2024' but not '2023', '02024', '2024.0' or True.

Default-bundle snapshot, re-pinned from CI run 36244118301. In it, the full test surface had 1 failure (this snapshot) and 1,267 passes. Changes from origin/main:

• tax_year:2023 63,342 → 36,449 and tax_year:2022 41,270 → 68,158: the 26,888 CD, state and IRA facts move off their ty2023 restamp.
• tax_year:2020 11 → 16: the 5 W-2 facts.
• semantic_duplicate_key_count 467 → 2,022. At TY2022 the CD file's state-total and US rows share semantic keys with the Historic Table 2 rows for the same cells. That is 1,560 new duplicate keys among the changed packages' own builds, for a bundle-wide net of +1,555. These are honest duplicates: two IRS publications of one cell, which consumers already select between.
• Every other count (facts, periods, sources, tables, geographies, warnings) is unchanged.

Chronicle Governance

• Approved Chronicle agent role:
  • ledger-source-ingestor for the package data (packages/**), chronicle/sources/specs.py and the tests.
  • The harness files chronicle/source_package.py and chronicle/bundle.py, and docs/agent-source-package-harness.md, are outside every approved role's allowed_paths. CODEOWNERS routes them to @PolicyEngine/core-developers for review.
• Deterministic checks run:
  • source-package validation: the guard scan over all 219 pinned packages flags 0 pairs; validate_source_package reports artifact_year_restamp in the new tests.
  • source-cell preservation: every artifact-year build is byte-identical before and after (facts and consumer rows). BEA regional's changed cells are read from the requested year's column, checked against the CSV.
  • consumer-contract validation and raw-facts boundary validation: run by the full test surface in CI.
  • Locally: tests/test_chronicle_artifact_year_restamp.py, tests/test_chronicle_source_cells.py and the new BEA test in tests/test_chronicle_source_package.py pass (61 passed); ruff check chronicle tests is clean.
• LLM judge verdicts:
  • ledger-source-fidelity: PASS. BEA column letters and guard columns match the CSV header; the 2023 and 2024 US values are checked; the IRS and ICI edits only write the year the pinned file covers. From an independent Opus 5.5 review on a Subfleet lane.
  • ledger-contract: PASS. There is no schema change; record ids and periods now name the true year; BEA gains one lineage key per fact.
  • ledger-boundary: PASS. Chronicle refuses to emit an unreported period and does no aging or alignment.

This PR changes the source-data boundary in one direction only. Chronicle now refuses to emit a fact at a period its publisher did not report. It does not align or age anything; that stays with consumers.

Follow-ups (not in this PR)

• Microcosm must re-derive chronicle_feed_scope.json when it re-pins to this commit. Its 2023 run no longer produces ty2023 rows for CD, state, IRA and W-2, so select_rows will stop with "scoped pairs produced no row".
• IRS has published TY2023 Historic Table 2 (23in54us.xlsx, 23in55cmcsv.csv) and IRA Tables 5/6 (23in05ira.xlsx, 23in06ira.xlsx); packaging them would give real ty2023 facts. No TY2023 W-2 or CD table exists yet.
• Draft PR Publish per-package consumer artifacts #232 (per-package consumer artifacts) resolves the manifest entry with artifact_year or year. Its fact builds should go through the guarded SourcePackage build methods.

🤖 Generated with Claude Code

[MaxGhenis]

Coordination from the TY2023 SOI packaging session:

• Add IRS SOI TY2023 state-data US totals and IRA Tables 5/6 #296 is stacked on this PR. It adds the TY2023 state US totals and IRA Tables 5/6, whose irs_soi.ty2023.* ids would collide with the restamped TY2022 packages on main.
  • Its snapshot delta on top of this branch is +8 facts (US, tax_year:2023, tax_unit), +3 packages and 0 new semantic-duplicate keys.
  • After this PR re-pins, I'll rebase Add IRS SOI TY2023 state-data US totals and IRA Tables 5/6 #296 and re-apply those absolute numbers.
• On this PR's CI failure (semantic_duplicate_key_count 467 to 2022): in-memory builds show that once soi-congressional-district-2022 is at ty2022, its state and US rows share semantic keys with the Historic Table 2 TY2022 state and national facts. Those are the same IRS quantities published in two tables, so this is legitimate cross-table overlap and only a warning. A plain re-pin is right.
• Merge order: Add IRS SOI Historic Table 2 TY2023 national, state broad and state EITC facts #295 (HT2 TY2023, stacked on Add IRS SOI Historic Table 2 TY2023 state AGI-band facts #291) adds 1,553 duplicates against the restamped CD ty2023 facts on main, which carry TY2022 values. Add IRS SOI Historic Table 2 TY2023 national, state broad and state EITC facts #295 should therefore merge after this PR.

🤖 Generated with Claude Code

[MaxGhenis]

Merge audit (2026-09-28 open-PR sweep): gh pr checks exit 0 at head 364f6fd; mergeable=MERGEABLE; not draft; no CHANGES_REQUESTED. Independent review: GPT-6 Astra via Subfleet (job07), verdict APPROVE — Both checks pass; MERGEABLE, isDraft=false, reviewDecision empty.; The per-cell guard, bundle failure behavior, publisher-year corrections and BEA header guards implement the stated change; no blocking defect, secret or debug leftover found.; Regression coverage includes mixed fixed/year-selected cells, column precedence, sheet semantics, 1,280 synthetic combinations and representative publisher artifacts.; No superseding change found in recent merges or local origin/main.; Tests reviewed but no. Repo rules: Require passing CI, pre-merge recheck, approved source roles, deterministic validation and source-fidelity/boundary judges; body records PASS verdicts. Merge commits are allowed and match recent histo