Read the ASEC at-interview coverage recodes the 2022/2023 inputs lack from the pinned Census archives (#720)

[MaxGhenis]

Layer 1 of #720, built into the base stage from inputs that are already pinned. Stacked on #959 (us-spm-role-stage), because route A (epic #956) builds on #959's role stage. This PR unblocks route A's release, which refused at the #744 gate.

Root cause

The base pools three processed ASEC inputs (us_runtime/asec_sources.py, #952): census_cps_2022.h5 7ccca976…, census_cps_2023.h5 cb578173… and census_cps_2024.h5 ec36604c…. The 2022 and 2023 files carry 2 of the 18 NOW_* at-interview recodes and lack A_EXPRRP, PTOTVAL, A_ENRLW and A_FTPT, all of which the 2024 file carries. asec_pool.pool_asec_sources concatenates the three per-year person tables, so every 2022/2023 row gets NaN in those columns, and every reader treats NaN as "no" or 0:

• cps_carried._fill_health_coverage_inputs sets seven reported-coverage flags to False. The Add per-vintage reported-coverage signal gate (#720) #744 gate reports 14 failures, which is what refused route A.
• asec_pool._with_relationship_recode derives A_EXPRRP from line, spouse and parent pointers instead of using the Census code.
• microunit's tax-unit construction reads PTOTVAL as 0 and finds no student enrollment.
• eligibility_inputs sets is_full_time_college_student False for everyone in those two years.

The fix

The official Census archives are already pinned build inputs: --asec-education-source and spm_role_source.ASEC_SPM_ROLE_SOURCES, with the archive SHA-256 and the member's name, size and SHA-256.

The new module us_runtime/asec_census_person_columns.py runs in asec_pool._prepare_year_input, straight after each H5 is read and before the relationship recode and the smoke-household limit. It appends every reviewed column that H5 lacks. It reads them from that year's pinned person member through the reader the SPM role stage already uses, spm_role_source.read_pinned_asec_person_columns, and joins on exact PERIDNUM.

That reader now takes a column list. Its default read, checks and messages are unchanged for the role stage. New tests cover the default read from both the CSV and the archive, and every refusal on a custom column list. All existing SPM role and role-stage tests pass.

AsecSource gains census_person_source. build_us_puf_support_base.py and build_us_asec_pooled_source_base.py bind every pooled year to its file: the --asec-education-source mapping if one is given, otherwise the pinned archive fetched and verified. They resolve it the same way the SPM role stage does (resolve_asec_spm_role_source_paths).

Column review

The starting point was the 24 columns in the 2026-08-23 offline fix (receipt_720.json). A column is kept only if something in the build reads it. test_every_restored_column_cites_readers_that_really_read_it resolves every citation below and requires the cited function or constant to name the column.

column	restored	build reader / reason
NOW_MCAID, NOW_NONM, NOW_CHAMPVA, NOW_MIL, NOW_VACARE, NOW_OTHMT, NOW_IHSFLG	yes	cps_carried._fill_health_coverage_inputs → the seven reported-coverage flags the #744 gate checks
A_EXPRRP	yes	asec_pool._with_relationship_recode; frame.units.MICROUNIT_REQUIRED_COLUMNS → microunit _prepare_household_people (tax-unit head); acs_transfer._person_head_feature; us_late_producer_registry._transfer_input_inventory
PTOTVAL	yes	frame.units._is_microunit_optional_column → microunit _precompute_tax_unit_inputs
A_ENRLW	yes	same → microunit _is_full_time_student
A_FTPT	yes	same; eligibility_inputs.derive_us_eligibility_inputs_from_manifest (is_full_time_college_student)
NOW_CAID, NOW_COV, NOW_DIR, NOW_MCARE, NOW_MRKS, NOW_MRKUN, NOW_PCHIP, NOW_PRIV, NOW_PUB	no	Nothing under packages/*/src or tools/ reads them. A test fails if code starts to.
A_FAMTYP, A_FAMREL, PECOHAB	no	Only the SPM role derivation's optional cross-check reads them. That derivation reads this same member itself, so restoring them would check the member against itself.
LKWEEKS	no	weeks_unemployed.fill_asec_2022_weeks_unemployed_source already restores it for 2022, from the same pinned pppub23.csv. The 2023 and 2024 inputs carry it.

Refusals (AsecCensusPersonColumnsError, a ValueError)

The restoration never predicts, imputes or fills a default. It refuses when:

• The archive or member fails its pins, or the person count is off.
• The member lacks a reviewed column, a reviewed column is not integer (blank, fractional or text), or a value falls outside its reviewed Census codes.
• A key is not 22 digits, or is duplicated on either side.
• An H5 person is missing from the member, or the member has a person the H5 lacks. The join must be one-to-one and total.
• PH_SEQ, P_SEQ, A_LINENO or A_AGE disagree after the join.
• The H5 already carries a reviewed column and it has NaN or differs from the member. A column the H5 carries is never overwritten.
• Any existing H5 column changes position, dtype or value. This is checked explicitly, not with assert.

On 2024, where the H5 carries all eleven columns, the join is therefore verified on real data: 142,125 persons, all eleven columns equal.

Provenance

Each vintage's pool metadata records census_person_columns: the source form and path, the archive URL and SHA-256, the member name, SHA-256 and size, H5, member and joined row counts, columns added, columns verified equal, and the member's code counts. It flows into the source-construction receipt, the raw-stage checkpoint's source_receipt, and the base summary's base_source.

Real-data evidence (source construction only; the full base and release were not run)

Route A's exact base inputs and flags (route-a/run-47976be6ce75/base-config.json) were run with --stage source_construction. The run took 69 s and peaked at 11.2 GB RSS. The inputs matched their pins: H5s 7ccca976…, cb578173…, ec36604c…; archives d2e00025…, cdb39cda…, 318845a2…. The receipts are in experiments/us-asec-720-census-person-columns/, and the review and results are written up in docs/us-asec-census-person-columns.md.

#744 gate on derive_us_cps_carried_inputs of the raw-stage frame:

	Medicaid reporters 2022 / 2023 / 2024	gate
before (route A raw stage, 47976be6c)	0 / 0 / 24,844	FAIL, 14 failures
after (this branch)	28,184 / 26,367 / 24,844	PASS, 0 failures, 0 nulls
2026-08-23 receipt	28,184 / 26,367 / 24,844	pass

The other flags after the fix, 2022 / 2023: non-marketplace direct purchase 8,666 / 8,435, TRICARE 3,882 / 4,318, VA 1,271 / 1,178, IHS 687 / 651, CHAMPVA 260 / 335, other means-tested 275 / 331. All 2024 counts, and has_esi and marketplace in every year, are unchanged.

Equality with the 2026-08-23 corrected inputs:

• Restoring each pinned H5 reproduces the corrected H5s' eleven columns exactly (value, dtype and row order, 2022 and 2023), and every pre-existing column is identical.
• The whole raw-stage frame equals the 8/23 corrected raw-stage frame on every entity table (row counts and all shared columns, including the 226,102 tax units), except exactly the 12 columns this PR does not restore. LKWEEKS 2022 is also equal, on all 146,133 rows.

Structural effect (intended, but downstream should know):

• 2022/2023 now use Census's A_EXPRRP. It changes for 20,595 / 20,561 persons, and code 13 (partner/roommate) now appears (4,312 / 4,387).
• Tax units fall from 231,007 to 226,102: 2022 goes 78,221 → 75,797 and 2023 goes 78,089 → 75,608, while 2024 is unchanged at 74,697. Tax units per person move to 0.519 / 0.524, next to 2024's 0.526.
• Household, family, SPM-unit, marital-unit and person counts are unchanged.

What is and isn't covered

• Covered: every build_us_puf_support_base.py run that pools --asec-h5 (staged, monolithic, or with a support-spine spec), and build_us_asec_pooled_source_base.py.
• Covered only once regenerated: build_us_multispine_pool.py reads the SHA-pinned asec_raw_stage.checkpoint.h5 written by source construction, so a checkpoint built before this PR still has the hole. The Add per-vintage reported-coverage signal gate (#720) #744 gate would refuse a release built on it.
• Not affected: --base-h5 builds and the enrichment and qualification lanes, which start from published parents. ACS-spine rows come from ACS PUMS, not these H5s.
• Unchanged: the Add per-vintage reported-coverage signal gate (#720) #744 gate itself; poverty and SPM outcomes, which are neither targets nor gates; and the reviewed is_unmarried_partner_of_household_head exclusion. That exclusion describes the inputs and is still true of them. Code 13 now appears in every vintage, but it still conflates partners and roommates.
• Layer 2 of Pooled ASEC 2022/2023 vintages lack the NOW_* coverage recodes: reported Medicaid at interview thins to 24.6M vs ~58M survey under 65 on the certified artifact #720 (re-extracting the full Census person set, Ingest ASEC 2024 and 2025 into the US support pool (spine is two vintages behind) #296) stays open.

Checks run locally

• ruff check ., and ruff format on the files this PR touches.
• tools/ci_test_groups.py --verify passes; the new test file lands in the fast and engine groups.
• test_us_spine_blindness passes with the new module classified in _OTHER_US_RUNTIME_MODULES, the way Materialize the Census SPM independence role in US builds #959 classified spm_role_source. The pinned pool-tool import graph stays at 73.
• No attested kernel module or generated manifest changed.
• Targeted suites pass: the new restoration tests (32), SPM role source and role stage, the base builder, the ASEC pool, checkpoint and sources tests, the release gate preflight, and every other test that imports a changed module.

🤖 Generated with Claude Code