Materialize the Census SPM independence role in US builds

[MaxGhenis]

ASEC bases can lose the Census relationship fields needed to classify minor-only SPM units. Restore is_spm_independent_minor_role through an explicit source stage that joins the pinned complete ASEC person files by income year and PERIDNUM, using the existing reviewed derivation. The base builder runs it before support cloning; releases from older ASEC bases rederive and reconcile it.

The adapter persists the engine-declared source input without manufacturing a default. Source mismatches, missing observations, stale provenance and unresolved units refuse the build. Gate details now survive JSON/checkpoint serialization; existing nonconstant role columns cannot bypass source checks.

Targets main (its original base, #948, merged). This does not qualify ACS-origin rows: prepared stacked pools without a sourced role fail before calibration. The ACS source/partition work and national/CD release qualification remain separate requirements.

Re-level on main (22–23 September)

Three merges of origin/main (f69389ee0 at 4d773a478, 0d3c40a03 at e6dede2e4, a222b904c at e66d6f6c8), then two fixes for contracts main gained while this branch was stacked.

Textual conflicts and resolutions:

• release_gate_preflight.py / spm_composition.py: this branch moved the SPM composition check into spm_composition.py so build stages can import it; us: name the SPM composition defect in seconds, and refuse it by name in the release #948's review round hardened it in place. The check stays in spm_composition.py with main's hardened bodies (every shared definition is AST-identical to main's except the remedy text, which now names this stage); the preflight module re-exports from there.
• Generated spec-engine pins (field usage, inventory digests, us-f0-coverage.json, the loader golden vector, the pool tool's bundle spec_sha256): re-derived at the merged tree, not picked from either side. generate_us_bundle_from_constants.py --check gives spec_sha256=1c46dda9…; spec_engine_coverage.py --check gives 42,180/42,180 fields and 41/41 inventory checks.
• Release input coverage (with Require the ASEC reported-receipt inputs in the US release coverage gate (#978 option 1) #980): union. is_spm_independent_minor_role (this PR) and receives_wic/receives_snap/receives_tanf (Local-area publish refused: the donor must be a published release, so fold the reported-receipt columns into the national build #978 option 1) are all required with no reviewed exclusion: 167 required, 7 reviewed exclusions, 174 total. Regenerating the manifest gives no diff.
• PROGRESS-spm-composition.md: both sides' notes kept, with a dated note.
• Make Build O's state SOI contract the ACS local default; totals and full are opt-in #984, Accept a restated age-band Ledger filter only where it agrees with the compiled bounds #986 and Declare the US SPM measurement universe from the source record type #977 merged without textual conflicts.

Fixes for main's newer contracts (121ea4119, ec5d701e6, 4ef635715):

• Serializer registry (frame_serializer_registry, e7bccc3): main forbids a bare DataFrame.to_hdf sink in production code. The stage's scratch parent H5 for derive_spm_role_source is now written by _write_role_projection (pd.HDFStore plus put_frame_table, fixed format). It is registered as spm_role_derivation_projection and has a nullable-boolean round-trip adapter, so the registry now lists ten serializers. The derivation still reads the projection with pd.read_hdf. The re-level did not change spm_role_source.py. The archive reader below changed it later, in 47976be6c.
• Spine blindness: person.get(NATIVE_SPM_ROLE) failed closed because the scanner does not resolve imported constants. It is now an explicit column subscript. The multispine pool tool's runtime import graph is re-pinned from 70 to 73 modules, adding spm_independence_role, spm_role_source and spm_composition, all three already classified and scanned. Both failures were already present at the pre-merge head 96761c635: a scan of that tree shows the same offender and 73 modules.
• H3 graph parity pin (microcosm-graph): CI engine-us (3.14, us-am) on ec5d701e6 failed test_h3_us_post_transfer_parity. The pinned us_post_transfer.json carries multispine_pool's remaining-stage manifest and engine-input projection/defaults digests, which this PR re-pins when the pool projection learns the role is a required source input. It was regenerated with tools/graph_us_post_transfer_fixture.py. Only those three digests change; sources/, expected.csv, NORMALIZATION.md and PRODUCED_BY.txt are byte-identical.

Archive reader (47976be6c) and producer identity

The base stage passes the official Census archives (asecpub23/24/25csv.zip) through --asec-education-source, which has always accepted either the archive or its extracted CSV. spm_role_source._load_source read only a CSV, so it would have refused those archives at the CSV byte-length check. 47976be6c adds _read_archive_member, which does the following:

• requires the archive to match the pinned SHA-256;
• requires exactly one member with the pinned name, whose size and streamed SHA-256 equal the CSV pins;
• reads that member with the same pd.read_csv call the CSV path uses;
• re-hashes the archive after reading.

The pins are the existing ASEC_EDUCATION_ASSISTANCE_ARCHIVES values. The extracted-CSV path is unchanged statement for statement, and everything after the read is shared and unchanged. In a local review run on all three vintages, the archive and the CSV gave equal frames and equal source checks.

Consequence: spm_role_source.py is in source_enrichment.PRODUCER_SOURCE_FILES. From a checkout with this change, _check_producer_source_identity refuses to certify or publish a native-SPM-role candidate recorded before it. Such a candidate must be re-validated from a checkout at its recorded producer commit. This is not a new requirement for populace-us-2024-spm-20260915: it recorded 743683b0e, and source_enrichment.py and contract.py have both changed on main since. Its 2026-09-15 recertification already ran certify, validate and publish from a clean worktree detached at 743683b0e. See docs/us-spm-role-stage.md §2.

Follow-up on 23 September (behaviour-neutral)

• Docs and docstrings now describe the archive reader. They no longer claim spm_role_source.py is byte-identical to main.
• SPM_COMPOSITION_REMEDY no longer offers the head/spouse fallback as a remedy. A release now requires the source role; only the message text changed.
• Dated notes in docs/us-spm-role-for-a-fresh-base.md and docs/us-spm-role-stage.md §4 record what this PR changed.
• The two derivation receipts now report the count of offending SPM units instead of 20 internal unit ids. Each records its pre-redaction SHA-256, and the proof script emits the count.
• Merged origin/main at b78f5a88e (Preflight a new-lineage US release without a selection source #988) with no conflicts. The merged release_gate_preflight.py is main's file, less the definitions this PR moved to spm_composition.py (AST-checked).
• The non-doc diff in packages/ and tools/ since 47976be6c has two sources. The first is Preflight a new-lineage US release without a selection source #988's three files, identical to main's change. The second is docstring and comment edits plus the SPM_COMPOSITION_REMEDY string: an AST comparison with docstrings stripped shows no other difference in spm_role_source.py, spm_independence_role.py or spm_composition.py.
• Unit-id lists removed from the two receipts remain in this branch's git history.

Validation

Re-level (local, policyengine-us 2.2.1, Python 3.14):

• 46 test files pass on their latest local run, with 0 failures. They are the test files this PR touches plus the tests that import the modules it touches: the role stage (75 tests), base builder, fiscal-refresh builder, spine blindness, serializer registry, release input coverage, spec-engine, bundle and multispine suites, SPM role source/enrichment, preflight and the microcosm-data importers. Every file that exercises the role stage, the registry or the blindness scanner was re-run after the last code change (ec5d701e6); test_acceptance_h_parity.py passes at 4ef635715.
• ruff check . is clean. ruff format --check passes on every changed file except test_spec_engine_field_usage.py and test_us_bundle_core_contracts.py, whose drift is identical on origin/main and outside this PR's hunks. CI lint runs ruff check only. tools/ci_test_groups.py --verify passes, and the generated-variable audit --check is current.
• PR CI is the gate. Local checks do not replace it.

Follow-up (23 September, merged tree at the new head):

• 1,880 tests across 25 files pass, with 0 failures. They cover the role stage, role source, composition engine, preflight (including Preflight a new-lineage US release without a selection source #988's tests), Build P enrichment builder and source_enrichment, h5_enrichment, spine blindness, serializer registry, release input coverage, the base builder, the multispine pool tool, the spec-engine seed, coverage and inventory tests, bundle pins, and the fiscal-refresh SPM and gate subset.
• ruff check . is clean, and ruff format --check passes on the changed Python files. ci_test_groups.py --verify, the bundle --check (spec_sha256=1c46dda9…), the generated-variable audit --check and spec_engine_coverage.py --check (42,180/42,180) all pass.

Original (pre-re-level):

• Actual wrapper and signal gate on pinned Build P (166,321 people) and base-q3 (907,382): all roles agree with the original derivation; unresolved SPM units fall from 28 and 222 to zero. Existing tables, columns, weights, strata, mass logs and input files are preserved. Final receipts also serialize the full gate details.
• Fable's second source review closed its eight initial findings and found no new high/medium defect. Dtype-sensitive binding, pre-clone metadata scope and JSON-only provenance limitations are documented in docs/us-spm-role-stage.md.
• No calibration result, incumbent no-worse comparison, release certification or publication is claimed. No end-to-end base-stage run with this stage has been made; the wrapper receipts above ran the stage on existing base outputs.

Aggregate receipts and reproducible proof scripts are committed under experiments/; population files are not.

🤖 Generated with Claude Code