Handle the capital-gains tail clone in release stages on a PUF-support base

[MaxGhenis]

Part of #956 (route A: tools/build_us_puf_support_base.py → tools/build_us_fiscal_refresh_release.py --base-h5, no selection source). Stacked on #959 (us-spm-role-stage). Route A will build from a commit that merges this branch and #991 (us-asec-720-coverage-recodes); this PR does not touch #991's files.

The failure

Route A's full-scale base (base_populace_us_2024_puf_support.h5, sha256 cb1bd1e6…, built at #959 head 47976be6c) keeps every support copy, because no selection source prunes it. Read directly from the H5's person table:

clone index	channel	persons
0	asec	432,523
1	puf_tax_detail	432,523
2	puf_tax_detail	42,336

Clone index 2 is the capital-gains own-tail copy (us_runtime/puf_capital_gains_tail.py, _clone_and_transfer). It copies a primary PUF-detail household, shifts its IDs, sets clone_index = 2, and keeps person_source_id and the puf_tax_detail channel. Each of the 42,336 tail persons has a clone-0 and a clone-1 twin. (person_source_id, clone_index) is unique; (person_source_id, role) repeats 42,336 times. The base carries no person_spine_source_id, so has_assembled_support_metadata is False, and release stages took their non-assembled branch. That branch identified a copy by (source ID, role). A diagnostic release (only the #720 gate waived) failed 199 s in:

US SIPP Head Start source units carry duplicated same-role rows; invalid source role(s): [('4', 'puf_tax_detail'), ...]

Voluntary filing has the same refusal and would have failed next.

Design decision

What the tail copy is. The tail transfer's receipt names its method selective_post_qrf_household_weight_split. Clone 2 takes part of a clone-1 household's weight. The transfer writes onto it only the capital-gains vector, new IDs and tail-provenance columns. The base builder then runs its post-clone stages (weeks_unemployed_post_clone, childcare_post_clone, and others) after transfer_puf_capital_gains_tail in STAGE_BOUNDARIES. They re-impute every PUF-role row, the tail copy included, row by row. The semantics follow:

• Source-person decisions (Head Start, voluntary filing, SSI take-up, stable-key draws) are made once per source unit and fan out to every copy, the tail copy included. The canonical predictor row is the lowest surviving clone index: the native ASEC copy. If a selection kept only PUF-role copies, it is the primary PUF-detail copy (clone 1), not the tail copy.
• Role-keyed row treatments (ASEC reconciliation, PUF-role QRF replacement) give the tail copy its clone-1 twin's role. support_role_series already maps any nonzero clone index to the PUF role. Row-level stochastic imputations give it its own draw, exactly as the base's own post-transfer stages and assembled pools do. They are not forced equal to the twin.
• Uniqueness is (source ID, clone index), never (source ID, role). A repeated pair is a genuinely duplicated copy and still fails closed.

Scope: the stages, not the base. The alternative was to make the base look assembled by giving it person_spine_source_id. That is wrong. On an assembled frame, support_gate_source_channel_series reads the channel column as the physical source, so puf_tax_detail clone-1 rows would be treated as a non-ASEC spine. That would change the ASEC-source masks in Medicare, retirement, weeks-unemployed and alimony. It would also require an assembly manifest this base does not have, and it would change base bytes and force a rebuild. The stage rule is smaller and uniform. A new provenance-owner helper, support_copy_rank_series, returns the validated clone index whenever the clone-index column is present, assembled or not. Only channel-only fixtures, which cannot express a tail copy, keep the historical two-role ranking (ASEC 0, PUF 1). Each affected stage now keys and orders copies by (source ID, copy rank). The helper sits beside support_role_series and inherits its validation (clone index must be a nonnegative integer that agrees with the historical channel), so malformed metadata still fails closed.

Per-stage audit

Every stage _main runs on a --base-h5 PUF-support base before calibration was traced from the release tool to its module. Each module was searched for support_role_series, has_assembled_support_metadata, clone-index and source-ID use, duplicated(, drop_duplicates( and cumcount. Modules with hits were read.

Stage (release order)	Old behaviour on the clone-2 copy	New behaviour	Test
SIPP Head Start (with_us_sipp_head_start_input)	Non-assembled branch keyed copies by (source, role) and refused the base (the reported failure)	Keys copies by (source, copy rank); one prediction per source person from the lowest surviving rank, fanned to all copies; repeated (source, clone index) refused	test_historical_tail_copy_fans_the_source_decision_to_every_copy, test_historical_puf_only_survivor_predicts_from_the_primary_detail_copy, test_historical_duplicate_clone_index_still_fails_closed; channel-only refusal test_imputer_rejects_identical_duplicate_same_role_source_rows (message updated)
Voluntary filing (with_us_voluntary_filing_input)	Same (source, role) refusal ("duplicated same-role rows"), next in line to fail; its clone-agreement summary put the tail copy in its own group, so a disagreeing tail copy passed the gate	Same (source, copy rank) rule at tax-unit grain; the summary compares every copy of a source unit	test_historical_tail_copy_fans_the_source_decision_to_every_copy, test_historical_puf_only_survivor_predicts_from_the_primary_detail_copy, test_historical_duplicate_clone_index_still_fails_closed; channel-only refusal test_duplicate_same_role_source_rows_fail_closed (message updated)
Prior-year income gate (us_prior_year_income_signal_gate; the base runs it too)	Paired copies by per-role occurrence: the tail copy was occurrence 1 and never compared, so its availability could disagree silently	Every copy of a source person is compared on clone-indexed frames	test_clone_availability_checks_the_historical_tail_copy. Base cb1bd1e6: 0 availability mismatches under both groupings, computed from the H5
SSI disability criteria	Imputation accepted the copy (PUF-role stream, no ASEC reporter anchor, like clone 1); the divergence diagnostic left the tail copy out	Imputation unchanged; the diagnostic groups every copy by source person (reporting only, not a gate failure)	test_historical_tail_copy_predicts_in_the_puf_role_stream, test_historical_tail_copy_divergence_joins_its_source_group
SSI take-up (with_us_ssi_take_up, reporter lineage)	Already keys lineage by (source, channel, clone index) when the clone index is present; assignment is per source person	Unchanged	iter1 passed on real data
WIC claim	Draws keyed by stable source identity (year, household, person), shared by all copies; the clone-mismatch gate groups by that key	Unchanged	iter1 passed on real data
Take-up, SNAP take-up, SNAP state take-up, Medicaid take-up	Draws keyed by stable source identity, shared by all copies	Unchanged	iter1 passed on real data
Pregnancy	Clone disagreement grouped by person_source_id across all copies	Unchanged	iter1 passed on real data
Weeks unemployed; workers' compensation, alimony, retirement contributions and distributions, Medicare gates; childcare, energy subsidy, education, other health insurance and the other role-split signal gates (farm, domestic production, educator, Form 4952, SALT refund, capital-gain details, child support, disability benefits)	Split rows by role via support_role_series (clone 2 has the PUF role); no copy-uniqueness assumption	Unchanged	iter1 passed on real data
QBI reconciliation	Clone-index-aware ACS universe resolver	Unchanged	iter1 passed on real data
Hours, immigration, relationships, SPM role, eligibility, SNAP discretionary, SCF wealth and auto loans, SIPP vehicles and tips, ORG wages, ACA, ACS predictor join (disabled: no ACS rows)	No support-provenance keying (household- or row-grain)	Unchanged	iter1 passed on real data

The post-calibration modules the release uses (l0_refit_export, exact_k_ladder, release_input_coverage, release_target_parity, reform_coverage_smoke, register_consistency, input_mass, nonzero_shares) contain no support-provenance references. The release-gate battery re-runs the summaries fixed above.

Real-data evidence

Throwaway local branch (never pushed): this PR plus the #720 gate waiver 4494fa249 cherry-picked. Run under supervise.py with the diagnostic config's limits, on base cb1bd1e6, with the exact diagnostic argv (own --out/--checkpoint-root, release ID populace-us-2024-DIAGNOSTIC-clone2-…).

• iter1 (head 1fbc4d962; artifacts under _recovered/scratch-backup/893/overnight-20260923/clone2-fix/iter1/) got past SIPP Head Start, where the old run died at 199 s. It then passed SSI take-up, SCF auto loans, SIPP vehicles, voluntary filing, SIPP tips, ORG wages, ACA, Medicaid and SNAP eligibility and other health insurance; no earlier gate failed. It failed at 1,742 s wall, peak RSS 64.8 GB (60.4 GiB), at the pre-solve degenerate-input gate. The message was "strike_benefits: every observed value equals the engine default (0)". This is not a clone issue. In the base, raw OI_OFF never equals 12 in any source year (codes seen: 0 to 10, 13, 19, 20), and alimony.py derives strike_benefits as OI_VAL where OI_OFF == 12. The main session has since settled it: the ASEC dictionary defines OI_OFF 12 as strike benefits, so the mapping is correct and the column is genuinely zero. The reviewed exclusion lives on us-strike-benefits-reviewed-exclusion, not in this PR.
• Full release evidence will come from route A's run on integration head 700bf0423. That head merges Read the ASEC at-interview coverage recodes the 2022/2023 inputs lack from the pinned Census archives (#720) #991, this PR at 888c53cef and the strike_benefits exclusion, and runs without the Pooled ASEC 2022/2023 vintages lack the NOW_* coverage recodes: reported Medicaid at interview thins to 24.6M vs ~58M survey under 65 on the certified artifact #720 waiver. No second diagnostic iteration was run from this branch.

Provenance hardening (review rounds 6 to 8)

Review found that copy ranking could accept assembled or malformed support tables that the base refused. A base-vs-head differential harness now runs every entry point this PR touches over 416 synthetic tables: each missing-column combination, clone indices 0/1/2, 3, 7, 262 and 263, fractional, negative and NaN indices, and missing or NaN source IDs. Round 8's final run:

• Zero fail-open divergences. The six fail-open cases reproduced before round 8 are closed.
• Unchanged intended behaviour. All 35 intended base-to-head changes keep their exact outputs.
• No change on valid tables. A positive matrix on real Frame objects with synthetic records (14 historical and 36 assembled cases, 10 entry points each) matches round 7 exactly and refuses nothing.
• Every remaining divergence is fail-closed or diagnostic. The head refuses malformed input the base accepted, or a gate that already failed reports a different reason.

What rounds 6 to 8 added:

Path	Now
Assembled table (raw spine ID present) missing channel, clone index or <entity>_source_id, or with a null source ID	Refused by require_assembled_support_provenance at the entry of every consumer this PR touches, and of the SPM universe reader, before any channel-only or one-row-per-source fallback. Spine assembly writes all four columns for every entity.
Historical table (no spine ID) with a clone index outside {0, 1, 2}	Refused. Only these are produced: 0 by spine assembly and puf_support, 1 by puf_support, 2 by the capital-gains tail (PUF_CAPITAL_GAINS_TAIL_CLONE_INDEX). Assembled tables keep the full nonnegative int64 domain, as before.
Clone index that is fractional, negative, NaN or at least 2**63	Refused before any cast (_validated_clone_indices).
Voluntary-filing gate without support metadata but with a repeated or missing tax_unit_source_id	Flags clone_metadata_missing, matching the receiver's refusal.
Voluntary-filing gate on a channel-only table with a repeated (source, role) pair	Refused, as the receiver refuses it. Every copy of a source is compared.
SPM universe reader	Validates assembled provenance before its native-copy fallback and reads clone indices through the shared int64 validator.

The attested prior_year_income.py and ssi_disability_criteria.py bytes, and the US bundle digest 3667e30e…, are unchanged. Their paths close through the shared readers they already call: the prior-year gate refuses a missing source ID through support_role_series.

Deferred. These need new seed receipts, because the spec engine's source inventory hashes these modules and editing them moves the bundle digest:

• Sharing the tail constant with puf_capital_gains_tail (the same value, 2).
• Validating acs_transfer._pregnancy_source_groups (pre-existing and outside this PR's consumers).
• Channel-only (source, role) uniqueness in the attested prior-year and SSI summaries (pre-existing).
• Validation on the direct-call early return of derive_us_prior_year_income_from_manifest. Its stage path is already closed, because the next operation reads roles through the validator.

Not covered

• The strike_benefits degenerate-input stop is handled outside this PR (see above). The throwaway run waived it locally only.
• Row-level stochastic imputations give the tail copy its own draw rather than copying its clone-1 twin's value. This matches the base's post-transfer stages and assembled pools, and this PR does not change it. Making the tail copy inherit every non-capital-gains value from its twin would be a separate design change in both base and release.
• Channel-only frames keep two-role ranking. They cannot express a tail copy.
• Calibration, export and the post-calibration gate battery were not run on real data. The diagnostic stops at calibration by design.
• ruff format --check still flags pre-existing formatting in several touched files (unchanged lines). CI runs ruff check only, which passes.

Verification

• uv run pytest on the touched and neighbouring suites: Head Start, voluntary filing, prior-year income, SSI disability, multispine PUF clone, spine blindness, capital-gains tail, SSI take-up, WIC, weeks unemployed, fiscal-refresh builder, fiscal-refresh checkpoint inspector, PUF-support base builder, stacked spine, multispine pool, pool input consumers, take-up contract, eligibility, Early Head Start exclusion, release input coverage, spine assembly and agreement, late-producer DAG. All pass.
• The new Head Start, voluntary-filing, prior-year and SSI-divergence tests fail when run against the base commit 517d9a407. The SSI disability imputation test also passes there: it pins behaviour that was already right. The copy-rank helper test imports a function the base commit does not have.
• uv run ruff check .; uv run python tools/ci_test_groups.py --verify; uv run python tools/graph_acceptance_burndown.py --verify: all ok.
• Test fixture tests/us_tail_clone_fixtures.py builds the clone-2 copy with the production _clone_and_transfer, not a hand-written imitation.

Producer-hash note. support_provenance.py is a hashed producer file for the ACS donor receipt qualification. The committed 2026-09-22 receipt pins its origin/main bytes, and this PR changes the file. Valid inputs produce identical outputs, but that receipt lineage must be re-qualified after merge (tools/build_us_acs_donor_receipt_qualification.py) or certified from a pre-#992 checkout.

🤖 Generated with Claude Code

[MaxGhenis]

Real-data check of round 8 (8c59fe2) on the Route A base H5 from run 8f63bf0 (base_populace_us_2024_puf_support.h5, 2,354,504,214 bytes). Each entity's id and provenance columns were read, and the head's shared readers were run on them.

Entity	Rows	Clone indices	Channels	Null source IDs	Duplicate (source, clone)	Readers (require_assembled_support_provenance, support_role_series, support_copy_rank_series)
person	908,023	0, 1, 2	asec, puf_tax_detail	0	0	accept; ranks 432,523 / 432,523 / 42,977
tax_unit	467,432	0, 1, 2	asec, puf_tax_detail	0	0	accept
household	352,932	0, 1, 2	asec, puf_tax_detail	0	0	accept
spm_unit	367,306	0, 1, 2	asec, puf_tax_detail	0	0	accept
family	394,051	0, 1, 2	asec, puf_tax_detail	0	0	accept
marital_unit	722,729	0, 1, 2	asec, puf_tax_detail	0	0	accept

• Every table is historical (no raw spine ID) and carries source, channel and clone-index columns.
• The voluntary-filing summary therefore takes its clone-index branch, whose grouping round 8 does not change. The new no-metadata and channel-only refusals cannot fire on this base.
• The SPM universe reader _person_clone_copies accepts the person table: 432,523 / 432,523 / 42,977 copies.
• The probe took 67 s with a 4.0 GiB peak RSS.