Carry the ESI employer premium on the ACS local release lane (#454)

[MaxGhenis]

Summary

The ACS local release lane stopped building once the meps_esi_premiums stage landed (#1189): its donor release carries employer_sponsored_insurance_premiums, its ACS rows did not, and fill_reviewed_nulls refuses a frame whose premium is null on some rows. The lane also had no verdict on the column: its input_coverage gate entry was a fixed "passed": True.

This PR puts the premium on every row of the lane's frame, holds it to the donor's scale, and grades it with both ESI gates at staging and on the calibrated artifact. Part of #454.

Stacked on #1199 (esi-premiums-pool-operator, which is stacked on #1189). It uses #1199's with_us_esi_premium_pool_anchor, us_esi_premiums_household_mass_share and the pooled grading in both gates, and changes none of them.

No data release was built or published.

The three decisions

1. How the lane gets the column: a lane-local transfer, fitted on ASEC observations

acs_local_esi_premium_transfer_target_families() is a one-family plan (source_operator_esi_premiums) that only the lane's staging build uses, the same shape as the #765 usual-hours repair. I did not add the column to _DECLARED_ACS_TRANSFER_TARGET_FAMILIES:

• The lane's main transfer fits every declared family on the PUF-detail role (ACS_DONOR_CHANNEL_AUTO), the wrong donor for this column (below).
• Under an existing declared family the column would also leave the stacked pool's source_operator_esi_premiums family (multispine_pool.pool_transfer_target_families() adds a source_operator_* family only for outputs the declared plan does not own), and it would enter the frozen v11 reconstruction (stacked_spine._legacy_stacked_authority_receipt() drops post-v11 targets by (entity, family)), which validate_stacked_post_puf_transfer_receipt compares an attested schema-9 pool against. A new declared family named source_operator_esi_premiums would avoid those two, not the donor problem.
• spine_agreement uses the declared plan as its default metric registry.

Taking the lane's ACS rows from the stacked pool would replace the lane rather than repair it: both tools read the legacy spine tags, legacy support roles and the staging summary's null register, and the hours repair refuses an assembled frame.

Donor. The transfer fits on the donor's ASEC observation role (support clone 0), not on the PUF-detail role that ACS_DONOR_CHANNEL_AUTO gives the lane's tax-detail families. On the observation role employment_income_before_lsr is the measured CPS wage. PUF support imputation writes the clone's wages (PUF_TAX_DETAIL_DEFAULT_PERSON_OUTPUTS), while the clone's premium is a copy of its source person's, so on those rows the premium no longer follows the wage. The recipient side is the measured ACS wage (WAGP).

What that does to the structural zero, computed on the test fixture (synthetic; 320 ASEC households cloned once, 280 ACS people of whom 112 report no wages; only wages tell the employed from the rest):

	ASEC observations (as built)	PUF-detail role
Donor rows with a premium and no wages	0%	42.7%
ACS people without wages who draw a premium	0.0%	31.3%
ACS workers who draw a premium	41.7%	31.0%
ACS rows the anchor clears	0	35 (41% of the transferred premium mass)
Anchor scale factor on the rest	1.34	1.77
ACS-over-donor mean premium per holder, after the anchor	1.23	1.67

With the PUF-detail role the fit cannot tell workers from non-workers (31.3% against 31.0%), the anchor then clears two fifths of what was drawn and loads the total onto fewer workers. Both variants end on the same lane total, because the anchor holds it. These are fixture numbers, not real data: no certified donor release carries the column yet.

2. Scaling

with_optional_acs_spine multiplies every donor household weight by 1 - acs_share and gives ACS the rest of the donor's household mass (half each by default). After pooling, with_acs_local_esi_premium_anchor runs the stacked pool's kernel, with_us_esi_premium_pool_anchor, which reads each side's share from the live frame. On the ACS rows it clears the premium where the person reports no wages and scales the rest by one factor. It never rewrites a donor row.

Three lane-specific points:

• Row kind and missing wages. The kernel tells the donor rows from the ACS rows by the CPS record id (PERIDNUM) and the seven raw coverage fields, and it refuses a transferred row whose wage is missing. ACS PUMS leaves WAGP blank below age 15, outside its earnings universe; the legacy lane keeps that blank in the staging frame, where the reviewed-null register fills it for the engine pass. The lane does not write zeros into the frame. In the anchor's view only, a transferred row whose mapped wage is null, whose age is below 15 and whose raw WAGP is blank reads as zero wages under acs_income_universe's named rule (acs_2024_pums_wagp_age_15_plus). Any other missing wage stays missing and stops the build. The donor qualifier also requires PERIDNUM.

• Source ids. The kernel finds a support clone's wages by person_source_id across the whole frame. The stacked pool makes that id unique at assembly. This lane does not: base_pool._prepared_source_view gives ACS rows their pre-remap ids as source ids, and those can repeat a donor's. Run on the lane frame directly, the kernel raises (source records repeat a 'person_source_id'). Every ACS row here is its own clone-0 source record, so the lane hands the kernel a view without the lookup's two columns and it reads each row's own wages. The wrapper refuses a transferred row that is not clone 0.

• Memory. The kernel copies every table it is given and each gate copies the person table. The lane runs them on a narrow view (ids, memberships and the columns they read) and rebuilds the pooled frame once. By the accounting estimate_optional_acs_pool_peak_bytes uses for pooling (both input frames plus two copies of the pooled tables), that one rebuild holds the donor plus two copies of the pooled frame, which is below the pooling peak the existing guard already budgets for.

3. Gates

• Staging (tools/_legacy/build_us_acs_multispine_base.py): the donor is qualified before the ACS sources are fetched (column, raw coverage columns, wages, legacy support roles, the stage's signal gate). Both gates run on the pooled frame before the staging H5 is written; a red gate fails the build. Verdicts and receipts go in the summary under local_esi_premiums.
• Finalize: both gates run on the calibrated artifact and are recorded as esi_premiums_signal and esi_premiums_anchor, bound to the artifact's SHA-256. Either one red fails finalize. input_coverage.passed is now the conjunction of the two instead of a fixed True.
• Package: refuses a report without both verdicts, with a verdict bound to other bytes, or with a red verdict, before a release directory exists.
• --allow-esi-premium-gaps on both tools records a red verdict without failing, for a diagnostic build from a donor that predates the stage. A release build leaves it unset.
• fill_reviewed_nulls keeps its refusal as a backstop for a staging H5 built before this change.

Invariants

For every lane frame (any acs_share, any household weights on either spine, multi-person ACS households, children with blank wages, any transfer draws):

1. Conservation. Lane-wide weighted premium = donor rows' weighted premium ÷ their share of household mass (1 - acs_share) = the donor release's own weighted premium.
2. Equal intensity. Both spines carry the same weighted premium per unit of household mass, so moving mass from one spine to the other without changing the frame's mass leaves the total unchanged.
3. Donor rows are immutable. No donor cell, weight, stratum or other column changes.
4. Structural zero. An ACS person without wages carries no premium.
5. Proportionality. Every other ACS premium is one common multiple of what the transfer drew.
6. No invented zeros. A blank wage at 15 or older is refused, and the frame's own wage blanks are never overwritten; a person below 15 carries no premium.
7. Idempotence. An anchored frame is returned unchanged (the same object).
8. Identity independence. ACS ids that repeat the donor's change no value.

Intended non-invariant: the lane total is not pinned to the stage's national anchor. It is the donor release's total, and a calibrated release sits within the release tolerance of the anchor, not on it.

Tests

packages/microcosm-build/tests/engine_free/us/test_us_acs_local_esi_premiums.py (51 tests):

• Hypothesis properties for invariants 1 to 8 over the share, both spines' weights and the draws, pooled through the real with_optional_acs_spine.
• Differential tests: the lane total against the donor release's own total; the narrow-view wrapper against the kernel on the whole frame where ids are unique; both gates on the narrow view against the whole frame (verdict, failures and every detail).
• The real orchestration end to end (build_optional_acs_multispine with the real QRF): every row carries the premium, the donor spine is the release byte for byte, both gates pass, and refuse_unassigned_us_esi_premiums has nothing to refuse.
• The donor-role contrast in the table above.
• The wage-universe rule: a blank WAGP below 15 reads as zero in the anchor only; at 15 or older, at an unknown age, or with a non-blank raw WAGP it is refused; end to end, ACS children get no premium and keep their blank wage.
• Refusals: donors without the column, raw fields, PERIDNUM, wages or legacy roles, or with a row whose raw fields lack the record id; assembled donors; a donor failing the signal gate; transferred clones; a missing anchor input; an untransferred spine.
• H5 round trip through the lane's writer and loader.

Also: staging-tool tests (qualification before fetch, waiver, coverage plan, red gates abort before export), release-tool tests (finalize records and enforces both gates; package refuses a missing, unbound, stale or red verdict), and an engine-contract test that runs the real fill_reviewed_nulls on the lane frame against policyengine-us 2.2.1 (the pass runs and leaves the premium; an untransferred spine is still refused).

Verification

This PR is stacked on #1199, so CI does not run until #1189 and #1199 merge and it is retargeted to main (both workflows trigger on pull_request to main only). Local runs at head 6ced20bb9, Python 3.14:

Run	Result
test_us_acs_local_esi_premiums.py (engine-free)	51 passed, exit 0
The lane's tool, registry and prerequisite ESI suites: test_us_acs_local_release_tool.py, test_us_acs_multispine_legacy_builder.py, test_us_spine_blindness.py, test_us_acs_multispine.py, test_us_acs_local_hours.py, test_us_acs_multispine_base_builder.py, test_us_modal_stage_plan_tool.py, test_us_esi_premiums_pool.py, test_us_esi_premiums.py	1,051 passed, 1 skipped, exit 0
engine_contract/us/test_us_acs_local_release_tool.py against policyengine-us 2.2.1	2 passed, exit 0
ruff check ., ruff format --check on touched files, tools/ci_test_plan.py verify	clean
Full packages/microcosm-build/tests/engine_free/us	6,254 passed, 21 skipped, exit 0
Full packages/microcosm-build/tests/engine_contract/us against policyengine-us 2.2.1	178 passed, 3 skipped, exit 0
Full packages/microcosm-build/tests/engine_free/shared	2,648 passed, 2 skipped, 1 failed (exit 1); the failure is environmental, see below

The one shared failure is test_telemetry_emitter.py::test_subprocess_exchanges_ambient_token_and_delivers_events at assert emitter.available. The emitter waits DEFAULT_STARTUP_TIMEOUT_SECONDS (3.0 s) for its service subprocess to become ready, and on this host under tonight's load (load average over 300) the service's cold start took about 13 s. The test fails identically at the prerequisite head db5f656d2 (same line, 4.83 s), and this PR touches neither the emitter nor its test.

Anchor (d1317). Max ruled on #1189 that the stage's scale anchor moves from CMS NHE Table 24 to BEA NIPA 7.8 (B4923C). That change lands in #1189. This PR reads the anchor from the stage (EMPLOYER_PREMIUM_ANCHOR) and restates no figure, so the lane's scaling and gates follow it unchanged. Commit a943ddc5a removed the NHE naming from this PR's comments, help text and test strings. The gate table row in the doc keeps #1189's wording, and #1189 will update it.

Independent review: GPT-6.1 Sol (subfleet run --task review --tier hard), four rounds. Round 1 requested changes: the Modal stage plan did not forward the new waiver, and one doc claim overstated what declaring the column would break. Both were fixed. Rounds 2 to 4 approved, the last at 6ced20bb9 after the adaptation to #1199's anchor rules (record id, missing-wage refusal, clone reset).

Not in this PR

• No real-data measurement of the lane's fill: it needs a certified donor release that carries the column.
• The fill does not use ACS's own employer-coverage, class-of-worker or employment-status items.
• The gates recompute the MEPS-IC assignment on the donor rows only; they compare the ACS rows with those rows.

axiom: n/a: data-pipeline change in microcosm (a microsimulation input on one release lane); no PolicyEngine country-model policy changed.

🤖 Generated with Claude Code

[MaxGhenis]

The base of this PR, #1199, is rebased onto #1189's newest head (dc8578d05, which merges main at 123927679) and answers its second independent review. Its head is now 167793776. Rebase with git rebase --onto origin/esi-premiums-pool-operator <the #1199 head this branch sits on> <this branch>.

Behavior changes since db5f656d2 that can break a caller:

1. The ESI pool anchor needs raw ACS wage evidence. with_us_esi_premium_pool_anchor resolves the ACS earnings universe (resolve_acs_pums_earnings_universe) over the transferred rows. It needs WAGP, age, person_tax_unit_id, person_support_channel, person_support_clone_index and person_source_id.
   • An eligible ACS person (15 or older) with a blank raw WAGP is refused, even when the mapped wage was filled.
   • A person under 15 must carry the named universe zero. apply_acs_pums_earnings_universe_zeros writes those.
2. Household masses are summed per side. The anchor target is source_total * transferred_mass / source_mass. The receipt gains source_household_mass, transferred_household_mass and transferred_wage_evidence. us_esi_premiums_household_mass_share keeps its signature.
3. New constant US_ESI_PREMIUMS_CPS_RECORD_COLUMN = "PERIDNUM".
4. The base's anchor is BEA NIPA 7.8 line 17 ($977.0B for 2024; Produce ESI employer premiums scaled to BEA B4923C from restored ASEC policyholder fields (#454) #1189, d1317).

Counts this branch may pin:

• remaining-stage manifest: 1,079 rows (54 derive), sha 2776a3dd…
• spec_sha256: 62ea14bb…
• field usage: 42,419

A third review of #1199 is running, so one more change is possible. If anything above moves again, I'll post it here.