Derive raw-PUF partnership self-employment earnings with the archived formula

[MaxGhenis]

What was wrong

puf_support._person_source_values resolves each requested donor output from the PUF arrays. For partnership_self_employment_net_earnings its last resort was a raw-column fallback that returned E25960 + E26180.

A few lines above, the same function builds partnership_income = E25980 - E25960 and s_corp_income = E26190 - E26180. So E25960 and E26180 are the partnership and S corporation loss legs, and the fallback defined partnership self-employment net earnings as partnership losses plus S corporation losses. That is not the engine's variable (policyengine-us 2.2.1 documents it as "partnership net earnings allocated to self-employment tax and earned-income calculations, generally from Schedule K-1 Box 14"), and it let an S corporation amount into a self-employment column.

The fallback arrived in ddb7d81 (#182, 2026-06-26).

Which branch production reaches

Production does not reach the fallback, so this PR changes no production value.

• The two production callers are load_puf_tax_unit_donor (us_runtime/puf_donor_io.py, used by tools/build_us_multispine_pool.py) and _puf_tax_unit_donor_from_h5 in tools/build_us_puf_support_base.py. Both pass every root dataset of one processed PUF H5 to puf_tax_unit_donor_from_arrays.
• The certified default's file is the processed_puf source pinned in us/spec/sources.yaml (sha256 7669f5b5…d6df, 316,939,164 bytes). tools/route_a/route_a.sh and docs/us-modal-stage-route-a-base-plan.json name the same hash.
• I hashed a local copy to that pin and listed its root datasets: 97 datasets, none E-coded. It has partnership_se_income and partnership_s_corp_income. It does not have partnership_self_employment_net_earnings, partnership_income or s_corp_income.
• So for this leaf production takes the partnership_se_income alias branch (puf_support.py, the branch just above the raw-column ones). The column is both-signed in that file (6.4% of rows positive, 3.6% negative), which a sum of two loss legs could not be.
• The task brief said the donor carries the engine-named column. It carries the older name, partnership_se_income; the conclusion is the same.

Before this PR, one caller reached E25960 + E26180: the test test_puf_tax_unit_donor_derives_partnership_and_s_corp_split_from_raw_fields, which asserted the loss sum.

The fix

Port the archived derivation, and fail closed when a raw PUF lacks any input.

The archived pipeline (PolicyEngine/policyengine-us-data at 42ed5d45, datasets/puf/puf.py lines 723-746) computes:

gross_se        = (E30400 + E30500) / 0.9235
partnership_se  = gross_se - (E00900 + E02100)   where E25940 + E25980 - E25920 - E25960 != 0
                = 0                               otherwise

It matches part_se in Yale Budget Lab Tax-Data's src/process_puf.R (lines 279-281 at 2f55fa97). The archived pipeline has used this formula for the column since policyengine-us-data#485 (2026-01-26), then under the name partnership_se_income.

I ported it and did not remove the fallback because every other raw-column fallback in the function reproduces the archived derivation of its leaf: partnership_income, s_corp_income, the E17500 medical split, qualified_tuition_expenses and the E02400 Social Security carry. This one was the exception.

All eight columns are required. With any absent, _person_source_values returns None and puf_tax_unit_donor_from_arrays raises PUF donor cannot derive requested output(s), as it does for any other underivable output. A raw PUF carrying only E25960 and E26180 now raises.

The column meanings in the new docstring are the archived pipeline's own comments. I did not check them against the PUF documentation, and the docstring says so.

Invariants

Stated in test_us_puf_partnership_se_derivation.py and checked with Hypothesis (200 examples each):

1. Equal to the archived formula bit for bit, on float columns with missing values and on integer columns. The reference is the archived lines copied into the test.
2. Exactly zero when net partnership income is exactly zero, whatever the Schedule SE, C and F amounts.
3. On a record with partnership activity, (result + E00900 + E02100) × 0.9235 = E30400 + E30500.
4. No S corporation column changes the result.
5. Rows are derived independently: permuting the rows permutes the result.
6. On integer-dollar amounts it is homogeneous of degree one under power-of-two rescaling, bit for bit.
7. With any of the eight raw columns absent it derives nothing, whatever other raw columns are present.

Two behaviours follow from porting the archive faithfully and are intended:

• The result can be negative. It is a residual, so a record whose Schedule C and F amounts exceed its grossed-up Schedule SE amounts gets a negative value.
• The gate is on net partnership income, so a record whose partnership income and losses cancel exactly gets zero.

Example tests cover hand arithmetic for six records, missing values, the source precedence (stored leaf, then partnership_se_income, then raw columns), the donor-builder seam, and a refusal for each of the eight columns removed in turn.

Do the tests catch wrong formulas?

An in-memory mutation check (mutation_check.py, not committed) swapped the helper for 13 wrong versions and ran every test in the new file against each:

Mutant	Tests that fail
the old E25960 + E26180	9
no 0.9235 gross-up; multiply instead of divide; factor 0.9265	6 each
no partnership gate; gate on active legs only; gate on positive net only; sign error in the gate	6, 5, 4, 5
drop the spouse's E30500; drop Schedule F	6, 5
add S corporation income	2
NaN instead of zero for inactive records	5
accept four of the eight raw columns	3

Every mutant was caught, and the unmutated helper passed all tests.

Pins

puf_support.py is on _DIRECT_KERNEL_MODULES in spec_engine/seeds.py, so its bytes are attested. Main moved these pins twice while this PR was open, once in #1186, which also edited this module. So the branch merges main taking main's pins, and the last commit recomputes them for this PR's bytes on the merged tree (main at 977ac983a):

Pin	main	this PR
EXPECTED_HASHES["seed_protocol"]	8d028354…	0e4c8a4a…
EXPECTED_HASHES["seed_map"]	bdd33986…	41a64b88…
US spec_sha256 (engine_workflow/us/test_us_multispine_pool_tool.py, coverage JSON)	638cd60e…	3080fe93…
minimal-loader golden (test_spec_engine_loader.py)	aadf7c8c…	0c0c8eea…

docs/evidence/spec-engine/us-f0-coverage.json was regenerated with tools/spec_engine_coverage.py. Afterwards tools/spec_engine_coverage.py --check and tools/generate_us_bundle_from_constants.py --check both exit 0.

Nothing else names the old digests. The module defines no graph kernel class, and no graph parity fixture refers to it. No schema or source_stages.json changed, so the frozen manifest hash stays put.

Tests run locally

On head 7556a00cf, US-engine venv, pytest -n 4: 341 passed, 0 failed. The run covered:

• test_us_puf_partnership_se_derivation.py (new), test_us_puf_support.py, test_us_qbi_inputs.py, test_us_puf_donor_io.py;
• both test_us_multispine_pool_tool.py files;
• test_spec_engine_inventory_coverage.py, test_spec_engine_coverage_tool.py, test_spec_engine_loader.py;
• test_us_plan.py::TestUsSources::test_no_incumbent_data_package_references_in_live_tree.

That last guard failed CI on an earlier head because the new docstring named the retired data package. The code now cites it as "the retired enhanced-CPS repository at 42ed5d45", as the other ports do.

ruff check and ruff format --check on the touched files are clean, and python3 tools/ci_test_plan.py verify passes.

Review

GPT-6.1 Sol reviewed the PR (Subfleet, review/hard) and returned APPROVE with one P3: the docstring credited the nonnumeric-to-zero coercion to the archive's fillna(0), but _numeric_array does it. The docstring now says so.

The reviewer also confirmed:

• The derivation matches the archived formula bit for bit over 10,000 float64 rows, missing values, signed zero and infinities.
• No surviving fixture depends on the old loss sum.
• Both production callers read processed H5 arrays.
• The re-pinned identities recompute exactly.

A round-2 delta review on head 7556a00cf (GPT-6.1 Sol) returned APPROVE with no findings. It covered the P3 fix, the retired-package wording, the merge with main at 977ac983a, and the re-pins. It confirmed:

• No tracked file outside the test's allowlist names the retired data package.
• Each of the four identities recomputes exactly, and swapping in only main's puf_support.py bytes restores main's value for each.
• The merge kept both sides: main's _preserve_cps_reported_social_security is byte-identical to main, and this PR's derivation is unchanged since round 1.

Both review outputs are kept outside the repository, at ~/reviews/microcosm-raw-puf-partnership-se-2026-10-11/.

Notes

• No dataset build or release is part of this PR.
• Attribute self-employed health premiums to farm and partnership businesses #1208 re-pins the same EXPECTED_HASHES entries and coverage JSON. Whichever lands second re-pins on the merged tree.
• axiom: n/a: data-pipeline donor derivation; no policy rule changes.

🤖 Generated with Claude Code

[MaxGhenis]

Merged at reviewed head 7556a00cf as e8db4fa2e, with a merge commit.

Gates at merge time:

• gh pr checks exited 0 on run 38319581442. All 10 checks passed: engine-free, engine-us and engine-uk on 3.13 and 3.14, plus integration-uk, lint, select-countries and wheels.
• Mergeable and not a draft, with no change requests.
• Independent review by GPT-6.1 Sol through Subfleet review/hard. Round 1 at 5dd439b returned APPROVE with one P3, which is fixed. Round 2 at 7556a00cf, this exact head, returned APPROVE with no findings.
• Main moved to 8fd39a21b (Match the SOI congressional-district file by family, at any vintage #1244) after CI started. Those commits touch none of the attested kernel modules and none of the pin files, so the re-pinned identities still hold on the merged tree.
• No production value changes, because the certified default reads the processed partnership_se_income column. No dataset build or release is involved.