diff --git a/PROGRESS.md b/PROGRESS.md index a0394ddf09..d87e1c8064 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,4 +1,193 @@ -# Gate-failed base-pool release lane +# Weeksgate: stacked release gates and integer-week provenance + +## State + +Complete on 2026-08-27. Real-pool provenance has refuted the proposed +post-transfer amount-mapping mechanism: every fractional week is an ACS-origin +non-native clone prediction outside the calibration's clone-0 recipient scope. +The source codec, weeks-gate architecture, source-scope, clone-layout, and +stable-identity repairs are implemented and focused-tested. The complete +release-call roster is classified; six archived-model input assumptions require +owner rulings and are deliberately reported instead of guessed. Repository-wide +Ruff, the CI inventory verifier, and all five full pytest shards pass in their +required independent processes. The completed provenance, audit, verification, +judgment calls, and host-owned checkpoint-rerun consequence are in `out.md`. +No network access, artifact build, publication, push, pool build, or release +build is in scope. + +## Done + +- Read `CLAUDE.md` and the GitNexus debugging workflow. +- Confirmed branch `stacked-release-gate-alignment` is clean at `4f453746`. +- Confirmed the local GitNexus CLI is installed but the repository is not yet + indexed. Its offline analyzer parsed the repository but could not register + the index because the sandbox forbids writes to `~/.gitnexus`; the generated + local index was moved out of the worktree to `/private/tmp`. +- Recorded the four requested workstreams: fractional-week provenance and PUF + misclassification; integer-support calibration repair; stacked/legacy weeks + gate alignment; and the full release-side gate archaeology sweep. +- Recorded the required verification boundary: repository Ruff plus one pytest + process per shard, with no pool/release builds. +- Read the fixed-format HDF5 blocks directly and classified all 369 noninteger + `weeks_unemployed` rows: 360 are ACS clone 1 (355 UC=0, 5 UC>0) and 9 are ACS + clone 2 (all UC=0); all are positive, all 369 values are distinct, and the + exact range is 1.0003521955067698--37.796501228614694. +- Confirmed zero nonintegers on ASEC rows and ACS clone 0. The receipted + calibration covers exactly the 856,626 ACS clone-0 rows, maps 8,419 carrier + amounts onto observed ASEC support with zero donor-support violations, and + records QED 0.5882352941176471 to 0.0. +- Reproduced the 5,218-row false "PUF" classification: the legacy role helper + calls every clone index above zero `puf_tax_detail`, regardless of raw source + channel. The rows are all ACS-origin clones: 4,733 integer clone-1 rows, 355 + fractional clone-1 rows, 121 integer clone-2 rows, and 9 fractional clone-2 + rows with nonzero weeks while UC is nonpositive. +- Traced the actual fractional mechanism to the ACS transfer target codec: + PolicyEngine-US declares `weeks_unemployed` as physical `float`, so the + generic QRF path treats it as continuous even though its reviewed source + contract is integer-supported. The later calibration repairs clone 0 only. +- Bound every final fractional value bit-for-bit to the late-transfer target + bank. Its raw QRF draw contains 711 nonintegers: 342 ACS clone 0, 360 clone 1, + and 9 clone 2. Post-transfer calibration changes 13,417 clone-0 rows and + eliminates all 342 clone-0 fractions; the 369 out-of-scope clone fractions + pass through unchanged. +- Added `weeks_unemployed` to the ACS transfer's authority-bound discrete + numeric target registry. The existing codec now snaps every prediction to + actual observed ASEC donor support, and the execution-contract identity + receipts the policy alongside the two mortgage-year targets. +- Added a focused ACS-transfer regression proving weeks predictions retain + observed integer support and the execution contract declares the target. +- Regenerated the checked-in US imputation authority so + `transfer_execution.discrete_numeric_targets` includes `weeks_unemployed`. + The generator's compile and byte-staleness checks pass at bundle spec SHA + `821d5838da3ac368170e61e017f1a72648f93e8a011aa40e33b8c2b4b14511f3` + at that source-fix step; the later runtime/seed identity repin below + supersedes this intermediate digest. The spec-bundle and imputation- + semantics suites pass. +- Passed the complete ACS-transfer test file (65 tests), the complete + post-transfer calibration receipt-contract file (47 tests), focused Ruff, + and `git diff --check` using the prebuilt `.venv` directly. A task-local + `UV_CACHE_DIR` later allowed the required `uv run --no-sync` commands to run + against that same prebuilt environment without accessing `~/.cache/uv`. +- Modernized the weeks summary/gate to derive its roster from actual assembled + source channels, while retaining the legacy ASEC/PUF role path. The ASEC + source-validity scope, direct native-clone reconciliation scope, and reviewed + UC-constraint scope are now distinct and explicitly receipted in details. +- Kept all four plausibility-band tuples and every numeric threshold unchanged; + non-ASEC assembled channels use the unchanged legacy recipient band. +- Added stacked ASEC+ACS and legacy ASEC+PUF fixtures covering roster detection, + raw-source scoping, native reconciliation, and UC constraint ownership. The + complete weeks file passes (26 passed, 1 skipped) with focused Ruff. +- Replayed the updated gate over the supplied pool's exact live arrays and + weights. It now reports 234,133 valid ASEC source rows, 108,073 exact native + reconciliation rows, 982,686 UC-constrained rows, zero source/UC failures, + both unchanged channel bands passing, and only the genuine 369 nonintegers. +- Added a centralized provenance-owner API that distinguishes validated + physical source channels from legacy clone-operator roles. The weeks gate + now consumes that API rather than reading provenance columns directly, and + both repository source-blindness tripwires pass alongside the stacked and + legacy provenance/weekly-signal suites. +- Made WIC's deterministic draw key prefer the assembly-unique + `person_source_id` on multispine frames, before the source-local raw identity + triple. Distinct ASEC/ACS records can no longer collide when their vintages + align, while all support clones of one assembled person retain the same draw + and the legacy key order remains unchanged. +- Completed the release-tool call-roster audit. It found unambiguous physical + ASEC scoping repairs in SSI take-up, workers' compensation, alimony, + retirement contributions/distributions, and Medicare; clone-2 layout fixes + in Head Start, voluntary filing, and prior-year income; and a stable-key fix + in WIC. Archived ASEC-only predictor assumptions in SSI disability, SCF + wealth, SCF auto loans, and SIPP vehicles require explicit model-owner + rulings and will be reported rather than guessed. +- Reworked the assembled Head Start and voluntary-filing receiver layouts to + key by assembly-unique source ID plus explicit clone index, accept clone 2 + and later, reject duplicate source/clone rows, choose clone 0 (or the lowest + surviving clone) deterministically, and fan one source-level decision to all + clones. Their legacy role-only duplicate contract remains unchanged. +- Replaced occurrence-pair clone diagnostics in voluntary filing and + prior-year income with all-clone grouping on assembled source IDs, so a + clone-2-only divergence is now detected. Focused tests and Ruff passed for + all three module/test pairs. +- Scoped Medicare and retirement release diagnostics to physical ASEC source + rows, separating raw-source validity from native direct-carry reconciliation + where transferred clones intentionally differ. Kept the producer kernels + origin-blind: the authenticated-pool release path skips those producers, and + indirect physical-source routing would violate the repository's population- + operator boundary. Their 60 focused module/source-blindness tests and Ruff + pass after that review correction. +- Scoped alimony and workers' compensation raw validity to every physical ASEC + clone and exact source-carry checks to physical ASEC native rows, leaving all + clone-operator plausibility bands unchanged. Stacked ASEC+ACS and legacy + ASEC+PUF fixtures pass (28 alimony and 21 workers' compensation tests). +- Changed SSI reporter-lineage capture to validate `SSI_VAL` only on physical + ASEC rows and accept null ACS raw-source cells. Assignment remains source- + blind: it consumes the source-ID set captured before L0, or obtains that set + through the reporter helper when no explicit set is supplied. The complete + SSI take-up file passes (71 tests). +- Fixed SSI-disability's non-fatal clone-divergence diagnostic to group every + assembled clone by source person, so clone-2-only divergence is reported. + The existing decision not to make divergence gate-fatal remains unchanged + for an owner ruling; the complete focused file and source-blindness checks + pass. +- Narrowed physical-channel resolution to a gate/reporter-only provenance API, + removed its general runtime/PUF-support re-exports, and added a static exact- + caller contract (including internal mask-helper callers). A future derive, + impute, or wrapper use now fails the source-blindness suite instead of + passing through indirection. +- Replayed the repaired release gates read-only against the supplied pool. + Alimony, Medicare, retirement contributions/distributions, workers' + compensation, and SSI reporter capture pass; the weeks gate now fails only + on the genuine 369 fractional values. Prior-year income remains outside its + unchanged availability band and WIC finds pregnant nonfemale rows, both + genuine data/spec outcomes rather than stacked-layout archaeology. +- Completed an adversarial review of the repaired code and focused tests with + no additional implementation defect found. It confirmed six owner-ruling + items: SSI disability criteria, SCF wealth, SCF auto loans, SIPP vehicles, + SIPP tips, and ORG wages/FLSA all consume ASEC-only archived predictors on a + frame whose 1,736,840 physical ACS rows carry null source cells. ORG is + guaranteed to fail its unchanged race/occupation bands; SIPP tips' unchanged + tipped-occupation band passes while the ACS channel is dead. +- Passed repository-wide Ruff and the CI test-group inventory verifier. The + calibrate, data, fit, and frame shards pass in four independent pytest + processes. +- Corrected the reviewed WIC seed protocol to match the implemented assembled- + multispine key precedence: assembly-unique `person_source_id` first, then the + unchanged legacy raw/support/person fallbacks. An exact seed-grammar test now + binds that order. +- Re-pinned the fail-closed spec-engine proof after adding one authored + transfer-execution field and one resolved seed-protocol field: 42,122 total + fields (32,352 authored and 9,770 resolved), complete exact-pointer claims, + and all 41 inventory checks. Regenerated the committed coverage report and + validated the final US spec SHA + `5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371`. +- Verified in a detached `origin/main` worktree, using the same prebuilt venv, + that the old BE, UK, and minimal-spec golden vectors still pass there. Their + current repins therefore reflect this branch's attested runtime and seed- + protocol changes rather than environment drift. All 102 tests in the eight + directly affected spec-engine files pass; generated-bundle and coverage- + report byte checks, focused Ruff, and `git diff --check` also pass. +- Ran the full build shard after that coherent spec repin. It reached 100% with + exactly one failure and no errors: the multispine constants-adapter fixture + still expected the former live US spec SHA. Updated only that live-binding + expectation to the regenerated final SHA; the separate arbitrary checkpoint + identity fixture remains deliberately unchanged. +- Passed the complete multispine-pool-tool file after that correction and + committed the coherent fixture repin as `12a918ed`. +- Reran the entire build shard from zero in one process: 6,608 tests collected, + 100% reached, and pytest exited 0 with expected skips only. All five full + package shards, repository-wide Ruff, generated-artifact checks, the spec + coverage proof, the CI test inventory, and `git diff --check` are green. +- Wrote the final provenance tables, mechanism verdict, per-file rationale, + exhaustive release-gate audit, owner-ruling list, verification evidence, and + judgment calls to `out.md`. + +## Next + +- Host session: rerun `late_transfer -> simulated -> terminal-gates` from the + candidate checkpoints because the discrete weeks codec changes pool content. +- Review the six archived-model owner rulings in `out.md`; do not reinterpret + their missing ACS predictors through a gate-only threshold/scope change. + +# Historical: gate-failed base-pool release lane ## State diff --git a/changelog.d/stacked-release-gate-alignment.fixed.md b/changelog.d/stacked-release-gate-alignment.fixed.md new file mode 100644 index 0000000000..4fe48f4692 --- /dev/null +++ b/changelog.d/stacked-release-gate-alignment.fixed.md @@ -0,0 +1 @@ +Preserve integer support for transferred ``weeks_unemployed`` values and align US release-side source, clone, and deterministic-identity checks with stacked ASEC+ACS pools while retaining legacy ASEC+PUF behavior and every existing plausibility threshold. diff --git a/docs/evidence/spec-engine/us-f0-coverage.json b/docs/evidence/spec-engine/us-f0-coverage.json index 68f70290fb..c3f6bd0a93 100644 --- a/docs/evidence/spec-engine/us-f0-coverage.json +++ b/docs/evidence/spec-engine/us-f0-coverage.json @@ -6,7 +6,7 @@ "country": "us", "documentation_sha256": "4b39450dbdb8dafb83c3b627123b8026c6f82c660b66fe76f341a67c4f37c77b", "field_usage": { - "authored_normative_field_count": 32351, + "authored_normative_field_count": 32352, "claim_count": 49, "claims": [ { @@ -220,8 +220,8 @@ ], "mode": "legacy_behavior", "pointer_class": "all", - "pointer_count": 79, - "pointer_sha256": "1575513251c72be60edc75f378221fa878ea76e009512f1352a111586d0a60f8", + "pointer_count": 80, + "pointer_sha256": "10bd811f86dae7941035533cfe0d51d81cd632f5995c1adb80234bc7ecbf6565", "rationale": null, "relative_sink_prefix": null, "source_prefix": "/authored/spec~1imputation.yaml/transfer_execution", @@ -736,8 +736,8 @@ "legacy_sinks": [], "mode": "compiler_semantic", "pointer_class": "all", - "pointer_count": 823, - "pointer_sha256": "c05ebb88538195798e9365f325381c14d92b74007f846f1fef50a08268e6d27d", + "pointer_count": 824, + "pointer_sha256": "7537385c3fd399a2dbb7dcd8ed7cf1ff2481ed336db621eafcbfd741d5792f40", "rationale": null, "relative_sink_prefix": null, "source_prefix": "/resolved/seed_protocol", @@ -772,21 +772,21 @@ "verifier": "vintages" } ], - "configuration_field_count": 42120, - "consumed_field_count": 42120, + "configuration_field_count": 42122, + "consumed_field_count": 42122, "generation0_effect_counts": { - "legacy_behavior": 38443, - "no_generation0_effect": 3677 + "legacy_behavior": 38444, + "no_generation0_effect": 3678 }, "mode_counts": { - "compiler_semantic": 27699, + "compiler_semantic": 27700, "front_end_validation": 348, "identity_only": 103, - "legacy_behavior": 13970 + "legacy_behavior": 13971 }, "multiple_primary_use_field_count": 0, - "pointer_inventory_sha256": "bc4a948ab632191954600da8474c5b011f977a65e24c399d126f3dc4a79f23e5", - "resolved_binding_field_count": 9769, + "pointer_inventory_sha256": "ae7881c07c6137037f43a5992b3e8fffbbdbd5ddf2e1b70956986cb9df043f71", + "resolved_binding_field_count": 9770, "unused_field_count": 0 }, "inventory_coverage": { @@ -1361,12 +1361,12 @@ ], "expected": { "producer_count": 38, - "sha256": "3850554eb804fde5e4f86a34ac1bb8a7a07aafff7e8b48396a3d5fca844798e8" + "sha256": "ef491637933945e28645479da70646d9c50366213b6f70477de445ab9e4bb1cf" }, "failures": [], "observed": { "producer_count": 38, - "sha256": "3850554eb804fde5e4f86a34ac1bb8a7a07aafff7e8b48396a3d5fca844798e8" + "sha256": "ef491637933945e28645479da70646d9c50366213b6f70477de445ab9e4bb1cf" }, "status": "covered" }, @@ -1656,13 +1656,13 @@ "compiler_ir.node_slices" ], "expected": { - "map_sha256": "36a9d819ef196c312888591936d49c025b0407df9928440cceefabc5458f72af", - "protocol_sha256": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246" + "map_sha256": "ee1ad8d5196ca48d70657675237294c2dd925a520900d4d476d256a1b514b283", + "protocol_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41" }, "failures": [], "observed": { - "map_sha256": "36a9d819ef196c312888591936d49c025b0407df9928440cceefabc5458f72af", - "protocol_sha256": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246" + "map_sha256": "ee1ad8d5196ca48d70657675237294c2dd925a520900d4d476d256a1b514b283", + "protocol_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41" }, "status": "covered" }, @@ -1677,7 +1677,7 @@ "compiler_ir.seed_stream_map" ], "expected": { - "implementation_sha256": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246", + "implementation_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1698,7 +1698,7 @@ }, "failures": [], "observed": { - "implementation_sha256": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246", + "implementation_sha256": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1736,7 +1736,7 @@ }, "failures": [], "observed": { - "sha256": "99710bf212c3801cfd0434d56d2d1d2e4902a2a958978cdbd91cd9acce81d2fb", + "sha256": "a0366cd518f42a240b589af3ef6580aac44ad7ed70969bcc669b365d315146ab", "sites": 53 }, "status": "covered" @@ -1887,7 +1887,7 @@ "alpha", "zeta" ], - "sha256": "a128a85f877fb32def9382b841b8b340f974e8a9148ac029c1f04becdc956c18" + "sha256": "1fa30ae9213c19607168df3cf0035478b9c96f815e1541cc97f4e1b8e7246935" }, "failures": [], "observed": { @@ -1911,7 +1911,7 @@ "alpha", "zeta" ], - "sha256": "a128a85f877fb32def9382b841b8b340f974e8a9148ac029c1f04becdc956c18" + "sha256": "1fa30ae9213c19607168df3cf0035478b9c96f815e1541cc97f4e1b8e7246935" }, "status": "covered" }, @@ -1975,7 +1975,7 @@ "take_up_contract", "us_qbi_reconciliation_contract" ], - "sha256": "91c65a9ff36839d575036264c4bf57ffde6457e8fd180a0856f9be712ada371d" + "sha256": "001a2bdc700615cd96771fc258abf5f769d67ea41687efd8cc2ad30361619057" }, "status": "covered" }, @@ -2599,7 +2599,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5378bb9189aec96f50da22aac71e5bd2c3d919e9795f6ef2147e0bc9c739dd8e" + "spec_sha256": "5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371" } }, "report_schema_version": 3, @@ -2609,7 +2609,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5378bb9189aec96f50da22aac71e5bd2c3d919e9795f6ef2147e0bc9c739dd8e" + "spec_sha256": "5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371" }, "status": "pass" } diff --git a/out.md b/out.md index df65be1589..6b9d8437a1 100644 --- a/out.md +++ b/out.md @@ -1,233 +1,560 @@ -# Gate-failed base-pool release lane: final report +# Weeksgate: stacked release gates and fractional-weeks final report -Date: 2026-08-26 +Date: 2026-08-27 -Branch: `release-from-gate-failed-pool` +Branch: `stacked-release-gate-alignment` -Base: `origin/main` at `2263df36` +Lane base: `4f453746` (`origin/main` at kickoff; includes #786) ## Outcome -The legacy `--base-h5` release and preflight paths now fail closed when the H5 -identifies as a US multispine pool. They require and authenticate the canonical -sibling pool manifest, bind it to the exact requested H5, and reject a red -terminal agreement battery by default. The new explicit opt-in is: - -`--allow-gate-failed-base-pool` - -The flag is valid only with `--base-h5` and only for an authenticated current -stacked pool with `status=gate_failed` and `simulation_ready=false`. It is -rejected for generic H5 inputs, green pools, and the exact-k manifest arm. A -release built with the flag carries the full authenticated red verdict so a -reader can see `battery: red, N failures` without fetching the pool. - -Publication preflight authenticates the same pool and displays the red receipt -as prominent human-review evidence. The red battery alone does not change the -existing preflight PASS/AT_RISK/FAIL calculation or exit code. Publication -remains a separate human-gated operation. - -No network access, pool build, release build, publication, push, battery -threshold change, or gate-logic change was performed. - -## Diff summary and rationale - -- `CLAUDE.md`: documents the authenticated `--base-h5` boundary, explicit red - opt-in, verdict carriage, strict exact-k arm, and separate publication step. -- `PROGRESS.md`: maintains the required state/done/next journal from kickoff - through the final verified handoff while preserving the prior lane history. -- `changelog.d/gate-failed-pool-release.fixed.md`: records the closed receipt - bypass and explicit opt-in under the repository changelog convention. -- `packages/microcosm-build/src/microcosm/build/us_runtime/h5_io.py`: adds the - shared pool classifier, exact-H5 binding check, release-specific authenticated - loader, and normalized self-contained release receipt. The existing strict - simulation-ready and scoring loader contracts were not weakened. -- `packages/microcosm-build/src/microcosm/build/us_runtime/release_gate_preflight.py`: - authenticates identified base pools through the shared seam, records their - receipt in `PreflightReport`, and renders authenticated red evidence before - the ordinary check table without including it in exit-code calculation. -- `tools/build_us_fiscal_refresh_release.py`: adds - `--allow-gate-failed-base-pool`, authenticates pool-like legacy base H5s before - the generic loader can run, uses the authenticated frame/H5 identity, and - passes the receipt into both generated manifests. The exact-k arm remains - simulation-ready-only. -- `tools/preflight_us_release_gates.py`: adds the same explicit opt-in and an - optional `--release-manifest`; validates the carried full verdict, requires - its receipt to exactly equal the pool authenticated from `--base-h5`, emits a - large red/human-review banner, and adds machine-readable carried evidence. - Existing required `--base-h5` and `--selection-source-manifest` arguments and - all other preflight behavior remain intact. -- `packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py`: exercises - real stamped H5 classification, strict/default refusal, explicit red - acceptance, redundant-green refusal, receipt contents, identity binding, and - malformed aggregate/nested verdict refusal. PyTables-dependent cases use the - sibling-test `importorskip` idiom. -- `packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py`: covers the - old bare-H5 bypass directly, red-manifest default refusal, flag routing, - generic/non-pool refusal when the flag is present, both-manifest carriage, - exact-k isolation, and AST-level ordinary base-pool wiring guards. -- `packages/microcosm-build/tests/test_us_release_gate_preflight.py`: covers - bare/missing/red receipt authentication, prominent text and JSON carriage, - nonblocking PASS/AT_RISK/FAIL exit semantics, required static inputs, and - exact mismatch refusal across manifest/H5/publication/gates/full-verdict - identities. -- `out.md`: this final implementation, audit, verification, and handoff report. - -## Flag and manifest contract - -The exact flag is `--allow-gate-failed-base-pool`. The release builder permits -it only on its legacy `--base-h5` arm. Publication preflight uses the same flag -to authenticate and inspect the same red pool; neither invocation is an -authorization to publish. - -The build manifest stores the receipt at top-level `base_pool`. The release -manifest stores the same receipt at `build.base_pool`: - -- `artifact_kind` -- `status` -- `simulation_ready` -- `manifest_sha256` -- `publication_run_id` -- `pool_h5_sha256` -- `pool_h5_size_bytes` -- `allow_gate_failed_base_pool` -- `agreement_gate_reference.battery_status` -- `agreement_gate_reference.passed` -- `agreement_gate_reference.gates_json_sha256` -- `agreement_gate_reference.failure_count` -- `agreement_gate_reference.failures`, a flattened list of `{gate, message}` -- `agreement_gate_reference.verdict`, the complete agreement-gate verdict - -The receipt accepts only coherent pairs: simulation-ready/green/passed or -gate-failed/red/failed. Every nested gate must carry a boolean `passed` and a -coherent failure list; the nested aggregate must match the terminal status. -For a red pool the failure list must be nonempty. SHA-256, publication run ID, -H5 size, full verdict, and all other receipt fields are bound through exact -preflight equality. - -The preflight report also exposes the authenticated receipt as `base_pool`. -When `--release-manifest` supplies a red carried receipt, JSON adds -`carried_base_pool_agreement_battery`, including `battery_status=red`, the -failure count/list, gates digest, full agreement reference, -`publication_decision=human_review_required`, and `affects_exit_code=false`. - -## §4 consumer audit - -The requested exhaustive source audit used -`simulation_ready|gate_failed|load_simulation_ready` across `tools/` and -`packages/`, followed by source tracing of every non-test match. - -- `h5_io.py` owns the authentication boundary. The new release wrapper selects - the unchanged strict loader by default and the existing private terminal - loader only when its required explicit boolean is true. The public scoring - and strict simulation-ready contracts remain unchanged. -- `tools/build_us_exact_k_ladder_release.py` remains deliberately strict: it - uses the simulation-ready loader, validates config/release identities, checks - `agreement_gate.passed`, and never forwards the new flag. -- `tools/score_us_release_head_to_head.py` retains its existing authenticated - terminal-evidence exception. It requires a pool manifest and terminal gates; - it cannot authorize a release or accept a naked pool H5. -- `tools/build_us_multispine_pool.py` and - `us_runtime/multispine_pool.py` are producers of the status, readiness, H5 - stamp, manifest, diagnostics, and terminal verdict rather than downstream - release consumers. -- `tools/_legacy/build_us_acs_multispine_base.py` writes its own pre-calibration - `simulation_ready=false` state; it is not a current terminal-pool consumer. -- `tools/build_us_acs_local_release.py` uses its own calibrated-release - readiness summary. A derivative keeps donor release/revision provenance but - does not project the nested `build.base_pool` verdict into its own manifest. -- The microcosm-data release contract, loader, TRACE conversion, and publisher - tolerate but otherwise ignore the additive `build.base_pool` object. The - designated publication preflight is therefore the prominent human-facing - red-verdict surface. - -Report-only interactions left unchanged, as directed: - -1. `tools/build_us_multispine_pool.py::_stacked_manifest_payload` still labels - `calibration.consumer` as `k-ladder` and says - `requires_manifest_simulation_ready=true`. That metadata is now incomplete - for the explicit legacy base-H5 red-pool route, but it is informational and - unenforced. -2. A red pool producer writes its H5, manifest, gates evidence, and failed - Logbook row, then returns status 1. Existing `set -e` candidate chains stop - there, so an operator must deliberately start the separate release command - to use the new opt-in. -3. ACS-local derivative manifests retain fetchable donor provenance but do not - self-contain the nested red receipt. -4. Generic data/release consumers accept the additive receipt but do not - surface it. This is not a publication bypass because the publication tool's - designated preflight now authenticates and displays it. - -No audit-only consumer was modified. +The 369 fractional `weeks_unemployed` values are genuine pool-content +defects, but the proposed post-transfer-calibration mechanism is refuted. They +were emitted earlier by the ACS-transfer QRF because PolicyEngine-US physically +types this integer-supported variable as `float`. The calibration maps ACS +clone 0 onto actual observed ASEC support and removes all 342 fractions in that +scope; it never touches the 360 ACS clone-1 and 9 ACS clone-2 fractions that +survive into the pool. + +The source fix registers `weeks_unemployed` as a discrete numeric ACS-transfer +target. Predictions now snap deterministically to actual donor support, with +lower-support tie breaking, and are returned on integer support. The transfer +execution contract and generated imputation authority receipt the policy. +The strict post-transfer calibration receipt and validator remain unchanged +because that kernel did not cause the defect; their complete contract suite +still passes. + +The weeks release gate now distinguishes physical source channel from +clone-operator role. It derives an assembled roster (`asec+acs`) or retains a +legacy roster (`asec+puf_tax_detail`), validates raw `LKWEEKS` only where an +ASEC source exists, reconciles direct carries only on native ASEC rows, and +checks the UC rule only on rows owned by that constraint. Every plausibility +band and numeric threshold is unchanged. + +The full release-side sweep fixed all unambiguous physical-source, +clone-layout, and stable-identity archaeology. Six archived-model predictor +contracts remain owner decisions and are reported precisely below. In +particular, ORG wages/FLSA is guaranteed to fail on the stacked pool, while +SIPP tips' global tipped-occupation component passes despite a dead ACS +channel. Those are not safe gate-only edits. + +No network access, pool build, release build, artifact publication, push, +plausibility-band change, or by-origin battery change was performed. Issue +#782's weeks-incidence-band adjudication was not touched. + +## Task 1: provenance of all 369 noninteger weeks + +### Evidence read + +The analysis read the supplied fixed-format pool directly: + +`/Users/maxghenis/PolicyEngine/_buildo-runtime/out/candidate-25/pool/pool.h5` + +It contains 1,970,973 person rows: 234,133 physical ASEC rows and 1,736,840 +physical ACS rows. It also read the exact pre-calibration late-transfer target +bank: + +`/Users/maxghenis/PolicyEngine/_buildo-runtime/out/candidate-25/pool/checkpoints/stacked/3847d30e1488ae18891780a7a5de6a1b75d1329f2491796e35639fc9e802c26e/acs-transfer/0914d1b858dff5171f7743f0e535349be4459fb99b3386bbc2e3be72d40dd5fc/late_producer_dag/person/source_operator_weeks_unemployed/targets/000__weeks_unemployed.h5` + +The raw-draw value SHA-256 is +`6ea20782bcb53fa730bb7e7045c7d79ab096dfe69bc5ceb540158d93f6672db4`. +The final fractional-row-index SHA-256 is +`3431befe89f007c353296b0c59792bc31e45a1031e62a6f9c99e003df286305f`; +the sorted 369-value SHA-256 is +`314ba4459bb57169eaf492dd28c8ea04af460104b962bb29285104e70eee976c`. + +### Counts by physical channel, clone, and UC + +| Physical channel | Clone index | All rows | Fractional, UC=0 | Fractional, UC>0 | +|---|---:|---:|---:|---:| +| ASEC | 0 | 108,073 | 0 | 0 | +| ASEC | 1 | 108,073 | 0 | 0 | +| ASEC | 2 | 17,987 | 0 | 0 | +| ACS | 0 | 856,626 | 0 | 0 | +| ACS | 1 | 856,626 | 355 | 5 | +| ACS | 2 | 23,588 | 9 | 0 | +| **Total** | | **1,970,973** | **364** | **5** | + +All 369 values are positive, have null ASEC `LKWEEKS`, and are distinct at +their exact IEEE-754 representation. Every fractional row maps one-to-one by +`person_source_id` to a unique ACS clone-0 sibling, and +`unemployment_compensation` is identical across that sibling group. Of the +364 UC-zero rows, the clone-0 sibling has zero weeks. The five UC-positive +rows have clone-0 integer weeks in `{2, 4, 40, 48, 50}`. + +The five fractional value / annual-UC pairs are: + +| Fractional weeks | Unemployment compensation | +|---:|---:| +| 3.3297787140375217 | 10,800 | +| 3.948788299342264 | 20,000 | +| 17.23699569220113 | 7,200 | +| 18.64447048256536 | 10,800 | +| 21.97077954067273 | 4,400 | + +### Value distribution + +| Statistic | Value | +|---|---:| +| Distinct values | 369 | +| Minimum | 1.0003521955067698 | +| p10 | 2.306610095577396 | +| p25 | 6.226837695984924 | +| Median | 12.319331262236448 | +| Mean | 14.643462963265243 | +| p75 | 23.618846787450813 | +| p90 | 27.706623369972394 | +| Maximum | 37.796501228614694 | + +| Weeks interval | All | ACS clone 1, UC=0 | ACS clone 1, UC>0 | ACS clone 2, UC=0 | +|---|---:|---:|---:|---:| +| [1, 5) | 74 | 70 | 2 | 2 | +| [5, 10) | 85 | 84 | 0 | 1 | +| [10, 15) | 45 | 44 | 0 | 1 | +| [15, 20) | 40 | 36 | 2 | 2 | +| [20, 25) | 60 | 58 | 1 | 1 | +| [25, 30) | 39 | 39 | 0 | 0 | +| [30, 35) | 16 | 15 | 0 | 1 | +| [35, 40) | 10 | 9 | 0 | 1 | + +The nine clone-2 values are +`1.3112775468533615`, `3.155651627690991`, +`8.163243626548894`, `13.03279351472897`, +`15.40528737169489`, `15.477036717133474`, +`24.60208367015022`, `31.658750512333974`, and +`37.10384792802646`. + +### Mechanism verdict: calibration hypothesis refuted + +The late-transfer target bank contains 711 fractional predictions: + +| ACS clone | Fractional before calibration | Fractional in final pool | Rows changed by calibration | +|---:|---:|---:|---:| +| 0 | 342 | 0 | 13,417 | +| 1 | 360 | 360 | 0 | +| 2 | 9 | 9 | 0 | + +Every surviving fractional bit is identical between the target bank and final +pool. The stacked calibration explicitly defines ASEC clone 0 as reference and +ACS clone 0 as recipient in +`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:8975-8985`. +Its amount mapping selects values from the sorted observed donor array rather +than interpolating in +`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:577-680`. + +The pool receipt confirms the calibration ran: + +- 8,419 allowed-carrier/addition-candidate rows within 856,626 mutable ACS + clone-0 rows; +- 13,417 changed clone-0 rows, including 4,998 cleared and 8,290 added; +- `capacity_limited=true`; +- 8,419 mapped positive amounts; +- reference quantiles `[2, 6, 12, 26, 36]`; +- recipient quantiles `[2, 8, 22, 24, 32]` before and + `[2, 6, 12, 26, 36]` after; +- amount QED `0.5882352941176471 -> 0.0`; +- zero donor-support violations. + +Weeks' declared post-transfer calibration spec is at +`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:237-263`. +The actual cause precedes calibration. Before this branch, +`_target_encoding` treated numeric `weeks_unemployed` as continuous because +the engine metadata says `float`; a quantile-regression-forest prediction may +therefore interpolate between integer observations. The relevant encoding and +decoding seam is +`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:3037-3120` +and `:3240-3270`. Calibration then repaired only clone 0 by design. + +### Why the gate said “5,218 PUF rows” + +There is no physical PUF channel in this pool. The compatibility helper +`support_role_series` deliberately labels clone 0 as the ASEC-compatible +operator role and every clone above zero as the PUF-tax-detail operator role, +independent of physical source. See +`packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py:390-489`. +The old summary confused that operator role with physical channel. + +The 5,218 rows are all ACS-origin, UC-zero, positive-week non-native clones: + +| Actual rows | Integer weeks | Fractional weeks | Total | +|---|---:|---:|---:| +| ACS clone 1 | 4,733 | 355 | 5,088 | +| ACS clone 2 | 121 | 9 | 130 | +| **Total** | **4,854** | **364** | **5,218** | + +The UC-zero constraint is not defined for those ACS non-native QRF clones. +The updated gate preserves the compatibility role for operator plausibility +but uses physical source and clone provenance to own source/UC checks. + +## Task 2: source fix and receipt contract + +`weeks_unemployed` is now in `_DISCRETE_NUMERIC_TARGETS` at +`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:133-141`. +The existing discrete codec: + +1. rejects nonintegral donor support; +2. snaps finite predictions to the nearest value in actual observed support; +3. resolves exact-distance ties to the lower support value; +4. emits integer/nullable-integer output. + +The snap is at +`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:3327-3339`. +The sorted discrete-target roster is part of the transfer execution identity at +`:271`; the generated authority records it at +`packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml:408-411`. +The final regenerated bundle spec SHA-256 is +`5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371`. +This invalidates affected target-bank identities instead of silently changing +content under an old receipt. + +The release-gate and WIC edits also change exact source bytes attested by the +shared legacy seed protocol. Its reviewed WIC grammar now explicitly records +assembled `person_source_id` before the unchanged legacy fallbacks at +`packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py:720-737`. +The fail-closed spec proof was coherently re-pinned to 42,122/42,122 claimed +fields (32,352 authored and 9,770 resolved) and 41/41 inventory checks. The +committed coverage report, producer-semantics digest, seed protocol/map, +checkpoint identity, pointer inventory, and affected country-envelope goldens +all move together. A detached `origin/main` comparison confirmed that the +BE/UK/minimal-envelope changes come from this branch's attested source and seed +grammar, not environment drift. + +The regression at +`packages/microcosm-build/tests/test_us_acs_transfer.py:1391-1430` sends a +fraction-producing mean QRF through the real target codec, verifies that output +lies on observed integer donor support, and verifies that the execution +contract declares `weeks_unemployed`. + +Because the hypothesis was refuted, no change was made to +`post_transfer_calibration.py` or its strict validator. Its observed-support +amount mapping, before/after margin fidelity, hashes, anchors, capacity status, +and donor-support validation remain mandatory; the complete 47-test receipt +contract passes. + +**Artifact consequence:** pool content changes. The host session must rerun +`late_transfer -> simulated -> terminal-gates` from checkpoints for the +candidate pool. This lane did not run any pool or release build. + +## Task 3: stacked-aware weeks release gate + +The gate now uses the narrow, read-only physical-source accessor +`support_gate_source_channel_series` at +`packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py:496-524`. +That accessor is restricted to reviewed gates/reporters by the static +source-blindness contract. + +| Frame layout | Plausibility roster | Raw LKWEEKS validity | Exact LKWEEKS carry | UC-zero consistency | +|---|---|---|---|---| +| No provenance | ASEC | all rows | all rows | none, matching legacy behavior | +| Legacy ASEC+PUF roles | ASEC + PUF | ASEC role | ASEC role | PUF role | +| Assembled ASEC+ACS | actual physical channels | every physical ASEC clone | physical ASEC clone 0 | physical ASEC non-native clones + non-ASEC clone 0 | + +The assembled scope logic is at +`packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py:1320-1380`; +summary and failure construction are at `:1204-1317` and `:1383-1435`. +The ASEC plausibility bands and the legacy recipient/PUF bands remain +byte-identical, and every non-ASEC physical channel uses the unchanged +recipient band. + +A read-only replay on the supplied pool reports: + +- roster `asec, acs`; +- 234,133 raw ASEC source rows, zero invalid; +- 108,073 direct native reconciliation rows, zero mismatches; +- 982,686 UC-constrained rows, zero mismatches; +- ASEC positive share `0.01956717285126748`, weighted mean + `0.32416674108944377`, weighted weeks `52,743,981.18128455`; +- ACS positive share `0.00800370929739283`, weighted mean + `0.1376988711272782`, weighted weeks `21,987,101.082573153`; +- the unchanged channel bands pass; +- the gate fails only on the genuine 369 noninteger values. + +Stacked ASEC+ACS and legacy ASEC+PUF fixtures cover roster derivation, raw +source scoping, native reconciliation, and UC ownership. The complete focused +weeks file passed: 26 passed, 1 skipped. + +## Task 4: complete release-side archaeology audit + +The audit started from every release call in +`tools/build_us_fiscal_refresh_release.py:8690-11726`, traced each gate and +its immediately preceding producer/wrapper, and classified physical-source, +operator-role, clone-layout, entity-layout, and raw-column assumptions. The +table groups gates only where they share the same architectural conclusion; +every release gate in the roster is named. + +| Gate/stage | Stacked assumption found | Disposition | +|---|---|---| +| Validation input coverage; register consistency; release target parity; target-profile coverage; base-population scale | Manifest, registry, target, or aggregate-weight contracts; no legacy per-row source roster | Already compatible; no change | +| PUF capital-gains-tail presence and post-selection preservation | Authenticated tail-column presence/preservation checks, not a demand for a physical PUF channel (`build_us_fiscal_refresh_release.py:8837-8841,8937-8941`) | Already compatible; no change | +| Exact-k PUF tail support; exact-k frozen-register fit; fiscal-target materialization/skip/zero-support; critical/SOI fit | Calibrated diagnostics and authenticated exact-k inputs, not physical source labels | Already compatible; no change | +| Weeks input and post-selection weeks input | Used clone operator roles as physical ASEC/PUF channels and applied raw/UC checks to the wrong rows | **Fixed** with physical roster and separate validity/reconciliation/UC scopes | +| QBI input | Already has an explicit stacked ACS path | Already compatible; no change | +| Workers' compensation; alimony; Medicare; retirement contributions; retirement distributions | Raw ASEC-only columns were validated/reconciled through legacy operator roles | **Fixed**: all physical ASEC clones own raw validity; native physical ASEC rows own exact direct-carry reconciliation where applicable | +| SSI reporter capture/take-up | Strict `SSI_VAL` validation treated null ACS raw-source cells as invalid | **Fixed**: capture reporters from physical ASEC; assignment remains source-blind and consumes the captured ID set | +| SIPP Head Start; voluntary filing | Assumed exactly an occurrence pair / two clones | **Fixed**: explicit assembly clone index, arbitrary clone count, duplicate source+clone refusal, clone-0-or-lowest canonical decision, fanout to every clone | +| Prior-year income clone diagnostic | Compared only the first two occurrences | **Fixed**: group all assembled clones; existing availability band unchanged | +| SSI disability clone-divergence diagnostic | Compared only a pair and missed clone-2-only divergence | **Fixed** for all assembled clones; whether divergence becomes fatal still requires owner ruling | +| WIC deterministic draw | Raw source-local identity could collide across physical ASEC and ACS | **Fixed**: assembled frames prefer unique `person_source_id`; clones remain draw-stable and legacy precedence is unchanged | +| Farm business; domestic production; child support; disability benefits; educator expenses; Form 4952; SALT refund; capital-gain details; energy subsidy; housing; other health insurance | These use clone-operator roles for reviewed producer/plausibility semantics, not as claims about physical source | Intentionally unchanged; stacked-compatible role diagnostics | +| Childcare; casualty loss; miscellaneous itemized; immigration; generic take-up; hours; SNAP take-up; relationship; eligibility; education; pregnancy; reported-coverage vintage; SNAP discretionary exemption | Output/input signal checks do not assume a physical PUF channel or require raw ASEC cells on all rows | Already compatible; no change | +| Local health input; Medicaid take-up; SNAP state take-up; SSI final/delivery | Output or diagnostics-based checks | Already compatible after SSI reporter fix | +| Input-mass reference; degenerate input; eCPS parity; release input coverage; export input mass; QRF tail concentration; reform-coverage smoke; source coverage | Export-wide schema/mass/tail/simulation contracts, not legacy source rosters | Already compatible; no change | +| Export count-calibrated take-up staleness | Export consistency against recorded count-calibration receipts (`build_us_fiscal_refresh_release.py:11702-11726`), independent of physical source roster | Already compatible; no change | +| SCF wealth | Archived predictors silently map ACS-null CPS race/Hispanic source cells to `Other` | **Owner ruling required**, detailed below | +| SSI disability criteria | Archived receiver strictly requires six ASEC disability predictors and `SSI_VAL` on every row | **Owner ruling required**, detailed below | +| SCF auto loans | Imports the SCF wealth CPS-race mapping, silently mapping ACS recipients to `Other` | **Owner ruling required**, detailed below | +| SIPP vehicles | Missing ACS `SPM_TENMORTSTATUS` defaults every ACS recipient to tenant code 3 | **Owner ruling required**, detailed below | +| SIPP tips | Missing ACS `PEIOOCC` defaults to non-tipped; global band can conceal a dead ACS channel | **Owner ruling required**, detailed below | +| ORG wages/FLSA | Missing ACS race/ethnicity/occupation defaults to zero; unchanged global gates are guaranteed to fail | **Owner ruling required**, detailed below | + +### Unambiguous repairs + +- `support_provenance.py:331-354,496-524` centralizes detection of assembled + metadata and physical channels. General derive/imputation code cannot call + the physical accessor: `test_us_spine_blindness.py` pins the exact reviewed + call graph. +- `alimony.py:282-327` and `workers_compensation.py:560-604` split physical + source validity from direct-carry reconciliation while preserving all role + bands. +- `medicare_take_up.py:134-143,248-270`, + `retirement_contributions.py:172-199,602-628`, and + `retirement_distributions.py:274-301,709-739` scope release diagnostics + without routing their population operators by physical origin. +- `ssi_take_up.py:515-550` captures reporters from physical ASEC and accepts + ACS nulls; the assignment consumes captured lineage rather than reading + origin. +- `sipp_head_start.py:579-621` and + `voluntary_filing.py:749-783` use explicit clone indices on assembled + frames. `prior_year_income.py:744-765` and + `ssi_disability_criteria.py:1119-1152` inspect all clones. +- `wic_claim.py:337-376` prefers the assembly-unique identity only when + assembled metadata exists. + +### Owner rulings required + +1. **SSI disability criteria.** Release call + `tools/build_us_fiscal_refresh_release.py:9840-9847`. The receiver + requires and strictly evaluates all six `PEDIS*` predictors on every row + at + `packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py:755-859`, + then strictly evaluates `SSI_VAL` at the same file's `:1023-1034`; the + support-role prediction loop is `:965-1005`. Each input is finite on all + 234,133 physical ASEC rows and null on all 1,736,840 ACS rows, so the stage + fails before its gate. + Required decision: transfer/map canonical ACS disability inputs and + reporter-anchor semantics, or revise/retrain the archived recipient model. + +2. **SCF wealth.** Release call + `tools/build_us_fiscal_refresh_release.py:9798-9808`. + `packages/microcosm-build/src/microcosm/build/us_runtime/scf_wealth.py:654-667` + requires `PRDTRACE`, `PRDTHSP`, `A_MARITL`, `PEPAR1/2`, `PH_SEQ`, and + `A_LINENO`; `:670-691` selects the archived reference person, and + `:694-740` builds race/marriage/children predictors. Only + `PRDTRACE/PRDTHSP` are ACS-null; `:611-620` maps those to zero/`Other`. + The other structural fields are finite and mapped on every ACS row. All + three asset leaves are all-null and `net_worth` is absent, so the wrapper's + recompute branch at `:1109-1159` is certain and every ACS recipient is + silently forced to `Other`. Required decision: authoritative ACS + race/Hispanic mapping or transfer, or a reviewed fallback/retrained model. + +3. **SCF auto loans.** Release call + `tools/build_us_fiscal_refresh_release.py:9990-9996`. + `packages/microcosm-build/src/microcosm/build/us_runtime/scf_auto_loans.py:125-137` + requires the archived layout; `:320-362` selects the reference person and + `:365-417` supplies race/marriage/children inputs. `A_LINENO` is required + by the selector's presence check but is not subsequently used; the mapped + structural fields are finite. The actual defect is the imported SCF-wealth + CPS-race mapping of ACS-null `PRDTRACE/PRDTHSP` to `Other`. All three + auto-loan outputs are absent, so `:475-513` certainly takes the QRF path. + Required decision: the same authoritative mapping or reviewed model change + as SCF wealth. + +4. **SIPP vehicles.** Release call + `tools/build_us_fiscal_refresh_release.py:10026-10033`. + `packages/microcosm-build/src/microcosm/build/us_runtime/sipp_vehicles.py:150-157` + declares the archived layout; `:577-622` reads person + `SPM_TENMORTSTATUS` and fills missing values to tenant code 3, while + `:649-763` uses marriage, `A_LINENO` reference selection, and tenure. + The pool's mapped `A_LINENO` is finite, but person + `SPM_TENMORTSTATUS` is ACS-null and the household field is absent. Both + vehicle outputs are absent, so `:872-917` certainly recomputes and + classifies ACS tenure as tenant. Required decision: canonical ACS tenure + mapping/transfer and whether assembled `A_LINENO` preserves the archived + reference-person semantics, or model revision/retraining. + +5. **SIPP tips.** Release call + `tools/build_us_fiscal_refresh_release.py:10103-10109`. + `packages/microcosm-build/src/microcosm/build/us_runtime/sipp_tips.py:249-263` + maps null `PEIOOCC` to tipped code 0; recipient use is `:363-396`, wrapper + branching is `:450-473`, and the gate is `:484-549`. The pool lacks both + tip output columns, so the wrapper takes its imputation branch. + `PEIOOCC` is ASEC-finite and ACS-null. The ASEC-conditional tipped-code + share is `0.0700204513`; ACS is exactly zero; the global share is + nevertheless `0.0353393976`, inside the unchanged `[0.02, 0.15]` band. + This proves that gate component conceals the dead ACS channel, not that the + separate tip-income component necessarily passes. Required decision: + map/transfer ACS occupation or select a reviewed alternate model, then add + channel-aware diagnostics once those semantics are owned. + +6. **ORG wages/FLSA.** Release call + `tools/build_us_fiscal_refresh_release.py:10144-10150`. + `packages/microcosm-build/src/microcosm/build/us_runtime/org_wages.py:535-559` + fills null `PRDTRACE/PRDTHSP/POCCU2` to zero; these enter features at + `:581-625`; wrapper `:901-945` always recomputes; gate `:959-1034`. All + three raw fields are ASEC-finite and ACS-null. On real weights, CPS-race + nonzero share is `0.5047010838` versus unchanged minimum `0.95`, and + detailed-occupation nonzero share is `0.4155259611` versus minimum `0.65`. + The next release is guaranteed to fail here. Required decision: + authoritative ACS race/ethnicity/occupation mapping or a revised model + contract, not threshold weakening. + +### Read-only real-pool replay after the fixes + +The repaired alimony, Medicare, retirement-contribution, +retirement-distribution, workers'-compensation, and SSI-reporter checks pass. +Their physical ASEC source count is 234,133; direct-carry gates use 108,073 +native ASEC rows where applicable. The weeks gate fails only on 369 fractional +values. + +Two other observed failures are genuine data/spec outcomes, not row-label +archaeology: + +- prior-year-income weighted availability is `0.042839`, outside the + unchanged `[0.05, 0.50]` band; +- WIC finds `is_pregnant=true` on nonfemale rows, with example row positions + `129405, 167076, 171133, 192443, 195546`. + +No threshold was changed to hide either outcome. + +## Diff summary by file + +| File | Rationale | +|---|---| +| `PROGRESS.md` | Maintains the required state/done/next journal from kickoff through verified handoff while preserving historical lanes | +| `changelog.d/stacked-release-gate-alignment.fixed.md` | Records the user-visible integer-support and stacked-gate fix under repository convention | +| `out.md` | This provenance, audit, verification, and handoff report | +| `us/spec/imputation.yaml` | Generated authority now receipts `weeks_unemployed` in the discrete numeric transfer roster | +| `us_runtime/acs_transfer.py` | Declares weeks integer-supported through the existing deterministic observed-support codec | +| `us_runtime/support_provenance.py` | Adds assembled-metadata detection and a narrow read-only physical-source accessor | +| `us_runtime/__init__.py`, `us_runtime/puf_support.py` | Re-export assembled-layout detection while keeping the physical accessor confined to its owner | +| `us_runtime/weeks_unemployed.py` | Derives stacked/legacy rosters and separates source validity, native reconciliation, and UC scopes | +| `us_runtime/alimony.py`, `us_runtime/workers_compensation.py` | Scope raw-source and exact-carry checks to their physical/native owners | +| `us_runtime/medicare_take_up.py`, `us_runtime/retirement_contributions.py`, `us_runtime/retirement_distributions.py` | Make release summaries source-aware while retaining source-blind producer kernels | +| `us_runtime/ssi_take_up.py` | Captures SSI reporters from physical ASEC and permits null ACS source cells without origin-routing assignment | +| `us_runtime/sipp_head_start.py`, `us_runtime/voluntary_filing.py` | Replace pair/occurrence layouts with explicit arbitrary-clone assembled layouts | +| `us_runtime/prior_year_income.py`, `us_runtime/ssi_disability_criteria.py` | Detect divergence across all assembled clones | +| `us_runtime/wic_claim.py` | Uses assembly-unique person identity for cross-origin deterministic draws | +| `spec_engine/seeds.py` | Receipts assembled `person_source_id` as WIC's first seed-key source | +| `spec_engine/field_usage.py`, `spec_engine/inventory_coverage.py`, `tools/spec_engine_coverage.py` | Re-pin the exact two-field expansion and downstream producer/seed/checkpoint/pointer identities | +| `docs/evidence/spec-engine/us-f0-coverage.json` | Regenerates the closed 42,122-field, 41-item coverage attestation | +| `tests/test_us_acs_transfer.py` | Proves fractional QRF output is snapped to observed integer weeks support and receipted | +| `tests/test_us_weeks_unemployed.py` | Covers stacked ASEC+ACS and legacy ASEC+PUF roster/scope contracts | +| `tests/test_us_alimony.py`, `test_us_workers_compensation.py`, `test_us_medicare_take_up.py`, `test_us_retirement_contributions.py`, `test_us_retirement_distributions.py`, `test_us_ssi_take_up.py` | Cover physical ASEC source validity, native reconciliation, legacy behavior, and SSI reporter lineage | +| `tests/test_us_sipp_head_start.py`, `test_us_voluntary_filing.py`, `test_us_prior_year_income.py`, `test_us_ssi_disability_criteria.py` | Cover clone 2+, duplicate clone refusal, deterministic canonical rows/fanout, and all-clone divergence | +| `tests/test_us_wic_claim.py` | Covers cross-origin identity collision avoidance and clone-stable draws | +| `tests/test_us_multispine_puf_clone.py`, `test_us_spine_blindness.py` | Cover assembled provenance and pin physical-source access to the reviewed gate/reporter call graph | +| `tests/test_spec_engine_seeds.py` | Pins WIC's assembled-first seed grammar | +| `tests/test_spec_engine_field_usage.py`, `test_spec_engine_coverage_tool.py`, `test_spec_engine_country_bundles.py`, `test_spec_engine_loader.py` | Pin the reviewed field totals, report identities, and source-attested envelope goldens | +| `tests/test_us_multispine_pool_tool.py` | Re-pins the constants adapter's live US spec identity after regeneration while retaining the separate arbitrary checkpoint identity fixture | + +All source paths in the table are under +`packages/microcosm-build/src/microcosm/build/`; all abbreviated test paths +are under `packages/microcosm-build/tests/`. ## Judgment calls -- Pool detection uses either a canonical sibling manifest whose artifact kind - is the pool kind or the H5's own artifact-metadata stamp. Either positive - identity requires the sidecar; sidecar existence alone is never trusted. -- A dedicated release loader with a required - `allow_terminal_gate_failure` argument keeps the strict and scoring APIs - semantically stable. -- The opt-in is rejected for a green pool rather than silently accepted, so - every recorded flag use has one unambiguous meaning. -- Preflight requires exact equality between the release-carried receipt and the - freshly authenticated base receipt. This prevents displaying release A's red - evidence while checking release B's pool. -- The earlier salvage branch was used as a reference and selected commits were - replayed only after line-by-line review. The final tree corrected its - manifest-only preflight expansion, loader naming/contract ambiguity, - redundant-green behavior, receipt binding, nested-verdict coherence, and an - inaccurate exact-k diagnostic. Full verification ran on the corrected tree. +- The post-transfer calibration was not modified merely because its receipt is + visible near the symptom. Checkpoint bits prove the surviving values predate + it, and its mapping already uses actual donor support. +- The integer contract is explicit by target rather than inferred from the + current PolicyEngine physical dtype. That dtype is `float` and caused the + bug; reviewed domain semantics are the authority. +- Snapping to observed donor support was preferred over generic rounding. It + cannot invent an unsupported week count, is deterministic, and reuses an + existing receipted codec. +- Physical source identity is exposed only to read-only gates/reporters. + Medicare and retirement producer changes considered during review were + reverted: routing population treatment by origin would violate the + source-blind operator boundary. The authenticated release path consumes + already-produced pool surfaces. +- Raw validity covers all physical ASEC clones because they carry the raw + source. Exact reconciliation covers only native ASEC where transferred + non-native clones are intentionally allowed to differ. +- Operator roles remain appropriate for producer/plausibility bands. They are + not aliases for physical ASEC/ACS source, which was the old weeks bug. +- Clone divergence in SSI disability remains diagnostic rather than newly + fatal; changing that policy needs an owner ruling. +- The six archived-model cases were not “fixed” with zero fills, gate + rescoping, or weaker thresholds. Each requires substantive decisions about + ACS predictors and model semantics. ## Verification evidence -All commands ran offline against the pre-built environment with -`uv run --no-sync` (and a task-local UV cache where required). Each pytest +All commands ran offline against the prebuilt environment, with +`UV_CACHE_DIR=/private/tmp/microcosm-weeksgate-uv-cache` so the required +`uv run --no-sync` command could operate inside the sandbox. Each pytest shard ran in one independent process. - `uv run --no-sync ruff check .`: PASS, `All checks passed!` -- `uv run --no-sync pytest -q packages/microcosm-build/tests`: PASS, 6,545 - passed, 45 skipped, 2,351 warnings in 3,326.74 seconds. -- `uv run --no-sync pytest -q packages/microcosm-calibrate/tests`: PASS, 203 - passed, 2 warnings in 11.39 seconds. -- `uv run --no-sync pytest -q packages/microcosm-data/tests`: PASS, 318 passed, - 2 skipped in 9.11 seconds. -- `uv run --no-sync pytest -q packages/microcosm-fit/tests`: PASS, 93 passed, - 1 warning in 30.27 seconds. -- `uv run --no-sync pytest -q packages/microcosm-frame/tests`: PASS, 295 - passed, 36 skipped, 1 warning in 89.30 seconds. -- Full aggregate: 7,454 passed, 83 skipped. -- `.venv/bin/python tools/ci_test_groups.py --verify`: PASS, - `tracked_test_files=309`, `verification=ok`. -- Focused changed-file suites plus exact-k E2E/launcher and data - contract/release/publish-guard regressions: PASS; the full shard runs above - subsequently covered the same tests on the final code tree. -- `git diff --check origin/main...HEAD`: PASS. - -The build-shard warnings are expected numerical, pandas chained-assignment, -PolicyEngine division, and intentionally fragmented-frame test warnings; no -warning is a test failure and none originates in the new receipt path. +- `uv run --no-sync python tools/ci_test_groups.py --verify`: PASS, + `tracked_test_files=309`, `verification=ok` +- Eight directly affected spec/receipt files: PASS, 102 tests +- `tools/generate_us_bundle_from_constants.py --check`: PASS, final US spec + SHA-256 + `5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371` +- `tools/spec_engine_coverage.py --check`: PASS, 42,122/42,122 fields and + 41/41 inventory checks +- `uv run --no-sync pytest packages/microcosm-calibrate/tests -q`: PASS, + 203 passed +- `uv run --no-sync pytest packages/microcosm-data/tests -q`: PASS, + 318 passed, 2 skipped +- `uv run --no-sync pytest packages/microcosm-fit/tests -q`: PASS, + 93 passed +- `uv run --no-sync pytest packages/microcosm-frame/tests -q`: PASS, + 295 passed, 36 skipped +- `uv run --no-sync pytest packages/microcosm-build/tests -q`: PASS, + 6,608 tests collected, 100% reached, exit code 0, expected skips only +- Complete ACS-transfer file: PASS, 65 tests +- Complete post-transfer calibration receipt-contract file: PASS, 47 tests +- Complete weeks file: PASS, 26 passed, 1 skipped +- Complete alimony/workers'/SSI files: PASS, 28 / 21 / 71 tests +- Medicare + retirement contribution/distribution focused files: PASS, 65 tests +- Repository `git diff --check`: PASS + +The shard warnings observed are existing numerical, sparse-tensor, +joblib core-detection, and PolicyEngine divide warnings; none is a failure and +none originates in the new transfer/source-scope paths. ## Commit inventory -Implementation and journal commits before this final report carrier: - -1. `34375fe6` Document gate-failed release lane kickoff -2. `5be9e49a` Preserve prior progress journal history -3. `c5a24c0f` Record release containment design review -4. `72274910` feat: allow explicit gate-failed pool release inputs -5. `e3d847d6` feat: surface carried red pool verdict in preflight -6. `46811fdb` fix: preserve default exact-k receipt fixtures -7. `dd1ad19a` Add shared multispine base-pool authentication seam -8. `271ee1fe` Close legacy base-H5 pool receipt bypass -9. `92134322` Authenticate and surface gate-failed pools in preflight -10. `cd39f756` Document gate-failed base-pool release boundary -11. `9db4694a` Tighten the authenticated red-pool release opt-in -12. `72c4c7a1` Bind carried pool verdicts to preflight inputs -13. `59c5759d` Record completed release containment work -14. `af20d3f5` Lock ordinary base-pool release wiring in tests - -Commits after the list carry this report, the final `PROGRESS.md` state, and -their handoff cleanup. A report cannot embed the hash of the commit containing -its own final bytes, so `git log --oneline origin/main..HEAD` is the -authoritative complete inventory. At final handoff the branch is 17 commits -ahead of `origin/main` with no uncommitted paths. +Implementation and evidence commits preceding the final report carrier: + +1. `a3331db6` Start stacked release gate alignment journal +2. `9979d101` Record fractional weeks provenance +3. `d7ad753e` Preserve integer support for transferred weeks +4. `0c5b05a8` Receipt integer weeks transfer support +5. `c22a6799` Align weeks gate with stacked source roles +6. `9bbd6dfe` Centralize stacked source channel scopes +7. `49052e6e` Scope alimony source checks to physical ASEC +8. `17ed9a0b` Key WIC draws by stacked person identity +9. `c7dd53ad` Support stacked Head Start clone layouts +10. `3f934688` Scope Medicare source gate to stacked ASEC +11. `5852f4ba` Scope retirement contribution source diagnostics +12. `e40d5db7` Support stacked voluntary filing clones +13. `148881c6` Check all stacked prior-year income clones +14. `52b27f78` Separate alimony source validation and carry scopes +15. `058f27b5` Record stacked clone-layout repairs +16. `97de22e7` Scope workers compensation sources in stacked pools +17. `74e96187` Scope retirement distribution source gates +18. `41007992` Anchor SSI reporters to physical ASEC sources +19. `eb5cd4b6` Keep SSI assignment source blind +20. `3dc5df51` Keep stacked source repairs gate scoped +21. `9b36720b` Record stacked raw-source gate repairs +22. `9520c03f` Report divergence across all SSI disability clones +23. `b8e04e9c` Confine physical source access to release gates +24. `b2de92b9` Document stacked release gate alignment +25. `cd8c55a7` Repin spec engine for integer weeks +26. `12a918ed` Repin multispine live spec fixture + +The final report/journal carrier follows this inventory and is necessarily +self-excluded; the clean-worktree handoff lists it in the final response. diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py index a5dd282ab8..ad66f291ad 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py @@ -26,9 +26,9 @@ ) from .schemas import load_schema_registry -EXPECTED_AUTHORED_FIELD_COUNT = 32_351 -EXPECTED_RESOLVED_BINDING_FIELD_COUNT = 9_769 -EXPECTED_CONFIGURATION_FIELD_COUNT = 42_120 +EXPECTED_AUTHORED_FIELD_COUNT = 32_352 +EXPECTED_RESOLVED_BINDING_FIELD_COUNT = 9_770 +EXPECTED_CONFIGURATION_FIELD_COUNT = 42_122 class FieldUsageError(AssertionError): @@ -429,8 +429,8 @@ def _path_inventory(rows: Sequence[tuple[str, object]]) -> tuple[int, str]: "1b9f7ffb88d31d2bd365c092d04a96038b2d5a2b03f4f2a79f42e95472b213d6", ), "imputation_transfer_execution": ( - 79, - "1575513251c72be60edc75f378221fa878ea76e009512f1352a111586d0a60f8", + 80, + "10bd811f86dae7941035533cfe0d51d81cd632f5995c1adb80234bc7ecbf6565", ), "imputation_waiver_records": ( 70, @@ -457,8 +457,8 @@ def _path_inventory(rows: Sequence[tuple[str, object]]) -> tuple[int, str]: "6a781915fd491d2c4b16d2b7d482f69cf362c904130093c59f9629f7a319269b", ), "resolved_seed_protocol": ( - 823, - "c05ebb88538195798e9365f325381c14d92b74007f846f1fef50a08268e6d27d", + 824, + "7537385c3fd399a2dbb7dcd8ed7cf1ff2481ed336db621eafcbfd741d5792f40", ), "resolved_seed_site_bindings": ( 277, diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py index b48afbf581..f0dd2529c5 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py @@ -350,17 +350,17 @@ "acs_person_predictors": "878c788a6f037d7aca12b3586ea034eff04f3034ffa11935a736493042551f25", "authority": "3a980927227704d0589f246eef9cd825c2ae84f3a4134ac835e0e5ed39a563ac", "early_families": "4aa9f736fd76e83955477ad1667e58f48f264783f05bdc7f0102cd32d61323bd", - "full_checkpoint": "a128a85f877fb32def9382b841b8b340f974e8a9148ac029c1f04becdc956c18", + "full_checkpoint": "1fa30ae9213c19607168df3cf0035478b9c96f815e1541cc97f4e1b8e7246935", "gap_fill_schedule": "1c31f9868f7884347cc19cf1ff65da43f950b9114941a715bab168246db414a7", "graph_nodes": "7125ad28ae2c69f22094a574bbf6ed2ddf1682a2c2c3b416f8f49304b7016ce7", "geography_assignment": "f49425ca8734ac559c73cf44f6458d86d3162a48956b98a27e6e758959361585", "late_families": "d91f9ff0eb52f43e7b6eed3d5c58c37abe1620c3a11021da15dae9c10e16d382", - "late_resource_semantics": "3850554eb804fde5e4f86a34ac1bb8a7a07aafff7e8b48396a3d5fca844798e8", + "late_resource_semantics": "ef491637933945e28645479da70646d9c50366213b6f70477de445ab9e4bb1cf", "late_schedule": "dcf3c6d2eade3449836c49a1dc4d3b8cd395aab9142db700c3c60598fa9c1c79", "ownership": "5f64f0aac49e2313177564f71876bffc8c81b3ded4df701e70930e60e9c98356", "primary_tuples": "987b501c695e31f45521c4a178528f75ab3df22c09bc407b182213b2de99ee57", - "seed_map": "36a9d819ef196c312888591936d49c025b0407df9928440cceefabc5458f72af", - "seed_protocol": "15840b380329410a7094f60b0f1dad453457fc785859f0c372f2c8e2d59b0246", + "seed_map": "ee1ad8d5196ca48d70657675237294c2dd925a520900d4d476d256a1b514b283", + "seed_protocol": "c7df307cba6b20cc26f4aeeb1ee5eae246362318cef885cd747c64349e08cc41", "source_manifest": "cd5ba8924d64da5425ee14cca82a774e3f4b2bb5aabe06df291cc3cc457287a9", "take_up": "fa186daea0f8dd641cc470e41d1a2953f887d45282ec990201298f47bedf8d4d", "tail": "ac92829c88a1a4fb6460d61190918d5d99c6c377fc8dd8f62f02b332d09bf59c", diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py b/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py index 36e6c27e85..04fb87ad8c 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/seeds.py @@ -726,6 +726,7 @@ def _stable_site( "wic_claim_assignment", salt="would_claim_wic", key_grammar=( + "source:person_source_id_if_assembled_multispine", "source_year:source_household_id:source_person_id_if_complete", "else_support:person_support_source_id", "else_person:person_id", diff --git a/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml b/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml index f0a7ed69c9..99d853a9ac 100644 --- a/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml +++ b/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml @@ -408,6 +408,7 @@ transfer_execution: discrete_numeric_targets: - first_home_mortgage_origination_year - second_home_mortgage_origination_year + - weeks_unemployed predictor_bindings: person_required: acs_person_required person_optional: diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py index e9e7564735..518baacb64 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/__init__.py @@ -593,6 +593,7 @@ PufTaxDetailChainInputs, clone_us_frame_for_puf_support, finalize_us_puf_tax_detail_predictions, + has_assembled_support_metadata, has_support_role_metadata, impute_us_puf_tax_detail_support, prepare_us_puf_tax_detail_chain_inputs, @@ -1953,6 +1954,7 @@ "validate_spine_agreement_registry", "assemble_spines", "PUF_TAX_DETAIL_CLONE_INDEX", + "has_assembled_support_metadata", "has_support_role_metadata", "puf_tax_detail_clone_mask", "spine_source_id_column", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py index f791823a0f..8688d84683 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py @@ -126,14 +126,15 @@ } ) -# These inputs are numeric in current dense artifacts, but their domain is a -# finite set of years. QRF quantile interpolation is valid for continuous -# amounts and invalid for a year code, so predictions are snapped to observed -# donor support exactly as the existing PUF support stage does. +# These inputs are numeric in current dense artifacts, but their domain has +# finite integer support. QRF quantile interpolation is valid for continuous +# amounts and invalid for year codes or counts, so predictions are snapped to +# observed donor support exactly as the existing PUF support stage does. _DISCRETE_NUMERIC_TARGETS = frozenset( { "first_home_mortgage_origination_year", "second_home_mortgage_origination_year", + "weeks_unemployed", } ) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py b/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py index 5ab6c1995c..8ae7fe8347 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/alimony.py @@ -23,7 +23,9 @@ from microcosm.build.source_manifest import SourceStageSpec, load_source_manifest from microcosm.build.us_runtime.support_provenance import ( BASE_ASEC_SUPPORT_CHANNEL, + has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -291,18 +293,41 @@ def us_alimony_signal_gate(frame: Frame) -> GateResult: details=summary, ) source_mask = np.ones(len(person), dtype=bool) + source_reconciliation_mask = source_mask.copy() if has_support_role_metadata(person, entity="person"): source_mask = ( - support_role_series(person, entity="person") + support_gate_source_channel_series(person, entity="person") .eq(BASE_ASEC_SUPPORT_CHANNEL) .to_numpy() ) - amounts = pd.to_numeric(person["OI_VAL"], errors="coerce").to_numpy( + source_reconciliation_mask = source_mask.copy() + if has_assembled_support_metadata(person, entity="person"): + source_reconciliation_mask &= ( + support_role_series(person, entity="person") + .eq(BASE_ASEC_SUPPORT_CHANNEL) + .to_numpy() + ) + raw_amounts = pd.to_numeric(person["OI_VAL"], errors="coerce").to_numpy( dtype=np.float64 - )[source_mask] - codes = pd.to_numeric(person["OI_OFF"], errors="coerce").to_numpy( + ) + raw_codes = pd.to_numeric(person["OI_OFF"], errors="coerce").to_numpy( dtype=np.float64 - )[source_mask] + ) + source_invalid = source_mask & ( + ~np.isfinite(raw_amounts) | ~np.isfinite(raw_codes) + ) + summary["asec_source_rows"] = int(np.count_nonzero(source_mask)) + summary["asec_source_invalid"] = int(np.count_nonzero(source_invalid)) + summary["asec_source_reconciliation_rows"] = int( + np.count_nonzero(source_reconciliation_mask) + ) + if bool(source_invalid.any()): + failures.append( + "ASEC OI_VAL/OI_OFF contain nonfinite source values on " + f"{int(np.count_nonzero(source_invalid))} row(s)." + ) + amounts = raw_amounts[source_reconciliation_mask] + codes = raw_codes[source_reconciliation_mask] if bool(np.isfinite(amounts).all() and np.isfinite(codes).all()): integer_codes = codes.astype(np.int64) expected_alimony = np.where( @@ -321,13 +346,16 @@ def us_alimony_signal_gate(frame: Frame) -> GateResult: amounts, ) actual_alimony = pd.to_numeric( - person.loc[source_mask, "alimony_income"], errors="coerce" + person.loc[source_reconciliation_mask, "alimony_income"], + errors="coerce", ).to_numpy(dtype=np.float64) actual_strike_benefits = pd.to_numeric( - person.loc[source_mask, "strike_benefits"], errors="coerce" + person.loc[source_reconciliation_mask, "strike_benefits"], + errors="coerce", ).to_numpy(dtype=np.float64) actual_miscellaneous = pd.to_numeric( - person.loc[source_mask, "miscellaneous_income"], errors="coerce" + person.loc[source_reconciliation_mask, "miscellaneous_income"], + errors="coerce", ).to_numpy(dtype=np.float64) alimony_mismatch = ~np.isclose(actual_alimony, expected_alimony) strike_benefits_mismatch = ~np.isclose( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py b/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py index d93ae50665..ba40e479b4 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/medicare_take_up.py @@ -27,7 +27,10 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + BASE_ASEC_SUPPORT_CHANNEL, + has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -128,6 +131,20 @@ def _source_codes(person: pd.DataFrame, source: str) -> np.ndarray: return numeric.astype(np.int8) +def _asec_source_mask(person: pd.DataFrame) -> np.ndarray: + """Select physical ASEC rows without changing legacy source semantics.""" + + if not has_assembled_support_metadata(person, entity="person"): + return np.ones(len(person), dtype=bool) + source_channels = support_gate_source_channel_series(person, entity="person") + mask = source_channels.eq(BASE_ASEC_SUPPORT_CHANNEL).to_numpy() + if not mask.any(): + raise SourceRuntimeError( + "US Medicare take-up support has no physical ASEC source rows." + ) + return mask + + def derive_us_medicare_take_up_from_manifest( frame: pd.DataFrame | None, operation: SourceOperationSpec, @@ -228,10 +245,14 @@ def us_medicare_take_up_summary(frame: Frame) -> dict[str, object]: "missing_count": int(person[_OUTPUT].isna().sum()), } if _SOURCE in person: - source_values = _source_codes(person, _SOURCE) == _ENROLLED_CODE + source_mask = _asec_source_mask(person) + source_values = ( + _source_codes(person.loc[source_mask], _SOURCE) == _ENROLLED_CODE + ) summary["source_mismatch_count"] = int( - np.count_nonzero(values != source_values) + np.count_nonzero(values[source_mask] != source_values) ) + summary["source_rows"] = int(np.count_nonzero(source_mask)) if has_support_role_metadata(person, entity="person"): channel_shares: dict[str, float] = {} channels = support_role_series(person, entity="person").to_numpy() diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py b/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py index 254181ca52..b03340266d 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py @@ -39,6 +39,7 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, support_role_series, without_support_role_metadata, @@ -750,7 +751,9 @@ def _share(mask: np.ndarray) -> float: } ) group_columns = ["_source_id"] - if has_support_role_metadata(person, entity="person"): + if has_support_role_metadata( + person, entity="person" + ) and not has_assembled_support_metadata(person, entity="person"): clone_work["_role"] = support_role_series( person, entity="person" ).to_numpy() diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py index c359eaeed9..d78c70a235 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/puf_support.py @@ -41,6 +41,7 @@ BASE_ASEC_SUPPORT_CHANNEL, PUF_TAX_DETAIL_CLONE_INDEX, PUF_TAX_DETAIL_SUPPORT_CHANNEL, + has_assembled_support_metadata, has_support_role_metadata, puf_tax_detail_clone_mask, spine_source_id_column, @@ -73,6 +74,7 @@ "bind_puf_clone_attachment_tail_descendant", "clone_us_frame_for_puf_support", "finalize_us_puf_tax_detail_predictions", + "has_assembled_support_metadata", "has_support_role_metadata", "impute_us_puf_tax_detail_support", "puf_tax_detail_clone_mask", diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py index 753110d3e7..3265139ab8 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_contributions.py @@ -38,7 +38,9 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -167,6 +169,30 @@ def _numeric_source(frame: pd.DataFrame, column: str) -> np.ndarray: return values +def _asec_source_mask(frame: pd.DataFrame) -> np.ndarray: + """Select physical ASEC rows while retaining the legacy all-row source.""" + + if not has_assembled_support_metadata(frame, entity="person"): + return np.ones(len(frame), dtype=bool) + source_channels = support_gate_source_channel_series(frame, entity="person") + mask = source_channels.eq(_BASE_ASEC_SUPPORT_CHANNEL).to_numpy() + if not mask.any(): + raise SourceRuntimeError( + "US retirement-contribution support has no physical ASEC source rows." + ) + return mask + + +def _source_reconciliation_mask(frame: pd.DataFrame) -> np.ndarray: + """Select direct ASEC operator rows whose allocation remains source-exact.""" + + source_mask = _asec_source_mask(frame) + if not has_assembled_support_metadata(frame, entity="person"): + return source_mask + roles = support_role_series(frame, entity="person").to_numpy() + return source_mask & (roles == _BASE_ASEC_SUPPORT_CHANNEL) + + def derive_us_retirement_contributions_from_manifest( frame: pd.DataFrame | None, operation: SourceOperationSpec, @@ -569,11 +595,16 @@ def us_retirement_contributions_summary(frame: Frame) -> dict[str, object]: ) for column in US_RETIREMENT_CONTRIBUTION_OUTPUT_COLUMNS } - source = ( - _numeric_source(person, "RETCB_VAL") - if "RETCB_VAL" in person - else np.zeros(len(person), dtype=np.float64) - ) + source = np.full(len(person), np.nan, dtype=np.float64) + source_mask = np.zeros(len(person), dtype=bool) + reconciliation_mask = np.zeros(len(person), dtype=bool) + if "RETCB_VAL" in person: + source_mask = _asec_source_mask(person) + source[source_mask] = _numeric_source( + person.loc[source_mask], + "RETCB_VAL", + ) + reconciliation_mask = _source_reconciliation_mask(person) combined = np.sum(np.column_stack(tuple(contributions.values())), axis=1) def _share(values: np.ndarray) -> float: @@ -590,7 +621,7 @@ def _share(values: np.ndarray) -> float: column: int(np.count_nonzero(values < 0)) for column, values in contributions.items() } - source_positive = source > 0 + source_positive = reconciliation_mask & (source > 0) allocation_mismatch = source_positive & ~np.isclose( combined, source, @@ -608,9 +639,13 @@ def _share(values: np.ndarray) -> float: "nonzero_share_band": list(_NONZERO_SHARE_BAND), "nonfinite": nonfinite, "negative": negative, + "source_rows": int(np.count_nonzero(source_mask)), + "source_reconciliation_rows": int(np.count_nonzero(reconciliation_mask)), "source_positive_rows": int(np.count_nonzero(source_positive)), "allocation_mismatch_rows": int(np.count_nonzero(allocation_mismatch)), - "source_total": float(np.sum(source * weights)), + "source_total": float( + np.sum(np.where(reconciliation_mask, source, 0.0) * weights) + ), "allocated_total": float(np.sum(np.nan_to_num(combined) * weights)), } diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py index 7f8d1427c0..7d9659af0d 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/retirement_distributions.py @@ -43,7 +43,9 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -269,6 +271,30 @@ def _derived_outputs( return outputs +def _asec_source_mask(frame: pd.DataFrame) -> np.ndarray: + """Select physical ASEC rows while retaining the legacy all-row source.""" + + if not has_assembled_support_metadata(frame, entity="person"): + return np.ones(len(frame), dtype=bool) + source_channels = support_gate_source_channel_series(frame, entity="person") + mask = source_channels.eq(_BASE_ASEC_SUPPORT_CHANNEL).to_numpy() + if not mask.any(): + raise SourceRuntimeError( + "US retirement-distribution support has no physical ASEC source rows." + ) + return mask + + +def _source_reconciliation_mask(frame: pd.DataFrame) -> np.ndarray: + """Select direct operator rows whose outputs remain measured-source exact.""" + + compare = _asec_source_mask(frame).copy() + if has_support_role_metadata(frame, entity="person"): + roles = support_role_series(frame, entity="person").to_numpy() + compare &= roles == _BASE_ASEC_SUPPORT_CHANNEL + return compare + + def derive_us_retirement_distributions_from_manifest( frame: pd.DataFrame | None, operation: SourceOperationSpec, @@ -674,17 +700,26 @@ def us_retirement_distributions_summary(frame: Frame) -> dict[str, object]: } source_mismatches: dict[str, int] = {} + source_rows = 0 + source_reconciliation_rows = 0 if all( column in person for column in US_RETIREMENT_DISTRIBUTION_REQUIRED_SOURCE_COLUMNS ): - expected = _derived_outputs(person, _EXPECTED_OUTPUT_BY_ACCOUNT_CODE) - compare = np.ones(len(person), dtype=bool) - if has_support_role_metadata(person, entity="person"): - compare = ( - support_role_series(person, entity="person").to_numpy() - == _BASE_ASEC_SUPPORT_CHANNEL - ) + source_mask = _asec_source_mask(person) + source_expected = _derived_outputs( + person.loc[source_mask], + _EXPECTED_OUTPUT_BY_ACCOUNT_CODE, + ) + expected = { + column: np.full(len(person), np.nan, dtype=np.float64) + for column in US_RETIREMENT_DISTRIBUTION_OUTPUT_COLUMNS + } + for column, source_values in source_expected.items(): + expected[column][source_mask] = source_values + compare = _source_reconciliation_mask(person) + source_rows = int(np.count_nonzero(source_mask)) + source_reconciliation_rows = int(np.count_nonzero(compare)) source_mismatches = { column: int( np.count_nonzero( @@ -722,6 +757,8 @@ def _share(array: np.ndarray) -> float: "negative": { column: int(np.count_nonzero(array < 0)) for column, array in values.items() }, + "source_rows": source_rows, + "source_reconciliation_rows": source_reconciliation_rows, "source_mismatches": source_mismatches, } diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_head_start.py b/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_head_start.py index e7c61f01a5..5d45b4764f 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_head_start.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_head_start.py @@ -35,7 +35,9 @@ from microcosm.build.gates import GateResult from microcosm.build.source_manifest import SourceStageSpec, load_source_manifest from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_clone_index_column, support_role_series, ) from microcosm.build.us_runtime.voluntary_filing import ( @@ -481,7 +483,7 @@ def _support_group_keys( person: pd.DataFrame, source_id: pd.Series, ) -> tuple[pd.Series, pd.Series]: - """Return clone-pair keys without consulting source-spine channels.""" + """Return source-person keys and validated legacy-compatible roles.""" if not has_support_role_metadata(person, entity="person"): return ( @@ -571,30 +573,58 @@ def _recipient_predictors(frame: Frame) -> tuple[pd.DataFrame, pd.Series, np.nda "US SIPP Head Start source clones disagree on age for " f"person_source_id(s): {inconsistent[:5]}." ) - role_rows = pd.DataFrame({"source_id": source_id, "role": roles}) - duplicate_roles = role_rows.duplicated( - ["source_id", "role"], - keep=False, - ) - if duplicate_roles.any(): - bad = ( - role_rows.loc[duplicate_roles, ["source_id", "role"]] - .drop_duplicates() - .itertuples(index=False, name=None) - ) - raise ValueError( - "US SIPP Head Start source units carry duplicated same-role rows; " - f"invalid source role(s): {list(bad)[:5]}." - ) - order = pd.DataFrame(index=person.index) order["source_id"] = source_id order["source_key"] = source_key - order["role_priority"] = roles.map({_ASEC_CHANNEL: 0, _PUF_CHANNEL: 1}) + if has_assembled_support_metadata(person, entity="person"): + clone_column = support_clone_index_column("person") + clone_indices = pd.to_numeric(person[clone_column], errors="raise").astype( + "int64" + ) + clone_rows = pd.DataFrame( + {"source_id": source_id, "clone_index": clone_indices} + ) + duplicate_clones = clone_rows.duplicated( + ["source_id", "clone_index"], + keep=False, + ) + if duplicate_clones.any(): + bad = ( + clone_rows.loc[ + duplicate_clones, + ["source_id", "clone_index"], + ] + .drop_duplicates() + .itertuples(index=False, name=None) + ) + raise ValueError( + "US SIPP Head Start assembled source units carry duplicated " + f"clone-index rows; invalid source clone(s): {list(bad)[:5]}." + ) + order["clone_priority"] = clone_indices + priority_column = "clone_priority" + else: + role_rows = pd.DataFrame({"source_id": source_id, "role": roles}) + duplicate_roles = role_rows.duplicated( + ["source_id", "role"], + keep=False, + ) + if duplicate_roles.any(): + bad = ( + role_rows.loc[duplicate_roles, ["source_id", "role"]] + .drop_duplicates() + .itertuples(index=False, name=None) + ) + raise ValueError( + "US SIPP Head Start source units carry duplicated same-role " + f"rows; invalid source role(s): {list(bad)[:5]}." + ) + order["role_priority"] = roles.map({_ASEC_CHANNEL: 0, _PUF_CHANNEL: 1}) + priority_column = "role_priority" order["person_key"] = person["person_id"].astype(str) canonical_index = ( order.sort_values( - ["source_id", "role_priority", "person_key"], + ["source_id", priority_column, "person_key"], kind="mergesort", ) .drop_duplicates("source_key", keep="first") diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py index c90d8169b5..4a6b566113 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_disability_criteria.py @@ -42,6 +42,7 @@ from microcosm.build.gates import GateResult from microcosm.build.source_manifest import SourceStageSpec, load_source_manifest from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, support_role_series, ) @@ -1137,12 +1138,16 @@ def us_ssi_disability_criteria_summary(frame: Frame) -> dict[str, object]: "value": values, } ) - clone_table["source_occurrence"] = clone_table.groupby( - ["source_id", "role"], sort=False - ).cumcount() - unique = clone_table.groupby( - ["source_id", "source_occurrence"], sort=False - )["value"].nunique(dropna=False) + if has_assembled_support_metadata(person, entity="person"): + clone_groups = ["source_id"] + else: + clone_table["source_occurrence"] = clone_table.groupby( + ["source_id", "role"], sort=False + ).cumcount() + clone_groups = ["source_id", "source_occurrence"] + unique = clone_table.groupby(clone_groups, sort=False)["value"].nunique( + dropna=False + ) clone_divergence_source_people = int((unique > 1).sum()) reporter_mismatches = 0 diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py index 0b8b035cfe..86f2089600 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/ssi_take_up.py @@ -77,6 +77,7 @@ PUF_TAX_DETAIL_SUPPORT_CHANNEL, has_support_role_metadata, support_clone_index_column, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -530,17 +531,20 @@ def us_ssi_take_up_reporter_source_ids(frame: Frame) -> frozenset[str]: if person[_SOURCE_ID].isna().any(): raise ValueError("US SSI take-up reporter lineage requires provenance.") source_ids = _decoded_strings(person[_SOURCE_ID]) - roles = support_role_series(person, entity="person") + source_channels = support_gate_source_channel_series(person, entity="person") + asec_source = source_channels.eq(_ASEC_CHANNEL).to_numpy() reported = pd.to_numeric(person[US_SSI_TAKE_UP_ANCHOR], errors="coerce").to_numpy( dtype=np.float64 ) - if source_ids.str.strip().eq("").any() or not np.isfinite(reported).all(): + if source_ids.str.strip().eq("").any() or not np.isfinite( + reported[asec_source] + ).all(): raise ValueError( "US SSI take-up reporter lineage requires nonblank identities and " - "finite SSI_VAL values." + "finite SSI_VAL values on physical ASEC source rows." ) reporter_ids = frozenset( - source_ids[roles.eq(_ASEC_CHANNEL).to_numpy() & (reported > 0.0)] + source_ids[asec_source & (reported > 0.0)] ) if not reporter_ids: raise ValueError("US SSI take-up found no direct ASEC SSI reporters.") @@ -576,8 +580,6 @@ def _source_table( weights = np.asarray(frame.resolve_weights("person").values, dtype=np.float64) if not np.isfinite(age).all() or (age < 0).any(): raise ValueError("US SSI take-up ages must be finite and nonnegative.") - if not np.isfinite(reported).all(): - raise ValueError("US SSI take-up SSI_VAL anchors must be finite.") if not np.isfinite(potential).all(): raise ValueError("US SSI take-up uncapped_ssi values must be finite.") if not (np.isfinite(weights) & (weights >= 0)).all() or weights.sum() <= 0: @@ -602,21 +604,19 @@ def _source_table( f"unsupported {sorted(observed_channels - _KNOWN_CHANNELS)}." ) - direct_anchor = (reported > 0.0) & channels.eq(_ASEC_CHANNEL).to_numpy() if reporter_source_ids is None: - anchored_source_ids = frozenset(source_ids[direct_anchor]) + anchored_source_ids = us_ssi_take_up_reporter_source_ids(frame) else: anchored_source_ids = frozenset(str(value) for value in reporter_source_ids) if not anchored_source_ids: raise ValueError("US SSI take-up reporter lineage cannot be empty.") if any(not value.strip() for value in anchored_source_ids): raise ValueError("US SSI take-up reporter source identities are nonblank.") - omitted_direct = sorted(set(source_ids[direct_anchor]) - anchored_source_ids) - if omitted_direct: - raise ValueError( - "US SSI take-up reporter lineage omitted direct ASEC anchors; " - f"examples {omitted_direct[:5]}." - ) + anchored_rows = source_ids.isin(anchored_source_ids).to_numpy() + if not np.isfinite(reported[anchored_rows]).all(): + raise ValueError( + "US SSI take-up SSI_VAL anchors must be finite on anchored source rows." + ) rows = pd.DataFrame( { @@ -627,9 +627,9 @@ def _source_table( "weight": weights, "candidate": potential > 0.0, # Capture lineage on the full support before L0. When pruning keeps - # only a PUF clone, the explicit source-ID set still preserves the - # underlying direct ASEC measurement without promoting PUF-only - # SSI_VAL copies into independent anchors. + # only a donor-role clone, the explicit source-ID set still + # preserves the underlying physical ASEC measurement without + # promoting non-ASEC SSI_VAL values into independent anchors. "anchor": source_ids.isin(anchored_source_ids).to_numpy(), }, index=person.index, diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py b/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py index da5bf9e08b..ec5a51eaf2 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/support_provenance.py @@ -16,6 +16,7 @@ "PUF_TAX_DETAIL_CLONE_INDEX", "PUF_TAX_DETAIL_SUPPORT_CHANNEL", "SPINE_ASSEMBLY_MANIFEST_KEY", + "has_assembled_support_metadata", "has_support_role_metadata", "puf_tax_detail_clone_mask", "spine_assembly_manifest", @@ -25,6 +26,7 @@ "support_channel_column", "support_clone_index_column", "support_role_series", + "support_gate_source_channel_series", "support_source_id_column", "us_reported_coverage_vintage_signal_gate", "validate_assembly_provenance", @@ -326,6 +328,21 @@ def has_support_role_metadata( ) +def has_assembled_support_metadata( + table: pd.DataFrame, + *, + entity: str, +) -> bool: + """Return whether support metadata belongs to a multispine assembly. + + The raw spine-record ID is the assembly discriminator. Historical frames + used the support-channel column for the two operator roles instead, so the + presence of a channel or clone index alone is not sufficient. + """ + + return spine_source_id_column(entity) in table + + def spine_source_id_column(entity: str) -> str: """Return the entity-prefixed raw spine-record ID metadata column.""" @@ -476,6 +493,37 @@ def support_role_series( ) +def support_gate_source_channel_series( + table: pd.DataFrame, + *, + entity: str, +) -> pd.Series: + """Return physical source channels for read-only gates and reporters. + + Assembled frames retain their receipt-declared physical channels (for + example, ``asec`` and ``acs``). Historical frames have no raw spine ID and + use the exact ``asec``/``puf_tax_detail`` operator roles as their channels. + This deliberately narrow accessor must not route population treatments. + Keeping resolution in the provenance owner lets reporting and release gates + inspect source coverage without teaching operators how to read provenance + columns directly. A static call-site contract pins its reviewed consumers. + """ + + roles = support_role_series(table, entity=entity) + if not has_assembled_support_metadata(table, entity=entity): + return pd.Series( + roles.to_numpy(dtype=object, copy=True), + index=table.index, + name=f"{entity}_source_support_channel", + ) + channels = table[support_channel_column(entity)] + return pd.Series( + channels.to_numpy(dtype=object, copy=True), + index=table.index, + name=f"{entity}_source_support_channel", + ) + + def puf_tax_detail_clone_mask( table: pd.DataFrame, *, diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py b/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py index e814cb476e..b19d32dda1 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py @@ -38,7 +38,9 @@ from microcosm.build.gates import GateResult from microcosm.build.source_manifest import SourceStageSpec, load_source_manifest from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_clone_index_column, support_role_series, ) from microcosm.frame import Frame @@ -742,53 +744,77 @@ def _source_receiver_rows( rows["_support_role"] = support_role_series( tax_unit, entity="tax_unit" ).to_numpy() - role_counts = rows.groupby( - ["_source_id", "_support_role"], - sort=False, - ).size() - duplicated_roles = role_counts[role_counts > 1] - if not duplicated_roles.empty: - bad = duplicated_roles.index.tolist() - raise ValueError( - "US voluntary-filing support source units carry duplicated " - f"same-role rows; invalid source role(s) {bad[:5]}." - ) rows["_source_key"] = rows["_source_id"] - asec_counts = ( - rows["_support_role"] - .eq(_BASE_ASEC_SUPPORT_CHANNEL) - .groupby(rows["_source_key"]) - .sum() - ) - if asec_counts.gt(1).any(): - bad = asec_counts.index[asec_counts.gt(1)].tolist() - raise ValueError( - "US voluntary-filing support source units carry duplicated " - f"ASEC rows; invalid source unit(s) {bad[:5]}." - ) - # Prefer each unit's ASEC row, but a frozen-support selection may - # legitimately keep only a unit's PUF clone (the L0-survivor case the - # SSI reporter lineage also handles — Build M's certified 57,240 - # selection does exactly this). Clones carry the unit's source - # predictors, so the surviving row predicts identically; pick it - # deterministically by channel then tax-unit id. ordered_rows = rows.copy() - ordered_rows["_asec_rank"] = ( - ~ordered_rows["_support_role"].eq(_BASE_ASEC_SUPPORT_CHANNEL) - ).astype(int) - source_rows = ( - ordered_rows.sort_values( - [ - "_source_id", - "_asec_rank", - "_support_role", - "_tax_unit_id", - ], - kind="stable", + if has_assembled_support_metadata(tax_unit, entity="tax_unit"): + clone_column = support_clone_index_column("tax_unit") + ordered_rows["_clone_index"] = pd.to_numeric( + tax_unit[clone_column], errors="raise" + ).to_numpy(dtype=np.int64) + clone_counts = ordered_rows.groupby( + ["_source_id", "_clone_index"], + sort=False, + ).size() + duplicated_clones = clone_counts[clone_counts > 1] + if not duplicated_clones.empty: + bad = duplicated_clones.index.tolist() + raise ValueError( + "US voluntary-filing assembled source units carry " + "duplicated clone-index rows; invalid source clone(s) " + f"{bad[:5]}." + ) + source_rows = ( + ordered_rows.sort_values( + ["_source_id", "_clone_index", "_tax_unit_id"], + kind="stable", + ) + .drop_duplicates("_source_key", keep="first") + .drop(columns="_clone_index") + ) + else: + role_counts = rows.groupby( + ["_source_id", "_support_role"], + sort=False, + ).size() + duplicated_roles = role_counts[role_counts > 1] + if not duplicated_roles.empty: + bad = duplicated_roles.index.tolist() + raise ValueError( + "US voluntary-filing support source units carry duplicated " + f"same-role rows; invalid source role(s) {bad[:5]}." + ) + asec_counts = ( + rows["_support_role"] + .eq(_BASE_ASEC_SUPPORT_CHANNEL) + .groupby(rows["_source_key"]) + .sum() + ) + if asec_counts.gt(1).any(): + bad = asec_counts.index[asec_counts.gt(1)].tolist() + raise ValueError( + "US voluntary-filing support source units carry duplicated " + f"ASEC rows; invalid source unit(s) {bad[:5]}." + ) + # Prefer each unit's ASEC row, but a frozen-support selection may + # legitimately keep only a unit's PUF clone. Clones carry the + # unit's source predictors, so the surviving row predicts + # identically; pick it deterministically by role then tax-unit id. + ordered_rows["_asec_rank"] = ( + ~ordered_rows["_support_role"].eq(_BASE_ASEC_SUPPORT_CHANNEL) + ).astype(int) + source_rows = ( + ordered_rows.sort_values( + [ + "_source_id", + "_asec_rank", + "_support_role", + "_tax_unit_id", + ], + kind="stable", + ) + .drop_duplicates("_source_key", keep="first") + .drop(columns="_asec_rank") ) - .drop_duplicates("_source_key", keep="first") - .drop(columns="_asec_rank") - ) else: if rows["_source_id"].duplicated().any(): duplicates = rows.loc[ @@ -1015,10 +1041,13 @@ def us_voluntary_filing_summary(frame: Frame) -> dict[str, object]: "value": values, } ) - clone_table["source_occurrence"] = clone_table.groupby( - ["source_id", "role"], sort=False - ).cumcount() - clone_groups = ["source_id", "source_occurrence"] + if has_assembled_support_metadata(tax_unit, entity="tax_unit"): + clone_groups = ["source_id"] + else: + clone_table["source_occurrence"] = clone_table.groupby( + ["source_id", "role"], sort=False + ).cumcount() + clone_groups = ["source_id", "source_occurrence"] sizes = clone_table.groupby(clone_groups, sort=False).size() clone_source_units = int((sizes > 1).sum()) unique = clone_table.groupby(clone_groups, sort=False)["value"].nunique( diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py b/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py index bd156ecbcf..dd7e366411 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py @@ -41,7 +41,10 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_clone_index_column, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -1220,14 +1223,18 @@ def us_weeks_unemployed_summary(frame: Frame) -> dict[str, object]: in_range = integer & (values >= 0.0) & (values <= 52.0) positive = in_range & (values > 0.0) - if has_support_role_metadata(person, entity="person"): - channel = support_role_series(person, entity="person").to_numpy() - else: - channel = np.full(len(person), _ASEC_CHANNEL, dtype=object) + ( + channel, + channel_roster, + source_rows, + source_reconciliation_rows, + uc_constraint_rows, + ) = _weeks_unemployed_gate_scopes(person) channels: dict[str, dict[str, float | int]] = {} - for name in (_ASEC_CHANNEL, _PUF_CHANNEL): + for name in channel_roster: mask = channel == name channel_weight = float(weights[mask].sum()) + plausibility_role = _ASEC_CHANNEL if name == _ASEC_CHANNEL else _PUF_CHANNEL channels[name] = { "rows": int(np.count_nonzero(mask)), "positive_rows": int(np.count_nonzero(mask & positive)), @@ -1243,8 +1250,12 @@ def us_weeks_unemployed_summary(frame: Frame) -> dict[str, object]: if channel_weight > 0.0 else 0.0 ), - "positive_share_band": list(_CHANNEL_POSITIVE_SHARE_BANDS[name]), - "weighted_mean_weeks_band": list(_CHANNEL_WEIGHTED_MEAN_WEEKS_BANDS[name]), + "positive_share_band": list( + _CHANNEL_POSITIVE_SHARE_BANDS[plausibility_role] + ), + "weighted_mean_weeks_band": list( + _CHANNEL_WEIGHTED_MEAN_WEEKS_BANDS[plausibility_role] + ), } source_missing = _SOURCE not in person @@ -1253,25 +1264,32 @@ def us_weeks_unemployed_summary(frame: Frame) -> dict[str, object]: source_raw = pd.to_numeric(person[_SOURCE], errors="coerce").to_numpy( dtype=np.float64 ) - asec_mask = channel == _ASEC_CHANNEL source_valid = np.isfinite(source_raw) & (source_raw == np.floor(source_raw)) source_valid &= (source_raw == -1.0) | ( (source_raw >= 0.0) & (source_raw <= 52.0) ) - source_invalid = int(np.count_nonzero(asec_mask & ~source_valid)) + source_invalid = int(np.count_nonzero(source_rows & ~source_valid)) expected = np.where(source_raw == -1.0, 0.0, source_raw) source_mismatch = int( - np.count_nonzero(asec_mask & source_valid & finite & (values != expected)) + np.count_nonzero( + source_reconciliation_rows + & source_valid + & finite + & (values != expected) + ) ) - puf_uc_zero_mismatch = 0 + uc_constraint_mismatch = 0 if _OPTIONAL_UC_PREDICTOR in person: uc = pd.to_numeric(person[_OPTIONAL_UC_PREDICTOR], errors="coerce").to_numpy( dtype=np.float64 ) - puf_mask = channel == _PUF_CHANNEL - puf_uc_zero_mismatch = int( + uc_constraint_mismatch = int( np.count_nonzero( - puf_mask & np.isfinite(uc) & (uc <= 0.0) & finite & (values != 0.0) + uc_constraint_rows + & np.isfinite(uc) + & (uc <= 0.0) + & finite + & (values != 0.0) ) ) @@ -1284,15 +1302,86 @@ def us_weeks_unemployed_summary(frame: Frame) -> dict[str, object]: "positive_share": float(weights[positive].sum() / weights.sum()), "weighted_weeks": float(np.dot(weights, np.nan_to_num(values))), "source_missing": source_missing, + "source_rows": int(np.count_nonzero(source_rows)), + "source_reconciliation_rows": int( + np.count_nonzero(source_reconciliation_rows) + ), "source_invalid": source_invalid, "source_mismatch_count": source_mismatch, - "puf_uc_zero_mismatch_count": puf_uc_zero_mismatch, + "uc_constraint_rows": int(np.count_nonzero(uc_constraint_rows)), + "uc_constraint_mismatch_count": uc_constraint_mismatch, + "channel_roster": list(channel_roster), "channels": channels, } +def _weeks_unemployed_gate_scopes( + person: pd.DataFrame, +) -> tuple[np.ndarray, tuple[str, ...], np.ndarray, np.ndarray, np.ndarray]: + """Resolve source channels and the two reviewed constraint scopes.""" + + rows = len(person) + if not has_support_role_metadata(person, entity="person"): + all_rows = np.ones(rows, dtype=bool) + return ( + np.full(rows, _ASEC_CHANNEL, dtype=object), + (_ASEC_CHANNEL,), + all_rows, + all_rows.copy(), + np.zeros(rows, dtype=bool), + ) + + # Validate complete channel/clone metadata even when an assembled frame's + # raw source channels, rather than its compatibility roles, own the gate. + roles = support_role_series(person, entity="person") + assembled = has_assembled_support_metadata(person, entity="person") + if not assembled: + channel = roles.to_numpy(dtype=object, copy=True) + roster = tuple( + name for name in (_ASEC_CHANNEL, _PUF_CHANNEL) if np.any(channel == name) + ) + source_rows = channel == _ASEC_CHANNEL + if not np.any(source_rows): + raise ValueError("US weeks-unemployed support has no ASEC source rows.") + return ( + channel, + roster, + source_rows, + source_rows.copy(), + channel == _PUF_CHANNEL, + ) + + clone_column = support_clone_index_column("person") + source_channel = support_gate_source_channel_series( + person, + entity="person", + ).to_numpy(dtype=object, copy=True) + observed = {str(value) for value in source_channel} + if _ASEC_CHANNEL not in observed: + raise ValueError("US weeks-unemployed support has no ASEC source rows.") + roster = (_ASEC_CHANNEL, *sorted(observed - {_ASEC_CHANNEL})) + clone_index = pd.to_numeric(person[clone_column], errors="raise").to_numpy( + dtype=np.int64 + ) + native = clone_index == 0 + source_rows = source_channel == _ASEC_CHANNEL + # Native ASEC rows are the direct LKWEEKS carry. Non-native ASEC rows are + # the reviewed PUF-detail imputation; native non-ASEC rows are the reviewed + # post-transfer calibration scope. Other non-native source rows have no UC + # zero rule in either producer contract. + source_reconciliation_rows = source_rows & native + uc_constraint_rows = (source_rows & ~native) | (~source_rows & native) + return ( + source_channel, + roster, + source_rows, + source_reconciliation_rows, + uc_constraint_rows, + ) + + def us_weeks_unemployed_signal_gate(frame: Frame) -> GateResult: - """Require exact ASEC carry and integer, nondefault signal on both halves.""" + """Require exact ASEC carry and integer signal on every support channel.""" person = frame.table("person") if _OUTPUT not in person: @@ -1330,15 +1419,12 @@ def us_weeks_unemployed_signal_gate(frame: Frame) -> GateResult: f"{_OUTPUT} has {summary['source_mismatch_count']} ASEC source " "reconciliation mismatch(es)." ) - if int(summary["puf_uc_zero_mismatch_count"]): + if int(summary["uc_constraint_mismatch_count"]): failures.append( - f"{_OUTPUT} has {summary['puf_uc_zero_mismatch_count']} PUF row(s) " - "positive without unemployment compensation." + f"{_OUTPUT} has {summary['uc_constraint_mismatch_count']} row(s) " + "positive outside the unemployment-compensation constraint." ) - has_support_roles = has_support_role_metadata(person, entity="person") - required_channels = ( - (_ASEC_CHANNEL, _PUF_CHANNEL) if has_support_roles else (_ASEC_CHANNEL,) - ) + required_channels = tuple(summary["channel_roster"]) channels = summary["channels"] for name in required_channels: channel = channels[name] diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/wic_claim.py b/packages/microcosm-build/src/microcosm/build/us_runtime/wic_claim.py index 48accbcefc..f6434a7414 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/wic_claim.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/wic_claim.py @@ -46,6 +46,7 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, support_role_series, ) @@ -108,6 +109,7 @@ # protocol attests this historical salt separately from the output column. _DRAW_SALT = "would_claim_wic" _PERSON_WEIGHT_COLUMN = "person_weight" +_PERSON_SOURCE_ID_COLUMN = "person_source_id" _PERSON_SUPPORT_SOURCE_ID_COLUMN = "person_support_source_id" _SOURCE_IDENTITY_COLUMNS = ( "source_year", @@ -344,6 +346,25 @@ def _wic_categories(person: pd.DataFrame) -> np.ndarray: def _stable_person_keys(person: pd.DataFrame) -> pd.Series: + # Multispine assembly makes this ID unique across physical source + # channels while retaining it across every support clone. Prefer it before + # the historical source-local triple, whose namespaces can overlap once + # ASEC and ACS vintages align. + assembled = has_assembled_support_metadata(person, entity="person") + if assembled and _PERSON_SOURCE_ID_COLUMN not in person.columns: + raise SourceRuntimeError( + "US WIC claim multispine identity requires person_source_id." + ) + if assembled: + source_id = person[_PERSON_SOURCE_ID_COLUMN] + if source_id.isna().any(): + rows = np.flatnonzero(source_id.isna().to_numpy())[:5].tolist() + raise SourceRuntimeError( + "US WIC claim assembly source identity contains missing values at " + f"row(s): {rows}." + ) + return "source:" + source_id.astype(str) + present = [column in person.columns for column in _SOURCE_IDENTITY_COLUMNS] if any(present) and not all(present): missing = [ diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py b/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py index b571a1226d..80426c5e2b 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/workers_compensation.py @@ -28,7 +28,9 @@ run_source_stage, ) from microcosm.build.us_runtime.support_provenance import ( + has_assembled_support_metadata, has_support_role_metadata, + support_gate_source_channel_series, support_role_series, ) from microcosm.frame import Frame @@ -570,16 +572,29 @@ def us_workers_compensation_summary(frame: Frame) -> dict[str, object]: dtype=np.float64 ) source_mask = np.ones(len(person), dtype=bool) + source_reconciliation_mask = source_mask.copy() if has_support_role_metadata(person, entity="person"): source_mask = ( - support_role_series(person, entity="person").to_numpy() + support_gate_source_channel_series( + person, entity="person" + ).to_numpy() == _BASE_ASEC_SUPPORT_CHANNEL ) + source_reconciliation_mask = source_mask.copy() + if has_assembled_support_metadata(person, entity="person"): + source_reconciliation_mask &= ( + support_role_series(person, entity="person").to_numpy() + == _BASE_ASEC_SUPPORT_CHANNEL + ) source_valid = np.isfinite(source) & (source >= 0.0) + summary["source_rows"] = int(np.count_nonzero(source_mask)) + summary["source_reconciliation_rows"] = int( + np.count_nonzero(source_reconciliation_mask) + ) summary["source_invalid"] = int(np.count_nonzero(source_mask & ~source_valid)) summary["source_mismatch_count"] = int( np.count_nonzero( - source_mask + source_reconciliation_mask & source_valid & finite & ~np.isclose(values, source, rtol=0.0, atol=0.0) diff --git a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py index cc672a1599..bcac668ecc 100644 --- a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py +++ b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py @@ -32,7 +32,7 @@ [ ( "be", - "86143c1c3f98980e34490c75706fe7dbf72e96e8accb8d7d4bbd2c1ae1a29b65", + "f7fd69bf4964faa2eeea1094323ce3a5ace180d76acfff0c30ed0d592ebc129e", { "household.household_id", "person.person_id", @@ -42,7 +42,7 @@ ), ( "uk", - "0c85845b4d463638ae3e5c5a25e17de8b720794e3653c5991dc4f069d95762d3", + "e040368ecf8fb89da14ff0b8f2b1f1c8f6845a18423dfe9792c0c2abc094bbbf", { "benunit.benunit_id", "household.household_id", diff --git a/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py b/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py index 7a4e443bf0..0bd070285d 100644 --- a/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py +++ b/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py @@ -52,22 +52,22 @@ def test_us_coverage_is_exact_complete_and_honest( assert_coverage_complete(coverage_report) assert coverage_report["status"] == "pass" fields = coverage_report["field_usage"] - assert fields["configuration_field_count"] == 42_120 - assert fields["authored_normative_field_count"] == 32_351 - assert fields["resolved_binding_field_count"] == 9_769 - assert fields["consumed_field_count"] == 42_120 + assert fields["configuration_field_count"] == 42_122 + assert fields["authored_normative_field_count"] == 32_352 + assert fields["resolved_binding_field_count"] == 9_770 + assert fields["consumed_field_count"] == 42_122 assert fields["unused_field_count"] == 0 assert fields["multiple_primary_use_field_count"] == 0 assert fields["claim_count"] == 49 assert fields["mode_counts"] == { - "legacy_behavior": 13_970, - "compiler_semantic": 27_699, + "legacy_behavior": 13_971, + "compiler_semantic": 27_700, "front_end_validation": 348, "identity_only": 103, } assert fields["generation0_effect_counts"] == { - "legacy_behavior": 38_443, - "no_generation0_effect": 3_677, + "legacy_behavior": 38_444, + "no_generation0_effect": 3_678, } inventory = coverage_report["inventory_coverage"] diff --git a/packages/microcosm-build/tests/test_spec_engine_field_usage.py b/packages/microcosm-build/tests/test_spec_engine_field_usage.py index 77fad49422..fb6b525b42 100644 --- a/packages/microcosm-build/tests/test_spec_engine_field_usage.py +++ b/packages/microcosm-build/tests/test_spec_engine_field_usage.py @@ -92,22 +92,22 @@ def _mutated_bundle( def test_exact_complete_ledger_has_one_primary_mode_per_pointer(field_ledger) -> None: - assert len(field_ledger.fields) == EXPECTED_CONFIGURATION_FIELD_COUNT == 42_120 + assert len(field_ledger.fields) == EXPECTED_CONFIGURATION_FIELD_COUNT == 42_122 assert field_ledger.source_counts == { - "authored": 32_351, - "resolved_bindings": 9_769, + "authored": 32_352, + "resolved_bindings": 9_770, } assert field_ledger.mode_counts == { - "legacy_behavior": 13_970, - "compiler_semantic": 27_699, + "legacy_behavior": 13_971, + "compiler_semantic": 27_700, "front_end_validation": 348, "identity_only": 103, } assert field_ledger.generation0_effect_counts == { - "legacy_behavior": 38_443, - "no_generation0_effect": 3_677, + "legacy_behavior": 38_444, + "no_generation0_effect": 3_678, } - assert len({field.pointer for field in field_ledger.fields}) == 42_120 + assert len({field.pointer for field in field_ledger.fields}) == 42_122 def test_eligibility_concepts_are_validation_not_generation0_behavior( diff --git a/packages/microcosm-build/tests/test_spec_engine_loader.py b/packages/microcosm-build/tests/test_spec_engine_loader.py index a52518ee63..8a3e0e2321 100644 --- a/packages/microcosm-build/tests/test_spec_engine_loader.py +++ b/packages/microcosm-build/tests/test_spec_engine_loader.py @@ -235,7 +235,7 @@ def test_semantic_hash_has_golden_vector_and_surface_separation(tmp_path) -> Non # Pin the domain separator, normalization rules, schema-set receipt, and # exact normative projection as one reviewable golden vector. assert first.spec_sha256 == ( - "1ff676072985f104b9d80e3e5fa6e2078778969246246cf1371e982c772550cc" + "2473f29cdd4aef89cb7998e7effeeb14272f4aaaa91f277f48163d8f5137d1de" ) second_root = _rich_minimal(tmp_path / "xy", note="second", store="local:b") diff --git a/packages/microcosm-build/tests/test_spec_engine_seeds.py b/packages/microcosm-build/tests/test_spec_engine_seeds.py index ec1a50ab1c..c0e181a931 100644 --- a/packages/microcosm-build/tests/test_spec_engine_seeds.py +++ b/packages/microcosm-build/tests/test_spec_engine_seeds.py @@ -292,6 +292,16 @@ def test_exact_blake2b_salts_keys_candidates_and_absence_conditions() -> None: assert count.draw_condition == "only_when_declared_draw_column_is_absent" assert joint.draw_condition == "only_when_declared_draw_column_is_absent" + wic = LEGACY_V1_PROTOCOL.site("wic_claim_assignment") + assert wic.seed_material == ( + "build_model_seed", + "literal_salt=would_claim_wic", + "source:person_source_id_if_assembled_multispine", + "source_year:source_household_id:source_person_id_if_complete", + "else_support:person_support_source_id", + "else_person:person_id", + ) + vectors = ( (0, "snap_take_up", "2024:10:2", 9_193_979_365_434_741_258), (0, "immigration:ead_workers", "2024:2", 3_846_788_339_087_460_008), diff --git a/packages/microcosm-build/tests/test_us_acs_transfer.py b/packages/microcosm-build/tests/test_us_acs_transfer.py index a436f50a86..f08c84c30a 100644 --- a/packages/microcosm-build/tests/test_us_acs_transfer.py +++ b/packages/microcosm-build/tests/test_us_acs_transfer.py @@ -1388,6 +1388,48 @@ def test_discrete_year_predictions_snap_to_observed_donor_support( assert pd.api.types.is_integer_dtype(values.dtype) +def test_integer_supported_weeks_predictions_snap_to_observed_donor_support( + monkeypatch: pytest.MonkeyPatch, +) -> None: + donor_support = {0.0, 1.0, 3.0, 8.0, 13.0} + donor = _with_columns( + _donor_frame(), + "person", + { + "weeks_unemployed": [ + 0.0, + 0.0, + 0.0, + 1.0, + 3.0, + 3.0, + 8.0, + 13.0, + ] + }, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + _recipient_frame(), + donor, + target_families={ + "person": {"weeks": ("weeks_unemployed",)}, + }, + seed=2, + n_estimators=1, + ) + + values = result.frame.table("person")["weeks_unemployed"] + assert set(values).issubset(donor_support) + assert pd.api.types.is_integer_dtype(values.dtype) + contract = acs_transfer_module.acs_transfer_execution_contract_identity( + targets=("weeks_unemployed",), + ) + assert "weeks_unemployed" in contract["discrete_numeric_targets"] + + def test_engine_boolean_metadata_restores_primary_qrf_float_h5_donor( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, diff --git a/packages/microcosm-build/tests/test_us_alimony.py b/packages/microcosm-build/tests/test_us_alimony.py index d52ab7be9d..25aefd15c8 100644 --- a/packages/microcosm-build/tests/test_us_alimony.py +++ b/packages/microcosm-build/tests/test_us_alimony.py @@ -374,6 +374,96 @@ def test_signal_gate_rejects_discarded_asec_strike_benefits() -> None: assert any("does not conserve OI_VAL" in failure for failure in result.failures) +def _stacked_alimony_person() -> pd.DataFrame: + source_count = 500 + source_numbers = np.repeat(np.arange(source_count), 2) + clone_indices = np.tile([0, 1], source_count) + source_channels = np.where(source_numbers < 250, "asec", "acs") + asec_source = source_channels == "asec" + + codes = np.full(len(source_numbers), np.nan) + amounts = np.full(len(source_numbers), np.nan) + codes[asec_source] = 0.0 + amounts[asec_source] = 0.0 + reported_alimony = source_numbers == 10 + reported_strike = source_numbers == 20 + codes[reported_alimony] = 20.0 + amounts[reported_alimony] = 2_000.0 + codes[reported_strike] = 12.0 + amounts[reported_strike] = 700.0 + + alimony_income = np.where(reported_alimony, amounts, 0.0) + alimony_expense = np.where(source_numbers == 300, 3_000.0, 0.0) + strike_benefits = np.where(reported_strike, amounts, 0.0) + return pd.DataFrame( + { + "person_spine_source_id": source_numbers, + "person_support_channel": source_channels, + "person_support_clone_index": clone_indices, + "OI_OFF": codes, + "OI_VAL": amounts, + "alimony_income": alimony_income, + "alimony_expense": alimony_expense, + "strike_benefits": strike_benefits, + "miscellaneous_income": np.zeros(len(source_numbers)), + } + ) + + +def test_signal_gate_reconciles_physical_asec_rows_in_stacked_pool() -> None: + person = _stacked_alimony_person() + + result = us_alimony_signal_gate(_PersonFrame(person)) # type: ignore[arg-type] + + assert result.passed, result.failures + + asec_clone = ( + person["person_support_channel"].eq("asec") + & person["person_support_clone_index"].eq(1) + ) + person.loc[asec_clone.idxmax(), "miscellaneous_income"] = 1.0 + transferred_clone = us_alimony_signal_gate( # type: ignore[arg-type] + _PersonFrame(person) + ) + assert transferred_clone.passed, transferred_clone.failures + + asec_native = ( + person["person_support_channel"].eq("asec") + & person["person_support_clone_index"].eq(0) + ) + person.loc[asec_native.idxmax(), "miscellaneous_income"] = 1.0 + mismatch = us_alimony_signal_gate(_PersonFrame(person)) # type: ignore[arg-type] + assert not mismatch.passed + assert any("does not conserve OI_VAL" in failure for failure in mismatch.failures) + + person.loc[asec_native.idxmax(), "miscellaneous_income"] = 0.0 + person.loc[asec_clone.idxmax(), "OI_VAL"] = np.nan + invalid_source = us_alimony_signal_gate( # type: ignore[arg-type] + _PersonFrame(person) + ) + assert not invalid_source.passed + assert invalid_source.details["asec_source_invalid"] == 1 + + +def test_signal_gate_preserves_legacy_asec_puf_source_scope() -> None: + person = _stacked_alimony_person().drop(columns=["person_spine_source_id"]) + person["person_support_channel"] = np.where( + person["person_support_clone_index"].eq(0), + "asec", + "puf_tax_detail", + ) + asec = person["person_support_channel"].eq("asec") + person.loc[asec, ["OI_OFF", "OI_VAL"]] = person.loc[ + asec, ["OI_OFF", "OI_VAL"] + ].fillna(0.0) + puf = person["person_support_channel"].eq("puf_tax_detail") + person.loc[puf, ["OI_OFF", "OI_VAL"]] = np.nan + + result = us_alimony_signal_gate(_PersonFrame(person)) # type: ignore[arg-type] + + assert result.passed, result.failures + + @pytest.mark.parametrize( "person", [ diff --git a/packages/microcosm-build/tests/test_us_medicare_take_up.py b/packages/microcosm-build/tests/test_us_medicare_take_up.py index 81182315b6..0ab8d2202e 100644 --- a/packages/microcosm-build/tests/test_us_medicare_take_up.py +++ b/packages/microcosm-build/tests/test_us_medicare_take_up.py @@ -109,6 +109,21 @@ def _operation() -> SourceOperationSpec: ) +def _stacked_frame() -> Frame: + derived = with_us_medicare_take_up_input(_frame(), seed=0, time_period=2024) + stacked = clone_us_frame_for_puf_support(derived) + person = stacked.table("person") + source_record = np.tile(np.arange(5, dtype=np.int64), 2) + person["person_spine_source_id"] = source_record + person["person_support_channel"] = np.where( + source_record < 3, + "asec", + "acs", + ) + person.loc[person["person_support_channel"].eq("acs"), _SOURCE] = np.nan + return stacked + + def _derive(person: pd.DataFrame) -> pd.DataFrame: return derive_us_medicare_take_up_from_manifest(person, _operation(), None) @@ -261,6 +276,26 @@ def test_support_cloning_preserves_measured_values_on_both_channels(self) -> Non } assert us_medicare_take_up_signal_gate(cloned).passed + def test_stacked_gate_reconciles_only_physical_asec_source_rows(self) -> None: + stacked = _stacked_frame() + + gate = us_medicare_take_up_signal_gate(stacked) + + assert gate.passed, gate.failures + assert gate.details["source_rows"] == 6 + assert gate.details["source_mismatch_count"] == 0 + assert gate.details["channel_weighted_enrolled_shares"] == { + "asec": pytest.approx(0.2), + PUF_TAX_DETAIL_SUPPORT_CHANNEL: pytest.approx(0.2), + } + + person = stacked.table("person") + asec = person["person_support_channel"].eq("asec") + person.loc[person.index[asec][0], _SOURCE] = np.nan + failed = us_medicare_take_up_signal_gate(stacked) + assert not failed.passed + assert any("MCARE" in failure for failure in failed.failures) + def test_gate_rejects_missing_constant_bad_share_and_mismatch(self) -> None: missing = _frame() assert not us_medicare_take_up_signal_gate(missing).passed diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py index d17303b3b1..473daf3644 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py @@ -2465,7 +2465,7 @@ def capture_equality(expected: object, actual: object) -> None: "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "5378bb9189aec96f50da22aac71e5bd2c3d919e9795f6ef2147e0bc9c739dd8e", + "spec_sha256": "5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371", }, } diff --git a/packages/microcosm-build/tests/test_us_multispine_puf_clone.py b/packages/microcosm-build/tests/test_us_multispine_puf_clone.py index 2e0663c054..d9b3dc7b1b 100644 --- a/packages/microcosm-build/tests/test_us_multispine_puf_clone.py +++ b/packages/microcosm-build/tests/test_us_multispine_puf_clone.py @@ -10,6 +10,7 @@ BASE_ASEC_SUPPORT_CHANNEL, PUF_TAX_DETAIL_SUPPORT_CHANNEL, clone_us_frame_for_puf_support, + has_assembled_support_metadata, has_support_role_metadata, prepare_us_puf_tax_detail_chain_inputs, puf_tax_detail_clone_mask, @@ -19,7 +20,10 @@ support_role_series, support_source_id_column, ) -from microcosm.build.us_runtime.support_provenance import spine_assembly_manifest +from microcosm.build.us_runtime.support_provenance import ( + spine_assembly_manifest, + support_gate_source_channel_series, +) from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights @@ -111,6 +115,13 @@ def test_puf_clone_preserves_source_spines_and_routes_by_clone_index() -> None: PUF_TAX_DETAIL_SUPPORT_CHANNEL, PUF_TAX_DETAIL_SUPPORT_CHANNEL, ] + assert has_assembled_support_metadata(table, entity=entity) + assert support_gate_source_channel_series(table, entity=entity).tolist() == [ + "asec", + "acs", + "asec", + "acs", + ] assert puf_tax_detail_clone_mask(table, entity=entity).tolist() == [ False, False, @@ -176,6 +187,11 @@ def test_support_role_legacy_fallback_is_closed_to_known_roles() -> None: ) assert has_support_role_metadata(legacy, entity="person") + assert not has_assembled_support_metadata(legacy, entity="person") + assert support_gate_source_channel_series(legacy, entity="person").tolist() == [ + BASE_ASEC_SUPPORT_CHANNEL, + PUF_TAX_DETAIL_SUPPORT_CHANNEL, + ] assert support_role_series(legacy, entity="person").tolist() == [ BASE_ASEC_SUPPORT_CHANNEL, PUF_TAX_DETAIL_SUPPORT_CHANNEL, diff --git a/packages/microcosm-build/tests/test_us_prior_year_income.py b/packages/microcosm-build/tests/test_us_prior_year_income.py index eb20c37eb8..c4fdbcd3a4 100644 --- a/packages/microcosm-build/tests/test_us_prior_year_income.py +++ b/packages/microcosm-build/tests/test_us_prior_year_income.py @@ -410,6 +410,42 @@ def test_signal_gate_accepts_signed_source_signal_and_rejects_defaults() -> None assert "availability" in " ".join(failing.failures) +def test_clone_availability_checks_all_assembled_clones_and_legacy_pairs() -> None: + assembled = _frame( + pd.DataFrame( + { + "person_source_id": [10, 10, 10, 20, 20, 20], + "person_spine_source_id": [1, 1, 1, 2, 2, 2], + "person_support_channel": ["acs"] * 6, + "person_support_clone_index": [0, 1, 2, 0, 1, 2], + "self_employment_income_last_year": [10, 10, 10, -5, -5, -5], + "previous_year_income_available": [True, True, False] + [False] * 3, + } + ) + ) + assembled_summary = module.us_prior_year_income_summary(assembled) + assert assembled_summary["clone_availability_mismatches"] == 1 + assembled_gate = us_prior_year_income_signal_gate(assembled) + assert any("1 source person" in failure for failure in assembled_gate.failures) + + legacy = _frame( + pd.DataFrame( + { + "person_source_id": [10, 10, 20, 20], + "person_support_channel": [ + BASE_ASEC_SUPPORT_CHANNEL, + PUF_TAX_DETAIL_SUPPORT_CHANNEL, + ] + * 2, + "self_employment_income_last_year": [10, 10, -5, -5], + "previous_year_income_available": [True, True, False, False], + } + ) + ) + legacy_summary = module.us_prior_year_income_summary(legacy) + assert legacy_summary["clone_availability_mismatches"] == 0 + + def test_source_reconciliation_detects_plausible_but_wrong_asec_carry() -> None: derived = with_us_prior_year_income_inputs( _source_frame(), seed=0, time_period=2024 diff --git a/packages/microcosm-build/tests/test_us_retirement_contributions.py b/packages/microcosm-build/tests/test_us_retirement_contributions.py index db8c97d5af..c40f22d66e 100644 --- a/packages/microcosm-build/tests/test_us_retirement_contributions.py +++ b/packages/microcosm-build/tests/test_us_retirement_contributions.py @@ -14,6 +14,7 @@ derive_us_retirement_contributions_from_manifest, us_retirement_contributions_signal_gate, us_retirement_contributions_stage_spec, + us_retirement_contributions_summary, with_us_retirement_contribution_inputs, ) from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights @@ -87,6 +88,26 @@ def _derive(frame: pd.DataFrame) -> pd.DataFrame: return derive_us_retirement_contributions_from_manifest(frame, operation, None) +def _stacked_frame() -> Frame: + direct = with_us_retirement_contribution_inputs( + _frame(), + seed=0, + time_period=2024, + ) + stacked = clone_us_frame_for_puf_support(direct) + person = stacked.table("person") + source_record = np.tile(np.arange(4, dtype=np.int64), 2) + person["person_spine_source_id"] = source_record + person["person_support_channel"] = np.where( + source_record < 2, + "asec", + "acs", + ) + acs = person["person_support_channel"].eq("acs") + person.loc[acs, ["RETCB_VAL", "WSAL_VAL", "SEMP_VAL"]] = np.nan + return stacked + + def test_stage_manifest_pins_sources_operations_and_five_desired_leaves() -> None: spec = us_retirement_contributions_stage_spec() @@ -160,6 +181,25 @@ def test_with_inputs_materializes_signal_and_preserves_reported_total() -> None: assert allocated.iloc[3] == 0.0 +def test_stacked_gate_validates_physical_source_and_reconciles_direct_role() -> None: + stacked = _stacked_frame() + + gate = us_retirement_contributions_signal_gate(stacked) + + assert gate.passed, gate.failures + assert gate.details["source_rows"] == 4 + assert gate.details["source_reconciliation_rows"] == 2 + assert gate.details["allocation_mismatch_rows"] == 0 + + person = stacked.table("person") + asec_puf_role = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[person.index[asec_puf_role][0], "RETCB_VAL"] = np.nan + with pytest.raises(SourceRuntimeError, match="RETCB_VAL"): + us_retirement_contributions_summary(stacked) + + def test_puf_half_uses_qrf_predictions_and_applies_income_constraints( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_retirement_distributions.py b/packages/microcosm-build/tests/test_us_retirement_distributions.py index 5ac1b0495d..8921b521ed 100644 --- a/packages/microcosm-build/tests/test_us_retirement_distributions.py +++ b/packages/microcosm-build/tests/test_us_retirement_distributions.py @@ -28,6 +28,7 @@ derive_us_retirement_distributions_from_manifest, us_retirement_distributions_signal_gate, us_retirement_distributions_stage_spec, + us_retirement_distributions_summary, with_us_retirement_distribution_inputs, ) from microcosm.build.us_runtime.source_runtime import us_source_operation_handlers @@ -125,6 +126,29 @@ def _derive(frame: pd.DataFrame) -> pd.DataFrame: ) +def _stacked_frame() -> Frame: + direct = with_us_retirement_distribution_inputs( + _frame(), + seed=0, + time_period=2024, + ) + stacked = clone_us_frame_for_puf_support(direct) + person = stacked.table("person") + source_record = np.tile(np.arange(8, dtype=np.int64), 2) + person["person_spine_source_id"] = source_record + person["person_support_channel"] = np.where( + source_record < 4, + "asec", + "acs", + ) + acs = person["person_support_channel"].eq("acs") + person.loc[ + acs, + list(US_RETIREMENT_DISTRIBUTION_REQUIRED_SOURCE_COLUMNS), + ] = np.nan + return stacked + + def _sha256(path: Path) -> str: digest = sha256() with path.open("rb") as stream: @@ -274,6 +298,25 @@ def test_frame_integration_gate_and_idempotence() -> None: ) +def test_stacked_gate_validates_physical_source_and_reconciles_direct_role() -> None: + stacked = _stacked_frame() + + gate = us_retirement_distributions_signal_gate(stacked) + + assert gate.passed, gate.failures + assert gate.details["source_rows"] == 8 + assert gate.details["source_reconciliation_rows"] == 4 + assert all(value == 0 for value in gate.details["source_mismatches"].values()) + + person = stacked.table("person") + asec_puf_role = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[person.index[asec_puf_role][0], "DST_SC1"] = np.nan + with pytest.raises(SourceRuntimeError, match="account codes"): + us_retirement_distributions_summary(stacked) + + def test_puf_half_uses_qrf_and_asec_half_remains_exact( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_sipp_head_start.py b/packages/microcosm-build/tests/test_us_sipp_head_start.py index 010a1d2f30..52f13aa02d 100644 --- a/packages/microcosm-build/tests/test_us_sipp_head_start.py +++ b/packages/microcosm-build/tests/test_us_sipp_head_start.py @@ -429,6 +429,42 @@ def test_imputer_rejects_identical_duplicate_same_role_source_rows( impute_us_sipp_head_start(duplicate, _donor(), seed=0) +def test_assembled_clone_two_uses_lowest_clone_and_fans_to_every_clone( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(module, "QRF", _FakeQRF) + frame = _frame( + [10, 10, 10, 20, 20], + ages=[4, 4, 4, 5, 5], + female=[True, False, False, False, True], + channels=["acs"] * 5, + ) + person = frame.table("person").copy() + person["person_spine_source_id"] = [100, 100, 100, 200, 200] + person["person_support_clone_index"] = [0, 1, 2, 1, 2] + assembled = _replace_person(frame, person) + + predicted = impute_us_sipp_head_start(assembled, _donor(), seed=7) + by_source = pd.DataFrame( + { + "source": person["person_source_id"], + "value": predicted, + } + ).groupby("source")["value"] + assert (by_source.nunique() == 1).all() + assert by_source.first().to_dict() == {10: True, 20: False} + + materialized = person.copy() + materialized[_OUTPUT] = predicted.to_numpy() + summary = us_sipp_head_start_summary(_replace_person(frame, materialized)) + assert summary["clone_group_count"] == 2 + assert summary["clone_mismatch_count"] == 0 + + materialized.loc[materialized["person_support_clone_index"].eq(2), _OUTPUT] ^= True + mismatch = us_sipp_head_start_summary(_replace_person(frame, materialized)) + assert mismatch["clone_mismatch_count"] == 2 + + def test_wrapper_heals_stale_output_and_is_exactly_idempotent( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_spine_blindness.py b/packages/microcosm-build/tests/test_us_spine_blindness.py index aa6b801623..98c59af52a 100644 --- a/packages/microcosm-build/tests/test_us_spine_blindness.py +++ b/packages/microcosm-build/tests/test_us_spine_blindness.py @@ -174,6 +174,36 @@ "workers_compensation.py", ) +_GATE_SOURCE_CHANNEL_ACCESSOR = "support_gate_source_channel_series" +_GATE_SOURCE_CHANNEL_CALLERS = { + "alimony.py": frozenset({"us_alimony_signal_gate"}), + "medicare_take_up.py": frozenset({"_asec_source_mask"}), + "retirement_contributions.py": frozenset({"_asec_source_mask"}), + "retirement_distributions.py": frozenset({"_asec_source_mask"}), + "ssi_take_up.py": frozenset({"us_ssi_take_up_reporter_source_ids"}), + "weeks_unemployed.py": frozenset({"_weeks_unemployed_gate_scopes"}), + "workers_compensation.py": frozenset({"us_workers_compensation_summary"}), +} +_GATE_SOURCE_SCOPE_HELPER_CALLERS = { + ("medicare_take_up.py", "_asec_source_mask"): frozenset( + {"us_medicare_take_up_summary"} + ), + ("retirement_contributions.py", "_asec_source_mask"): frozenset( + {"_source_reconciliation_mask", "us_retirement_contributions_summary"} + ), + ( + "retirement_contributions.py", + "_source_reconciliation_mask", + ): frozenset({"us_retirement_contributions_summary"}), + ("retirement_distributions.py", "_asec_source_mask"): frozenset( + {"_source_reconciliation_mask", "us_retirement_distributions_summary"} + ), + ( + "retirement_distributions.py", + "_source_reconciliation_mask", + ): frozenset({"us_retirement_distributions_summary"}), +} + # Every runtime module must be deliberately classified. This allowlist does # not exempt a module from the all-runtime AST scan below; it only records # modules outside the migrated population-treatment registry. Keeping the @@ -3071,6 +3101,46 @@ def _called_function_names(source: str) -> set[str]: } +def _function_callers(source: str, callee: str) -> tuple[tuple[str, int], ...]: + """Return function names and lines that call one imported or local name.""" + + tree = ast.parse(source) + aliases = {callee} + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom): + continue + aliases.update( + alias.asname or alias.name + for alias in node.names + if alias.name == callee + ) + + class CallerVisitor(ast.NodeVisitor): + def __init__(self) -> None: + self.function_stack: list[str] = [] + self.callers: list[tuple[str, int]] = [] + + def visit_FunctionDef(self, node: ast.FunctionDef) -> None: + self.function_stack.append(node.name) + self.generic_visit(node) + self.function_stack.pop() + + def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None: + self.function_stack.append(node.name) + self.generic_visit(node) + self.function_stack.pop() + + def visit_Call(self, node: ast.Call) -> None: + if _call_name(node) in aliases: + caller = self.function_stack[-1] if self.function_stack else "" + self.callers.append((caller, node.lineno)) + self.generic_visit(node) + + visitor = CallerVisitor() + visitor.visit(tree) + return tuple(visitor.callers) + + def _imported_us_runtime_modules(source: str) -> tuple[str, ...]: """Return statically imported, flat ``us_runtime`` module filenames.""" @@ -3267,6 +3337,48 @@ def test_registered_population_operators_do_not_read_any_source_channel() -> Non ) +def test_physical_source_accessor_is_confined_to_reviewed_gates() -> None: + """Origin-aware diagnostics must not become population treatments.""" + + actual: dict[str, frozenset[str]] = {} + call_details: dict[str, tuple[tuple[str, int], ...]] = {} + for path in sorted(_US_RUNTIME.glob("*.py")): + if path.name == "support_provenance.py": + continue + callers = _function_callers(path.read_text(), _GATE_SOURCE_CHANNEL_ACCESSOR) + if callers: + actual[path.name] = frozenset(caller for caller, _line in callers) + call_details[path.name] = callers + assert actual == _GATE_SOURCE_CHANNEL_CALLERS, ( + "The physical support-channel accessor is restricted to reviewed " + "release gates and reporter capture; population derivation, imputation, " + f"or wrapper use is forbidden. Found callers: {call_details}" + ) + + for (module_name, helper), expected_callers in ( + _GATE_SOURCE_SCOPE_HELPER_CALLERS.items() + ): + callers = _function_callers((_US_RUNTIME / module_name).read_text(), helper) + actual_callers = frozenset(caller for caller, _line in callers) + assert actual_callers == expected_callers, ( + f"Physical source-scope helper {module_name}:{helper} may be called " + "only by its reviewed gate-summary chain; " + f"expected={sorted(expected_callers)}, found={callers}." + ) + + +def test_physical_source_accessor_guard_rejects_a_derivation_call() -> None: + source = """ +from owner import support_gate_source_channel_series as channels + +def derive_population(frame): + return channels(frame, entity=\"person\") +""" + assert _function_callers(source, _GATE_SOURCE_CHANNEL_ACCESSOR) == ( + ("derive_population", 5), + ) + + def test_pool_build_tool_import_graph_is_source_spine_blind() -> None: """The wired CLI and every runtime operator it reaches remain blind.""" diff --git a/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py b/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py index 5c52c49afa..adf0e19ca4 100644 --- a/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py +++ b/packages/microcosm-build/tests/test_us_ssi_disability_criteria.py @@ -584,6 +584,23 @@ def test_signal_gate_requires_each_channel_but_allows_clone_divergence( ) +def test_stacked_clone_divergence_diagnostic_checks_clone_two() -> None: + stacked = _replace_person( + _frame(3), + **{ + "person_source_id": np.asarray([10, 10, 10]), + "person_spine_source_id": np.asarray([1, 1, 1]), + "person_support_channel": np.asarray(["acs", "acs", "acs"]), + "person_support_clone_index": np.asarray([0, 1, 2]), + _OUTPUT: np.asarray([False, False, True]), + }, + ) + + summary = us_ssi_disability_criteria_summary(stacked) + + assert summary["clone_divergence_source_people"] == 1 + + def test_gate_requires_complete_support_provenance( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_ssi_take_up.py b/packages/microcosm-build/tests/test_us_ssi_take_up.py index 8ae286b8c7..5819e63024 100644 --- a/packages/microcosm-build/tests/test_us_ssi_take_up.py +++ b/packages/microcosm-build/tests/test_us_ssi_take_up.py @@ -150,6 +150,29 @@ def _frame(*, stale_output: bool = False) -> tuple[Frame, np.ndarray]: return frame, np.asarray(potential, dtype=np.float64) +def _stacked_frame() -> tuple[Frame, np.ndarray]: + """Convert the legacy operator fixture to physical ASEC/ACS channels.""" + + frame, potential = _frame() + person = frame.table("person").copy() + source_number = person["person_source_id"].str.rsplit(":").str[-1].astype(int) + legacy_roles = person["person_support_channel"].copy() + person["person_spine_source_id"] = person["person_source_id"] + person["person_support_clone_index"] = np.where( + legacy_roles.eq("asec"), + 0, + 1, + ) + person["person_support_channel"] = np.where( + source_number.le(6), + "asec", + "acs", + ) + acs_source = person["person_support_channel"].eq("acs") + person.loc[acs_source, US_SSI_TAKE_UP_ANCHOR] = np.nan + return _replace_person(frame, person), potential + + def _replace_person(frame: Frame, person: pd.DataFrame) -> Frame: tables = {entity: frame.table(entity).copy() for entity in frame.entities} tables["person"] = person @@ -309,6 +332,45 @@ def test_puf_only_ssi_value_is_not_promoted_to_reporter_anchor() -> None: assert diagnostics["age_bands"] == baseline_diagnostics["age_bands"] +def test_stacked_asec_sources_own_reporters_across_operator_clones() -> None: + frame, potential = _stacked_frame() + + reporter_source_ids = us_ssi_take_up_reporter_source_ids(frame) + expected = frozenset( + f"{band}:{source_number}" + for band in _AGES + for source_number in (0, 6) + ) + assert reporter_source_ids == expected + + result, diagnostics = with_us_ssi_take_up( + frame, + uncapped_ssi=potential, + seed=17, + targets=_TARGETS, + ) + assert diagnostics["reporter_anchor_lost_count"] == 0 + assert result.table("person").loc[ + result.table("person")["person_source_id"].isin(expected), _OUTPUT + ].all() + + person = frame.table("person").copy() + asec_clone = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[asec_clone.idxmax(), US_SSI_TAKE_UP_ANCHOR] = np.nan + invalid = _replace_person(frame, person) + with pytest.raises(ValueError, match="physical ASEC source rows"): + us_ssi_take_up_reporter_source_ids(invalid) + with pytest.raises(ValueError, match="physical ASEC source rows"): + with_us_ssi_take_up( + invalid, + uncapped_ssi=potential, + seed=17, + targets=_TARGETS, + ) + + def test_reporter_lineage_survives_when_l0_keeps_only_the_puf_clone() -> None: full, potential = _frame() reporter_source_ids = us_ssi_take_up_reporter_source_ids(full) diff --git a/packages/microcosm-build/tests/test_us_voluntary_filing.py b/packages/microcosm-build/tests/test_us_voluntary_filing.py index b3460662b4..304f4d63db 100644 --- a/packages/microcosm-build/tests/test_us_voluntary_filing.py +++ b/packages/microcosm-build/tests/test_us_voluntary_filing.py @@ -652,6 +652,55 @@ def test_duplicate_same_role_source_rows_fail_closed() -> None: impute_us_voluntary_filing(expanded, _donor(), seed=17) +def test_assembled_clone_two_uses_explicit_index_and_checks_every_clone() -> None: + tax_unit = pd.DataFrame( + { + "tax_unit_id": [100, 101, 102, 200, 201], + "tax_unit_source_id": [10, 10, 10, 20, 20], + "tax_unit_spine_source_id": [1, 1, 1, 2, 2], + "tax_unit_support_channel": ["acs"] * 5, + "tax_unit_support_clone_index": [0, 1, 2, 1, 2], + } + ) + receiver = pd.DataFrame( + { + predictor: np.arange(5, dtype=np.float64) + offset + for offset, predictor in enumerate( + SIPP_VOLUNTARY_FILING_MODEL_PREDICTORS + ) + }, + index=tax_unit["tax_unit_id"], + ) + + class TaxUnitFrame: + def table(self, entity: str) -> pd.DataFrame: + assert entity == "tax_unit" + return tax_unit + + prediction_rows, fan_keys = module._source_receiver_rows( + TaxUnitFrame(), receiver + ) + assert prediction_rows.index.tolist() == ["10", "20"] + assert fan_keys.tolist() == ["10", "10", "10", "20", "20"] + # Source 10 prefers clone 0; source 20 has no native survivor and picks + # the lowest surviving clone index, clone 1. + assert prediction_rows.iloc[:, 0].tolist() == [0.0, 3.0] + + summary_frame = _replace_tax_unit( + _frame(3), + **{ + _OUTPUT: np.asarray([False, False, True]), + "tax_unit_source_id": np.asarray([7, 7, 7]), + "tax_unit_spine_source_id": np.asarray([70, 70, 70]), + "tax_unit_support_channel": np.asarray(["acs", "acs", "acs"]), + "tax_unit_support_clone_index": np.asarray([0, 1, 2]), + }, + ) + summary = us_voluntary_filing_summary(summary_frame) + assert summary["clone_source_units"] == 1 + assert summary["clone_mismatch_source_units"] == 1 + + def test_real_qrf_recomputation_is_deterministic() -> None: frame = _frame(14) donor = _donor(120) diff --git a/packages/microcosm-build/tests/test_us_weeks_unemployed.py b/packages/microcosm-build/tests/test_us_weeks_unemployed.py index bd6e334195..479b114906 100644 --- a/packages/microcosm-build/tests/test_us_weeks_unemployed.py +++ b/packages/microcosm-build/tests/test_us_weeks_unemployed.py @@ -243,6 +243,46 @@ def _gate_frame() -> Frame: ) +def _stacked_gate_frame() -> Frame: + frame = _gate_frame() + person = frame.table("person").copy() + asec_rows = 1_000 + asec_native_rows = 500 + acs_native_rows = 1_000 + person["person_support_channel"] = ["asec"] * asec_rows + ["acs"] * 2_000 + person["person_support_clone_index"] = np.concatenate( + [ + np.zeros(asec_native_rows, dtype=np.int64), + np.ones(asec_rows - asec_native_rows, dtype=np.int64), + np.zeros(acs_native_rows, dtype=np.int64), + np.ones(2_000 - acs_native_rows, dtype=np.int64), + ] + ) + person["person_spine_source_id"] = np.concatenate( + [ + np.arange(asec_native_rows, dtype=np.int64), + np.arange(asec_native_rows, dtype=np.int64), + np.arange(10_000, 10_000 + acs_native_rows, dtype=np.int64), + np.arange(10_000, 10_000 + acs_native_rows, dtype=np.int64), + ] + ) + weeks = np.zeros(len(person), dtype=np.float64) + weeks[:18] = 17.0 + weeks[asec_native_rows : asec_native_rows + 12] = 17.0 + weeks[asec_rows + acs_native_rows : asec_rows + acs_native_rows + 12] = 16.0 + person[_OUTPUT] = weeks + source = np.full(len(person), np.nan, dtype=np.float64) + source[:asec_rows] = 0.0 + source[:18] = 17.0 + person["LKWEEKS"] = source + unemployment_compensation = np.zeros(len(person), dtype=np.float64) + unemployment_compensation[ + asec_native_rows : asec_native_rows + 12 + ] = 100.0 + person["unemployment_compensation"] = unemployment_compensation + return module._replace_person_table(frame, person) + + def test_public_stage_contract_is_exactly_manifest_pinned() -> None: spec = us_weeks_unemployed_stage_spec() @@ -665,6 +705,74 @@ def test_signal_gate_requires_exact_asec_and_nondefault_integer_both_channels() assert any("puf_tax_detail" in failure for failure in gate.failures) +def test_signal_gate_derives_legacy_asec_puf_roster_and_constraint_scope() -> None: + frame = _gate_frame() + summary = module.us_weeks_unemployed_summary(frame) + + assert summary["channel_roster"] == ["asec", "puf_tax_detail"] + assert summary["source_rows"] == 1_000 + assert summary["source_reconciliation_rows"] == 1_000 + assert summary["uc_constraint_rows"] == 2_000 + assert summary["uc_constraint_mismatch_count"] == 0 + + person = frame.table("person").copy() + first_puf_carrier = person.index[ + person["person_support_channel"].eq("puf_tax_detail") + & person[_OUTPUT].gt(0.0) + ][0] + person.loc[first_puf_carrier, "unemployment_compensation"] = 0.0 + gate = us_weeks_unemployed_signal_gate(module._replace_person_table(frame, person)) + + assert not gate.passed + assert any("unemployment-compensation constraint" in item for item in gate.failures) + + +def test_signal_gate_derives_stacked_asec_acs_roster_and_reviewed_scopes() -> None: + frame = _stacked_gate_frame() + gate = us_weeks_unemployed_signal_gate(frame) + summary = gate.details + + assert gate.passed + assert summary["channel_roster"] == ["asec", "acs"] + assert set(summary["channels"]) == {"asec", "acs"} + assert summary["source_rows"] == 1_000 + assert summary["source_reconciliation_rows"] == 500 + assert summary["source_invalid"] == 0 + assert summary["source_mismatch_count"] == 0 + assert summary["uc_constraint_rows"] == 1_500 + assert summary["uc_constraint_mismatch_count"] == 0 + + +def test_signal_gate_stacked_source_and_uc_checks_ignore_unowned_rows() -> None: + frame = _stacked_gate_frame() + person = frame.table("person").copy() + acs_clone = person["person_support_channel"].eq("acs") + person.loc[acs_clone, "LKWEEKS"] = 99.0 + ignored = module._replace_person_table(frame, person) + assert us_weeks_unemployed_signal_gate(ignored).passed + + asec_clone_one = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[person.index[asec_clone_one][0], "LKWEEKS"] = 99.0 + invalid_source = us_weeks_unemployed_signal_gate( + module._replace_person_table(frame, person) + ) + assert not invalid_source.passed + assert invalid_source.details["source_invalid"] == 1 + + person = frame.table("person").copy() + acs_native = person["person_support_channel"].eq("acs") & person[ + "person_support_clone_index" + ].eq(0) + person.loc[person.index[acs_native][0], _OUTPUT] = 1.0 + uc_mismatch = us_weeks_unemployed_signal_gate( + module._replace_person_table(frame, person) + ) + assert not uc_mismatch.passed + assert uc_mismatch.details["uc_constraint_mismatch_count"] == 1 + + def test_signal_gate_rejects_collapsed_puf_share_and_weighted_weeks() -> None: frame = _gate_frame() person = frame.table("person").copy() diff --git a/packages/microcosm-build/tests/test_us_wic_claim.py b/packages/microcosm-build/tests/test_us_wic_claim.py index 1375153fee..3b307e0635 100644 --- a/packages/microcosm-build/tests/test_us_wic_claim.py +++ b/packages/microcosm-build/tests/test_us_wic_claim.py @@ -47,6 +47,7 @@ RESTORED_REFERENCE_ECPS_REQUIRED_INPUTS, ) from microcosm.build.us_runtime.source_runtime import us_source_operation_handlers +from microcosm.build.us_runtime.wic_claim import _stable_person_keys from microcosm.frame import US_SCHEMA, EntitySchema, Frame, WeightKind, Weights policyengine_us_installed = importlib.util.find_spec("policyengine_us") is not None @@ -333,6 +334,43 @@ def test_draws_are_reproducible_and_keyed_by_source_identity(self) -> None: _derive(many, seed=2)[_OUTPUT].to_numpy(), ) + def test_multispine_identity_precedes_source_local_identity(self) -> None: + person = _frame( + [ + { + "person_source_id": 10, + "person_spine_source_id": 1, + "source_year": 2024, + "source_household_id": 50, + "source_person_id": 7, + }, + { + "person_source_id": 20, + "person_spine_source_id": 2, + "source_year": 2024, + "source_household_id": 50, + "source_person_id": 7, + }, + { + "person_source_id": 10, + "person_spine_source_id": 1, + "source_year": 2024, + "source_household_id": 50, + "source_person_id": 7, + }, + ] + ).table("person") + + assert _stable_person_keys(person).tolist() == [ + "source:10", + "source:20", + "source:10", + ] + + person.loc[person.index[0], "person_source_id"] = np.nan + with pytest.raises(SourceRuntimeError, match="assembly source identity"): + _stable_person_keys(person) + @pytest.mark.parametrize("column", US_WIC_CLAIM_REQUIRED_SOURCE_COLUMNS) def test_missing_source_columns_fail_closed(self, column: str) -> None: person = _frame([{}]).table("person").drop(columns=[column]) diff --git a/packages/microcosm-build/tests/test_us_workers_compensation.py b/packages/microcosm-build/tests/test_us_workers_compensation.py index 1e0fdabe90..786e28614e 100644 --- a/packages/microcosm-build/tests/test_us_workers_compensation.py +++ b/packages/microcosm-build/tests/test_us_workers_compensation.py @@ -532,6 +532,77 @@ def test_signal_gate_rejects_either_dead_support_channel(dead_channel: str) -> N assert any(dead_channel in failure for failure in gate.failures) +def _stacked_workers_compensation_frame() -> Frame: + direct = with_us_workers_compensation(_frame(), seed=0, time_period=2024) + expanded = clone_us_frame_for_puf_support(direct) + tables = {entity: expanded.table(entity).copy() for entity in expanded.entities} + person = tables["person"] + physical_asec = person["person_source_id"].le(50) + person["person_spine_source_id"] = person["person_source_id"] + person["person_support_channel"] = np.where(physical_asec, "asec", "acs") + person.loc[~physical_asec, "WC_VAL"] = np.nan + return Frame( + tables, + expanded.schema, + { + entity: expanded.weights_for(entity) + for entity in expanded.weighted_entities + }, + expanded.strata, + mass_log=expanded.mass_log, + metadata=expanded.metadata, + ) + + +def test_signal_gate_reconciles_physical_asec_rows_in_stacked_pool() -> None: + frame = _stacked_workers_compensation_frame() + + gate = us_workers_compensation_signal_gate(frame) + + assert gate.passed, gate.failures + assert gate.details["source_invalid"] == 0 + assert gate.details["source_mismatch_count"] == 0 + + person = frame.table("person") + asec_clone = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(1) + person.loc[asec_clone.idxmax(), _OUTPUT] = 1.0 + transferred_clone = us_workers_compensation_signal_gate(frame) + assert transferred_clone.passed, transferred_clone.failures + assert transferred_clone.details["source_mismatch_count"] == 0 + + asec_native = person["person_support_channel"].eq("asec") & person[ + "person_support_clone_index" + ].eq(0) + person.loc[asec_native.idxmax(), _OUTPUT] = 1.0 + mismatch = us_workers_compensation_signal_gate(frame) + assert not mismatch.passed + assert mismatch.details["source_mismatch_count"] == 1 + + person.loc[asec_native.idxmax(), _OUTPUT] = person.loc[ + asec_native.idxmax(), "WC_VAL" + ] + person.loc[asec_clone.idxmax(), "WC_VAL"] = np.nan + invalid_source = us_workers_compensation_signal_gate(frame) + assert not invalid_source.passed + assert invalid_source.details["source_invalid"] == 1 + + +def test_signal_gate_preserves_legacy_asec_puf_source_scope() -> None: + direct = with_us_workers_compensation(_frame(), seed=0, time_period=2024) + legacy = clone_us_frame_for_puf_support(direct) + person = legacy.table("person") + puf = person["person_support_channel"].eq("puf_tax_detail") + person.loc[puf, "WC_VAL"] = np.nan + + gate = us_workers_compensation_signal_gate(legacy) + + assert gate.passed, gate.failures + assert gate.details["source_invalid"] == 0 + assert gate.details["source_mismatch_count"] == 0 + + @requires_us def test_policyengine_us_1_819_0_contract_and_positive_annual_behavior() -> None: from policyengine_us import CountryTaxBenefitSystem, Simulation diff --git a/tools/spec_engine_coverage.py b/tools/spec_engine_coverage.py index 7a7a1786b2..5bd0d07d38 100644 --- a/tools/spec_engine_coverage.py +++ b/tools/spec_engine_coverage.py @@ -41,7 +41,7 @@ REPORT_SCHEMA_VERSION = 3 EXPECTED_POINTER_INVENTORY_SHA256 = ( - "bc4a948ab632191954600da8474c5b011f977a65e24c399d126f3dc4a79f23e5" + "ae7881c07c6137037f43a5992b3e8fffbbdbd5ddf2e1b70956986cb9df043f71" ) DEFAULT_REPORT_PATH = ( Path(__file__).resolve().parents[1]