Materialize the Census SPM independence role in US builds

PROGRESS-spm-composition.md

@@ -31,6 +31,17 @@ review of #948):
   raises the "cannot be evaluated" `ValueError` (SKIPPED) instead of reporting
   every unit as offending.
 
+> Historical note (2026-09-18): the state and "Next" sections below describe
+> the first session. Part 2's design note shipped in PR #948
+> (`docs/us-spm-role-for-a-fresh-base.md`); Max chose its Option 1 (shape b)
+> on 2026-09-18. The current state is in "Step 2" at the end of this file.
+
+> Historical note (2026-09-22): #948 merged into main on 2026-09-22 and #959
+> was retargeted to main and re-levelled on it. The 2026-09-21 hardening
+> listed above (age bands, report cap, join guard) now lives in
+> `us_runtime/spm_composition.py`, where #959 moved the check; the preflight
+> runbook re-exports it. Check GitHub for #959's current state.
+
 ## Measured on the phase-2 base (read-only, `~/PolicyEngine/_buildq-runtime/out/base-q3/`)
 
 352,932 households / 907,382 persons / 367,306 SPM units.
@@ -101,6 +112,120 @@ review of #948):
 branch. Item 2 was not decided here; it is the open question, and the tracking
 issue — not this file — is where its verdict belongs.*
 
-1. Part 2 design note `docs/us-spm-role-for-a-fresh-base.md`.
-2. Decide (a) declared-parent generalisation vs (b) source stage; implement (a)
-   in this PR if it closes without a decision from Max.
+1. Part 2 design note `docs/us-spm-role-for-a-fresh-base.md`. — shipped in #948.
+2. Decide (a) declared-parent generalisation vs (b) source stage. — Max chose
+   (b) on 2026-09-18; implemented on branch `us-spm-role-stage`, PR #959 (see
+   Step 2 below).
+
+# Step 2 — the role as a build-stage input leaf (2026-09-18)
+
+> Historical note (2026-09-19): this section records the earlier session.
+> Implementation and actual-wrapper acceptance have since advanced; see the
+> dated continuation at the end. Check GitHub for current PR/CI status.
+
+Branch `us-spm-role-stage`, cut from `spm-composition-preflight` at
+`0e4b20de7` (PR #948's head, CI green on run 35346454375, mergeable).
+
+## State
+
+Step 1 (make #948 green) was already complete on arrival: the first session's
+`0e4b20de7` fixed the seven red jobs' one cause (the gate-failure test's fake
+frame lacked a schema; repaired in the fake, not the guard), and the rerun is
+green on every job. Step 2 design reading done; proof scripts written; design
+note next, then the stage.
+
+## Done
+
+- [x] Verified `gh pr checks 948` all pass on head `0e4b20de7`; mergeable.
+- [x] Read at this head: `spm_role_source.py` (byte-identical to main),
+      `asec_pool.py`, `relationship_inputs.py`, `education_inputs.py` (sidecar
+      pattern), the adapter's two classification paths, the engine's
+      `DATASET_SOURCE_INPUTS` declaration, the enrichment lane's pins, the
+      coverage manifest generator, the bundle generator's frozen digests, the
+      raw ASEC inputs' columns (no vintage carries `SPM_HEAD`; only 2024
+      carries `A_FAMTYP`/`A_FAMREL`).
+- [x] Located every artifact the proofs need on disk: the three pinned Census
+      person CSVs (`~/.cache/microcosm/cps/asec_education/`), the Build P
+      parent (HF blob named by its pinned digest), the certified reference
+      evidence CSV (digest matches pin 5).
+- [x] `experiments/spm_role_stage_proof.py` written (base + buildp receipts).
+
+## Next
+
+1. Run the two proofs (foreground), commit receipts.
+2. `docs/us-spm-role-stage.md`.
+3. Stage module + manifest + registries + adapter carve-out + coverage manifest
+   + export list + base/release tool wiring; regenerate every moved pin through
+   its generator; tests; changelog; draft PR.
+
+# Continuation — September 19, 2026
+
+Resumed Claude's `us-spm-role-stage` at `cff157729599c41f7ccfdadf32145d9679ffe796`
+and preserved its uncommitted fixture/doc/changelog work. Initial focused
+battery: 442 passed.
+
+Independent source review found missing operator-boundary registration and
+an impermissible synthesized default for the measured source role. Added
+boundary rejection and a regression that failed before the fix. Source inputs
+remain exportable but receive no generic default. The simulation projection
+refuses an incomplete role, and its audit records 926 inputs / 925 defaults.
+Existing role columns are now rederived and compared with the pinned Census
+source, rather than trusted because they are nonconstant. Missing provenance
+fails the signal gate. Regenerated engine/projection contracts with the
+repository generator.
+
+Revised relevant battery: **936 passed**, one country-model divide warning,
+167.45 seconds. CI test-group verification passes. Generated bundle/coverage
+checks and Fable review are underway at this journal entry.
+
+The new `experiments/spm_role_stage_wrapper_proof.py` invokes the actual stage
+and signal gate. Both pinned populations pass: Build P 166,321 persons,
+28 unresolved units to zero; base-q3 907,382 persons, 222 to zero. Every role
+matches the original derivation; Build P also reproduces the reference CSV
+bytes. Every existing column/table/weight/stratum/mass log and every input
+file remains unchanged. Receipts record exact source fingerprints and engine
+versions. The two original direct-derivation receipts are unchanged.
+
+Remaining: complete generated-artifact checks and review; push a stacked draft
+PR; integrate after acceptance. ACS-origin source roles, full-base calibration,
+incumbent comparisons and release certification remain open. These proofs do
+not establish a new released file.
+
+## Final wrapper verification and second review
+
+Fable reproduced a builder JSON failure caused by nested frozen provenance.
+The first failing serialization regression is preserved outside the repository;
+recursive conversion fixes it. The signal gate now checks provenance counts
+and an ordered person/age/unit/role binding; it rejects integer roles and
+returns a failed result for missing age. Both builder modes explicitly test
+source-path forwarding, and the prepared-pool path tests its real refusal.
+75 stage tests and 10 focused builder tests pass after these changes.
+
+The second Fable source review closed all eight original findings and reported
+no new high or medium defect. Remaining scope notes about dtype-sensitive
+bindings, historical pre-clone provenance and JSON-only metadata are in the
+design note. Review did not substitute for execution.
+
+Final v2 wrapper proofs ran at clean committed source `6a6d53b2f`: all 166,321
+Build P and 907,382 base-q3 roles match, both gates pass and serialize their
+complete details, all existing data are preserved. Peak RSS was 8.42 and
+13.28 GiB respectively. Earlier receipts remain unchanged. Generated tests
+found two stale golden counts: 38 source stages and 1,059 remaining input
+rows (992 simulation rows); the generated lock was already correct. These
+test expectations now follow the reviewed source addition.
+
+All nine selected generated-contract test files now pass across the focused
+reruns. The final three-file run passed **300 tests in 556.47 seconds**;
+earlier failure logs remain preserved and are not relabeled as green runs.
+The bundle generator's `--check` passes at spec SHA `c1893460afccc6c1c4291efe2fc4d6091f60a15079043882dfbfd2bbc09e7e00`;
+coverage is 42,174/42,174 fields and 41/41 inventory checks. Ruff and test-group
+verification pass. Draft #959 is stacked on #948; the main-only CI workflow
+does not run on that stacked base, so local checks are not a PR CI claim.
+
+> Historical note (2026-09-23): after the re-level, `47976be6c` added an
+> archive reader to `spm_role_source.py`. The note's claims that the file is
+> "byte-identical to main" and that "earlier receipts remain unchanged"
+> describe the state before that commit and before the 2026-09-23 unit-id
+> redaction of the two derivation receipts. `docs/us-spm-role-stage.md` §2
+> ("The archive reader") and §6 record both. Check GitHub for #959's current
+> state.

changelog.d/893-spm-independence-role-stage.added.md

@@ -0,0 +1 @@
+Add the `spm_independence_role` US build stage: restore the engine's one declared dataset source input, `is_spm_independent_minor_role`, on a base built from raw sources by exact income-year/PERIDNUM identity from the pinned complete Census ASEC person files through the certified `derive_spm_role_source`, fail closed on a missing source column, a unit with more or fewer than one SPM head, an unmatched person, or a unit left without a classified adult, and classify the role as an input leaf in both engine-adapter paths from `policyengine_us.spm.DATASET_SOURCE_INPUTS` so the export persists it.

docs/evidence/spec-engine/us-f0-coverage.json

@@ -4,9 +4,9 @@
     "version": 1
   },
   "country": "us",
-  "documentation_sha256": "177ac92f6c70d9eca47fbce8642ecf0d8d80eb9a02745824bfa91fc39089d102",
+  "documentation_sha256": "911b21d13cc214601cfb537c538fed2851e5166e8f72de5ebfce5f2918af43d3",
   "field_usage": {
-    "authored_normative_field_count": 32390,
+    "authored_normative_field_count": 32400,
     "claim_count": 49,
     "claims": [
       {
@@ -514,8 +514,8 @@
         ],
         "mode": "legacy_behavior",
         "pointer_class": "all",
-        "pointer_count": 1702,
-        "pointer_sha256": "be3e24b53955b9dabccfc7e779212b1e97daf0c0992be3c183e391bf695ed4ee",
+        "pointer_count": 1712,
+        "pointer_sha256": "a46d5e657d919db23ee7752d844bd8bbe59c15321dec07102837452d7994d8dd",
         "rationale": null,
         "relative_sink_prefix": "/source_manifest/stages",
         "source_prefix": "/authored/spec~1sources.yaml/stages",
@@ -722,8 +722,8 @@
         ],
         "mode": "legacy_behavior",
         "pointer_class": "all",
-        "pointer_count": 8606,
-        "pointer_sha256": "3f20975597d93f7313583a944eeb9d6437651c4ff20e67628bf6bf4c5aa9f004",
+        "pointer_count": 8614,
+        "pointer_sha256": "66eec1216f404e43f7605fba0016d1cd666e71ebe3e35278d14a944d8b48b619",
         "rationale": null,
         "relative_sink_prefix": null,
         "source_prefix": "/resolved/generated_authorities",
@@ -772,21 +772,21 @@
         "verifier": "vintages"
       }
     ],
-    "configuration_field_count": 42162,
-    "consumed_field_count": 42162,
+    "configuration_field_count": 42180,
+    "consumed_field_count": 42180,
     "generation0_effect_counts": {
-      "legacy_behavior": 38476,
+      "legacy_behavior": 38494,
       "no_generation0_effect": 3686
     },
     "mode_counts": {
       "compiler_semantic": 27717,
       "front_end_validation": 354,
       "identity_only": 103,
-      "legacy_behavior": 13988
+      "legacy_behavior": 14006
     },
     "multiple_primary_use_field_count": 0,
-    "pointer_inventory_sha256": "8b9791cd74909a425279206bddab17c8c9c1f7a5600c75c4bcf81de35311acf5",
-    "resolved_binding_field_count": 9772,
+    "pointer_inventory_sha256": "21b6fe2f6b96307f8ec7d20d4162ee2e0702f363bb28ebd3fa47a4cf7fc12ed4",
+    "resolved_binding_field_count": 9780,
     "unused_field_count": 0
   },
   "inventory_coverage": {
@@ -820,7 +820,7 @@
       "seed_sites": 53,
       "seed_streams": 14,
       "source_operators": 16,
-      "source_stages": 37,
+      "source_stages": 38,
       "stacked_checkpoint_full_components": 13,
       "stacked_checkpoint_pool_code_components": 19,
       "stacked_checkpoint_static_components": 10,
@@ -1361,12 +1361,12 @@
         ],
         "expected": {
           "producer_count": 38,
-          "sha256": "895ffd419de464a761669a5c3bf68db66e3d1a273c47631de80778a6c7f73b57"
+          "sha256": "2c51a57c2265a41503e048c2acecd2ae84871e870f2b7e84d46c13129041e551"
         },
         "failures": [],
         "observed": {
           "producer_count": 38,
-          "sha256": "895ffd419de464a761669a5c3bf68db66e3d1a273c47631de80778a6c7f73b57"
+          "sha256": "2c51a57c2265a41503e048c2acecd2ae84871e870f2b7e84d46c13129041e551"
         },
         "status": "covered"
       },
@@ -1656,13 +1656,13 @@
           "compiler_ir.node_slices"
         ],
         "expected": {
-          "map_sha256": "2db648c8740ce0713f52e34b46e764056f91c59c1754b8d553998f41ba831a6b",
-          "protocol_sha256": "0f6c58a987c54f7a129c325d4ddef449890f0f73cb140ec859dff806ccbe4f1a"
+          "map_sha256": "a6feafaf62ed99b4f3fa9fee32e986f83919aed7335848df6470db6ecd17987d",
+          "protocol_sha256": "45f82554869c7a966b3c12b71edab43a6c887806b0fc6e70cd7598e62f375922"
         },
         "failures": [],
         "observed": {
-          "map_sha256": "2db648c8740ce0713f52e34b46e764056f91c59c1754b8d553998f41ba831a6b",
-          "protocol_sha256": "0f6c58a987c54f7a129c325d4ddef449890f0f73cb140ec859dff806ccbe4f1a"
+          "map_sha256": "a6feafaf62ed99b4f3fa9fee32e986f83919aed7335848df6470db6ecd17987d",
+          "protocol_sha256": "45f82554869c7a966b3c12b71edab43a6c887806b0fc6e70cd7598e62f375922"
         },
         "status": "covered"
       },
@@ -1677,7 +1677,7 @@
           "compiler_ir.seed_stream_map"
         ],
         "expected": {
-          "implementation_sha256": "0f6c58a987c54f7a129c325d4ddef449890f0f73cb140ec859dff806ccbe4f1a",
+          "implementation_sha256": "45f82554869c7a966b3c12b71edab43a6c887806b0fc6e70cd7598e62f375922",
           "protocol": "legacy-v1",
           "streams": [
             "build_model",
@@ -1698,7 +1698,7 @@
         },
         "failures": [],
         "observed": {
-          "implementation_sha256": "0f6c58a987c54f7a129c325d4ddef449890f0f73cb140ec859dff806ccbe4f1a",
+          "implementation_sha256": "45f82554869c7a966b3c12b71edab43a6c887806b0fc6e70cd7598e62f375922",
           "protocol": "legacy-v1",
           "streams": [
             "build_model",
@@ -1775,13 +1775,13 @@
           "legacy_adapter.imputation.late_producer_resource_semantics"
         ],
         "expected": {
-          "sha256": "d6849714bb9d6f64c000193a871c324b6b82850100db6cb3196f5f6e417dc494",
+          "sha256": "a87278445d928ea20e15935a8cc07312d6e819e34409b04e8c814f17910d828f",
           "stages": 37
         },
         "failures": [],
         "observed": {
-          "sha256": "d6849714bb9d6f64c000193a871c324b6b82850100db6cb3196f5f6e417dc494",
-          "stages": 37
+          "sha256": "a87278445d928ea20e15935a8cc07312d6e819e34409b04e8c814f17910d828f",
+          "stages": 38
         },
         "status": "covered"
       },
@@ -1887,7 +1887,7 @@
             "alpha",
             "zeta"
           ],
-          "sha256": "c66844ca246fb0277e648691a91fe7bea2a21c26980b398017af90f3128521c5"
+          "sha256": "25d41aa00362261774ec88bd87f65a1c72c24eb407fcf2c517b8795c7220bdb2"
         },
         "failures": [],
         "observed": {
@@ -1911,7 +1911,7 @@
             "alpha",
             "zeta"
           ],
-          "sha256": "c66844ca246fb0277e648691a91fe7bea2a21c26980b398017af90f3128521c5"
+          "sha256": "25d41aa00362261774ec88bd87f65a1c72c24eb407fcf2c517b8795c7220bdb2"
         },
         "status": "covered"
       },
@@ -1975,7 +1975,7 @@
             "take_up_contract",
             "us_qbi_reconciliation_contract"
           ],
-          "sha256": "7fc5e7d72ba465cae5b7cc80f042c1b82c756b0a74c1ba4ebc1ed3d99e88d10d"
+          "sha256": "1794b6cccfdc5a88c5eb9075fbff036c29259ec14bb5f6ad1422654dc7b2a4c9"
         },
         "status": "covered"
       },
@@ -2599,7 +2599,7 @@
       "country": "us",
       "schema_id": "country_spec",
       "schema_version": 1,
-      "spec_sha256": "77e8def60119df8f290b40b95ee477c3830848bc7f48b6ca3efd9f9d6fcf5414"
+      "spec_sha256": "1c46dda9cff4e53e5a44c6502a46fa695d02d92731f23d618616a892071d4841"
     }
   },
   "report_schema_version": 3,
@@ -2609,7 +2609,7 @@
     "country": "us",
     "schema_id": "country_spec",
     "schema_version": 1,
-    "spec_sha256": "77e8def60119df8f290b40b95ee477c3830848bc7f48b6ca3efd9f9d6fcf5414"
+    "spec_sha256": "1c46dda9cff4e53e5a44c6502a46fa695d02d92731f23d618616a892071d4841"
   },
   "status": "pass"
 }

docs/us-spm-role-for-a-fresh-base.md

@@ -4,6 +4,33 @@ Every claim here was read at `spm-composition-preflight`'s head, which is
 `origin/main` `d1196af10` plus this branch. The installed engine is
 policyengine-us 2.2.1 / spm-calculator 1.0.0 / policyengine-core 3.32.5.
 
+> Dated note (2026-09-23): this note records the state before PR #959, which
+> implements its recommendation, Option 1, shape (b). The current design is in
+> [`docs/us-spm-role-stage.md`](us-spm-role-stage.md). With #959 merged:
+>
+> - The `spm_independence_role` build stage derives the role for a base built
+>   from raw sources. It runs before support cloning, and the release tool
+>   re-derives and reconciles it for an older base.
+> - The adapter classifies the role as an input leaf, by the engine's
+>   `DATASET_SOURCE_INPUTS`. The table in §4 that marks the role not
+>   exportable, and the generated audit's `formula_owned=True`, describe the
+>   adapter before #959.
+> - The release requires the role. It is in the release input-coverage
+>   manifest with no reviewed exclusion, it is a required non-constant person
+>   source column in the export, and the role's signal gate must pass. No
+>   release ships on the head-only fallback. A frame without a sourced role is
+>   refused; for example, a prepared stacked pool fails the role's signal gate
+>   before calibration.
+> - "A requirement that is missing today" (§1) was already wrong when
+>   written: `_reconcile_units` has refused an unresolved zero-adult SPM unit
+>   since `43171405d`.
+> - "The existing Build P enrichment lane is untouched and stays
+>   byte-identical" (§5) no longer holds for one producer file.
+>   `spm_role_source.py` gained an archive reader in `47976be6c`. The rule,
+>   the join, the refusals and the extracted-CSV path are unchanged, but the
+>   file's hash is one the lane's producer-identity check compares. See
+>   `docs/us-spm-role-stage.md` §2, "The archive reader".
+
 ## The problem in one paragraph
 
 `spm-calculator` 1.0.0 refuses the **whole population's** SPM measurement if