Skip to content

Consolidate the US survey graph build and enrichment pipeline - #893

Closed
MaxGhenis wants to merge 294 commits into
mainfrom
microcosm-us-launch-integration-20260909
Closed

MaxGhenis wants to merge 294 commits into
mainfrom
microcosm-us-launch-integration-20260909

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Status (15 September, Max): draft until the certified default builds from raw sources and passes preflight and certification on this tree. The complete native dataset is not the merge gate. See docs/us-uk-release-path.md, "15 September decisions".

Construct one US population through ACS+ASEC harmonization, complete initial support clones, household Census-block assignment, derived geographies, financial enrichment and a PUF alternative on the combined survey frame. National, state and congressional-district analyses use this population with explicit calibration and pruning. The release path and review guide describe the architecture and remaining release work.

The graph retains source observations, reporting universes, unresolved values, design weights, relationships and geography. Prior wages are excluded. Financial and property-income stages precede the PUF alternative; subsequent amount, health and housing stages retain checked parent populations. Country models own benefit awards and caps. Unknown inputs remain unknown, and incomplete tax inputs refuse PUF qualification.

This integration includes optional child property-income completion, person-status and household-role stages. The completion host now distinguishes private donor/recipient support frames from the full US population while retaining frame, manifest and mutation checks. It also incorporates current main, including María's UK target changes in #904. The UK specification fingerprint was computed from the actual merged source/resource set before resolving the conflict. Shared calibration snapshots, best-iterate selection, fiscal measurement, geography primitives and UK lineage remain included.

Validation is scoped to the source and run recorded in the dated evidence:

  • Child property-income component: 58 pure tests and all 14 actual-owner cases pass across bounded invented-input runs. Corrected support custody: all 50 small cases pass, including mutation and permanent-revocation checks.
  • The combined 45-node cold/required-replay test stopped at its original 1,800-second CPU limit without a completion receipt. It remains unsuccessful; the 49/51-node variants are held. A separate real-function experiment passes seven correctness controls and measures per-call costs on 128/512/2,048 invented people. It does not establish whole-build attribution or an optimization benefit.
  • Property 35-node cold/replay and the corrected focused PUF handoff pass. The 38-node option passes 11 actual invented-source integration controls plus 40 smaller controls. Completion routing has 29 pure, 27 focused and three actual host tests; incomplete inputs remain incomplete.
  • Housing: 64 focused and nine actual invented full-host controls pass, including 277 replay hits and retained-population checks. Root and Fable approved that component. Earlier PUF/amount/health acceptance exercised 271 nodes and required all 271 cache hits; its 74-of-161 name inventory is not a release-coverage verdict.
  • Retirement detail: 62 controls; conservative candidate ledger: 80 controls, with no fiscal input assignment. UK lineage: 43 controls. Final observation mutation repair: five controls. Calibration parity repin: five tests covering 15 phases, followed by a byte-identical second repin.
  • Merged UK identity diagnostic and the separately executed maintained compiler/golden test each pass, with all 245 source files and 113 resource seals unchanged. The resolved UK fingerprint is 81099c1986b54ec92a3efceb72867938887cd4e7e3c9a81bb21630154824e1d9. The maintained case completed in 11.33 seconds wall time and 9.78 seconds CPU; its receipt is 2d81ad311965fb795038c1cbf9d7656b9f3e0566c44b3cc228e209ac0409b796. This validates compilation and identity, not a UK population release.
  • The installed-wheel fixture correction is incorporated but its focused runtime check remains unverified after a collection failure. The separate diagnostic/finalization repair passes 31 invented controls; that does not fix collection. Earlier six-wheel install and engine-free HDF smoke checks remain scoped evidence, not a substitute for the full wheel suite.
  • Mutation regression tests now recognize the node-population integrity check, which rejects changed inputs before the broader run-level checks. They retain the original mutation, refusal and restoration assertions. The test-only correction passed independent source review, Ruff and formatting; its expensive module fixtures were not rerun locally, and exact-head CI is pending.
  • CI file inventory verification passes for 544 tracked test files. Ruff and formatting pass on the eight files in the new correction set. Whole-module fixtures, existing process isolation and disjoint file shards remain intact. New-head CI is pending.

The latest native property/tax attempt also stopped at its 21,600-second CPU limit without a final acceptance receipt. Neither that attempt nor the 45-node test certifies a native PUF result. Repeated verification is under investigation; no larger native budget or automatic retry is implied.

Release work still includes measuring applicable missing-input routes, choosing any necessary completion models, the retirement measurement bridge, a complete small native build, held-out evaluation, national/state/CD calibration, progressive scale, full/compact exports and a dashboard tied to the accepted bytes. Historical native financial replay is useful evidence for the earlier prefix only. This draft PR publishes source and review evidence; it does not release a dataset, promote a default or deploy the site.

Reconcile with main (17 September)

Two merges bring the branch level with main at d1196af10 (74 commits, including the US engine lock at policyengine-us 2.2.1 / spm-calculator 1.0.0 / policyengine-core 3.32.5 from #936 and UK #927, #934, #937): 8ce9224e7 and e47adc6db. Six files conflicted, all identity pins or tests; pins were regenerated from the merged tree rather than taken from either side. Before the push: ruff clean; tools/ci_test_groups.py --verify ok; tools/spec_engine_coverage.py --check 42156/42156 fields and 41/41 inventory checks; the identity and pin test set 354 passed; the three lock-binding tests 13 passed (sha256(uv.lock) equals the approved lock digest in worker_identity.py and tools/spec_seed_identity_diagnostics.py).

An independent read-only review of the two merge commits found nothing lost from main (its non-text county_fips gate and tests are present beside this branch's own nine ladder tests; the branch carried no edits of its own under us/spec/, so taking main's values for those four inventory hashes dropped nothing), no 1.819 residue outside three historical comments, the eight engine parameter-file pins equal to the installed 2.2.1 RECORD, and the 15 September decisions text intact. The review's open notes are closed on the branch. graph_fiscal_measurement.py now binds the release SPM selection ({"geography_kind": "county"}, asserted equal to reform_validation.US_RELEASE_SPM_SELECTION) into the model contract and passes it to the engine adapter. The static dependency closure is blind to what the calculator-generated measurement variables read (spm_unit_spm_threshold has no input leaves in the index, although the provider reads the household county, the unit's tenure and each member's age), so a root that reaches one of those variables now declares age, county_fips and spm_unit_tenure_type as leaves: an undeclared one refuses the node declaration (UNDECLARED_MODEL_LEAF) and a missing value refuses the kernel, both before the engine runs. A test pins the measurement roster against the installed engine and calculator, so an upgrade that adds a consumer moves it. DISTRIBUTIONS in tools/spec_seed_identity_diagnostics.py now lists spm-calculator. The loader golden vector is a value neither side had because it binds the seed protocol implementation hash, which both sides moved; the test now asserts that tie.

MaxGhenis and others added 2 commits September 18, 2026 08:18
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Third merge of main into this branch, at integration head b041972
(8a7e1b1 plus this lane's journal section). main moved seventeen
commits past d1196af, all of them #938 (graph verify-once: run-scoped
source identities, streamed column digests, verification epoch on the
manifest, run-end and exception-path eviction). Nine files, every one
under packages/microcosm-graph/ plus changelog.d/graph-verify-once.changed.md.
uv.lock and every pyproject.toml are unchanged on main, so the approved
lock digest in worker_identity.py and spec_seed_identity_diagnostics.py
does not move.

One textual conflict, packages/microcosm-graph/src/microcosm/graph/executor.py:
both sides added a stdlib import at the same line (this branch `import
struct` for the exact-float digest short-circuit in _update_scalar; main
`import stat` for #938's source stat signatures). Both kept, sorted. The
resolved file is main's executor.py plus exactly this branch's seven
lines (the import and the `type(value) is float` branch), verified with
`git diff origin/main -- <file>`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
hua7450 pushed a commit to hua7450/microcosm that referenced this pull request Sep 20, 2026
…rces

The rule was decided on 15 September and written only on the PolicyEngine#893
integration branch, so main carried no authority for its own standing
check. This states the rule, and the four things a from-scratch attempt on
main measured between main and that build: a US Chronicle feed with no
dimension labels, no build stage that emits the SPM independence role, a
July selection that does not map onto a rebuilt base, and three raw inputs
that exist in one untracked directory. It records the 17 September
decisions on SPM composition and the stage times that were measured.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Sep 21, 2026
`_SPM_AGE_BANDS` was declared and never read — the vocabulary it documented
belongs in `_age_band`'s docstring, where it now lives with the reason exact
ages are not emitted.

The gate-site comment asserted "~0.05 s on a 900k-person pool" as if measured
here. It was measured by the #893 lane on the 907k-person phase-2 base pool and
recorded in the journal; the comment now says so instead of claiming it fresh.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@MaxGhenis MaxGhenis mentioned this pull request Sep 23, 2026
4 of 7 tasks
MaxGhenis added a commit that referenced this pull request Sep 23, 2026
Conflicts (release input coverage) resolved as a union of both sides:
is_spm_independent_minor_role (this branch, #893) and the three ASEC
reported-receipt inputs receives_wic/snap/tanf (#980, #978 option 1) are
all required with no reviewed exclusion; manifest counts 167 required /
174 total; derivation text names both additions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 25, 2026
atomic_geography.py, graph_atomic_geography.py and tests/test_atomic_geography.py are
the bytes at #893's 2300e56 (ebb8db8 introduced them, 1cebe35 added the gate's
validation artifact and emit_validation_artifact, which the UK node swap consumes);
they are blob-identical to #893's current head a402dfc, so #893 rebases over this
commit without conflict. Country-agnostic: their imports (keyed_uniform,
load_source_bytes, KernelRole, canonical) are on main. test_atomic_geography.py is
registered in the explicit shared-spec lane.

Verified: 37 tests passed; ci_test_groups --verify ok.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 25, 2026
Port uk_runtime/atomic_area_support.py, atomic_household_identity.py and
atomic_household_lineage.py, their tests test_uk_atomic_area_support.py and
test_uk_atomic_household_lineage.py, and docs/geography-assignment.md and
docs/uk-atomic-household-lineage.md byte-for-byte from #893 at 2300e56
(Max's b1626e1 and aaf3174, as amended on that line through 1cebe35).
The modules stay module-local imports: nothing on #932 reaches them through
uk_runtime/__init__.py, so main's eager package init is untouched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 26, 2026
atomic_geography.py, graph_atomic_geography.py and tests/test_atomic_geography.py are
the bytes at #893's 2300e56 (ebb8db8 introduced them, 1cebe35 added the gate's
validation artifact and emit_validation_artifact, which the UK node swap consumes);
they are blob-identical to #893's current head a402dfc, so #893 rebases over this
commit without conflict. Country-agnostic: their imports (keyed_uniform,
load_source_bytes, KernelRole, canonical) are on main. test_atomic_geography.py is
registered in the explicit shared-spec lane.

Verified: 37 tests passed; ci_test_groups --verify ok.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 26, 2026
Port uk_runtime/atomic_area_support.py, atomic_household_identity.py and
atomic_household_lineage.py, their tests test_uk_atomic_area_support.py and
test_uk_atomic_household_lineage.py, and docs/geography-assignment.md and
docs/uk-atomic-household-lineage.md byte-for-byte from #893 at 2300e56
(Max's b1626e1 and aaf3174, as amended on that line through 1cebe35).
The modules stay module-local imports: nothing on #932 reaches them through
uk_runtime/__init__.py, so main's eager package init is untouched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 26, 2026
atomic_geography.py, graph_atomic_geography.py and tests/test_atomic_geography.py are
the bytes at #893's 2300e56 (ebb8db8 introduced them, 1cebe35 added the gate's
validation artifact and emit_validation_artifact, which the UK node swap consumes);
they are blob-identical to #893's current head a402dfc, so #893 rebases over this
commit without conflict. Country-agnostic: their imports (keyed_uniform,
load_source_bytes, KernelRole, canonical) are on main. test_atomic_geography.py is
registered in the explicit shared-spec lane.

Verified: 37 tests passed; ci_test_groups --verify ok.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 26, 2026
Port uk_runtime/atomic_area_support.py, atomic_household_identity.py and
atomic_household_lineage.py, their tests test_uk_atomic_area_support.py and
test_uk_atomic_household_lineage.py, and docs/geography-assignment.md and
docs/uk-atomic-household-lineage.md byte-for-byte from #893 at 2300e56
(Max's b1626e1 and aaf3174, as amended on that line through 1cebe35).
The modules stay module-local imports: nothing on #932 reaches them through
uk_runtime/__init__.py, so main's eager package init is untouched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 28, 2026
atomic_geography.py, graph_atomic_geography.py and tests/test_atomic_geography.py are
the bytes at #893's 2300e56 (ebb8db8 introduced them, 1cebe35 added the gate's
validation artifact and emit_validation_artifact, which the UK node swap consumes);
they are blob-identical to #893's current head a402dfc, so #893 rebases over this
commit without conflict. Country-agnostic: their imports (keyed_uniform,
load_source_bytes, KernelRole, canonical) are on main. test_atomic_geography.py lives in
tests/engine_free/shared/ (the directory is the lane; re-stacked on #901 b2538d1).

Verified: 37 tests passed; ci_test_groups --verify ok.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 28, 2026
Port uk_runtime/atomic_area_support.py, atomic_household_identity.py and
atomic_household_lineage.py, their tests test_uk_atomic_area_support.py and
test_uk_atomic_household_lineage.py, and docs/geography-assignment.md and
docs/uk-atomic-household-lineage.md byte-for-byte from #893 at 2300e56
(Max's b1626e1 and aaf3174, as amended on that line through 1cebe35).
The modules stay module-local imports: nothing on #932 reaches them through
uk_runtime/__init__.py, so main's eager package init is untouched.

Re-stacked on #901 b2538d1: the two tests live in tests/engine_free/uk/, and the
invented three-system support helpers (parts, payloads) that the lineage test imported
from its sibling now live in test_support/microcosm_build/uk_atomic_area_support.py.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 28, 2026
atomic_geography.py, graph_atomic_geography.py and tests/test_atomic_geography.py are
the bytes at #893's 2300e56 (ebb8db8 introduced them, 1cebe35 added the gate's
validation artifact and emit_validation_artifact, which the UK node swap consumes);
they are blob-identical to #893's current head a402dfc, so #893 rebases over this
commit without conflict. Country-agnostic: their imports (keyed_uniform,
load_source_bytes, KernelRole, canonical) are on main. test_atomic_geography.py lives in
tests/engine_free/shared/ (the directory is the lane; re-stacked on #901 b2538d1).

Verified: 37 tests passed; ci_test_groups --verify ok.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 28, 2026
Port uk_runtime/atomic_area_support.py, atomic_household_identity.py and
atomic_household_lineage.py, their tests test_uk_atomic_area_support.py and
test_uk_atomic_household_lineage.py, and docs/geography-assignment.md and
docs/uk-atomic-household-lineage.md byte-for-byte from #893 at 2300e56
(Max's b1626e1 and aaf3174, as amended on that line through 1cebe35).
The modules stay module-local imports: nothing on #932 reaches them through
uk_runtime/__init__.py, so main's eager package init is untouched.

Re-stacked on #901 b2538d1: the two tests live in tests/engine_free/uk/, and the
invented three-system support helpers (parts, payloads) that the lineage test imported
from its sibling now live in test_support/microcosm_build/uk_atomic_area_support.py.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 28, 2026
atomic_geography.py, graph_atomic_geography.py and tests/test_atomic_geography.py are
the bytes at #893's 2300e56 (ebb8db8 introduced them, 1cebe35 added the gate's
validation artifact and emit_validation_artifact, which the UK node swap consumes);
they are blob-identical to #893's current head a402dfc, so #893 rebases over this
commit without conflict. Country-agnostic: their imports (keyed_uniform,
load_source_bytes, KernelRole, canonical) are on main. test_atomic_geography.py lives in
tests/engine_free/shared/ (the directory is the lane; re-stacked on #901 b2538d1).

Verified: 37 tests passed; ci_test_groups --verify ok.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 28, 2026
Port uk_runtime/atomic_area_support.py, atomic_household_identity.py and
atomic_household_lineage.py, their tests test_uk_atomic_area_support.py and
test_uk_atomic_household_lineage.py, and docs/geography-assignment.md and
docs/uk-atomic-household-lineage.md byte-for-byte from #893 at 2300e56
(Max's b1626e1 and aaf3174, as amended on that line through 1cebe35).
The modules stay module-local imports: nothing on #932 reaches them through
uk_runtime/__init__.py, so main's eager package init is untouched.

Re-stacked on #901 b2538d1: the two tests live in tests/engine_free/uk/, and the
invented three-system support helpers (parts, payloads) that the lineage test imported
from its sibling now live in test_support/microcosm_build/uk_atomic_area_support.py.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 29, 2026
atomic_geography.py, graph_atomic_geography.py and tests/test_atomic_geography.py are
the bytes at #893's 2300e56 (ebb8db8 introduced them, 1cebe35 added the gate's
validation artifact and emit_validation_artifact, which the UK node swap consumes);
they are blob-identical to #893's current head a402dfc, so #893 rebases over this
commit without conflict. Country-agnostic: their imports (keyed_uniform,
load_source_bytes, KernelRole, canonical) are on main. test_atomic_geography.py lives in
tests/engine_free/shared/ (the directory is the lane; re-stacked on #901 b2538d1).

Verified: 37 tests passed; ci_test_groups --verify ok.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 29, 2026
Port uk_runtime/atomic_area_support.py, atomic_household_identity.py and
atomic_household_lineage.py, their tests test_uk_atomic_area_support.py and
test_uk_atomic_household_lineage.py, and docs/geography-assignment.md and
docs/uk-atomic-household-lineage.md byte-for-byte from #893 at 2300e56
(Max's b1626e1 and aaf3174, as amended on that line through 1cebe35).
The modules stay module-local imports: nothing on #932 reaches them through
uk_runtime/__init__.py, so main's eager package init is untouched.

Re-stacked on #901 b2538d1: the two tests live in tests/engine_free/uk/, and the
invented three-system support helpers (parts, payloads) that the lineage test imported
from its sibling now live in test_support/microcosm_build/uk_atomic_area_support.py.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 29, 2026
atomic_geography.py, graph_atomic_geography.py and tests/test_atomic_geography.py are
the bytes at #893's 2300e56 (ebb8db8 introduced them, 1cebe35 added the gate's
validation artifact and emit_validation_artifact, which the UK node swap consumes);
they are blob-identical to #893's current head a402dfc, so #893 rebases over this
commit without conflict. Country-agnostic: their imports (keyed_uniform,
load_source_bytes, KernelRole, canonical) are on main. test_atomic_geography.py lives in
tests/engine_free/shared/ (the directory is the lane; re-stacked on #901 b2538d1).

Verified: 37 tests passed; ci_test_groups --verify ok.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 29, 2026
Port uk_runtime/atomic_area_support.py, atomic_household_identity.py and
atomic_household_lineage.py, their tests test_uk_atomic_area_support.py and
test_uk_atomic_household_lineage.py, and docs/geography-assignment.md and
docs/uk-atomic-household-lineage.md byte-for-byte from #893 at 2300e56
(Max's b1626e1 and aaf3174, as amended on that line through 1cebe35).
The modules stay module-local imports: nothing on #932 reaches them through
uk_runtime/__init__.py, so main's eager package init is untouched.

Re-stacked on #901 b2538d1: the two tests live in tests/engine_free/uk/, and the
invented three-system support helpers (parts, payloads) that the lineage test imported
from its sibling now live in test_support/microcosm_build/uk_atomic_area_support.py.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Closing as superseded: this head (a402dfc) is an ancestor of #1011's head (3de7348, branch native-integration-20260923), so nothing here is lost. The branch is kept. Whether the graph-native US line itself continues is decision US-2 in the 2026-10-09 consolidation review.

@MaxGhenis MaxGhenis closed this Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant