Epic: publish the US default from the graph-native build

Tracks the path to publishing a certified US default built by the graph-native line, and the acceleration work along it. Opened 18 September 2026 from the data weekly's ask for an updated graph timeline.

Definition of done

tools/publish_release.sh runs on a certified US release built by the graph-native line; #893 is merged into main only after a from-scratch build, preflight and certification have run on its own tree (the rule in docs/us-release-build-rule.md, #947); main carries the same build as a standing check.

Decided 27 September 2026: the certified build reads no artifact built by policyengine-us-data code. Every input is a raw public or licensed source file pinned by digest, or an artifact microcosm builds from those files. This replaces #983. On main today the legacy base stage (tools/build_us_puf_support_base.py) still reads five such artifacts:

puf_2024.h5 from release://policyengine/irs-soi-puf/1.8.0 (--puf-h5);
the 2022 ACS rent donor acs_2022.h5 (--acs-h5);
census_cps_2022.h5, census_cps_2023.h5 and census_cps_2024.h5, mirrored unchanged to policyengine/microcosm-us-sources (--asec-h5).

The graph-native branch already reads the raw 2024 ACS PUMS and builds its PUF donor from the raw 2015 file with its own growth recipe (us_runtime/puf_target2024_growth.py). Whether any native node still reads the processed ASEC files is not yet checked. The criterion is met when the build receipt's source roster names none of the five files above and no other policyengine-us-data output.

Where we are (18 September, evening)

The stacked line Consolidate the US survey graph build and enrichment pipeline #893 ← Verify native sources once per run, not once per access and per node #935 ← Lift the whole-roster receipt ceiling and vectorise the frame seal #945 ← Lift the seven row-count ceilings a full-source native build meets, under one rule #949 / Compare replayed populations by content seal instead of by retained object #950 is merged level with main; every PR in it is mergeable. Let a seal-only population observer skip the executor's snapshot detachment #951 (executor observer flag) and Record on main that main builds the certified US default from raw sources #947 (the rule and its four blockers) are merged.
Shared inputs: Pin the processed CPS ASEC inputs to their public mirror and verify them in the base build #952 (ASEC pins, one gate round from merge), Re-pin the US Chronicle feed to a labelled export rebuilt from targeted package runs #955 (labelled Chronicle feed re-pin, CI running), the SPM independence role as a build stage (six commits on us-spm-role-stage, lane in progress), fresh lineage accepted, build-bundle emits source_label concept alignments without authority; build-consumer-artifact refuses them under consumer_fact.v3 chronicle#277 filed for the consumer-artifact refusal.
Measured: the 19-node graph completes at 1/1000 (2,028 CPU-s, 84% of it US-runtime source authentication outside the graph; a required replay with every node a store hit still costs 1,664 CPU-s). Nothing above 1/1000 has completed. The executor runs nodes sequentially. Modal is not wired for US runs.

Steps

Type: T = agent-bound, C = compute-bound, H = a decision.

Serial sum about 3–5 weeks. Current estimate (my ledger, 18 September): 10% by 1 October, 35% by 15 October, 60% by 1 November; median about 18 October. A Codex second opinion is planned for 19 September.

Acceleration, by leverage

A. Publish from main's tools first. When Pin the processed CPS ASEC inputs to their public mirror and verify them in the base build #952, Re-pin the US Chronicle feed to a labelled export rebuilt from targeted package runs #955 and the SPM stage land, run the base stage (46 min measured) and the release tool on the labelled feed and publish that certified release. The graph's launch criterion then becomes reproducing the certified default's receipts. Needs the operator's word for the build.
B. Parallel executor. Independent DAG branches execute concurrently behind a flag, with node keys, receipts and store objects byte-identical to the sequential run. Converts step 6 from days per attempt toward the longest chain. Lane brief written; PR against main.
C. Cut the 84%. Attribute the outside-graph source-authentication cost precisely and memoize what is content-addressed across nodes and runs. Same lane as B.
D. Skip the 1/10 run (step 2) unless the 1/15 run says otherwise.
E. A second machine or Modal for full runs, so fix attempts run concurrently instead of serially on the one 128 GiB box. Modal is not wired for US runs; 1–2 days.
F. Lane capacity: every reactivated Claude account is a lane; Codex returning gives Astra for the hard reviews.

Where we are (22 September, morning)

Pin the processed CPS ASEC inputs to their public mirror and verify them in the base build #952 and us: name the SPM composition defect in seconds, and refuse it by name in the release #948 merged. Re-pin the US Chronicle feed to a labelled export rebuilt from targeted package runs #955 peer-approved (round 2, no findings) after its release build learned to hold the feed to the committed pin; re-levelled on main again and merging on green CI. Accept a restated Ledger filter bound only where it agrees with the compiled constraint #969 (restated-filter guard the labelled feed needs on main's compiler) stacked on it, to re-level after. Recover ACS usual hours and validate local release inputs #923 peer-approved and merging on green CI; Gate the ACS local-area finalize on the usual-weekly-hours surface (#765) #941 in its review gate, to re-level on Recover ACS usual hours and validate local release inputs #923.
A full-scale local ACS local-area build ran on 22 September on a tree carrying Recover ACS usual hours and validate local release inputs #923 + Gate the ACS local-area finalize on the usual-weekly-hours surface (#765) #941 (the acs-local: weekly_hours_worked_before_lsr defaults to 40, silently nulling SNAP ABAWD analysis #765 fix), the Medicaid hierarchy fix from Re-pin the US Chronicle feed to a labelled export rebuilt from targeted package runs #955, a donor qualified by Qualify a pinned Build P lineage donor with the reported-receipt inputs the ACS staging gate requires #972 and the package guard from Refuse to package a capped ACS local-area staging run #973: every finalize gate green at full geography; on the ACS spine usual weekly hours went from one distinct value (all 40, the published Build O) to 100 distinct values. Receipt: Record the full-scale local ACS hours rebuild of 2026-09-22 (#765 build evidence) #974 (experiments/us-acs-local-hours-rebuild-20260922/); the donor qualification tool is Qualify a pinned Build P lineage donor with the reported-receipt inputs the ACS staging gate requires #972 and the capped-smoke package guard is Refuse to package a capped ACS local-area staging run #973; the group-quarters housing take-up flag the build exposed is ACS group-quarters records draw a housing take-up flag in the local-area transfer #975. Not published; the target contract used SOI totals mode (1,247 targets), which needs a ruling before anything is called a release candidate. This is acceleration item A's "main's tools can build" half; the graph's launch criterion (reproduce the certified default's receipts) is unchanged.
SPM consumer scope, measured (counts only): the ACS spine carries 5,246 group-quarters one-person units of minors no role can classify (Census excludes group quarters from the ACS SPM universe); the published ACS-local file has the same units. Decision 3 below.
The 1/15 native run completed on 19 September (19 nodes, 8,015 wall-s, 46.7 GB peak).

Decisions

The certification claim at step 7. A "certified release" is available on the --base-h5 arm now. Promotion as the default under US base v2: one CPS+ACS+PUF-detail pool; datasets labeled by exact record count (dense = full pool; exact-k L0 selection) #578's improvement gate needs the --exact-k arm, which needs a Chronicle consumer artifact (build-bundle emits source_label concept alignments without authority; build-consumer-artifact refuses them under consumer_fact.v3 chronicle#277). Recommendation: publish as a certified release; let US base v2: one CPS+ACS+PUF-detail pool; datasets labeled by exact record count (dense = full pool; exact-k L0 selection) #578 promotion follow the Chronicle fix.
The SPM measurement universe. Ruled 21 September: the engine-side universe rule is the fix (Separate SPM measurement scope from ordinary income policyengine-us#9462, re-levelled on main 22 September; only the "published default declares no scope" failures remain). Microcosm side: docs/us-spm-measurement-universe-producer.md design (ACS TYPEHUGQ 1 → INCLUDED, 2/3 → OUTSIDE; every ASEC record INCLUDED). Engine sub-decision ruled 22 September (Max): strict UNRESOLVED default, with the package's DEFAULT_DATASET and its sha256 moving in the same release, so #9462 lands together with the first default that declares scope; no transition rule. Microcosm-side rulings the same day on Declare the US SPM measurement universe from the source record type #977: the producer module stays out of PRODUCER_SOURCE_FILES; the missing-SPM_ID refusal stays strict; puf_tax_detail stays refusing until its arm is established; every ASEC record is INCLUDED via a named, cited constant with its own refusal. Read the SPM poverty indicator as nullable in _person_rate #976 (nullable _person_rate) merged 22 September after M4a showed all 104 state SPM rates bit-for-bit unchanged.
Merge order for the stack (step 3): downward through gates as listed, or wait and merge Consolidate the US survey graph build and enrichment pipeline #893 with everything at once after the build. Recommendation: downward now, so each piece carries its own review and the final gate reviews only the build receipt.

Not in scope

The UK graph work (María's calibration-on-graph PR, paused on #893) and the geography reordering (deferred until after the Autumn Budget, one-off test first) are tracked separately.

🤖 Generated with Claude Code

September 19 continuation update, 16:00 UTC:

The native 1/1000 cold and required-cache-replay pair passed at source 9af56aa8a7a210e18674d484b11d36f417de6759. All 19 financial and 9 prefix replay nodes were cache hits. Manifest keys, canonical content, entity counts and all 5,869 store files matched: zero added, changed or removed files. Output: 3,168 households and 6,928 people. Cold took 1,999.41 CPU seconds / 2,081.53 wall seconds; replay took 1,652.08 CPU seconds / 1,716.61 wall seconds. Whole-process peak RSS was 11,900,780,544 bytes. Aggregate report SHA256: 4e7be8adfe472a684cd8ff86a7139cffb42e8a3699d8072d02775fdd9554976d. That job is complete.

Statistical CPU attribution puts graph calls at 317.74 seconds cold versus 27.12 seconds on replay; source preparation and other work outside graph calls still take 1,624.74 CPU seconds. Another task had a concurrent population calculation, so wall timings are observational. A narrow ACS parser improvement now reuses exact-string membership indices across chunks while preserving age, orphan and roster checks. Local commit fee4aac9d7be2643fe02ec6b855d231abe5980a4 passed 185 focused tests and independent review. An invented benchmark confirms the repeated representation work is removed; no native speedup is claimed yet.

Completion-compatible memory reduction is under final synthetic testing: private directional observation witnesses release intermediate snapshots while keeping live boundary objects, mutation checks, source reauthentication and complete union-state derivation. The default executor/store behavior is unchanged. The 45-, 49- and 51-node completion cases are running in separate small test processes. A full 51-node native comparison is being prepared with separate all/compact cold/required processes; it has not run. The older 1/15 experiment remains 211,638 households / 475,728 people, 8,015.20 seconds wall and 43.46 GiB peak RSS: one resume=auto result, not required replay or certification. Full-source memory fit remains unproven.

ACS SPM structural qualification now passes on actual pilot data. Source 4d567e2e04bb6494ae895638fe5f675bda4bc0f7 and exact calculator PR45 source reconstructed 1,183 people from 512 old units into 542 SPM units, preserving all 62 group-quarters units. Both partner-role sensitivities and both tenure policies match the saved membership/regroup goldens. Eight source-uncertain households remain explicit; 21 childcare ledger rows reconcile. Every primitive table and authenticated input/control file is unchanged. No country engine, network or unapproved population read occurred. Runtime was 9.739 seconds, peak RSS 366,772,224 bytes. Report SHA256: 2409e8b410706f9b9c6ff857f06ca7a4051eee11019e1a81046419677784cf1d.

The first attempt exposed a golden-comparison bug between pandas nullable Booleans and JSON nulls; its failure remains preserved. Aggregate diagnosis found zero value differences in all 1,183 × 10 membership cells. The comparison-only fix checks missing masks, distinguishes Boolean from numeric values, and preserves exact nonmissing values, including integer precision. All 28 focused harness tests pass. The unchanged helper matrix has 173 passes with reviewed PR45 source; actually absent and installed-1.0.0 assemblers each have 132 passes and 41 strict expected refusals, zero skips/unexpected failures. The four source/harness review findings are closed. Implementation Fable review remains pending; this is a development source proof, not annual-universe, country-consumer or release acceptance. The source and public aggregate proof are now draft #962 at 9024be75da1117050eaf774fc5b2efef9bd35df3; CI is running. That later head adds documentation; the actual passing run stays pinned to 4d567e2e. A separate opt-in source adapter is now being built to construct canonical SPM membership before the initial graph CREATE, preserving tax linkage and household weights; it has no actual-data result yet.

SPM source-role integration remains draft #959, stacked on #948, head 96761c63526eaca6a17e3bf0cc5109882edebe7f. Actual wrapper/gate proofs at implementation 6a6d53b2fb2cad7ac7b84634decc2a41125efa92 pass on Build P (166,321 people) and base-q3 (907,382 people): roles match the pinned Census derivation, 28 and 222 unresolved units become zero, and existing inputs/data remain unchanged. Gate JSON serialization is exercised. Fable closed eight prior findings with no new high/medium defect. All nine selected generated-contract files passed across focused runs. Its feature-branch base does not trigger main-only PR CI.

The country-model reconciliation now has a clean local source result at 6f6d7d33ca6a7e37508bb94427ec0721d1c02f82: 167 final affected tests, 44 YAML cases and 13 additional baseline YAML checks pass, with both independent reviews approving after a county-typing collision was fixed. It preserves main's separation of ordinary housing benefits from SPM valuation. Exact Microdf compatibility was checked on the same invented fixtures: 1.3.0 passes all 43 required checks, while locked 1.2.1 fails 10, principally missing-aware counts. Both already handle direct missing-aware means. Tied-income ranks change; the wrapper already locks 1.3.0 and documents that tie convention. A minimal isolated country pin/lock candidate and a bounded 512B consumer harness are being prepared. Generic Gini-shift and missing-quantile limitations remain separate and no blanket dependency/no-worse claim is made. The old conflicting #9462 has not been substituted or merged. Actual population/wrapper/universe qualification remains outstanding.

Chronicle publisher-authority fix PolicyEngine/chronicle#278 passed both exact-head CI runs and is ready for review, not merged. The valid US artifact contains 39,158 rows and 586 record-set/period pairs. Microcosm consumer draft #961, stacked on #955, is at 112a2d2069bf7a1bc8e83b75019cd9fd84566422. All fact values and 32,867 target identities/hierarchies remain unchanged; 994 fact rows and 166 targets have expected provenance changes. Facts SHA256 4d1dba8c1b6274877bf184fa6de5d99b13fc61f34709ccab1487db2b5c64a79f, manifest SHA256 38ec5bf1efe5a0bd017ec5279065e2ea7645b37da237197f03ae2fbca28cadac. Its 195-test integration battery, two reporting regressions and all 13 renewed real-artifact comparison checks pass. Fable reporting findings are closed. The stacked consumer has no main-only PR CI run.

The #958 owner continues #960 targets plus full-vector tail support; #960 now has green CI, but target-only adoption remains unqualified. The pre-existing ten-year tail calculation finished, freeing memory for the next native study. The #779 owner controls the separate humanitarian-status integration and affected schema/spec pins. No new certified file, incumbent no-worse result, promotion or publication is claimed. Release-critical work remains SPM consumer/dependency qualification, full native graph execution within local resources, calibration and matched national/congressional-district acceptance.

State on 2026-09-22 midday:

Merged since the last note: Pin the processed CPS ASEC inputs to their public mirror and verify them in the base build #952 (ASEC source pins), us: name the SPM composition defect in seconds, and refuse it by name in the release #948 (SPM composition preflight).
Approved by gate, merging on green CI (the org runner queue is slow today): Re-pin the US Chronicle feed to a labelled export rebuilt from targeted package runs #955 (labelled Chronicle feed pin, round 2 approve), Recover ACS usual hours and validate local release inputs #923 (native ACS usual hours, round 2 approve), Gate the ACS local-area finalize on the usual-weekly-hours surface (#765) #941 (hours finalize gate; round 4 approve after three fix rounds; the superseded review is dismissed as addressed per Max), Refuse to package a capped ACS local-area staging run #973 (package guard refuses capped smokes; round 1 approve). Recover ACS usual hours and validate local release inputs #923 is re-levelled on Gate the ACS local-area finalize on the usual-weekly-hours surface (#765) #941 locally (both gates kept on the same finalize seam) and pushes once Gate the ACS local-area finalize on the usual-weekly-hours surface (#765) #941 lands; Accept a restated Ledger filter bound only where it agrees with the compiled constraint #969 re-levels on Re-pin the US Chronicle feed to a labelled export rebuilt from targeted package runs #955 after it lands.
In gate: Qualify a pinned Build P lineage donor with the reported-receipt inputs the ACS staging gate requires #972 (donor receipt qualification; round 3 after a fail-closed fix), Record the full-scale local ACS hours rebuild of 2026-09-22 (#765 build evidence) #974 (the full-scale local hours rebuild record; round 3 after two rounds of file-backed corrections).
Full-scale local hours rebuild completed 2026-09-22 03:50: all eight finalize gates green at full geography on SOI totals mode (1,247 targets), usual hours on the ACS spine 100 distinct values against the published Build O's constant 40. Not published: a publish attempt was refused by the release contract because the donor is a receipt-qualified child, not a published release. Filed as its own issue with three fix paths (fold the receipt columns into the national build is the root-cause one).
SPM measurement universe: Declare the US SPM measurement universe from the source record type #977 (source producer, TYPEHUGQ-based) and Read the SPM poverty indicator as nullable in _person_rate #976 (nullable person rate) open as drafts from the design note; policyengine-us#9462 carries the engine-side rule and needs the compatibility ruling.
PolicyBench: a lane is building the comparison of a 100-record exact-k calibration against PolicyBench's PPS-sampled 100 on the same June 11 release (record PR to follow).

State on 2026-09-23, 05:30 EDT:

Merged since the last note: #969 (restated Ledger filters), #980, #985 (the reported-receipt enrichment as a source_enrichment operation), #984 (the ACS local default SOI surface is state, Build O/P's 4,459-target contract; totals and full are opt-in), #986 (restated age-band filter), #977 (SPM measurement-universe producer, not wired into any build), #988 (new-lineage mode for the release-gate preflight).

Published overnight, tag-only with --no-latest, after every gate passed (latest.json still points at populace-us-2024-spm-20260915):

Donor populace-us-2024-spm-receipts-20260923: the reported-receipt child of the national default (sha256 00946972…), certified against the four pinned wheels.
ACS local release populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z (sha256 769756c3…), built on that donor with the state surface: calibration loss 0.0155, 97.58% of targets within 10%, Kish ESS 13,631, and all eight finalize gates green. It fixes acs-local: weekly_hours_worked_before_lsr defaults to 40, silently nulling SNAP ABAWD analysis #765's constant 40-hour usual hours on the ACS spine. Tracked as the non-default local-area file.

Acceleration A (publish from main's tools) is running. The driver makes a local release candidate and never publishes.

The base stage ran from raw sources on Materialize the Census SPM independence role in US builds #959's head (main plus the SPM independence role stage) with the pinned feed: 50 min, 79 GB peak.
- The first attempt found that the role loader rejected the official Census ASEC archives the base stage passes. Fixed on Materialize the Census SPM independence role in US builds #959 (47976be6c): the loader reads the pinned member in the archive and checks the archive and member digests.
The new-lineage base preflight passed after Compile the preflight's target surface through the release tool's CD crosswalk #990 fixed it to compile targets through the release tool's congressional-district vintage crosswalk. Without the crosswalk, the TY2023 SOI district file's three pre-2022 West Virginia districts broke the compile. Results:
- capital-gains own-tail present (15,228 donors)
- zero-support preview clean
- smoke-probe support clean
- all 367,306 SPM units have a classified adult, with the role read from the source column
The release tool (--base-h5, no selection source) started at 05:16. Its runtime at this scale has not been measured before.

In review: #959 (SPM role stage; review found no code defect; doc fixes in progress; the merged code will be the code the route A build ran), #987 (a US build stage on Modal; hardening fixes in progress), #990 (the preflight crosswalk fix; peer-approved, merging on green CI), #989 (draft record: 100-record exact-k against PolicyBench's sampled 100 on the same release).

Modal (acceleration E) measured. The acceptance replay of the full-scale materialize was preempted twice (at 54 min and 2 h 4 min) and was stopped at about $3.62 list price without a receipt. Per chunk it ran 3.3 to 7.6 times slower than the local machine, bound to one core. #987 adds an opt-in non-preemptible placement at three times the list price. Uploading the restricted PUF inputs to Modal for the base stage awaits a decision.

SPM measurement scope (decision 3): no release can declare spm_unit_spm_universe_status yet. PolicyEngine/policyengine-us#9462 gives the variable a formula (households INCLUDED, datasets UNRESOLVED), and Microcosm's adapter does not export formula-owned variables. The design note's Q-A (an upstream source-declaration marker, with a Microcosm allowlist as a bridge) awaits a ruling, and until then #9462 cannot land with the default.

🤖 Generated with Claude Code

Merge main (6442030) and poverty-comparison-only (07b69be) into the n…

Bind terminal-gate evidence into the release manifests (Route A PR-3)

Score the post-export release gates over household batches (Route A P…

Fence reviewed exclusions at every vintage and restore the M-CHIP CHI…

Bind the staged frame into the target-frame checkpoint identity (Rout…

Batch the target materializer's base simulation (Route A PR-5, F-1)

Arm the batch-invariance guard for pre-sliced callers and every engin…

Navigation Menu

Repository navigation

Epic: publish the US default from the graph-native build #956

Definition of done

Where we are (18 September, evening)

Steps

Acceleration, by leverage

Where we are (22 September, morning)

Decisions

Not in scope

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Epic: publish the US default from the graph-native build #956

Description

Definition of done

Where we are (18 September, evening)

Steps

Acceleration, by leverage

Where we are (22 September, morning)

Decisions

Not in scope

Activity

MaxGhenis commented on Sep 19, 2026

MaxGhenis commented on Sep 22, 2026

MaxGhenis commented on Sep 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions