Repository navigation
US base v2: one CPS+ACS+PUF-detail pool; datasets labeled by exact record count (dense = full pool; exact-k L0 selection) #578
Description
Activity
UK parity audit (sol recon, 2026-07-30 — full receipts in the lane log)
Tier naming settled by lineage facts: the bundled UK suite is UKDS-licensed FRS-derived microdata in
populace-uk-private— there is no public UK entry in today's bundle at all. Token for the bundled suite:frs. The public basis is a separate Enhanced-CPS-to-UK transfer lineage (maps eligible US ECPS households into UK entities, calibrates to UK targets; its own docs say it does not represent native UK survey data) — per the provenance-token rule, that suite should becps-transfer, notpublic(a license adjective). Adoptingfrs/cps-transferunless overruled.June UK release facts (the current certified private default): 149 targets (national 22 / region 119 / country 8 — zero LA or constituency targets), 146/149 within 10%; 535,080 rows of which only 335,080 carry weight (the 200k SPI-synthetic stratum ships entirely at zero weight); Kish ESS 21,378; max weight 18,653. Two landmines the ladder work must clear:
- Region-consistency defect: June sampled English OAs country-wide, not within the household's calibrated region — only 11.97% of English records have matching FRS and OA-derived regions. Region targets fit; sub-region geography on the file is unreliable. The July OA-ladder artifact (239,023 OAs, 650 constituencies incl. NI) fixes this by design but is not wired into any release.
- Gate battery gap: the UK path has essentially none of the O/P battery — no weighted input-mass parity, no QRF tail-concentration gate, no delivered take-up gates, no whole-surface degenerate-input sweep, no failure-evidence machinery.
Exact-k readiness (both countries): not ready anywhere — the existing
target_recordsAPI is explicitly approximate (±5% or closest run), and June UK never invoked L0 at all. The #578 exact-k design (certainty units + fixed-size πps boundary draw + fixed-support refit +n_nonzero == kassertion + full battery per rung) is new machinery, as chartered.UK increment ordering (from the audit's effort-ranked backlog):
- Freeze naming/lineage (
frs,cps-transfer) — low. - Standardize diagnostics (ESS, ratios, top-1%, per-geo pass rates) — low.
- Consolidated terminal gates + degenerate sweep — low-medium.
- Port weighted integrity gates (mass parity, QRF tail) — medium.
- Delivered take-up gates on UK administrative anchors — medium-high.
- Wire the full OA ladder (fixes the region defect, adds NI's 18 seats, makes constituency coverage release-blocking) — high.
- Migrate remaining policyengine-uk-data backlog before archive #144 raw-only rebuild (the current tool still hash-gates the June processed H5 as its only accepted base) — very high; runs as its own increment.
- UK exact-k suite on the raw-only base — very high; final rung.
Items 1–6 are the launchable-parity path and don't wait on 7; the
frsk-ladder can bridge on the June-derived base with the battery enforced, with raw-only (#144) completing underneath and re-cutting the suite when done.Increment 1 (#395 ordering) is fully landed.
- Assemble spines first: pre-operator multispine seam, spine-blind operators, and the spine-agreement gate (#395, increment 1 of #578) #581 (merged 6c14a0a):
assemble_spinespre-operator seam, PUF rejected as a peer spine (clone-index routed as the tax-detail channel), 27 operators spine-blind with an AST guard, and the spine-agreement gate — incidence band [0.8, 1.25], quantile envelope 0.25, default registry covering the take-up contract + SSI. The ACS-multispine staging: SSI probe finds spine-inconsistent incidence (2.6% vs 1.9% baseline rate) and weight-composition skew (ASEC 1.5 persons/hh, ESS 0.3%) #403 repro fails the gate at ratio 0.714, as it should. - One-pool multispine builder: assemble-first build path with the agreement gate terminal (#578 increment 2) #583 (merged 136ffc9):
tools/build_us_multispine_pool.py— assemble-first build path (producer split at the pre-operator boundary), tombstone-first publication, agreement gate terminal; plus the spine-blindness guard hardened through 22 adversarial review rounds (guard battery grew 61 → 495 tests; final rounds verified by independent case oracles with zero divergences). Cumulative post-round-2 diff was guard-test-only.
Next: increment 2 — the ACS transfer re-run under the current machinery (#568/#570/#571-era), executed through the new pool builder with the agreement gate adjudicating the #403 defects on real data. First run stops at the gate (transfer + agreement adjudication, no calibration spend) so the gate verdict is the deliverable.
- Assemble spines first: pre-operator multispine seam, spine-blind operators, and the spine-agreement gate (#395, increment 1 of #578) #581 (merged 6c14a0a):
Demo-class ladder point set: k = 20,000 (Max, 2026-07-31).
The first exact-k ladder is now: k = N (full pool), k = 57,240 (today's contract), k = 20,000. No architecture is involved in adding it — k is a release parameter of the stage-2 selection (#585), and every ladder point runs the same frozen comparison register, invariant battery, and per-geography scoreboard. Whether 20,000 records carry the state surface is a number the scoreboard publishes, not a promise: the cell-coverage gates fail loudly if a k is below the feasibility floor for a target surface, per the charter.
- added a commit that references this issue
on Jul 31, 2026 Increment 3 machinery landed: exact-k selection (#585, merged 46e2e41).
select_exact_k— certainty units (π ≥ π_hi) + fixed-size Sampford πps boundary draw (exact O(Nk) dynamic program / accept-reject with fail-closed cap / complement design, all three paths cross-authenticated statistically and by mutation tests: stripping the Sampford tilt or substituting SRSWOR fails the committed joint-law and marginal tests), returning(support, six-scalar manifest receipt, aligned q). Infeasible designs error rather than clamp; complex/non-finite inputs rejected by name; deterministic given (π, k, π_hi, seed).The frozen-support refit takes the stage-2 support with the fail-closed
len(support) == kgate before and after subsetting, and benchmarks selected records at w/q (Horvitz–Thompson, unbiased for pool mass in design expectation) so the weight-cap's feasible region doesn't punish low-π records — the review's concrete adjudication case (a −16.67% target miss under a uniform rescale becoming +0.07% under w/q) is a committed test. Legacy paths byte-identical throughout.Review trail: 3 adversarial rounds + 3 fix lanes; enumerated-subset χ² verification of the joint law; 40k-case summation audits. Remaining for the increment: the k-ladder launcher + release plumbing, which land against the increment-2 pool once it exists (first ladder: k = N, 57,240, 20,000).
UK backlog items 1–3 landed (#588, merged 1de826a).
- Tier identity frozen: closed vocabulary
frs/cps-transferenforced fail-closed at every assembly surface (formatter, manifest, publication contract); quality adjectives rejected; canonical idspopulace-uk-<year>-<tier>-k<N>; exact-k ids reconcile record count/year/country across manifest, diagnostics, and target surface; the June release is grandfathered against its REAL schema-2 artifacts (committed real-shape fixtures, all 149 legacy selectors) and lineage-bound (tierfrsor absent —cps-transferon the June id rejects). - Standardized diagnostics: Kish ESS + fraction, weight-ratio stats, top-1% share, zero-weight rows by stratum (the 200k SPI-synthetic stratum is now routine emitted evidence), per-geography pass rates with LA/constituency slots ready for item 6.
- Consolidated terminal gates: one batched fail-closed report (input coverage, degenerate surface sweep, zero-weight strata, ESS floor 0.01, weight-ratio ≤ the incumbent June measurement 1151.2542195939373 as a no-regression bound purchased by the acceptance rule), report written before any raise, evidence-dependent gates present iff their stages ran — absent evidence is a named failure, absent stage is an omitted gate, never a fabricated pass. Reports are HMAC-attested (env-keyed) and identity-bound: the signed payload carries release id + diagnostics byte-hash, so a report cannot validate under any other release.
Review trail: 3 adversarial rounds + 2 fix lanes; the wheel gate caught a missing pytables skip guard post-approval (fixed per the tables-extra convention). Operational note: release automation needs
POPULACE_UK_TERMINAL_GATE_SIGNING_KEYminted before any UK release cut. Items 4–6 (weighted integrity gates, take-up anchors, OA-ladder wiring) are the next UK increments.- Tier identity frozen: closed vocabulary
- added 4 commits that reference this issue
on Aug 1, 2026 The pool input surface is now self-auditing (#597, merged 268594a; supersedes #594).
Three CI guards, all in the ordinary secrets-free PR gate, all mutation-authenticated:
- Dtype (ACS transfer accepts object-backed semantic booleans; restore hours-worked targets; receipt donor-less assets (#578 inc 2) #589): every transfer target's produced physical dtype verified against the validator's contract by running the real producers.
- Engine ownership (Transfer targets must be engine input leaves: drop formula-owned weeks_worked, fix medicare premiums leaf, add ownership guard (#578 inc 2) #590): every transfer target verified as a live input leaf of the pinned engine — formula-owned or unknown names fail by name.
- Consumer existence (Consumer-existence guard: every pool input must earn its imputation; drop deprecated marketplace leaf (#578 inc 2) #597): every pool input variable — 135 entries across transfer targets, deferred assets, primary-QRF targets, and take-up seeds — must have a demonstrated engine consumer (formula call, adds/subtracts, defined_for, parameter aggregations, constructed QBI names), or exactly one reviewed allowlist entry with a consumer class and justification (seeded with the two gate-consumed exceptions). Carrier/producer/manifest mirrors do not count as consumers. Runs in ~8–10s via a shared AST index of the engine (no engine import, fingerprint-gated, fail-closed on unreviewed engine versions and on unresolvable dynamic aggregation, both import styles).
Also in #597:
has_marketplace_health_coveragedropped from the plan (engine-deprecated, zero live consumers;_at_interviewemission unchanged) — plan is 114 targets + 3 receipted deferrals.Net effect: an engine pin bump that orphans a variable (as one did
weeks_worked), claims one, or changes a dtype now fails CI by name instead of failing a billed build. Drop receipts and the default-hole complement live in #591.7 remaining items
Filed #611: consolidate gating and reporting into one country-agnostic module.
Working item 4 (#609) surfaced a structural finding worth acting on before more gates land: the UK and US have converged on the same discipline by two different mechanisms, and a third country is about to need it.
Three gate-report formats exist today — UK
terminal_gates.json(schema-versioned, HMAC-attested, batched, membership-checked), US national per-gate keys insidecalibration_diagnostics.json(no aggregate verdict, no attestation, no membership check), and US local-areagate_summary.json(retiring withacs_local). The gap is not gate count; the US carries 66 distinct gate objects against the UK's 11. It is that a US gate's identity lives in three hand-maintained places, so nothing forces consolidation. The UK has the better container with fewer things in it, and the US has the better placement — each gate where its evidence appears, which is what lets a build fail before paying for a downstream stage.The reconciliation is a phase field: batched within a phase, fail-closed at the phase boundary.
SourceOperatorContract(multispine_pool.py:324-330) already does exactly this for operators; nobody has applied it to gates.Two things make now the moment rather than later:
- The seam already exists and is unused.
build/country_spec.pydefinesGateSelectionSpec(id, gate, criticality, parameters, notes)with anALLOWED_GATE_FUNCTIONSallowlist and per-country selection —test_country_spec.py:122already asserts BE deselectsparity/export_surfacefor having no incumbent. Nothing loads a spec and runs its gates.ALLOWED_GEOGRAPHY_SPINE_METHODSis documented in-file as "the UK OA pattern, generalized," so the module was written country-agnostic and has been waiting for a second consumer. - Belgium.
build/be/gates.jsonexists with 9 gates and needs per-country applicability by construction. Consolidating now costs one issue; discovering it afterpopulace-beships costs three migrations.
Increment 1 is the shared module with the UK as first consumer, and it is behaviour-preserving: the UK is already
phases = (preflight, terminal), so declaring that changes nothing about when gates run and the PR can assert identical per-gate verdicts against the committed June fixtures. #611 records the full consolidation goal — pool builder, then the national release tool, thengate_summary.jsonretirement — as named later increments.Two design calls in #611 worth flagging here since they touch shared doctrine:
- Write-then-block becomes an invariant with a test rather than a convention repeated three times. All three surfaces already write their report before raising; the pool builder additionally needs
marks_artifactsemantics (H5 written, manifest records the verdict) since its output is an intermediate, so the executor carries both modes. contract.pylearns battery membership from a pinnedspec.fingerprintper country, not a mirrored gate list. It cannot importpopulace-build(dependency cycle, and a verifier that derives expectations from what it verifies checks nothing), and a pinned digest keeps the verifier flat as countries are added. Same doctrine as the existing policy pin — "a threshold outside this hash is not attested" — applied to membership.
Against the UK 1–8 backlog: #611 sits under item 3's container and is a prerequisite for doing items 5–8 without re-cutting the battery each time. In particular item 5 (delivered take-up) lands as a declared
not_applicablewith a reason today — the UK has no take-up assignment surface at all (grep takes_upoveruk_runtime/returns nothing) — which turns a silent gap into a receipt. Items 1–4 are unaffected.- The seam already exists and is unused.
- added a commit that references this issue
on Aug 4, 2026 Increment 2's agreement-gate verdict is in: FAILED — and the failure is informative. Run 7c completed the full pool build (assembly → QRF → banked ACS transfer → derive/seed/simulate → agreement → publication; pool h5 + manifest + agreement diagnostics all on the volume,
simulation_ready: false).104 of 131 registered spine comparisons violate tolerance (incidence ratio bounds [0.8, 1.25], TVD ≤ 0.25, quantile envelope ≤ 0.25). The signature:
- The
puf_tax_itemizationfamily is effectively missing from the ACS spine: ASEC-over-ACS incidence ratios of 160,667× (taxable_interest_income), 9,336× (qualified_dividend_income), 4,169× (STCG), 2,244× (LTCG-before-response) — despite these being declared ACS-transfer targets whose per-target banks completed. The values die somewhere between the transfer bank and the assembled pool, or the transfer produced near-zeros. simulated_output/ssiinverts: ACS incidence ≈ 3.6× ASEC — consistent with hollow ACS income inflating simulated eligibility, i.e. plausibly downstream of the same root cause.- Scattered source-operator envelope violations (weeks_unemployed 5.1×, social_security_retirement 2.4×, ...).
A read-only forensic lane is on the artifacts now (bank-vs-pool value tracing for sentinel targets, SSI decomposition, and a review of whether the [0.8, 1.25] contract is the right registration for heavy-tailed QRF-transferred detail). Fix charter + phase-survival analysis (what banked state survives the fix) follows. The k=N ladder build holds until the pool passes.
- The
Increment-2 revision charter: the stacked spine (root cause in hand)
The forensic audit of run 7's agreement failure (full report in the review trail; receipts embedded) found the values never died in transit — the ACS transfer never ran for the PUF family:
- The primary PUF-QRF runs on both spines' clone-1 rows (2,097,745 ACS recipients alongside 231,007 ASEC), with financial predictors that are the target leaves themselves — present on ASEC, zero-filled on ACS — collapsing ACS clone-1 draws to degenerate ~$2 values. The honest native predictor (
acs_interest_dividend_rental_income, nonzero for 16.4% of ACS tax units) sits unused. - PUF finalization then globally converts remaining nulls to
0.0, so the ACS transfer's null-only activation judges all 58 PUF targets "complete" and silently skips the family — 59 of 117 targets banked, indices 25–76 and 105–110 never requested. - SSI's inversion is downstream of the hollow income surface (3.48× ACS excess of low-income elderly ≈ the 3.62× SSI gap; take-up, assets, and age mix affirmatively ruled out).
Every one of these lives in the two-spine / two-pipeline seam — clone selection ignoring source spine, one pipeline's zeros masquerading as the other's observations. Per Max's direction, the revision removes the seam rather than patching it:
Design: stack, gap-fill, one PUF pass
- Stack ASEC + ACS (pilot: seeded 10% ACS sample) into one spine with origin labels and survey-share weight harmonization (house precedent: the P-lineage pool is already a stacked-arm design; the ACS-as-agreement-spine construct was the deviation).
- Gap-fill survey-specific fields cross-survey, banked per-target (Per-target banking for the ACS transfer: preemptions mid-phase cost one target, not the phase (#578) #608 machinery): ACS←ASEC for detailed income/program fields using native ACS aggregates as predictors (the field the audit found unused); ASEC←ACS for housing. Absence stays null until filled — the null-vs-zero doctrine: no boundary may convert absence into an observed zero (the direct root cause; same doctrine class as ACS transfer accepts object-backed semantic booleans; restore hours-worked targets; receipt donor-less assets (#578 inc 2) #589's semantic booleans).
- One PUF pass over the stacked spine — imputation with predictors guaranteed present post-gap-fill, plus the PUF clone arm attached to a seeded sample of stacked households.
Gates (the statistics survive; the frame retires)
- By-origin battery replaces spine-vs-spine agreement: same comparisons, scoped to gap-filled families, with per-family declared metrics (explicit boolean incidence, support-aware rare-incidence checks, sign-separated monetary distribution checks) instead of dtype-dispatched contracts and one universal envelope. Still terminal. Tolerances are NOT widened to make anything pass — the 160,667× class fails under any profile.
- Pre-simulation completeness gate: every declared target is banked/imputed or carries an explicit source×role proof of authority — a registered family can never silently vanish from the active bank again.
Execution
Pilot at 10% ACS (fast pipeline validation, agreement-battery verdict same day) → scale per the memory envelope (observed peak 241 GB vs 330 GiB ceiling). Banked-state survival per the audit's matrix: assembled + primary-QRF banks survive where semantics allow via scoped identities; transferred/simulated rebuild (they were wrong anyway).
- The primary PUF-QRF runs on both spines' clone-1 rows (2,097,745 ACS recipients alongside 231,007 ASEC), with financial predictors that are the target leaves themselves — present on ASEC, zero-filled on ACS — collapsing ACS clone-1 draws to degenerate ~$2 values. The honest native predictor (
- added a commit that references this issue
on Aug 6, 2026 The stacked-spine revision is on main: #616 merged (86f6c83) after four adversarial rounds (r1: 3 critical fail-opens + 6 findings; r2: the caller-authority class; fix2: the canonical-authority doctrine; r3: APPROVE with every case re-executed independently and a stronger adversary — rebinding all fourteen module globals — failing closed). The architecture Max ratified is now real: one origin-labeled spine, banked cross-origin gap-fill under declared activation authority, the null-vs-zero doctrine with run 7's failure pinned byte-for-byte as a regression, one uniform PUF pass, the completeness gate, and the by-origin battery with per-column declared metrics under live-digested canonical authority.
Next: the production adoption lane (stacked path into the pool tool, the full battery registry, clone-2 tail, single
sample_fractionwith the #624 rungs), then the first 1% smoke build — local, sub-hour, end-to-end through export.- added a commit that references this issue
on Aug 7, 2026 - added a commit that references this issue
on Aug 7, 2026 - added a commit that references this issue
on Aug 10, 2026 - added a commit that references this issue
on Sep 18, 2026 - added a commit that references this issue
on Sep 21, 2026 Evaluation rule on the UK national line (microcosm#967, 2026-09-22): rule 1 (#578) is decided on the surface both artifacts can materialize. A target whose measure the incumbent cannot compute is pruned from BOTH arms only under a signed, in-force entry of the reviewed incumbent-unresolvable register (
uk/incumbent_unresolvable_measures.json); an unlisted measure refuses the evaluation. The receipt names every pruned row, its measure and the entry it stands on, and the release-cut certifier re-checks the pruned measures against the committed register. At v21 the six measures the enhanced FRS never carried remove 120 of 644 targets (UC administrative family 93, two-child limit 15, ONS household type 10, CGT asset type 2); the six entries are signed on the register as of 2026-09-22 (a8aa8684on #967), in force to 2026-12-22.(Cross-posted from #823; the register and its refusals live in the UK build package.)
Update (2026-09-22, after the rebase over #965): the signing commit on the branch is
2f97e087; the register ispackages/microcosm-build/src/microcosm/build/uk/incumbent_unresolvable_measures.json.- added a commit that references this issue
on Sep 22, 2026
The acceptance rule (stated once, enforced everywhere)
A candidate release ships iff:
Nothing else gates a ship. Adjudications and fences are receipts inside rule 2, not extra rules; the per-arm sprawl this epic retires was rule 2 metastasizing across two lineages. Loss is the objective the optimizer games — it can be rule 1 only because rule 2 checks what it can't say.
The design (ratified direction, 2026-07-29)
One base pool, one lineage, every dataset a record count:
puf_tax_detailchannel is already this).dense_no_l0.…-k57240,…-k250000, …), and the artifact carries exactly that many households, gate-asserted (len(support) == k). "sparse"/"dense" retire as dataset names; future sizes are just new k values.Exact-k selection (three stages, all seams existing)
populace-calibrate/gates.py) already yields a per-record open-probability π_i; today we threshold it. Instead, stop at π.calibrate_l0_refitsemantics). Cell-coverage gates catch any unlucky boundary draw; a k below the feasibility floor for the target surface fails loudly, never ships thin.Why now — the measured defects this must clear (the bar, not a hope)
The buildl ACS-multispine staging (#403; 1.61M households, ASEC+ACS) measured three defects that are the acceptance criteria for increment 2:
The transfer machinery these gates need did not exist in June and does now: donor joint-vector fidelity (#570's construction assertion + bijection), the deterministic mass-conserving tail-transfer stratum (#568), stage-attributable concentration gating (#571), reviewed-exclusion discipline (#564/#574), and arm-fence receipts in both manifests (#577).
Increments
Non-goals / doctrine carried forward
Context receipts
puf_tax_detailclones + 22,200asec(channel crosstab of the released H5); base pool 352,932 = 184,080 + 168,852 — zero ACS units in either arm;--acs-h5is input transfer only.Sequencing: starts after the dense-P3 diagnostic publishes (#567 close-out). Build-Q-scale epic.
Scope hardening (Max ratified 2026-07-29 evening — this section governs)
populace_us_2024_acs_local(and the Rebuild the US ACS local-area artifact on the certified Build O lineage (buildl 36de5d9a is stale) #512 lane) ends as a product: there is no "local file." Every release of the one pool carries the full geography ladder and geo targets; every file is local — the difference between releases is fidelity, and fidelity is published, not implied by a product name. The k-ladder's scoreboard reports within-tolerance by geography level (national / state / CD) per release, so choosing a k IS choosing a local sample size with its measured consequences in front of you.populace-us-2024-k57240,-k353000,-k5000— one lineage, one register, one battery, k values.populace-uk-2023-frs-k<N>(restricted distribution) andpopulace-uk-2023-public-k<N>. Rules: the tier token names what the spines are built from, never adjectives ("true", "full"); both tiers run the same register and battery; the public tier's scoreboard shows its fidelity gap to the frs tier as a published number. Tier token per country ratified with that country's onboarding. US ships tierless (single public-safe basis).Increment 4 is amended accordingly: not "overlay re-base" but overlay retirement — the acs_local consumer surface (policyengine.py named access) migrates to a k-release of the one pool, then the artifact family is closed out with a deprecation note in the bundle manifest.