Give Connecticut blocks their CBSA through 2022 planning regions

[MaxGhenis]

Summary

Every Connecticut block in the US block ladder had cbsa_code == 0. assemble_us_block_ladder looked each block's CBSA up by its 2020 county prefix (09001–09015). The OMB 2023 delineation file (list1_2023.xlsx) lists Connecticut by its nine 2022 planning regions instead (county-equivalents 09110–09190), so no CT block matched. Both local ladders carry the defect: the Route A input (7ba39b95…) and the PUMA-carrying rebuild for #1047 (9f2dc986…). Connecticut was the only state with an all-zero CBSA.

This PR maps each CT block to its planning region through its town, then looks the CBSA up by that county-equivalent:

• The 2020 BAF county-subdivision layer (BlockAssign_ST09_CT_MCD.txt, already inside the BAF zip the builder downloads) gives block → 2020 county + town. 2020 blocks nest in towns.
• Census's official crosswalk ct_cou_to_cousub_crosswalk.txt gives town → planning region. In that file every town keeps its county-subdivision FIPS, and the new file is downloaded, cached and sha-recorded like every other source.
• The OMB delineation then gives planning region → CBSA.

The CBSA layer's vintage string stays omb_2023_delineations, because the delineation vintage did not change.

Changes

packages/microcosm-build/src/microcosm/build/us_runtime/block_ladder_sources.py (not byte-hashed):

• parse_baf_county_subdivision_file parses the MCD layer. It refuses a row whose COUNTYFP disagrees with the block geoid.
• parse_ct_planning_region_crosswalk handles the file's layout: the quoted two-line header cells, BOM, trailing glossary, and the redundant old/new geoid columns, which are cross-checked. It skips the "County subdivisions not defined" rows (00000, coastal water). Fairfield County's is listed under two planning regions, so the code names no single region, and no populated 2020 block carries it.
• ct_planning_region_by_block refuses a CT block with no town, or a town the crosswalk lacks.
• assemble_us_block_ladder(..., cbsa_county_by_block=...) looks each block up by its delineation county-equivalent. That is its 2020 county unless the override map names another. County-equivalents stay int64 and the remap is applied with searchsorted, so the join no longer holds a Python string per national block.
• cbsa_delineated_states and us_block_ladder_cbsa_coverage_failures(ladder) are the loader-level check over a loaded UsBlockLadder. On a ladder this join just accepted the check cannot fail. Its job is ladders built elsewhere or before the join refused unreached county-equivalents: on the Route A artifact it reports exactly state 09.

tools/build_us_block_ladder_artifact.py:

• Parses CT's MCD layer and downloads the crosswalk when CT is in the build. A CT build without the MCD layer stops before writing and names the missing file.
• Records cbsa_delineated_states in the metadata.
• Runs the coverage check right after the existing load_us_block_ladder self-check.
• Adds the per-state CBSA population share to the summary.

geography_ladder.py (the loader and gate) and every other _DIRECT_KERNEL_MODULES entry are untouched.

Invariants

Each of these holds for every input and is enforced by a test:

1. The join refuses exactly when coverage is broken. For a remap that covers every populated block of its states, assemble_us_block_ladder raises if and only if some delineated county-equivalent in a built state is reached by no populated block. A partial remap is refused separately, by invariant 5. The pre-fix CT join now fails loudly instead of emitting zeros. Rows outside the built states (Puerto Rico, or CT in a smoke build of other states) are ignored. Tests: Hypothesis test_cbsa_join_invariants_hold_for_generated_ladders, plus test_assemble_refuses_ct_blocks_keyed_by_2020_county and test_assemble_ignores_delineation_rows_outside_the_built_states.
2. Lookup semantics. When the join accepts, cbsa_code[i] == delineation.get(county_equivalent(block_i), 0). Tested by the same Hypothesis test.
3. Isolation. Every array except cbsa_code is independent of the CBSA inputs, so the delineation and the override map change nothing else. Tested by the Hypothesis test, and checked on the real national rebuild below.
4. Builder implies loader. A ladder the join accepts, with its recorded cbsa_delineated_states, passes us_block_ladder_cbsa_coverage_failures. Tested by the Hypothesis test.
5. Whole-state override. A state with any cbsa_county_by_block entry must have one for every populated block, and every entry is a 5-digit county-equivalent in the block's own state. Tests: test_assemble_refuses_a_state_only_partly_remapped and test_assemble_refuses_malformed_county_equivalents.
6. Crosswalk round-trip. Any town table written in the Census layout parses back to itself: town geoid → 09 + new county. Tested by Hypothesis test_ct_crosswalk_round_trips_generated_towns.
7. Loader-level coverage. A state listed in cbsa_delineated_states whose blocks all have cbsa_code == 0 is a failure. Artifacts built before that record fall back to every state present, because OMB 2023 delineates territory in all 50 states and DC (computed from list1_2023.xlsx: 52 states including PR). Tests: test_cbsa_coverage_check_flags_the_legacy_connecticut_ladder and test_cbsa_coverage_check_follows_the_recorded_states.

Differentials.

• On real data, CT population by CBSA from the block join equals the same totals from Census's own town tabulation. That second path runs P.L. 94-171 summary level 060 through the crosswalk and the delineation. The two agree exactly on all 7 CBSAs, and 0 people live in "not defined" county subdivisions.
• The vectorized join is checked against a per-block Python reference in the Hypothesis test. On the national build, the first version (string keys) and the final version (int64 keys) wrote byte-identical artifacts (6840b990… both times).
• test_builder_output_matches_the_pre_fix_join_outside_connecticut runs the tool end to end and checks that the join outside CT equals the plain 2020-county lookup.

Mutation analysis. I ran 44 mutants, one per plausible regression: ignoring the remap, dropping the whole-state or reachability rule, looking up by 2020 prefix, coverage over the wrong states, crosswalk and MCD validation removed, a wrong return_inverse order, and so on. Each was monkeypatched in-process against test_us_block_ladder_sources.py and test_us_block_ladder_tool.py. All 44 are killed. Two of them had to be re-pointed at lines the vectorized rewrite changed (E3, F6), and the adapted versions are also killed. The first pass left 7 alive, and the tests above were added for them. An 8th survivor was equivalent (a newline re-append csv never needed) and its code was deleted.

Verification: national rebuild

Rebuilt with tools/build_us_block_ladder_artifact.py at this branch's code (nice -n 10, cached sources) and compared with the Route A artifact us_block_ladder_2020.npz (7ba39b959068…). The new artifact is 6840b990acdf….

Check	Result
block_geoid, population, congressional_district_geoid, sldu, sldl, place_fips	byte-identical, same dtypes, 5,769,942 rows
cbsa_code rows changed	42,008, all in CT; 0 outside CT
CT blocks with cbsa_code == 0	42,008 → 0
Loader-level coverage check	old: fails (state 09 only); new: passes
metadata_json	adds cbsa_delineated_states (all 51), the crosswalk source file and sha, and the cbsa layer's source text; vintage unchanged

CT CBSA population shares (2020 P.L. 94-171 block populations, CT total 3,605,944):

CBSA	Title	Population	Share
25540	Hartford-West Hartford-East Hartford, CT	1,150,473	31.90%
14860	Bridgeport-Stamford-Danbury, CT	946,327	26.24%
35300	New Haven, CT	570,487	15.82%
47930	Waterbury-Shelton, CT	450,376	12.49%
35980	Norwich-New London-Willimantic, CT	280,430	7.78%
45860	Torrington, CT (micro)	112,503	3.12%
39480	Putnam, CT (micro)	95,348	2.64%

All of CT is inside a CBSA under the 2023 delineation.

PUMA-carrying variant. I applied this fix to #1047's head 4324432e5; the conflicts were additive, and a patch is kept locally. I rebuilt with #1047's builder and compared the result with us_block_ladder_2020_puma.npz (9f2dc986bec7…). The new artifact is f84936637d49…. Every array, puma included, is byte-identical except cbsa_code, which changed only on the 42,008 CT blocks. The coverage check and the differential give the same results as above. The cbsa_code arrays of the two rebuilt artifacts are identical.

Downstream effects

• us_geography_ladder_gate (hashed, unchanged). Its CBSA check is the national nonempty weighted share against (0.75, 0.99). On ladder population the share moves from 0.9352 to 0.9461 (+1.09 pp, CT's population share), well inside the bounds. The docstring's "about 93% in a CBSA" anchor now reads low by about a point. The gate has no per-state CBSA check, which is why an all-zero CT passed it. Adding one would mean editing the hashed module.
• CBSA narrowing in us_runtime/block_location.py (US household location v1: one census block per household, every geography derived #1047, not on main).
  • The state_unidentified_counties_cbsa view excludes blocks with cbsa <= 0. Before this fix, a CT household narrowed to a CBSA had no candidate block, and the draw refuses an empty candidate set. After it, CT households have candidates in their CBSA.
  • cbsa_vintage_agreement builds its county → CBSA lookup from the first block of each 2020 county (np.unique(..., return_index=True)). That assumes a 2020 county sits in one CBSA, which stops holding for CT: 7 of its 8 counties now span 2–3 CBSAs (09001: 14860/47930; 09003: 25540/45860/47930; 09005: 14860/45860/47930; 09009: 35300/47930; 09011: 25540/35980/39480; 09013: 25540/39480; 09015: 35980/39480; only 09007 is single-CBSA). A CT source pair would then be compared with whichever CBSA the county's lowest block has. The fix belongs in US household location v1: one census block per household, every geography derived #1047: a pair should agree only when every ladder block in the county carries that CBSA.
  • This change composes with US household location v1: one census block per household, every geography derived #1047. Applied to US household location v1: one census block per household, every geography derived #1047's head 4324432e5, the conflicts are additive only, all 41 ladder-source tests pass, and the rebuilt PUMA ladder is described above.
  • Recomputed on real CPS pairs (GTCO > 0, ASEC 2023, 2024 and 2025, 280 pairs each) with that first-block rule, cbsa_vintage_agreement finds 20 disagreeing pairs against both the old and the rebuilt ladder, so narrowing stays off. The CT pairs move as follows: 09001 (CPS 14860) flips to agree. 09005 (CPS 0) flips to disagree, against its first block's 14860. 09009 (CPS 35300 vs 47930) and 09015 (CPS 49340, the 2013 Worcester delineation, vs 35980) disagree both before and after.
  • US household location v1: one census block per household, every geography derived #1047's note that its core arrays are "byte-identical to Route A's ladder (7ba39b95…)" stops holding for cbsa_code once it adopts a rebuilt ladder.
• policyengine-us. cbsa_code is a str household input with default "", defined only in variables/input/geography.py. I checked 1.764.6 and 2.2.1; no formula or parameter reads it, so no tax or benefit output moves. A rebuilt ladder changes CT households' cbsa_code from "" to a planning-region CBSA. Block draws are population-weighted within the CD and do not depend on CBSA, so no other column changes.
• Release consumer. tools/build_us_puf_support_base.py loads the ladder for the release path and does not run the new coverage check yet. US household location v1: one census block per household, every geography derived #1047 moves that load, so wiring it in waits until US household location v1: one census block per household, every geography derived #1047 lands.
• Route A. The Route A input artifact in _buildf-runtime is untouched, and adopting the rebuilt ladder there is a separate call. An in-flight checkpointed release should not pull this branch mid-run: the builder's builder_code_identity digests every package source, so its resume would refuse the changed run config.

Tests

• test_us_block_ladder_sources.py: 47 passed (34 new, including 2 Hypothesis property tests).
• test_us_block_ladder_tool.py (new): 4 passed. It runs main() offline on a synthetic cache: CT gets planning-region CBSAs, a build without CT never reads the crosswalk, a CT build without the MCD layer stops before writing, and the join outside CT is unchanged.
• test_us_geography_ladder.py: 21 passed.
• test_us_spine_blindness.py -k "classification or source_spine or import_graph": 9 passed.
• ruff check / ruff format --check: clean; tools/ci_test_plan.py verify: ok.
• Targeted runs only, because the shared build host is saturated. CI runs the full lanes.

Review

Four independent Opus 5.5 reviewers (source formats, invariants, downstream effects, conformance) each had every finding put to an adversarial verifier. Confirmed and fixed:

• inaccurate comments on the "not defined" crosswalk rows;
• a per-block string list in the join (about 330 MB of peak memory nationally), now int64;
• a hard-coded '09' in an error message.

An independent Opus 5.5 merge-gate review approved the change. Its own differential re-parsed the raw sources without the PR's parsers and found 0 mismatches on all 42,008 CT blocks and outside CT; the body corrections it asked for are applied.

Confirmed and deferred: the coverage check cannot fail inside the builder, and the release consumer doesn't call it yet (see Downstream). Refuted: "a refused ladder is left at --out" predates this change.

No byte-hashed module is in the diff, which touches 5 files. source_inventory_sha256 for _DIRECT_KERNEL_MODULES and _QRF_KERNEL_MODULES is 4734e4eb… and 9c4ba3bf… on this branch, as on main.

axiom: n/a: data infrastructure (geography ladder builder), no policy rules.

🤖 Generated with Claude Code