Declare the US SPM measurement universe from the source record type

[MaxGhenis]

What this does

PolicyEngine/policyengine-us#9462 makes the SPM measurement universe a
source declaration: a dataset supplies spm_unit_spm_universe_status ∈
{INCLUDED, OUTSIDE, UNRESOLVED} for every SPM unit, OUTSIDE units get
missing SPM amounts and poverty indicators, and an undeclared dataset raises
SPM_UNIVERSE_REQUIRED. Microcosm had no producer for that input.

This adds one: microcosm.build.us_runtime.spm_universe_source.

Source record	Status
ACS housing unit (TYPEHUGQ == 1)	INCLUDED
ACS group quarters (TYPEHUGQ ∈ {2, 3})	OUTSIDE
ASEC record	INCLUDED

Both arms come from Census SEHSD-WP2020-09 pages 6–7 and they are not
symmetric. The ACS exclusion is a data limitation Census states explicitly —
the public PUMS cannot distinguish noninstitutional group quarters, so the ACS
SPM sample is limited to people living in households. The ASEC ruling is the
opposite shape: the CPS ASEC sample frame already equals the CPS SPM universe,
so there is no ASEC record outside it to declare. That makes the ASEC ruling
necessarily spine-level, and it carries its own refusal so a future vintage
that admits a record-type field cannot be silently declared in-universe
(design note Q-E).

Status is stored as the enum member name string — "OUTSIDE", not 2,
not "SPMUniverseStatus.OUTSIDE", not the label text — because the export
adapter normalizes a stored cell to a member name and validates it against the
engine's own enum domain.

Never an inference

The ACS problem was discovered as SPM units with no classified adult, and
the tempting wrong design marks zero-adult units OUTSIDE. That would brand a
genuine household-unit data defect as out-of-universe and hide it, and it
would make the universe move whenever the role column moved. The producer
reads TYPEHUGQ, which the source fixes before any composition is computed.
test_a_zero_adult_housing_unit_is_included is the anti-inference test, with
the identical composition under TYPEHUGQ 2 as its negative control;
test_the_declaration_is_invariant_to_age_and_role pins it in both
directions.

Never a target, a gate, or an agreement surface

The status is not a calibration target, not a selection criterion and not a
release gate. Poverty in this repo is a comparison only. It is also
deliberately not a spine_agreement surface: it is meant to differ by
spine — that is its entire content — so registering it there would assert that
the ACS and ASEC arms should agree about a fact they are defined to disagree
about. The module docstring says so on the record.

OUTSIDE moves no weight, no membership and no row. The fiscal population is
unchanged; only its SPM measurability is declared.

Twelve refusals, all fail-closed

SPM_UNIVERSE_COLUMN_EXISTS, …_UNDECLARED_SPINE,
…_UNKNOWN_HOUSEHOLD_KIND, …_OFF_ARM_HOUSEHOLD_KIND, …_MIXED_HOUSEHOLD,
…_GQ_MULTI_PERSON, …_ORPHAN_MEMBER, …_EMPTY_UNIT,
…_UNIT_SPANS_KINDS, …_DEGRADED_PARTITION,
…_ASEC_RECORD_TYPE_UNREVIEWED, …_INVALID_CLONE_INDEX. UNRESOLVED is the engine's absence
sentinel, so a producer that cannot decide refuses — it never writes it.

Two are worth naming:

• SPM_UNIVERSE_UNIT_SPANS_KINDS is natively unreachable (ACS
  group-quarters households are one-person placeholders). It earns its keep
  the moment a future spine, a donor merge or a clone-index bug puts a
  group-quarters person into a multi-person unit, where picking a majority
  kind would be a fabricated universe decision.
• SPM_UNIVERSE_DEGRADED_PARTITION closes the largest silent-failure mode
  here. assign_us_unit_structure uses the native SPM_ID when fully present
  and falls back to household_id otherwise — any missing SPM_ID triggers
  that fallback frame-wide, and asec_pool._globalize_source_unit_column
  deliberately tolerates a missing SPM_ID. Declaring INCLUDED over a
  degraded partition would attest to a unit structure that does not exist, so
  the ASEC arm must prove its partition is a one-to-one densification of
  (support-clone copy, SPM_ID). The copy is part of the key because the PUF
  support clone copies SPM_ID verbatim under new unit ids (see the
  merge-prep review below). The check is ACS-exempt on purpose: acs_pums
  reaches assign_us_unit_structure with no SPM_ID at all, so one SPM unit
  per household is the native ACS partition, not a degradation.

What is wired, and what is not

Wired. The producer, its refusals, its tests, and a doc page.

Not wired, deliberately. No build path calls it. The export adapter is
untouched, so the column reaches no release H5 — the adapter change is blocked
upstream (design note §4 / Q-A). The enrichment lane does not carry it.
check_spm_composition and reform_validation are unchanged.

Registry entries this moves

• tests/test_us_spine_blindness.py — _OTHER_US_RUNTIME_MODULES
  (classification) and _SOURCE_SPINE_PROVENANCE_OWNERS. The second is
  not in the design note's checklist and is load-bearing: removing it fails
  test_runtime_population_operators_are_source_spine_blind with six
  contraband reads, because reading the provenance tag is the point of a
  per-arm universe ruling.
• CI test grouping — test_us_spm_universe_source.py resolves to us-qs by
  suffix; tools/ci_test_groups.py --verify is green and --list us-qs
  contains it.
• Changelog fragment.

Not moved, deliberately: source_enrichment.PRODUCER_SOURCE_FILES. The
design note lists it, but that tuple is the SPM-role enrichment producer's
reviewed source closure (mirrored in tools/build_us_spm_role_enrichment.py),
and validate_source_enrichment_candidate refuses a recorded inventory whose
key set differs from it. Adding a seventh entry would make the already-recorded
six-key provenance of the published role-enrichment candidate fail validation,
for a module that lane neither imports nor executes. It belongs in the
enrichment-lane generalization instead. Ruled 2026-09-22: keep it out (see
Rulings).

Tests

packages/microcosm-build/tests/test_us_spm_universe_source.py, 65 cases on
invented fixtures (48 as first built, 13 added in merge prep, 4 in gate round 1) — nothing opens a
build artifact, a pinned source or a release. One case is requires_us: it
runs the real clone_us_frame_for_puf_support over invented rows. Covering the rule on each arm and each TYPEHUGQ code, the ASEC
spine ruling, the anti-inference case with its negative control, clone
inheritance, the stored member-name encoding with three negative controls
(integer index, repr(), label text), every one of the twelve refusals with a
minimal frame that triggers exactly it, negative controls beside the
refusals that would otherwise over-refuse (a group-quarters placeholder with
a clone is accepted; the degraded-partition check does not fire on the ACS
arm; the ASEC record-type guard does not fire on an ACS-only frame), and the
frame wrapper leaving weights, membership, rows, metadata and mass log intact.

Three structural tests keep the module honest: the channel ruling table is
checked against each canonical constant by ast-parsing its defining module
(no import — each pulls the country engine: acs_pums took ~38 s and
stacked_spine ~36 s warm in separate processes on 2026-09-22, and the
authoring lane measured stacked_spine at ~90 s cold, which is why a leaf
producer names them instead), the spine column name is checked
against base_pool.spine_column's own f-string template, and every refusal
code the module actually raises is required to equal the declared tuple.

As first built, at b13dc8c (the merge-prep rerun is in its own section below):

uv run pytest test_us_spm_universe_source.py test_us_spine_blindness.py
545 passed in 64.33s        exit 0
uv run pytest test_release_input_coverage.py test_us_spec_bundle.py \
              test_spec_engine_inventory_coverage.py test_source_enrichment.py
267 passed in 467.73s       exit 0
uv run ruff check .                              All checks passed!   exit 0
uv run ruff format --check <touched> docs        5 files already formatted   exit 0
uv run python tools/ci_test_groups.py --verify   verification=ok      exit 0

The second run is the evidence for "this module moves no other registry": the
release input coverage manifest, the US spec bundle, the spec inventory
coverage and the source-enrichment contract are all green unchanged.

Context

• Engine PR: Separate SPM measurement scope from ordinary income policyengine-us#9462, fixing #9461.
• Design note: docs/us-spm-measurement-universe-producer.md on branch
  us-spm-measurement-universe-producer-design (§2 the rule, §3 the registry
  survey, §7 the plan). This is its PR 1.
• Companion: Read the SPM poverty indicator as nullable in _person_rate #976 (us-person-rate-nullable), the §6 _person_rate fix,
  merged 2026-09-22 and now on this branch through the main merge.
• The dataset must carry the status for every year SPM is requested; a
  single-year H5 resolves only its base year.

Draft. Merge prep is done (below); marking ready and merging are the main
session's call.

Rulings (Max, 2026-09-22)

Max ruled on the four open questions ("i approve all"). The branch follows
each one:

1. PRODUCER_SOURCE_FILES: keep the module out. It is not in
   source_enrichment.PRODUCER_SOURCE_FILES (source_enrichment.py:120-127
   on main), whose key set validate_source_enrichment_candidate requires to
   match exactly (source_enrichment.py:572-576).
2. Missing SPM_ID: the refusal stays strict. An absent or partly missing
   SPM_ID on the ASEC arm refuses with SPM_UNIVERSE_DEGRADED_PARTITION
   (_assert_native_asec_partition). The merge-prep fix below changes only
   how clone copies are keyed. It does not relax the missing-SPM_ID rule.
3. puf_tax_detail: it keeps refusing. It is not in
   SPM_UNIVERSE_CHANNEL_ARMS, so a household tagged only with that channel
   refuses with SPM_UNIVERSE_UNDECLARED_SPINE
   (test_refuses_a_channel_with_no_universe_ruling). A PUF clone inside a
   pooled frame still resolves through its household_spine tag, which
   takes precedence
   (test_the_spine_tag_takes_precedence_over_the_support_channel).
4. ASEC spine judgement: signed off as a named, cited constant.
   ASEC_SPM_INCLUDED_AUTHORITY, with SPM_UNIVERSE_METHODOLOGY citing
   Census SEHSD-WP2020-09 (Fox, Glassman and Pacas), pages 6–7. I re-read the
   quoted passage and Table 2 ("Poverty universe") on 2026-09-22. The
   constant's comment now records the sign-off.

Merge-prep review (2026-09-22)

Merged origin/main (63897ed) with no conflicts. I reviewed the diff
against the code and fixed one medium defect and two low ones. Each fix has
tests that fail without it.

• Medium, fixed. SPM_UNIVERSE_DEGRADED_PARTITION refused every
  support-cloned ASEC frame. puf_support._clone_entity_table
  deep-copies the person table and remaps only the id and membership columns
  (puf_support.py:2085-2120, _entity_id_columns at :2189-2195). Clone
  copy 1 therefore carries the native SPM_ID under new SPM unit ids. The
  check keyed on SPM_ID alone, so it read that clone as a degraded
  partition. I measured this by running the real
  clone_us_frame_for_puf_support on three invented ASEC persons at the
  pre-fix head: it refused with "3 native unit(s), 6 frame unit(s)". The
  native unit key is now (clone copy, SPM_ID). A degraded partition inside
  one copy still refuses, whether two units are collapsed or one is split.
• Low, fixed. A lost clone index let the GQ check pass without testing
  anything. _assert_group_quarters_are_single_native_records counted only
  persons with clone_index == 0, so a person with a NaN index was never
  counted. That can happen when a pool lacks the column on one arm. A missing
  index now reads as the native copy in both checks, which can only add
  collisions.
• Low, fixed. SPM_UNIVERSE_COLUMN_EXISTS checked only the SPM unit
  table. If another entity already carried the column, the frame
  constructor failed on its global-name rule instead. That was still a
  refusal, but unnamed. The check now covers every entity table.
• Test gaps, closed. The tests now cover the explicit channel_column
  path and both of its refusals, the spine tag's precedence over the support
  channel, H_LIVQRT as an ASEC refusal, and a record-type field on the
  person table.
• Doc claims updated. Read the SPM poverty indicator as nullable in _person_rate #976 has merged, so _person_rate now reads the
  poverty indicator as nullable. The graph-native spm_input_contract and
  acs_spm_scope references point to unmerged branches (first at
  774a900), and the comments now say so. Import timings are re-measured
  above.

Claims I checked against the code: TYPEHUGQ is staged on the household
table (acs_pums.py:63-71), and GQ rows are NP == 1, WGTP == 0
placeholders (acs_pums.py:394-414). The ACS arm reaches
assign_us_unit_structure with no SPM_ID (acs_pums.py:282-297). Any
missing SPM_ID falls back to households (units.py:125-129).
asec_pool._globalize_source_unit_column keeps a missing SPM_ID missing
(asec_pool.py:359-381). The Frame constructor refuses duplicated group ids
and orphaned or dangling membership (bundle.py:218-243). The export adapter
normalizes enum cells to member names and validates them against the engine
domain (policyengine_us.py:589-600, :1086-1113). check_spm_composition
has no universe predicate. I also re-proved the
_SOURCE_SPINE_PROVENANCE_OWNERS entry by removing it: the six contraband
reads return.

Deduced, not measured: the module is not wired to any build path, so today
no frame reaches it. The clone defect would have blocked the first ASEC-arm
call site rather than mislabel data.

uv run pytest test_us_spm_universe_source.py test_us_spine_blindness.py \
              test_release_input_coverage.py test_reform_validation.py
687 passed (57 of them this file, before the last 4 channel tests)   exit 0
uv run pytest test_us_spm_universe_source.py        65 passed        exit 0
uv run pytest test_us_spec_bundle.py test_spec_engine_inventory_coverage.py \
              test_source_enrichment.py     181 passed in 742s   exit 0
uv run ruff check .                                 All checks passed!
uv run ruff format --check <changed .py files>      already formatted
uv run python tools/ci_test_groups.py --verify      verification=ok
  (test_us_spm_universe_source.py in [fast] and in engine us-qs; not [defaulted])

Gate round 1 (2026-09-22)

The peer found one low defect: the merge-prep fix read a missing support-clone index as the native copy and claimed that could only make the checks stricter. It could not for the degraded-partition check: two ASEC rows in different clone copies that share one frame SPM unit refuse when their indices are present (two native keys, one frame unit) but would merge into one key and pass if both indices were missing. Fixed by refusing instead of guessing: when the person table carries person_support_clone_index, a missing, non-finite, negative or non-integral value now raises SPM_UNIVERSE_INVALID_CLONE_INDEX before either check runs, matching the maintained readers of that column (acs_transfer.py refuses a non-finite clone index; stacked_spine and puf_support parse it with errors="raise" or refuse a missing one). A frame with no clone-index column was never support-cloned and stays all-native. Tests: the peer's counterexample (refuses on the partition with indices present, on the clone index with them missing), three invalid values, and the two earlier missing-index tests retargeted to the new refusal; 65 pass.

🤖 Generated with Claude Code