Measure SPM poverty on legacy certified US sources (backward compatibility)

[anth-volk]

Fixes #597
Fixes #598

Backward compatibility for legacy certified sources. policyengine-us measures SPM poverty only on sources that declare SPM measurement scope and the SPM independence role. Certified sources published before those declarations existed, such as the ACS-local Build O release, carry neither, so every SPM calculation on them stops with SPM_COMPOSITION_REQUIRED. This PR derives the declarations from evidence those sources do carry, for registered legacy sources only. New sources must carry the declarations themselves. The model-override half switches itself off once policyengine-us can read the scope declaration (policyengine-us #9462). Runbook: docs/engineering/runbooks/us-legacy-spm-sources.md.

Problem

Every US SPM calculation on the certified ACS-local source stops with "SPM unit has no classified adult". This is the failure behind:

• the beta deploy verification after policyengine-sim-api #746 and #750, on state/ut;
• the real Utah image check in policyengine-sim-api #726.

A read-only scan of the full source found two causes:

1. No SPM measurement scope (Measure SPM poverty on legacy certified sources: ACS group quarters outside the SPM measurement #597). Of 183,206 single-person ACS group-quarters SPM units, 5,246 are children with no adult.
2. No independence role (Supply the certified SPM independence role for legacy ACS-local ASEC and PUF records #598). 28 Census-built ASEC/PUF units, each formed around an independent 15–17-year-old, are in 17 states. The build ran a few hours before Materialize the Census SPM independence role in US builds microcosm#959.

Design: policyengine.tax_benefit_models.us.legacy_spm

Module	Responsibility
registry	Typed registry of legacy sources, keyed by certified SHA-256, naming the declarations each needs. Build O is the only entry.
universe	Scope rule acs_household_population, following Census's ACS household-population method. ACS records with TYPEHUGQ 2 or 3 and the matching RELSHIPP (37 or 38) are OUTSIDE. ACS housing units and asec/puf_tax_detail records are INCLUDED. Age is never read, and unknown or inconsistent evidence raises.
independence	Joins the national release's certified independence list exactly (by person and SPM unit) onto ASEC/PUF people. ACS people keep the model's own fallback.
declarations	Writes both declarations as the ordinary inputs newer sources carry: spm_unit_spm_universe_status (INCLUDED/OUTSIDE, the Microcosm #977 contract) and is_spm_independent_minor_role.
record	A versioned LegacySPMRecord holding the LegacySPMProfile (registry entry plus ADAPTER_REVISION). Checks are total (no stored value makes them raise) and exact (the whole profile must match), following #590's year-file stamp.
runs	Picks a run's declarations from a year file, or derives them in memory for a run straight over a raw legacy file. A raw file is recognised by its evidence columns and its SHA-256, hashed once per process.
model_adapter	native mode: if policyengine-us defines spm_unit_spm_universe_status, the declaration is just an input. override mode (policyengine-us 2.2.1): a variable-only reform wraps the installed SPM chain. Outside units never reach the canonical provider, and every SPM outcome for them is NaN. The override wraps the installed formulas with one exception. spm_unit_income_decile ranks units, so it is recalculated over measured units with the installed method, and a contract test pins that method. The housing cap runs through the installed formula, because the allocation it reads is NaN for outside units. The reform checks the chain's shape and refuses to run on a chain it wasn't verified against.

Load paths covered: create_datasets/ensure_datasets year files, Simulation.run() on a raw legacy file, and managed_microsimulation.

Output contract:

• On legacy sources, poverty indicators are floats (1.0, 0.0, or NaN for "not measured").
• Poverty rates count measured people only.
• Poverty.rate is now None when nobody in the group is measured. Previously it was 0.0. This applies to every source, so it has its own changed changelog fragment and is documented in docs/outputs.md. examples/us_budgetary_impact.py handles it.
• Poverty refuses to sum NaN indicators onto a group entity.
• Outputs, release_bundle and run records carry legacy_spm_declarations, including the model_mode.

Not changed: country, core, calculator and dataset pins, and environment variables. The independence list comes from the same public Hugging Face repository as the sources, with no credentials needed. Ordinary tax, benefit and income calculations are unchanged, and nobody is dropped.

Validation

Tests:

• 49 unit tests, including Hypothesis properties:
  • the classifier against an independent statement of the rule (300 examples);
  • record totality and exactness over arbitrary stored JSON (200 examples);
  • unreadable-record cases;
  • registry pins checked against the bundle manifest;
  • every year-file declaration guard;
  • recognising the source by its URI as well as its name.
• 13 model-contract tests, including a pin on the installed decile method the override mirrors. These rebuild, from the installed policyengine-us, every variable depending on the SPM provider: 19 today, all covered by the override. They also check the chain shape, mode selection (native, override, idempotence, branch disagreement), and that the model does not yet read scope natively, which signals when the override can go.
• 24 integration tests on the real policyengine-us 2.2.1, covering:
  • provider calls (by spy);
  • measured-unit parity, and ordinary-output parity on the full population;
  • a reform's baseline;
  • Tlaib ECPA, with HUD abolition on and off;
  • regional filtering;
  • an empty measured group giving None;
  • cache reuse and refusal, and run records;
  • year files for 2025–2027;
  • regeneration after a deleted, garbage, or out-of-revision record;
  • raw-source runs and managed_microsimulation;
  • loading year files by the source's URI;
  • SPM_UNIVERSE_REQUIRED for a year with no declaration;
  • other sources being untouched.
• Before the review fixes, the full suite passed locally: 1,495 passed, 9 skipped. After them, the legacy SPM, poverty, year-file stamp, legacy-input and dataset suites pass (271 tests). One of Stamp US year files with the rename register they were cut under #590's stamp tests now also allows the legacy record node, which is a small JSON node and not a table. ruff format and ruff check are clean on new code; the remaining lint errors are already on main.

Full certified source, read-only: running the derivation on the real file gives:

• 183,206 SPM units outside the measurement and 1,408,308 inside;
• 166,321 roles from the list;
• 0 included units without a classified adult.

Real Utah partition, with its hash checked and served under the certified identity for validation only, for 2025–2027:

• all 39,679 people are kept and 1,617 group-quarters units are classified;
• every ordinary output is bit-identical to the unmodified model on the full population;
• every SPM value for measured units is identical to the unmodified model without group quarters;
• Utah's 2026 SPM poverty rate is 8.53% (deep: 2.41%) over 3.316M measured people;
• each run takes about 13 s with a peak around 2 GiB.

Follow-ups

• Unmanaged copies of a legacy source get declarations in Simulation.run(), which hashes the file, but not in create_datasets. Matching that would mean judging those year files by hashing the source, which Stamp US year files with the rename register they were cut under #590's staleness decision deliberately avoids. Source identity in year-file stamps (Tie US year files to the source dataset and package versions they were cut from #593) is the place to close this.

• Fold the record into the year-file and output stamps once policyengine.py Tie US year files to the source dataset and package versions they were cut from #593 and Stamp saved US outputs with the rename register they were calculated under #594 land.

• Carry None poverty rates through the simulation API (_number(), the gateway schema) and API v1. Today they collapse to 0.

• policyengine-sim-api #726 adopts this release, and policyengine-api #3868 re-pins.

🤖 Generated with Claude Code

[DTrim99]

PR Review: Measure SPM poverty on legacy certified US sources (backward compatibility)

Early feedback on the draft. The design is sound. Only registered sources get the declarations: the registry is keyed by certified SHA-256 and Build O is its one entry. The ACS household-population rule matches Census's method. The override wraps the installed SPM chain rather than forking it, and most failure modes are loud. I found no wrong-result or crash bugs against policyengine-us 2.2.1. The items below are gaps to close before this leaves draft: one formula really is copied (the docs say none is), one lookup works for a dataset's manifest name but not its URI, a few guard branches have no tests, and there is changelog and docs housekeeping.

Critical (0)

None found.

Should address (5)

1. The decile override copies the model formula, the docs say it doesn't, and nothing catches drift

• legacy_spm/model_adapter.py:295-323 (_income_decile_variable) re-implements policyengine-us 2.2.1's spm_unit_income_decile formula: the same weights (spm_unit_weight * nb_persons()) and the same MicroSeries(...).decile_rank(), restricted to measured units.
• Three places claim nothing is copied: the module banner at model_adapter.py:18 ("The override never copies a model formula"), the runbook (us-legacy-spm-sources.md:79, "No model formula is copied") and the PR body.
• _check_spm_chain (model_adapter.py:167-187) only checks that spm_unit_income_decile has a formula. Suppose a later pin changes the installed method, for example to negatives_in_zero or different weights. Deciles on legacy sources would then silently stop matching the model's method.
• Fix: add a contract test that pins the installed formula, for example by comparing the normalised AST or source of system.variables["spm_unit_income_decile"].formulas with the verified version. Correct the three "never copies" statements to name this one exception.

2. load_datasets / ensure_datasets only recognise the legacy source by its manifest name, not by its URI

• _managed_source_sha256 (us/datasets.py:650-661) matches only None, the default URI, or a manifest name.
• load_datasets documents its argument as "HuggingFace dataset paths" (datasets.py:685), and the region registry's dataset_path is exactly such a URI. I checked against the head:
  • resolve_region_dataset_path("us", "state", state_code="UT") gives hf://policyengine/populace-us/populace_us_2024_acs_local.h5@populace-us-2024-buildo-….
  • That equals resolve_dataset_reference("us", "populace_us_2024_acs_local") and maps to the same year-file stem.
  • But _expected_legacy_spm_profile(<that URI>) returns None, while the name returns the Build O profile.
• Consequences:
  • load_datasets([region_uri]) refuses a correctly cut ACS-local year file (datasets.py:709). The stated reason is false: "it records legacy SPM declarations that its source does not need".
  • ensure_datasets([region_uri]) marks every file stale (datasets.py:1404) and calls create_datasets. That raises "Explicit dataset URIs bypass the policyengine.py release bundle". With allow_unmanaged=True it instead re-cuts the files with no declarations, overwriting the good ones.
  • Relatedly, create_datasets identifies the source by the bundle entry's hash. A byte-identical unmanaged copy (a local path or hf:// with allow_unmanaged) therefore gets no declarations, while Simulation.run() over the same bytes does, because runs.raw_legacy_source hashes the file.
• Production sim-api translates region URIs into manifest names before calling ensure_datasets, so this does not block #726. It is still a wrong failure message on a documented input form.
• Fix: have _managed_source_sha256 also match a manifest entry whose resolve_dataset_reference equals the given reference. Optionally, identify the source in create_datasets by the materialised file's hash, as the raw-run path does. Add a test that loads a name-cut year file by its URI.

3. Tests do not cover the guards that keep failures loud

These checks are the PR's guarantee against silent fallback, and none of them has a test:

• SPM_UNIVERSE_REQUIRED (model_adapter.py:227-236): raised when an SPM unit has no scope declaration for the period (UNRESOLVED). This is reachable in practice. Auto carry-over only moves forward, so a managed Build O simulation asked for a year before its stored year, e.g. calculate("spm_unit_spm_threshold", 2023), has no declaration.
• check_year_file_frames (legacy_spm/declarations.py:145, 151, 158): a stored scope column with a value outside INCLUDED/OUTSIDE, a missing role column, and a role column with NaN or a non-boolean dtype. Only the missing-scope-column case is tested (test_a_declared_dataset_must_be_current_and_complete).
• Fix: add one targeted test per branch. Each needs only the existing synthetic fixtures.

4. Poverty.rate → None changes the public output for every source but is filed as a legacy-source fix

• outputs/poverty.py:142-146 now returns None instead of 0.0 for an empty group, for UK and US alike. The type was already Optional[float] (poverty.py:69), but the meaning changes for every caller.
• The only mention is inside changelog.d/597.fixed.md and the legacy section of docs/microsim.md.
• docs/outputs.md (the Poverty section, around line 155-167) does not mention it.
• Existing callers do arithmetic on rate: examples/us_budgetary_impact.py:152-154 formats bp.rate:.4f and subtracts rp.rate - bp.rate, which raise TypeError on None.
• Fix: move the cross-source change into its own changed fragment (or breaking, if maintainers see it that way). Document None in the Poverty section of docs/outputs.md, and make the example handle None.

5. Both changelog fragments will render as double bullets

• changelog.d/597.fixed.md and 598.fixed.md both begin with - . Towncrier adds its own bullet, so they come out as - - …. CHANGELOG.md already shows this twice, at lines 12 and 631 (for example 6.3.2's "- - Publish to PyPI…").
• Fix: remove the leading - from both fragments.

Suggestions (6)

1. A 200,000-character test ID. tests/test_us_legacy_spm.py:424-435 parametrizes on the raw string "[" * 100_000 + "]" * 100_000, so pytest uses the string itself as the test ID.

   • Locally on Windows this gives 39 passed and 2 errors: PYTEST_CURRENT_TEST exceeds the 32,767-character environment-variable limit.
   • On Linux CI, with addopts = "-v", it prints the full ID into the logs.
   • Stamp US year files with the rename register they were cut under #590's tests avoid this by passing DEEP_JSON inside tuples. Use ids=[...] or the same wrapping here.

2. The contract test only follows dependents of provider calls. _provider_dependents (tests/test_us_legacy_spm_model_contract.py:38, 71-88) starts only from variables that call policyengine_amount or masked_policyengine_amount.

   • Two masked variables do not call the provider: spm_unit_allocated_housing_subsidy and spm_unit_allocated_tenant_payment. A new consumer of either would get NaN without this test noticing.
   • References made through list-valued parameters (add(unit, period, p.some_list)) are not followed either.
   • A one-off scan against 2.2.1 found no such consumers today. Starting the search from all of MASKED_FORMULA_VARIABLES, and resolving list parameters, would keep it that way.

3. Stale docstrings. create_datasets, load_datasets and ensure_datasets don't mention the legacy SPM record. ensure_datasets still says "Only each file's stamp and renames record are read to decide" (datasets.py:1365-1366).

4. The public surface is large for a layer meant to be deleted. legacy_spm/__init__.py:88-120 exports 30 names, including the frozen dataclasses LegacySPMDeclarations and LegacySPMRun, which are not Pydantic models (AGENTS.md asks for Pydantic models on public classes). Keeping __all__ to what callers need (RECORD_KEY, LegacySPMRecord, maybe LEGACY_SPM_SOURCES), or making the package private, means removing it later won't be a breaking change.

5. Private helpers imported across modules. Several files import underscore names from legacy_inputs:

   • _ABSENT, _UNREADABLE, _read_json_dataset and _write_json_dataset in record.py:24-29;
   • _simulation_and_branches and _record_input_variables in declarations.py:27-31 and model_adapter.py:35-37;
   • _ABSENT (as _ABSENT_RECORD) in runs.py:23 and datasets.py.

   A small public helper module (sentinels plus reading and writing the JSON node) would make that contract explicit.

6. Where the independence list is downloaded. Raw runs and managed_microsimulation download it next to the source file (runs.py:89, model.py:578). That directory may be read-only or shared. Using the configured data directory would be more robust.

Verified correct

• Scope rule. Census's ACS SPM excludes everyone living in group quarters (Census working paper SEHSD-WP2020-09), so treating TYPEHUGQ 2 or 3 with the matching RELSHIPP (37 or 38) as OUTSIDE follows Census. Age is never read, and unknown, inconsistent or mixed evidence raises.
• Independence fallback. For ACS people it reproduces spm-calculator 1.0.0's formula exactly: is_household_head | is_household_spouse, both input-only in policyengine-us 2.2.1. is_spm_independent_minor_role is defined for all time (ETERNITY), so setting it once covers every year.
• The wrappers really replace the formulas.
  • When the reform updates a variable, core keeps none of the original's dated formulas, because the new undated formula starts at the earliest possible date. The wrapper still calls the installed formula for each period.
  • None of the masked variables uses defined_for.
  • The capped housing subsidy reads the masked allocation (NaN > 0 is false), so outside units never reach the provider.
  • A measured unit with no adult still raises SPM_COMPOSITION_REQUIRED.
• Ordinary outputs are untouched. In policyengine-us 2.2.1, no unmasked variable, contributed reform or parameter list consumes any masked variable (checked by a scan of the installed system). household_benefits uses housing_assistance, not the capped subsidy.
• Poverty denominators. With microdf 1.3, a weighted count() skips NaN, so rates cover measured people only. The new group-entity guard stops NaN being summed onto households.
• Backward compatibility.
  • Non-legacy year files, outputs and run records are unaffected: no record is expected and none is required.
  • Legacy year files with a stale record are cut again.
  • Simulation.ensure() treats a load() refusal as "run it again", so outdated outputs are recomputed rather than raised.
  • Country pins are exact (policyengine-us==2.2.1), so the native/override switch cannot flip without a pin bump, and the contract and integration tests trip when it does.
• Branches. The reform's baseline branch gets the override and the declarations.
• Repo conventions. The PR is a same-repo draft, the body starts with Fixes #597 / Fixes #598, and the fragment names follow the <issue>.fixed.md pattern.
• Local test runs (policyengine-us 2.2.1, policyengine-core 3.33.0, spm-calculator 1.0.0, Python 3.12):
  • tests/test_us_legacy_spm.py: 39 passed. The 2 errors are the Windows-only test-ID problem in Suggestion 1.
  • tests/test_us_legacy_spm_model_contract.py: 12 passed.
  • I did not run the integration suite locally because of memory limits; CI will run it.

CI status

Pending. "Verify release inputs and bundle packages" and "Test documentation builds" pass. Changelog check, Lint, Mypy, Install + smoke-import (3.11 to 3.14) and Test (3.11 to 3.14) have not finished. The CI test job installs .[dev], so the integration and contract tests will run there rather than skip.

Branch status

2 commits ahead and 0 behind main; mergeable; draft; the head branch is in PolicyEngine/policyengine.py.

Verdict: REQUEST_CHANGES

[DTrim99]

Early feedback on the draft (details in the comment above). The design is sound: the hash-gated registry, the ACS group-quarters rule that matches Census's ACS SPM, and the override wrapping the installed formulas. I found no wrong-result or crash bugs against policyengine-us 2.2.1. Five should items: (1) the decile override re-implements spm_unit_income_decile despite the 'never copies a model formula' claim, so pin it in a contract test; (2) Build O is recognised only by manifest name, so the region hf:// URI for the same file is refused and ensure_datasets falls through to create_datasets and raises; (3) the loud-failure guards (SPM_UNIVERSE_REQUIRED, the year-file frame checks) are untested; (4) Poverty.rate changing from 0.0 to None affects every source, sits in a .fixed fragment, and breaks examples/us_budgetary_impact.py formatting; (5) both changelog fragments start with '- ' and render as '- - '.