Measure SPM poverty on legacy certified US sources (backward compatibility)

changelog.d/597.fixed.md

@@ -0,0 +1 @@
+Measure SPM poverty on legacy certified US sources that predate SPM measurement-scope declarations, such as the ACS-local Build O release. Following Census's ACS household-population method, ACS group-quarters units are outside the SPM measurement: their SPM results are missing instead of stopping the run with `SPM_COMPOSITION_REQUIRED`, and poverty rates count measured people only. Ordinary tax and benefit results are unchanged.

changelog.d/598.fixed.md

@@ -0,0 +1 @@
+Supply the certified SPM independence role from the national release for the ASEC and PUF records of legacy certified US sources, so independent-teen SPM units are measured instead of stopping SPM calculations.

changelog.d/poverty-rate-unmeasured.changed.md

@@ -0,0 +1 @@
+`Poverty.rate` is now `None`, not `0.0`, when nobody in the group is measured (its weighted population with a poverty status is zero), for every source. `headcount` and `total_population` are unchanged, and the poverty DataFrame helpers report the rate as missing. Callers that format or subtract rates must handle `None`.

docs/engineering/runbooks/us-legacy-spm-sources.md

@@ -0,0 +1,122 @@
+# Legacy US SPM sources (backward compatibility)
+
+> **Backward compatibility, not a data contract.** This layer exists so that
+> certified US sources published *before* SPM declarations existed can still
+> be measured. New certified sources must carry the declarations themselves;
+> do not register them here.
+
+policyengine-us measures SPM poverty only when a source declares two things:
+
+- **Measurement scope:** which SPM units are inside the SPM measurement
+  population (`spm_unit_spm_universe_status`: `INCLUDED`/`OUTSIDE`, the
+  contract of PolicyEngine/microcosm#977).
+- **Independence role:** which 15–17-year-olds independently head their own
+  SPM unit (`is_spm_independent_minor_role`).
+
+A legacy source carries neither. Every SPM calculation on it stops with
+`SPM_COMPOSITION_REQUIRED` ("SPM unit has no classified adult").
+
+The package
+[`policyengine.tax_benefit_models.us.legacy_spm`](../../../src/policyengine/tax_benefit_models/us/legacy_spm/__init__.py)
+derives the missing declarations from evidence a legacy source does carry.
+Nobody is dropped: memberships, weights and every ordinary tax, benefit and
+income calculation are unchanged.
+
+## Registered sources
+
+Sources are keyed by certified SHA-256 in `legacy_spm/registry.py`.
+
+| Source | Scope rule | Independence list |
+| --- | --- | --- |
+| `populace_us_2024_acs_local` Build O (`769756c3…a7ec`) | `acs_household_population` | national release `source_spm_person_independence.csv` (`22b5968d…3ce5`) |
+
+### Scope rule `acs_household_population` (#597)
+
+Census's ACS SPM methodology measures the household population only.
+
+- ACS records with `TYPEHUGQ` 2 or 3 and the matching `RELSHIPP` (37 or 38) are
+  group quarters, so their SPM units are `OUTSIDE`.
+- ACS housing-unit records are `INCLUDED`.
+- Records from the `asec` and `puf_tax_detail` channels are `INCLUDED`.
+
+Age is never used. Unknown channels, missing or inconsistent evidence, and SPM
+units that mix group quarters with household people all raise.
+
+On the full Build O source, 183,206 SPM units are outside the measurement.
+
+### Independence list (#598)
+
+The source's ASEC/PUF records are the same records the certified national
+release carries with a source-backed independence role. The list is joined by
+`person_id` and `person_spm_unit_id`, and the join must be exact:
+
+- every ASEC/PUF person appears once, in the same SPM unit;
+- the list names nobody else.
+
+ACS people keep the model's own fallback (household head or spouse).
+
+On Build O, exactly 28 units change: Census-built units around an independent
+teen.
+
+## How runs get the declarations
+
+| Path | What happens |
+| --- | --- |
+| `create_datasets` / `ensure_datasets` | Year files store the declarations as ordinary input columns, with a `policyengine_legacy_spm_declarations` record. |
+| `Simulation.run()` over a raw legacy source file | The declarations are derived in memory. The file is recognised by its evidence columns and then its SHA-256, which is hashed once per process. |
+| `managed_microsimulation` | The declarations are derived from the loaded tables and set on the simulation and its branches. |
+
+`legacy_spm.model_adapter` then makes the model honour the scope:
+
+- **native:** policyengine-us defines `spm_unit_spm_universe_status`
+  (PolicyEngine/policyengine-us#9462). The declaration is just an input, and
+  nothing is overridden.
+- **override:** policyengine-us 2.2.1 cannot read it. A variable-only reform:
+  - adds the input;
+  - wraps the installed SPM chain so that outside units never reach the
+    canonical provider and every SPM outcome for them is NaN ("not measured").
+
+  Poverty indicators become floats (1.0, 0.0 or NaN). The override wraps the
+  installed formulas, with one exception: `spm_unit_income_decile` ranks
+  units, so it is recalculated over measured units with the installed method.
+  A contract test pins that method. The override refuses to run if the SPM
+  chain no longer has the verified shape.
+
+Poverty rates count measured people only. A group with nobody measured has a
+rate of `None`, not 0.
+
+## Staleness
+
+The record holds a `LegacySPMProfile`: the registry entry plus
+`ADAPTER_REVISION`. Checks are total and exact, like the year-file stamp:
+
+- `ensure_datasets` cuts a year file again, and `load_datasets` refuses it, when
+  its record is absent, unreadable, malformed, or from another profile.
+- `Simulation.load()` refuses an output saved under a different profile.
+
+Raise `ADAPTER_REVISION` whenever a change to derivation or the override means
+existing files must be made again.
+
+## When the model or data changes
+
+- **policyengine-us gains native scope (#9462).**
+  `test_the_installed_model_does_not_read_scope_natively_yet` fails. Check that
+  the native members include `INCLUDED` and `OUTSIDE`, and confirm the native
+  runs in the integration tests. Then delete the override half of
+  `model_adapter` (keep the derivation) and raise `ADAPTER_REVISION`.
+- **The SPM chain changes.** `test_the_override_covers_every_variable_that_depends_on_the_provider`,
+  `test_the_installed_chain_has_the_verified_shape` or
+  `test_the_recalculated_decile_matches_the_installed_method` fails. Re-verify
+  the override against the new chain before bumping the pin.
+- **A rebuilt source declares scope and role itself.** Point the bundle at it.
+  Keep the Build O registry entry only while anything still reads Build O,
+  then delete it.
+
+## Related
+
+- Microcosm #977: scope classification at source.
+- Microcosm #959: independence role in US builds.
+- policyengine-us #9462: native scope.
+- policyengine.py #519: nullable SPM summaries.
+- The year-file and output stamps (#590, and #593/#594 in flight). Fold this
+  record into them once those land.

docs/microsim.md

@@ -87,6 +87,24 @@ before passing them to the country model. Generic adult/child counts remain
 separate. The native source enrichment is not recalibration, and inherited
 schema-5 calibration diagnostics cannot be relabeled as schema 6.
 
+**Legacy SPM sources (backward compatibility).** Some certified US sources
+predate SPM declarations, for example the ACS-local Build O release. For those
+sources, .py derives the declarations from evidence they carry, following
+Census's ACS SPM methodology, which measures the household population only:
+
+- ACS group-quarters SPM units are outside the SPM measurement. Their SPM
+  thresholds, resources and poverty indicators are missing (NaN).
+- Poverty indicators on those sources are floats: 1.0, 0.0, or NaN for "not
+  measured".
+- Poverty rates count measured people only. A group with nobody measured has
+  a rate of `None`, not 0.
+- The sources' ASEC and PUF records take the certified SPM independence role
+  from the national release.
+
+Ordinary tax, benefit and income results are unchanged. Outputs record the
+declarations they used under `legacy_spm_declarations`. See
+[the runbook](engineering/runbooks/us-legacy-spm-sources.md).
+
 The wrapper's native pandas HDF loader accepts files that store only calibrated
 `household_weight`. It maps missing person and group weight columns through
 native membership IDs in memory, retaining supplied weight columns and the

docs/outputs.md

@@ -167,6 +167,12 @@ rate.run()
 rate.headcount, rate.total_population, rate.rate
 ```
 
+`rate.rate` is `None` when nobody in the group is measured: the group's
+weighted population with a poverty status is zero, for example a filter that
+matches no one, or a group whose members are all outside the SPM measurement
+on a legacy source. That is not a measured rate of zero, so handle `None`
+before formatting or subtracting rates.
+
 For all canonical poverty measures over one simulation:
 
 ```python

examples/us_budgetary_impact.py

@@ -32,6 +32,11 @@
 )
 
 
+def _format_rate(rate: float | None) -> str:
+    """Format a poverty rate; ``None`` means nobody in the group is measured."""
+    return "n/a" if rate is None else f"{rate:.4f}"
+
+
 def main():
     year = 2026
 
@@ -147,11 +152,15 @@ def main():
         analysis.reform_poverty.outputs,
         strict=True,
     ):
+        # A rate is None when nobody in the group is measured.
+        change = (
+            "n/a" if bp.rate is None or rp.rate is None else f"{rp.rate - bp.rate:+.4f}"
+        )
         print(
             f"  {bp.metric:30s}  "
-            f"baseline={bp.rate:.4f}  "
-            f"reform={rp.rate:.4f}  "
-            f"change={rp.rate - bp.rate:+.4f}"
+            f"baseline={_format_rate(bp.rate)}  "
+            f"reform={_format_rate(rp.rate)}  "
+            f"change={change}"
         )
 
     print("\n=== Inequality ===")

src/policyengine/core/run_record.py

@@ -193,6 +193,15 @@ def build_simulation_run_record_payloads(
     if renames is not None:
         results_payload["legacy_input_renames"] = dict(renames)
 
+    # BACKWARD COMPATIBILITY (#597, #598): the legacy SPM declarations a
+    # US run applied, which change its SPM results. See
+    # ``policyengine.tax_benefit_models.us.legacy_spm``.
+    legacy_spm_declarations = (
+        getattr(simulation.output_dataset, "metadata", None) or {}
+    ).get("legacy_spm_declarations")
+    if legacy_spm_declarations is not None:
+        results_payload["legacy_spm_declarations"] = dict(legacy_spm_declarations)
+
     return {"reform": reform, "input": input_payload, "results": results_payload}
 
 

src/policyengine/core/simulation.py

@@ -258,4 +258,12 @@ def release_bundle(self) -> dict[str, Any]:
         )
         if renames is not None:
             result["legacy_input_renames"] = dict(renames)
+        # BACKWARD COMPATIBILITY (#597, #598): the legacy SPM declarations a
+        # US run applied, which change its SPM results. See
+        # ``policyengine.tax_benefit_models.us.legacy_spm``.
+        legacy_spm_declarations = (
+            getattr(self.output_dataset, "metadata", None) or {}
+        ).get("legacy_spm_declarations")
+        if legacy_spm_declarations is not None:
+            result["legacy_spm_declarations"] = dict(legacy_spm_declarations)
         return result

src/policyengine/outputs/poverty.py

@@ -81,6 +81,21 @@ def run(self):
 
         # Map poverty variable to target entity if needed
         if poverty_var_obj.entity != target_entity:
+            source = getattr(
+                self.simulation.output_dataset.data, poverty_var_obj.entity
+            )
+            if (
+                target_entity != "person"
+                and pd.isna(source[self.poverty_variable]).any()
+            ):
+                # Mapping onto a group sums members and fills empty groups
+                # with zero, which would count a person outside the SPM
+                # measurement as measured and not poor.
+                raise ValueError(
+                    f"{self.poverty_variable} is missing for units outside "
+                    f"the poverty measurement; summarise it for people, not "
+                    f"for {target_entity} units."
+                )
             mapped = self.simulation.output_dataset.data.map_to_entity(
                 poverty_var_obj.entity,
                 target_entity,
@@ -121,8 +136,13 @@ def run(self):
         # Calculate results using weighted counts
         self.headcount = float((poverty_series == True).sum())  # noqa: E712
         self.total_population = float(poverty_series.count())
+        # ``None`` means nobody in the group is measured (for example, every
+        # member is outside the SPM measurement population), which is not
+        # a measured rate of zero.
         self.rate = (
-            self.headcount / self.total_population if self.total_population > 0 else 0.0
+            self.headcount / self.total_population
+            if self.total_population > 0
+            else None
         )
 
 

src/policyengine/tax_benefit_models/common/model_version.py

@@ -417,6 +417,14 @@ def load(self, simulation: Simulation) -> None:
                 RENAMES_RECORD_KEY,
                 read_renames_record,
             )
+            from policyengine.tax_benefit_models.us.legacy_spm import (
+                RECORD_KEY as LEGACY_SPM_RECORD_KEY,
+            )
+            from policyengine.tax_benefit_models.us.legacy_spm import (
+                expected_profile_for_dataset,
+                read_record_value,
+                record_problem,
+            )
 
             with h5py.File(filepath, "r") as stream:
                 raw = (
@@ -442,6 +450,17 @@ def load(self, simulation: Simulation) -> None:
                     "Saved US simulation predates the mapping of renamed "
                     "stored inputs (it has no record of them); run it again"
                 )
+            # BACKWARD COMPATIBILITY (#597, #598): an output is reused only
+            # under exactly the legacy SPM declarations its input needs.
+            recorded_legacy_spm = read_record_value(filepath)
+            legacy_problem = record_problem(
+                recorded_legacy_spm, expected_profile_for_dataset(simulation.dataset)
+            )
+            if legacy_problem is not None:
+                raise ValueError(
+                    f"Saved US simulation output is out of date: {legacy_problem}; "
+                    "run it again"
+                )
 
         if self.country_code == "uk":
             from policyengine.tax_benefit_models.uk.datasets import (
@@ -474,6 +493,10 @@ def load(self, simulation: Simulation) -> None:
             simulation.output_dataset.metadata.update(
                 {"spm_config": recorded["config"], RENAMES_RECORD_KEY: recorded_renames}
             )
+            if isinstance(recorded_legacy_spm, dict):
+                simulation.output_dataset.metadata[LEGACY_SPM_RECORD_KEY] = (
+                    recorded_legacy_spm
+                )
 
         if os.path.exists(filepath):
             simulation.created_at = datetime.datetime.fromtimestamp(