diff --git a/changelog.d/spm-universe.breaking.md b/changelog.d/spm-universe.breaking.md new file mode 100644 index 00000000000..d44df907c82 --- /dev/null +++ b/changelog.d/spm-universe.breaking.md @@ -0,0 +1,5 @@ +Require explicit SPM measurement-universe status in datasets, keep SPM amounts, resources and poverty outcomes missing outside that universe, and return nullable floating-point poverty, equivalised-income and income-decile outputs. + +Require Microdf 1.3.0 or later so weighted counts exclude missing SPM outcomes. + +Fail closed when a dataset stores an SPM measurement-universe value the status enum does not define, instead of measuring that unit as outside the universe. diff --git a/docs/engineering/spm-universe.md b/docs/engineering/spm-universe.md new file mode 100644 index 00000000000..0d36190c682 --- /dev/null +++ b/docs/engineering/spm-universe.md @@ -0,0 +1,112 @@ +# SPM measurement scope + +The country model separates Supplemental Poverty Measure (SPM) outputs from +ordinary income for records outside a dataset's declared measurement universe. +Dataset construction must supply the measurement universe. + +## Source decisions + +Supply `spm_unit_spm_universe_status` for every SPM unit and year: + +| Value | Measurement behavior | +| --- | --- | +| `INCLUDED` | Compute canonical amounts and poverty indicators. | +| `OUTSIDE` | Return missing SPM amounts, resources and poverty indicators; do not call the threshold provider for the unit. | +| `UNRESOLVED` | Raise `SPM_UNIVERSE_REQUIRED` when scope-dependent outputs are requested. | + +Store the declaration on the SPM-unit table, for every year, as either the +member name (`"INCLUDED"`, `"OUTSIDE"`, `"UNRESOLVED"`) or the member index +(`0`, `1`, `2`); both encodings resolve the same scope. Any other stored value +raises `SPM_UNIVERSE_REQUIRED`: a name the enum does not define is rejected on +load, and an index the enum does not define fails closed at measurement rather +than removing the unit from the universe. + +Dataset simulations default to `UNRESOLVED`. A household situation defaults to +`INCLUDED`. The model never infers scope from missing tenure, age, county or a +zero-adult measurement composition. A dataset must resolve scope for each year; +an earlier year's declaration does not supply a later year's declaration. +Automatic dataset extension drops these annual source declarations from its +generated future-year tables; explicitly supplied multi-year tables retain them. + +Scope is a source-owned input with a household fallback, so it is listed in +`DATASET_SOURCE_INPUTS` rather than rejected. Setting it invalidates dependent +cached results, because every scope-dependent measurement and summary is +downstream of it. + +Source universes differ. Census's ACS research excludes all group quarters +because the public ACS does not identify the required subtypes. CPS-based SPM +includes some noninstitutional group quarters. A dataset release must identify +its source universe and validate household/member linkage before declaring +units outside. See [Census's ACS SPM research, pages 6–7](https://www.census.gov/content/dam/Census/library/working-papers/2020/demo/SEHSD-WP2020-09.pdf#page=6) +and [Census group-quarters guidance](https://www.census.gov/topics/income-poverty/guidance/group-quarters.html). + +## Housing in ordinary income + +Ordinary income aggregates - general household benefits, CBO means-tested +transfers, California CPUC countable income and the contributed household +benefit overrides - add the modelled `housing_assistance` award, not the +Census SPM's capped housing valuation. That separation is independent of SPM +scope: `housing_assistance` is a program amount the model computes for every +unit, so these aggregates never consult the measurement universe and never +require a scope declaration. + +SPM resources keep the capped valuation through +`spm_unit_capped_housing_subsidy`, which caps the SPM-unit-allocated award at +shelter need less the allocated tenant payment. That variable is scoped: +included units receive the capped value, outside units receive `NaN`. Included +allocated assistance must be finite and nonnegative; otherwise the formula +raises `SPM_HOUSING_ASSISTANCE_INVALID`. + +Census first values housing assistance using market rent less tenant payment, +then caps the amount added to SPM resources at shelter need less tenant payment. +See the [SPM technical documentation, page 14](https://www2.census.gov/programs-surveys/supplemental-poverty-measure/datasets/spm/spm_techdoc.pdf#page=14). +That definition does not establish a housing valuation outside the SPM universe +or authorize changing CBO and CPUC conventions; those aggregates use the actual +award instead. + +## Nullable indicators, distribution outputs and aggregation + +The following outputs now use floating-point `0`, `1` and `NaN`: +`spm_unit_is_in_spm_poverty`, `spm_unit_is_in_deep_spm_poverty`, `in_poverty`, +`in_deep_poverty` and `person_in_poverty`. `NaN` identifies an outside record. +Included records with invalid resources or thresholds raise +`SPM_MEASUREMENT_INVALID`. +The indicators retain stock-quantity semantics: a monthly request returns the +annual classification without dividing a positive indicator by twelve. + +`spm_unit_oecd_equiv_net_income` and `spm_unit_income_decile` inherit the same +universe. Equivalised income is `NaN` outside the universe, and decile ranks +are computed over the included units alone, so an outside record cannot move +another record's rank. Both are float stock quantities and both reject +non-finite included inputs with `SPM_MEASUREMENT_INVALID`. + +Use `calculate(..., map_to="person")` and MicroSeries `count`, `sum` and `mean` +for person-weighted summaries. Keep outside records in full-population coverage +counts and exclude their missing indicators from the eligible denominator. +An empty eligible denominator yields a missing rate; serialize it as JSON +`null`. Do not cast these indicators to `bool`, negate them with `~`, or use +them directly as Boolean selection masks. Compare an observed indicator with +`1` when selecting poor units and separately retain its missingness mask. + +Dataset inputs and `set_input` cannot override formula-owned measurement +outputs. Preserve observed poverty reports under separate report-only names. + +The country consumer audit found only direct alias projections in production +Python; usage documentation uses MicroSeries means. The historical Utah +notebook divides one poverty mean by another and requires an explicit empty or +zero-baseline check before reuse. Wrapper serialization and summary consumers +require the companion nullable-summary change. + +## Qualification limits + +Synthetic tests exercise mixed included/outside records, canonical-provider +selection, included parity, missing outcomes, decile ranking, housing +consumers and contributed reforms. They do not certify a native population +build or a complete managed-wrapper run. + +Tiny country dataset fixtures also invoke dataset-dependent Medicaid allocation: +the model distributes statewide spending across the fixture's weighted enrolled +cost indices. Thus their absolute Medicaid totals do not represent household +estimates. This limitation reproduces on the unchanged country base and +requires representative population validation before interpreting fiscal +totals. diff --git a/docs/usage/microsimulation.md b/docs/usage/microsimulation.md index 76005fda62d..93cb0fe7c54 100644 --- a/docs/usage/microsimulation.md +++ b/docs/usage/microsimulation.md @@ -89,6 +89,11 @@ poverty_rate = person_poverty.mean() hh_eitc = sim.calc("eitc", map_to="household") ``` +SPM poverty indicators contain `0` and `1` for included records and `NaN` for +records outside the declared measurement universe. MicroSeries means exclude +missing indicators; an empty eligible population has a missing rate. Preserve +that rate as JSON `null`, and do not cast the indicators to Boolean values. + ## Available datasets ### The default build @@ -109,6 +114,11 @@ source-backed `is_spm_independent_minor_role` values, and it must not store formula-owned SPM outputs such as `spm_unit_spm_threshold`. Any observed Census measurement is retained under a separate report-only name. +The file must also declare `spm_unit_spm_universe_status` for every unit and +measurement year. See the +[SPM universe contract](../engineering/spm-universe.md) for source requirements +and nullable outputs. + The legacy files under `hf://policyengine/policyengine-us-data/` predate that contract, so SPM measurements are not available over them: diff --git a/policyengine_us/data/economic_assumptions.py b/policyengine_us/data/economic_assumptions.py index c72459f9833..2cc05e9028a 100644 --- a/policyengine_us/data/economic_assumptions.py +++ b/policyengine_us/data/economic_assumptions.py @@ -20,6 +20,9 @@ "calibration.gov.cbo.income_by_source.adjusted_gross_income" ) +# Each annual scope declaration requires an explicit source decision. +ANNUAL_SPM_SOURCE_DECLARATIONS = ("spm_unit_spm_universe_status",) + MICRODATA_UPRATING_OVERRIDES = { "american_opportunity_credit": DEFAULT_MICRODATA_UPRATING, "cdcc_relevant_expenses": DEFAULT_MICRODATA_UPRATING, @@ -92,7 +95,8 @@ def extend_single_year_dataset( If ``end_year`` is not provided, it defaults to the latest year covered by the CPI-U parameter (gov.bls.cpi.cpi_u). - Variables without an uprating parameter are carried forward unchanged. + Variables without an uprating parameter are carried forward unchanged, + except annual SPM scope declarations, which require explicit new-year inputs. """ if system is None: from policyengine_us.system import system as _system @@ -115,6 +119,13 @@ def extend_single_year_dataset( for year in range(start_year + 1, end_year + 1): next_year = dataset.copy(deep=False) next_year.time_period = str(year) + # The loader accepts source columns across entity-table layouts. + for table in next_year.tables: + table.drop( + columns=list(ANNUAL_SPM_SOURCE_DECLARATIONS), + errors="ignore", + inplace=True, + ) datasets.append(next_year) multi_year_dataset = USMultiYearDataset(datasets=datasets) diff --git a/policyengine_us/reforms/congress/tlaib/boost/boost_middle_class_tax_credit.py b/policyengine_us/reforms/congress/tlaib/boost/boost_middle_class_tax_credit.py index 72e3418adaa..a6618ae3774 100644 --- a/policyengine_us/reforms/congress/tlaib/boost/boost_middle_class_tax_credit.py +++ b/policyengine_us/reforms/congress/tlaib/boost/boost_middle_class_tax_credit.py @@ -1,4 +1,5 @@ from policyengine_us.model_api import * +from policyengine_us.spm import spm_universe_mask def create_boost_middle_class_tax_credit() -> Reform: @@ -100,6 +101,7 @@ class spm_unit_benefits(Variable): unit = USD def formula(spm_unit, period, parameters): + included = spm_universe_mask(spm_unit, period) BENEFITS = [ "social_security", "ssi", @@ -144,8 +146,8 @@ def formula(spm_unit, period, parameters): ) previous_benefits = add(spm_unit, period, BENEFITS) if p.administered_through_ssa is False: - return previous_benefits - return add(spm_unit, period, BENEFITS) + middle_class_tax_credit + return where(included, previous_benefits, np.nan) + return where(included, previous_benefits + middle_class_tax_credit, np.nan) class reform(Reform): def apply(self): diff --git a/policyengine_us/reforms/congress/tlaib/economic_dignity_for_all_agenda/edaa_end_child_poverty_act.py b/policyengine_us/reforms/congress/tlaib/economic_dignity_for_all_agenda/edaa_end_child_poverty_act.py index abdb9b17928..fee116b6a3b 100644 --- a/policyengine_us/reforms/congress/tlaib/economic_dignity_for_all_agenda/edaa_end_child_poverty_act.py +++ b/policyengine_us/reforms/congress/tlaib/economic_dignity_for_all_agenda/edaa_end_child_poverty_act.py @@ -1,4 +1,5 @@ from policyengine_us.model_api import * +from policyengine_us.spm import spm_universe_mask def create_ecpa_only() -> Reform: @@ -114,6 +115,7 @@ class spm_unit_benefits(Variable): unit = USD def formula(spm_unit, period, parameters): + included = spm_universe_mask(spm_unit, period) BENEFITS = [ "social_security", "ssi", @@ -153,7 +155,7 @@ def formula(spm_unit, period, parameters): if not parameters(period).gov.hud.abolition: BENEFITS.append("spm_unit_capped_housing_subsidy") - return add(spm_unit, period, BENEFITS) + return where(included, add(spm_unit, period, BENEFITS), np.nan) class income_tax_refundable_credits(Variable): value_type = float diff --git a/policyengine_us/reforms/congress/tlaib/end_child_poverty_act.py b/policyengine_us/reforms/congress/tlaib/end_child_poverty_act.py index c1d9cb13e40..cca363ed877 100644 --- a/policyengine_us/reforms/congress/tlaib/end_child_poverty_act.py +++ b/policyengine_us/reforms/congress/tlaib/end_child_poverty_act.py @@ -1,4 +1,5 @@ from policyengine_us.model_api import * +from policyengine_us.spm import spm_universe_mask def create_end_child_poverty_act() -> Reform: @@ -128,6 +129,7 @@ class spm_unit_benefits(Variable): unit = USD def formula(spm_unit, period, parameters): + included = spm_universe_mask(spm_unit, period) BENEFITS = [ "social_security", "ssi", @@ -165,7 +167,7 @@ def formula(spm_unit, period, parameters): BENEFITS.append("aca_ptc") if not parameters(period).gov.hud.abolition: BENEFITS.append("spm_unit_capped_housing_subsidy") - return add(spm_unit, period, BENEFITS) + return where(included, add(spm_unit, period, BENEFITS), np.nan) class income_tax_refundable_credits(Variable): value_type = float diff --git a/policyengine_us/reforms/states/tx/rebate/tx_rebate.py b/policyengine_us/reforms/states/tx/rebate/tx_rebate.py index 22019ce28df..cdc4da63337 100644 --- a/policyengine_us/reforms/states/tx/rebate/tx_rebate.py +++ b/policyengine_us/reforms/states/tx/rebate/tx_rebate.py @@ -13,6 +13,7 @@ """ from policyengine_us.model_api import * +from policyengine_us.spm import spm_universe_mask from policyengine_core.periods import period as period_ from policyengine_core.periods import instant @@ -50,6 +51,7 @@ class spm_unit_benefits(Variable): unit = USD def formula(spm_unit, period, parameters): + included = spm_universe_mask(spm_unit, period) BENEFITS = [ "social_security", "ssi", @@ -124,7 +126,7 @@ def formula(spm_unit, period, parameters): BENEFITS.append("assigned_aca_ptc") if not parameters(period).gov.hud.abolition: BENEFITS.append("spm_unit_capped_housing_subsidy") - return add(spm_unit, period, BENEFITS) + return where(included, add(spm_unit, period, BENEFITS), np.nan) class reform(Reform): def apply(self): diff --git a/policyengine_us/spm.py b/policyengine_us/spm.py index 4419a4432a5..73ef531c34e 100644 --- a/policyengine_us/spm.py +++ b/policyengine_us/spm.py @@ -12,6 +12,8 @@ from functools import lru_cache from inspect import ismethod, signature +import numpy as np + from policyengine_core import periods as periods_ from policyengine_core.simulations import Simulation as CoreSimulation from policyengine_core.taxbenefitsystems import TaxBenefitSystem @@ -19,6 +21,7 @@ from spm_calculator.policyengine_adapter import ( FORMULA_OWNED_INPUTS, PolicyEngineSPMProvider, + build_policyengine_variables, ) from spm_calculator.rolling_forecast import load_forecast @@ -40,10 +43,11 @@ # SPM resource total, and the canonical indicators ``poverty_line``, # ``poverty_gap``, ``spm_unit_is_in_spm_poverty`` and # ``spm_unit_is_in_deep_spm_poverty``. These names are country-side functions -# of those same values - ``in_poverty`` of ``poverty_gap``, +# of those same values - ``in_poverty`` of the canonical poverty indicator, # ``deep_poverty_line`` of ``poverty_line``, ``deep_poverty_gap`` of # ``deep_poverty_line`` and ``spm_unit_net_income``, ``in_deep_poverty`` of -# ``deep_poverty_gap``, and ``person_in_poverty`` of ``in_poverty`` - so a +# the canonical deep-poverty indicator, and ``person_in_poverty`` of +# ``in_poverty`` - so a # dataset column for any of them is an alias for a formula-owned output under a # name the calculator does not police. Accepting one lets the poverty # calculation documented in ``docs/usage/microsimulation.md`` and the canonical @@ -70,16 +74,27 @@ } ) +# Distribution outputs inherit the same source universe as SPM resources. +SPM_DISTRIBUTION_OUTPUTS = frozenset( + {"spm_unit_oecd_equiv_net_income", "spm_unit_income_decile"} +) + # Every name a dataset must not store, from either side of the boundary. REJECTED_DATASET_INPUTS = ( - frozenset(FORMULA_OWNED_INPUTS) | DERIVED_POVERTY_OUTPUTS | SPM_ALLOCATION_OUTPUTS + frozenset(FORMULA_OWNED_INPUTS) + | DERIVED_POVERTY_OUTPUTS + | SPM_ALLOCATION_OUTPUTS + | SPM_DISTRIBUTION_OUTPUTS ) # This source role has a head/spouse fallback for household situations. A # population producer must retain its observed boolean instead of treating the # fallback formula as ownership of the input. This declaration permits source # delivery; it does not permit synthesizing a default value when data are absent. -DATASET_SOURCE_INPUTS = frozenset({"is_spm_independent_minor_role"}) +# Annual measurement scope is also source-owned despite its household fallback. +DATASET_SOURCE_INPUTS = frozenset( + {"is_spm_independent_minor_role", "spm_unit_spm_universe_status"} +) COUNTY_FIPS_PATTERN = re.compile(r"[0-9]{5}") @@ -89,6 +104,87 @@ ) +def default_spm_universe_status(unit): + """Household requests describe included units; datasets must declare scope. + + Data origin is a scalar construction contract, not a demographic inference. + In particular, missing age or tenure never makes a unit outside the universe. + """ + values = unit.simulation.tax_benefit_system.variables[ + "spm_unit_spm_universe_status" + ].possible_values + status = values.UNRESOLVED if unit.simulation.is_over_dataset else values.INCLUDED + return unit.filled_array(status.index) + + +def spm_universe_mask(unit, period): + """Require a resolved source decision and return included unit positions.""" + status = unit("spm_unit_spm_universe_status", period).decode_to_str() + included = status == "INCLUDED" + # A dataset may store this enum as member indices, and core decodes an index + # outside the enum to "unknown" rather than rejecting it. Reading anything + # but a declaration as exclusion would silently drop a mis-encoded unit from + # the measurement, which is the inference from missing data this declaration + # exists to prevent, so only the two declared decisions resolve scope. + if not np.all(included | (status == "OUTSIDE")): + raise SPMInputError( + "SPM_UNIVERSE_REQUIRED", + "Declare INCLUDED or OUTSIDE for every SPM unit from the source " + "measurement universe; an unresolved or unreadable declaration " + "cannot be measured.", + ) + return included + + +def scoped_spm_amount(unit, period, field): + """Keep outside amounts missing and never pass those units to the provider.""" + included = spm_universe_mask(unit, period) + amounts = masked_policyengine_amount(unit, period, field, included) + return np.where(included, amounts, np.nan) + + +def country_spm_variables(): + """Apply country-owned scope to the calculator's registered amount fields.""" + fields = { + "spm_unit_reference_spm_threshold": "reference_threshold", + "spm_unit_unadjusted_spm_threshold": "unadjusted_threshold", + "spm_unit_geographic_adjustment": "geographic_factor", + "spm_unit_spm_threshold": "threshold", + "spm_unit_spm_threshold_housing_portion": "housing_portion", + } + + def formula_for(field): + def formula(unit, period, parameters): + return scoped_spm_amount(unit, period, field) + + return formula + + variables = build_policyengine_variables() + for variable in variables: + if variable.__name__ in fields: + # The calculator creates fresh classes for each system. Only the + # selection changes; its provider still owns every final amount. + variable.formula = formula_for(fields[variable.__name__]) + return variables + + +def nullable_spm_indicator(unit, period, income, threshold): + """Return 0/1 for measured units, NaN outside, and reject invalid inputs.""" + included = spm_universe_mask(unit, period) + if ( + not np.isfinite(income[included]).all() + or not np.isfinite(threshold[included]).all() + or np.any(threshold[included] <= 0) + ): + raise SPMInputError( + "SPM_MEASUREMENT_INVALID", + "Included units require finite resources and positive finite thresholds.", + ) + result = np.full(included.shape, np.nan) + result[included] = income[included] < threshold[included] + return result + + def is_county_fips(value): """Accept only a five-digit county FIPS code, supplied as text. @@ -171,6 +267,19 @@ def record_county_input_types(self, year, values): if not isinstance(value, (str, bytes)): untyped[(year, str(value))] = repr(value) + def with_county_input_types(self, year, values): + """Validate selected inputs without replacing simulation-wide receipts. + + An excluded numeric county can spell the same FIPS as an included text + county. Only the selected raw inputs constrain this measurement. Share + the canonical amount cache and provenance with the simulation, while + keeping this view's typing receipt private. + """ + selected = copy(self) + object.__setattr__(selected, "_untyped_counties", self._untyped_counties.copy()) + selected.record_county_input_types(year, values) + return selected + def require_county_input(self, year, county_fips): """Reject a county this provider cannot honour as a five-digit string.""" if self.geography_kind != "county": @@ -250,6 +359,7 @@ def masked_policyengine_amount(unit, period, field, mask): tenures = np.asarray(unit("spm_unit_tenure_type", period).decode_to_str())[mask] if bound.geography_kind == "county": counties = np.asarray(unit.household("county_fips", period))[mask] + bound = bound.with_county_input_types(int(period.start.year), counties) else: counties = [None] * len(adults) rows = [ @@ -939,6 +1049,10 @@ def set_input(self, variable_name, period, value): "Use primitive inputs and retain observed outputs under report-only names." ) result = super().set_input(variable_name, period, value) + if variable_name in DATASET_SOURCE_INPUTS: + # Scope and observed roles govern cached measurements and summaries. + # Core preserves explicit inputs while clearing dependent results. + self._invalidate_all_caches() if variable_name == "county_fips": self._record_county_input_types(period) return result diff --git a/policyengine_us/system.py b/policyengine_us/system.py index af9c644d584..06848632c01 100644 --- a/policyengine_us/system.py +++ b/policyengine_us/system.py @@ -43,11 +43,11 @@ ) from typing import Annotated -from spm_calculator.policyengine_adapter import build_policyengine_variables from policyengine_us.spm import ( SPMSimulationMixin, clone_spm_system, create_spm_provider, + country_spm_variables, ) COUNTRY_DIR = Path(__file__).parent @@ -108,7 +108,7 @@ def __init__( ): super().__init__(entities, reform=reform) self.spm_forecast_provider = create_spm_provider(spm) - self.add_variables(*build_policyengine_variables()) + self.add_variables(*country_spm_variables()) self.load_parameters(COUNTRY_DIR / "parameters") self.add_abolition_parameters() self.parameters = set_irs_uprating_parameter(self.parameters) diff --git a/policyengine_us/tests/core/test_income_decile_ties.py b/policyengine_us/tests/core/test_income_decile_ties.py new file mode 100644 index 00000000000..ea4bfb0c9af --- /dev/null +++ b/policyengine_us/tests/core/test_income_decile_ties.py @@ -0,0 +1,209 @@ +"""Invented person-weighted deciles under the shared upper-rank tie convention. + +Python is needed for independently shuffled entity tables, unequal weights and +member counts, and count-gated nullable group summaries. No population file or +formula-owned dataset output is used. Both decile formulas remain unmodified. +""" + +import socket + +import h5py +import numpy as np +import pandas as pd +import pytest +import requests +from policyengine_core.reforms import Reform +from policyengine_core.variables import Variable + +from policyengine_us import Microsimulation +from policyengine_us.data.dataset_schema import USSingleYearDataset +from policyengine_us.spm import spm_universe_mask + + +YEAR = 2024 +STATUS = "spm_unit_spm_universe_status" +EXPECTED_DECILES = {10: 1, 20: 7, 30: 7, 40: 10} +MEASURES = ( + pytest.param("household", "household_income_decile", id="household"), + pytest.param("spm_unit", "spm_unit_income_decile", id="spm"), +) + + +@pytest.fixture(autouse=True) +def no_population_data_reads(monkeypatch): + def forbidden(*args, **kwargs): + raise AssertionError("Decile tie tests cannot read population data or network") + + monkeypatch.setattr(requests.sessions.Session, "request", forbidden) + monkeypatch.setattr(socket.socket, "connect", forbidden) + monkeypatch.setattr(pd, "HDFStore", forbidden) + monkeypatch.setattr(h5py, "File", forbidden) + + +def _invented_income(entity_ids): + """Income labels attached to IDs, independent of source row order.""" + return np.select( + [entity_ids == 10, entity_ids == 20, entity_ids == 30, entity_ids == 40], + [50.0, 100.0, 100.0, 300.0], + default=1_000_000_000.0, + ) + + +class household_net_income(Variable): + def formula(household, period, parameters): + return _invented_income(household("household_id", period)) + + +class spm_unit_net_income(Variable): + def formula(spm_unit, period, parameters): + included = spm_universe_mask(spm_unit, period) + equivalised_income = _invented_income(spm_unit("spm_unit_id", period)) + # The four-member unit receives resources 200 and the actual OECD + # formula divides by sqrt(4), tying it with the one-member unit at 100. + resources = equivalised_income * np.sqrt(spm_unit.nb_persons()) + return np.where(included, resources, np.nan) + + +class InventedResources(Reform): + """Supply test resources through formulas, preserving dataset input guards.""" + + def apply(self): + self.update_variable(household_net_income) + self.update_variable(spm_unit_net_income) + + +def invented_dataset(*, shuffled=False, include_outside=False): + # ID, member count, household weight. The included person weights are + # 1, 2, 4 and 3, totaling 10. Below the tie is weight1; the tie contributes + # weight2+4, so BOTH tied units have upper cumulative rank7/10 -> decile7. + # These are hand-specified fixture expectations, not aggregated model weights. + units = [(10, 1, 1.0), (20, 1, 2.0), (30, 4, 1.0), (40, 1, 3.0)] + if include_outside: + units.append((50, 1, 1_000_000.0)) + people = [] + for identity, members, _ in units: + for member in range(members): + person_id = identity * 10 + member + 1 + people.append( + { + "person_id": person_id, + "age": 40, + "is_household_head": member == 0, + "is_tax_unit_head": member == 0, + "person_marital_unit_id": person_id, + **{ + f"person_{entity}_id": identity + for entity in ("household", "tax_unit", "family", "spm_unit") + }, + } + ) + ids = [identity for identity, _, _ in units] + tables = { + "person": pd.DataFrame(people), + "household": pd.DataFrame( + { + "household_id": ids, + "household_weight": [weight for _, _, weight in units], + "state_code": "CA", + "county_fips": "06037", + } + ), + "spm_unit": pd.DataFrame( + { + "spm_unit_id": ids, + STATUS: [ + "OUTSIDE" if identity == 50 else "INCLUDED" for identity in ids + ], + } + ), + "tax_unit": pd.DataFrame({"tax_unit_id": ids}), + "family": pd.DataFrame({"family_id": ids}), + "marital_unit": pd.DataFrame( + {"marital_unit_id": [person["person_id"] for person in people]} + ), + } + if shuffled: + # Different permutations ensure IDs, not matching table positions, link + # people and groups. Each permutation is deterministic. + tables = { + name: frame.sample(frac=1, random_state=17 + index).reset_index(drop=True) + for index, (name, frame) in enumerate(tables.items()) + } + return USSingleYearDataset(**tables, time_period=YEAR) + + +def simulate(**kwargs): + return Microsimulation(dataset=invented_dataset(**kwargs), reform=InventedResources) + + +def deciles_by_id(simulation, entity, variable): + ids = simulation.calc(f"{entity}_id", period=YEAR, map_to=entity) + deciles = simulation.calc(variable, period=YEAR, map_to=entity) + # Align scalar assertions by identity only; all weighted statistics below + # remain MicroSeries operations on model-projected results. + return {int(ids.iloc[index]): deciles.iloc[index] for index in range(len(ids))} + + +@pytest.mark.parametrize("entity,variable", MEASURES) +@pytest.mark.parametrize("shuffled", [False, True], ids=["ordered", "shuffled"]) +def test_equal_incomes_share_the_upper_person_weighted_decile( + entity, variable, shuffled +): + simulation = simulate(shuffled=shuffled) + assert deciles_by_id(simulation, entity, variable) == EXPECTED_DECILES + ids = simulation.calc(f"person_{entity}_id", period=YEAR, map_to="person") + deciles = simulation.calc(variable, period=YEAR, map_to="person") + for index in range(len(ids)): + assert deciles.iloc[index] == EXPECTED_DECILES[int(ids.iloc[index])] + assert deciles.count() == 10 + # A household-count-weighted estimator would put the tie at ceil(4/7*10)=6; + # ignoring survey weights would put it at ceil(6/7*10)=9. Neither is seven. + assert (deciles == 7).sum() == 6 + + +@pytest.mark.parametrize("entity,variable", MEASURES) +def test_independently_shuffled_tables_preserve_deciles_by_entity_id(entity, variable): + ordered = simulate() + shuffled = simulate(shuffled=True) + assert deciles_by_id(ordered, entity, variable) == deciles_by_id( + shuffled, entity, variable + ) + + +@pytest.mark.parametrize("shuffled", [False, True], ids=["ordered", "shuffled"]) +def test_dominant_outside_weight_is_excluded_from_spm_tie_ranks(shuffled): + simulation = simulate(shuffled=shuffled, include_outside=True) + standalone = simulate(shuffled=shuffled) + actual = deciles_by_id(simulation, "spm_unit", "spm_unit_income_decile") + expected = deciles_by_id(standalone, "spm_unit", "spm_unit_income_decile") + assert {identity: actual[identity] for identity in EXPECTED_DECILES} == expected + assert expected == EXPECTED_DECILES + assert pd.isna(actual[50]) + ranks = simulation.calc("spm_unit_income_decile", period=YEAR, map_to="person") + assert ranks.count() == 10 + assert (ranks == 7).sum() == 6 + assert simulation.calc("age", period=YEAR, map_to="person").count() == 1_000_010 + + +@pytest.mark.parametrize("entity,variable", MEASURES) +def test_empty_decile_groups_have_zero_counts_and_nullable_means(entity, variable): + simulation = simulate() + deciles = simulation.calc(variable, period=YEAR, map_to="person") + income_variable = ( + "household_net_income" + if entity == "household" + else "spm_unit_oecd_equiv_net_income" + ) + income = simulation.calc(income_variable, period=YEAR, map_to="person") + expected_nonempty = {1: (1, 50), 7: (6, 100), 10: (3, 300)} + for decile in range(1, 11): + group = income[deciles == decile] + count = group.count() + mean = group.mean() if count > 0 else None + if decile in expected_nonempty: + expected_count, expected_mean = expected_nonempty[decile] + assert count == expected_count + assert mean == expected_mean + else: + assert count == 0 + assert mean is None diff --git a/policyengine_us/tests/core/test_spm_production_parity.py b/policyengine_us/tests/core/test_spm_production_parity.py new file mode 100644 index 00000000000..694a0ff7119 --- /dev/null +++ b/policyengine_us/tests/core/test_spm_production_parity.py @@ -0,0 +1,322 @@ +"""Capture invented all-INCLUDED results in separately pinned model processes. + +This file has no model imports at collection. A guarded runner explicitly opts +in with SPM_PARITY_VARIANT, SPM_PARITY_COUNTRY_ROOT and +SPM_PARITY_FORECAST_SHA256. It runs this exact file against baseline and port +environments separately; no country module or source overlay is permitted. +The comparison node reads only the two resulting, externally admitted JSON files. +""" + +import hashlib +from importlib import metadata +import json +import math +import os +from pathlib import Path +import re + +import pytest + + +YEAR = 2024 +STATUS = "spm_unit_spm_universe_status" +DEPENDENCIES = { + "policyengine-core": "3.32.5", + "microdf-python": "1.3.0", + "spm-calculator": "1.0.0", + "numpy": "2.4.6", + "pandas": "3.0.3", +} +NULLABLE_INDICATORS = ( + "spm_unit_is_in_spm_poverty", + "spm_unit_is_in_deep_spm_poverty", + "in_poverty", + "in_deep_poverty", + "person_in_poverty", +) +VARIABLES = ( + "spm_unit_reference_spm_threshold", + "spm_unit_unadjusted_spm_threshold", + "spm_unit_spm_threshold", + "spm_unit_spm_threshold_housing_portion", + "spm_unit_geographic_adjustment", + "spm_unit_capped_housing_subsidy", + "spm_unit_benefits", + "spm_unit_net_income", + "poverty_line", + "deep_poverty_line", + "poverty_gap", + "deep_poverty_gap", + "spm_unit_oecd_equiv_net_income", + "spm_unit_income_decile", + *NULLABLE_INDICATORS, + "housing_assistance", + "spm_unit_allocated_housing_subsidy", + "spm_unit_allocated_tenant_payment", + "household_benefits", + "household_net_income", + "cbo_household_means_tested_transfers", +) + + +def _fixture_tables(): + """Four people, three SPM units and two households; no stored SPM outputs.""" + return { + "person": { + "person_id": [101, 102, 103, 201], + "age": [40, 8, 17, 50], + "employment_income": [6_000.0, 0.0, 10_000.0, 100_000.0], + "is_spm_independent_minor_role": [False, False, True, False], + "is_household_head": [True, False, False, True], + "is_tax_unit_head": [True, False, True, True], + "is_tax_unit_dependent": [False, True, False, False], + "person_household_id": [10, 10, 10, 20], + "person_spm_unit_id": [10, 10, 30, 20], + "person_tax_unit_id": [10, 10, 30, 20], + "person_family_id": [10, 10, 30, 20], + "person_marital_unit_id": [101, 102, 103, 201], + }, + "household": { + "household_id": [10, 20], + "household_weight": [2.0, 5.0], + "state_code": ["CA", "NY"], + "county_fips": ["06037", "36061"], + }, + "spm_unit": { + "spm_unit_id": [10, 30, 20], + "spm_unit_tenure_type": ["RENTER", "RENTER", "RENTER"], + "is_eligible_for_housing_assistance": [True, True, False], + "takes_up_housing_assistance_if_eligible": [True, True, False], + "hud_hap": [9_000.0, 0.0, 0.0], + # The unawarded minor's hypothetical TTP must not be allocated. + "hud_ttp": [900.0, 2_000.0, 0.0], + }, + "tax_unit": {"tax_unit_id": [10, 30, 20]}, + "family": {"family_id": [10, 30, 20]}, + "marital_unit": {"marital_unit_id": [101, 102, 103, 201]}, + } + + +def _json_digest(value): + raw = json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False) + return hashlib.sha256(raw.encode()).hexdigest() + + +def _compact_provenance(provenance): + """Authenticate large forecast receipts without repeating bundled tables.""" + small_fields = { + "forecast_id", + "forecast_sha256", + "scenario", + "geography_kind", + "runtime_versions", + "composition_method", + "storage_method", + } + assert set(provenance) == small_fields | {"years", "geographies"} + return { + **{key: provenance[key] for key in sorted(small_fields)}, + "full_sha256": _json_digest(provenance), + "year_count": len(provenance["years"]), + "years": { + year: _json_digest(value) for year, value in provenance["years"].items() + }, + "geographies": { + "count": len(provenance["geographies"]), + "sha256": _json_digest(provenance["geographies"]), + }, + } + + +def _finite_number(value): + result = float(value) + assert math.isfinite(result), "All-INCLUDED fixture returned a nonfinite value" + return result + + +def _values_by_id(simulation, variable, entity): + ids = simulation.calc(f"{entity}_id", period=YEAR, map_to=entity) + values = simulation.calc(variable, period=YEAR, map_to=entity) + assert len(ids) == len(values) + result = { + str(int(ids.iloc[index])): _finite_number(values.iloc[index]) + for index in range(len(ids)) + } + assert len(result) == len(ids), "Duplicate entity identity" + return result + + +@pytest.mark.parametrize("geography", ["county", "national"]) +def test_all_included_production_capture(geography): + variant = os.environ.get("SPM_PARITY_VARIANT") + if variant is None: + pytest.skip("Explicit separately pinned parity process required") + assert variant in {"baseline", "port"} + country_root = Path(os.environ["SPM_PARITY_COUNTRY_ROOT"]).resolve(strict=True) + forecast_sha = os.environ["SPM_PARITY_FORECAST_SHA256"] + assert re.fullmatch(r"[a-f0-9]{64}", forecast_sha) + artifact = Path(os.environ["BOUNDED_ARTIFACT_DIR"]).resolve(strict=True) + assert artifact.is_dir() + + # The caller's guard and exact origin checks must already be active here. + import pandas as pd + import policyengine_us + from policyengine_us import Microsimulation + from policyengine_us.data.dataset_schema import USSingleYearDataset + + assert Path(policyengine_us.__file__).resolve() == ( + country_root / "policyengine_us/__init__.py" + ) + versions = {name: metadata.version(name) for name in DEPENDENCIES} + assert versions == DEPENDENCIES + country_version = metadata.version("policyengine-us") + assert ( + country_version + == { + "baseline": "2.2.1", + "port": "2.2.1+spmuniverse.20260919", + }[variant] + ) + + primitive_tables = _fixture_tables() + primitive_sha = _json_digest(primitive_tables) + tables = {name: pd.DataFrame(data) for name, data in primitive_tables.items()} + if variant == "port": + tables["spm_unit"][STATUS] = "INCLUDED" + source = USSingleYearDataset(**tables, time_period=YEAR) + source_before = [table.copy(deep=True) for table in source.tables] + config = { + "forecast_content_sha256": forecast_sha, + "scenario": "ce_trend", + "geography_kind": geography, + "geography_id": None, + "county_vintage": "2020", + "as_of": "2026-09-09", + } + simulation = Microsimulation(dataset=source, spm=config) + assert simulation.spm_config == config + assert (STATUS in simulation.tax_benefit_system.variables) == (variant == "port") + if variant == "port": + assert list(simulation.calc(STATUS, period=YEAR)) == ["INCLUDED"] * 3 + + # Check both nonzero allocation and an unassisted recipient's phantom TTP. + assert _values_by_id(simulation, "housing_assistance", "spm_unit") == { + "10": 9_000.0, + "30": 0.0, + "20": 0.0, + } + subsidy = _values_by_id( + simulation, "spm_unit_allocated_housing_subsidy", "spm_unit" + ) + assert subsidy == { + "10": 6_000.0, + "30": 3_000.0, + "20": 0.0, + } + tenant_payment = _values_by_id( + simulation, "spm_unit_allocated_tenant_payment", "spm_unit" + ) + assert tenant_payment == { + "10": 600.0, + "30": 300.0, + "20": 0.0, + } + caps = simulation.calc("spm_unit_capped_housing_subsidy", period=YEAR) + assert (caps > 0).any(), "A zero-only fixture cannot qualify housing allocation" + + measures = {} + for name in VARIABLES: + entity = simulation.tax_benefit_system.variables[name].entity.key + values = simulation.calc(name, period=YEAR, map_to="person") + assert not values.isna().any(), name + assert values.count() == 11, name # Three people at weight2, one at weight5. + if name in NULLABLE_INDICATORS: + assert values.isin([0, 1]).all(), name + measures[name] = { + "entity": entity, + "person_dtype": str(values.dtype), + "entity_values": _values_by_id(simulation, name, entity), + "person_values": _values_by_id(simulation, name, "person"), + "weighted_count": _finite_number(values.count()), + "weighted_sum": _finite_number(values.sum()), + "weighted_mean": _finite_number(values.mean()), + } + for before, after in zip(source_before, source.tables, strict=True): + pd.testing.assert_frame_equal(before, after) + assert _json_digest(primitive_tables) == primitive_sha + provenance = simulation.spm_provenance() + assert set(provenance["years"]) == {str(YEAR)} + record = { + "schema": "spm-invented-production-parity/v2", + "variant": variant, + "country_root": str(country_root), + "country_version": country_version, + "dependency_versions": versions, + "helper_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + "primitive_tables_sha256": primitive_sha, + "year": YEAR, + "spm_config": config, + "measures": measures, + "spm_provenance": _compact_provenance(provenance), + "scope": "Invented all-INCLUDED fixture; no native or release qualification", + } + encoded = ( + json.dumps(record, sort_keys=True, indent=2, allow_nan=False) + "\n" + ).encode() + assert len(encoded) < 128 * 1024 + with (artifact / f"spm-parity-{variant}-{geography}.json").open("xb") as stream: + stream.write(encoded) + + +@pytest.mark.parametrize("geography", ["county", "national"]) +def test_compare_all_included_production_captures(geography): + if "SPM_PARITY_BASELINE_ARTIFACT_DIR" not in os.environ: + pytest.skip("Two completed, separately admitted capture artifacts required") + records = {} + for variant in ("baseline", "port"): + directory = Path(os.environ[f"SPM_PARITY_{variant.upper()}_ARTIFACT_DIR"]) + path = directory / f"spm-parity-{variant}-{geography}.json" + assert path.stat().st_size < 128 * 1024 + record = json.loads(path.read_text()) + assert record["schema"] == "spm-invented-production-parity/v2" + assert record["variant"] == variant + assert ( + record["helper_sha256"] + == hashlib.sha256(Path(__file__).read_bytes()).hexdigest() + ) + assert set(record["measures"]) == set(VARIABLES) + assert record["dependency_versions"] == DEPENDENCIES + assert record["year"] == YEAR + provenance = record["spm_provenance"] + assert provenance["runtime_versions"] == { + "policyengine": None, + "policyengine-us": record["country_version"], + "policyengine-core": DEPENDENCIES["policyengine-core"], + "spm-calculator": DEPENDENCIES["spm-calculator"], + } + assert provenance["year_count"] == 1 + assert set(provenance["years"]) == {str(YEAR)} + assert ( + provenance["forecast_sha256"] + == record["spm_config"]["forecast_content_sha256"] + ) + records[variant] = record + baseline, port = records["baseline"], records["port"] + assert baseline["primitive_tables_sha256"] == port["primitive_tables_sha256"] + assert baseline["spm_config"] == port["spm_config"] + assert baseline["spm_config"]["geography_kind"] == geography + assert baseline["country_root"] != port["country_root"] + for key in baseline["spm_provenance"]: + if key not in {"full_sha256", "runtime_versions"}: + assert baseline["spm_provenance"][key] == port["spm_provenance"][key], key + for name in VARIABLES: + expected, actual = baseline["measures"][name], port["measures"][name] + assert set(expected) == set(actual) + for key in expected: + if key == "person_dtype" and name in ( + *NULLABLE_INDICATORS, + "spm_unit_income_decile", + ): + assert actual[key].startswith("float"), name + continue + assert actual[key] == expected[key], (name, key) diff --git a/policyengine_us/tests/core/test_spm_source_input_contract.py b/policyengine_us/tests/core/test_spm_source_input_contract.py index 76e1e340079..4c20e353aae 100644 --- a/policyengine_us/tests/core/test_spm_source_input_contract.py +++ b/policyengine_us/tests/core/test_spm_source_input_contract.py @@ -1,6 +1,7 @@ """A household fallback formula must not hide an observed population role.""" -from policyengine_core.periods import ETERNITY +from policyengine_core.enums import Enum +from policyengine_core.periods import ETERNITY, YEAR import pytest from policyengine_us import Simulation @@ -9,13 +10,27 @@ def test_source_contract_is_disjoint_and_matches_registered_role(): - assert DATASET_SOURCE_INPUTS == {"is_spm_independent_minor_role"} + assert DATASET_SOURCE_INPUTS == { + "is_spm_independent_minor_role", + "spm_unit_spm_universe_status", + } assert not DATASET_SOURCE_INPUTS & REJECTED_DATASET_INPUTS variable = system.variables["is_spm_independent_minor_role"] assert variable.entity.key == "person" assert variable.value_type is bool assert variable.definition_period == ETERNITY assert variable.formulas # Household fallback remains available. + scope = system.variables["spm_unit_spm_universe_status"] + assert scope.entity.key == "spm_unit" + assert scope.value_type is Enum + assert scope.definition_period == YEAR + assert scope.default_value.name == "UNRESOLVED" + assert {value.name for value in scope.possible_values} == { + "INCLUDED", + "OUTSIDE", + "UNRESOLVED", + } + assert scope.formulas # A situation fallback is separate from the enum default. @pytest.mark.parametrize("source_role,adults", [(None, 1), (False, 1), (True, 2)]) diff --git a/policyengine_us/tests/core/test_spm_system.py b/policyengine_us/tests/core/test_spm_system.py index 54c94b2e2e2..d733c07bf86 100644 --- a/policyengine_us/tests/core/test_spm_system.py +++ b/policyengine_us/tests/core/test_spm_system.py @@ -23,6 +23,7 @@ from policyengine_us.spm import ( DERIVED_POVERTY_OUTPUTS, REJECTED_DATASET_INPUTS, + SPM_DISTRIBUTION_OUTPUTS, create_spm_provider, is_county_fips, ) @@ -559,10 +560,18 @@ def small_dataset(): "household_weight": [1.0, 1.0], } ), + # These two deliberate household units are included for this synthetic + # year's measurements. Real/default population files are not relabelled. + spm_unit=pd.DataFrame( + { + "spm_unit_id": [1, 2], + "spm_unit_spm_universe_status": ["INCLUDED", "INCLUDED"], + } + ), **{ entity: pd.DataFrame({f"{entity}_id": [1, 2]}) for entity in groups - if entity != "household" + if entity not in {"household", "spm_unit"} }, time_period=2024, ) @@ -588,7 +597,7 @@ def test_dataset_roles_receipts_and_formula_owned_rejection_preserve_source( assert np.all(simulation.calculate("spm_unit_spm_threshold", 2024) > 0) assert len(simulation.spm_provenance()["geographies"]) == 2 json.dumps(simulation.spm_provenance()) - for name in FORMULA_OWNED_INPUTS: + for name in REJECTED_DATASET_INPUTS: with pytest.raises(ValueError, match="formula-owned SPM output"): simulation.set_input(variable_name=name, period=2024, value=[1.0, 1.0]) for original, current in zip(before, source.tables): @@ -619,7 +628,9 @@ def test_microsimulation_rejects_stored_measurements_without_mutating_dataset( assert path.read_bytes() == content -@pytest.mark.parametrize("name", sorted(DERIVED_POVERTY_OUTPUTS)) +@pytest.mark.parametrize( + "name", sorted(DERIVED_POVERTY_OUTPUTS | SPM_DISTRIBUTION_OUTPUTS) +) def test_dataset_storing_a_derived_poverty_output_is_refused(name): """A poverty alias the calculator does not own must still be refused. @@ -659,9 +670,14 @@ def test_accepted_population_publishes_one_poverty_rate(): def test_rejection_set_extends_rather_than_replaces_the_calculator_contract(): for name in FORMULA_OWNED_INPUTS: assert name in REJECTED_DATASET_INPUTS - for name in DERIVED_POVERTY_OUTPUTS: + for name in DERIVED_POVERTY_OUTPUTS | SPM_DISTRIBUTION_OUTPUTS: # A rejected name must be computed, never a legitimate stored input. + assert name in REJECTED_DATASET_INPUTS assert system.variables[name].formula is not None + assert SPM_DISTRIBUTION_OUTPUTS == { + "spm_unit_oecd_equiv_net_income", + "spm_unit_income_decile", + } ENTITY_KEYS = frozenset( @@ -743,7 +759,7 @@ def test_every_pure_function_of_the_poverty_chain_is_rejected(): chain = { name for name in REJECTED_DATASET_INPUTS - if "poverty" in name or "threshold" in name + if "poverty" in name or "threshold" in name or name in SPM_DISTRIBUTION_OUTPUTS } unrejected_aliases = [] checked = [] @@ -762,6 +778,10 @@ def test_every_pure_function_of_the_poverty_chain_is_rejected(): ) # Guard the guard: a scan that silently matched nothing proves nothing. assert set(DERIVED_POVERTY_OUTPUTS) <= set(checked) + # These distribution outputs also read scope/weights, so they are not pure + # aliases captured by the subset scan. Their names must still be rejected + # explicitly, and extending the chain above covers aliases of either one. + assert SPM_DISTRIBUTION_OUTPUTS <= REJECTED_DATASET_INPUTS @pytest.fixture diff --git a/policyengine_us/tests/core/test_spm_universe.py b/policyengine_us/tests/core/test_spm_universe.py new file mode 100644 index 00000000000..44a296273fb --- /dev/null +++ b/policyengine_us/tests/core/test_spm_universe.py @@ -0,0 +1,1067 @@ +"""Synthetic mixed-universe contracts requiring the Python simulation API. + +These tests inspect +provider calls, missing outputs, independent entity membership, and preservation +of explicitly supplied source data; the YAML runner cannot express those checks. +No native population data, data build, or default dataset is used. +""" + +from importlib import import_module +import socket + +import h5py +import numpy as np +import pandas as pd +import pytest +import requests +from spm_calculator.errors import SPMInputError +from spm_calculator.policyengine_adapter import PolicyEngineSPMProvider + +from policyengine_us import Microsimulation, Simulation +from policyengine_core.reforms import Reform +from policyengine_core.variables import Variable + +from policyengine_us.data.dataset_schema import USMultiYearDataset, USSingleYearDataset +from policyengine_us.spm import REJECTED_DATASET_INPUTS +from policyengine_us.system import CountryTaxBenefitSystem, system + + +YEAR = 2024 +STATUS = "spm_unit_spm_universe_status" +POVERTY_INDICATORS = ( + "spm_unit_is_in_spm_poverty", + "spm_unit_is_in_deep_spm_poverty", + "in_poverty", + "in_deep_poverty", + "person_in_poverty", +) +MEASUREMENT_AMOUNTS = ( + "spm_unit_reference_spm_threshold", + "spm_unit_unadjusted_spm_threshold", + "spm_unit_spm_threshold", + "spm_unit_spm_threshold_housing_portion", + "spm_unit_geographic_adjustment", + "spm_unit_capped_housing_subsidy", + "spm_unit_benefits", + "spm_unit_net_income", + "poverty_line", + "deep_poverty_line", + "poverty_gap", + "deep_poverty_gap", + "spm_unit_oecd_equiv_net_income", +) +DISTRIBUTION_OUTPUTS = ("spm_unit_oecd_equiv_net_income", "spm_unit_income_decile") +ALL_MEASUREMENTS = (*MEASUREMENT_AMOUNTS, *POVERTY_INDICATORS, "spm_unit_income_decile") +ORDINARY_OUTCOMES = ( + "housing_assistance", + "spm_unit_allocated_housing_subsidy", + "spm_unit_allocated_tenant_payment", + "household_benefits", + "household_net_income", + "cbo_household_means_tested_transfers", +) + + +@pytest.fixture(autouse=True) +def no_population_data_reads(monkeypatch): + def forbidden(*args, **kwargs): + raise AssertionError("Mixed SPM tests must not read population data") + + monkeypatch.setattr(requests.sessions.Session, "request", forbidden) + monkeypatch.setattr(socket.socket, "connect", forbidden) + monkeypatch.setattr(pd, "HDFStore", forbidden) + monkeypatch.setattr(h5py, "File", forbidden) + + +@pytest.fixture +def mixed_source(): + """One ordinary parent/child unit and one source-marked GQ child unit.""" + groups = ("household", "tax_unit", "spm_unit", "family") + return USSingleYearDataset( + person=pd.DataFrame( + { + "person_id": [101, 102, 301], + "age": [40, 8, 12], + "employment_income": [6_000.0, 0.0, 0.0], + "is_spm_independent_minor_role": [False, False, False], + "is_household_head": [True, False, False], + "is_tax_unit_head": [True, False, True], + "is_tax_unit_dependent": [False, True, False], + **{f"person_{entity}_id": [10, 10, 30] for entity in groups}, + "person_marital_unit_id": [101, 102, 301], + } + ), + household=pd.DataFrame( + { + "household_id": [10, 30], + "state_code": ["CA", "CA"], + # The outside record deliberately lacks SPM geography. + "county_fips": ["06037", ""], + "household_weight": [2.0, 5.0], + } + ), + tax_unit=pd.DataFrame({"tax_unit_id": [10, 30]}), + family=pd.DataFrame({"family_id": [10, 30]}), + marital_unit=pd.DataFrame({"marital_unit_id": [101, 102, 301]}), + spm_unit=pd.DataFrame( + { + "spm_unit_id": [10, 30], + STATUS: ["INCLUDED", "OUTSIDE"], + "spm_unit_tenure_type": ["RENTER", "RENTER"], + "is_eligible_for_housing_assistance": [True, True], + "takes_up_housing_assistance_if_eligible": [True, True], + "hud_hap": [30_000.0, 30_000.0], + "hud_ttp": [1_000.0, 1_000.0], + } + ), + time_period=YEAR, + ) + + +def _only_included(source): + """A separately built comparison fixture with the identical ordinary unit.""" + return USSingleYearDataset( + person=source.person.loc[source.person.person_household_id == 10].copy(), + household=source.household.loc[source.household.household_id == 10].copy(), + tax_unit=source.tax_unit.loc[source.tax_unit.tax_unit_id == 10].copy(), + spm_unit=source.spm_unit.loc[source.spm_unit.spm_unit_id == 10].copy(), + family=source.family.loc[source.family.family_id == 10].copy(), + marital_unit=source.marital_unit.loc[ + source.marital_unit.marital_unit_id.isin([101, 102]) + ].copy(), + time_period=YEAR, + ) + + +def _capture_provider(monkeypatch): + calls = [] + original = PolicyEngineSPMProvider.calculate_unit + + def capture(self, **kwargs): + calls.append(kwargs.copy()) + assert kwargs["adults"] >= 1, "Outside child-only unit reached SPM provider" + assert kwargs.get("county_fips") != "", "Outside geography reached provider" + return original(self, **kwargs) + + monkeypatch.setattr(PolicyEngineSPMProvider, "calculate_unit", capture) + return calls + + +@pytest.mark.parametrize("simulation_type", [Simulation, Microsimulation]) +def test_mixed_measurement_retains_included_values_and_missing_outside( + mixed_source, monkeypatch, simulation_type +): + calls = _capture_provider(monkeypatch) + simulation = simulation_type(dataset=mixed_source) + included = simulation_type(dataset=_only_included(mixed_source)) + for variable in (*MEASUREMENT_AMOUNTS, *POVERTY_INDICATORS): + actual = simulation.calculate(variable, YEAR, map_to="person") + expected = included.calculate(variable, YEAR, map_to="person") + assert np.issubdtype(actual.dtype, np.floating), variable + np.testing.assert_array_equal(actual[:2], expected, err_msg=variable) + assert np.isnan(actual[2]), f"{variable} outside must be missing, never false" + if simulation_type is Microsimulation and variable in POVERTY_INDICATORS: + # The two included people inherit household weight 2; the outside + # person's weight 5 must not enter the measurement denominator. + assert actual.count() == 4 + assert actual.mean() == expected.mean() + assert calls, "Included units must execute the real canonical provider" + assert {call["year"] for call in calls} == {YEAR} + assert {call["county_fips"] for call in calls} == {"06037"} + np.testing.assert_array_equal( + simulation.calculate("person_id", YEAR), [101, 102, 301] + ) + np.testing.assert_array_equal(simulation.calculate("spm_unit_size", YEAR), [2, 1]) + np.testing.assert_array_equal( + simulation.calculate("spm_unit_count_children", YEAR), [1, 1] + ) + + +@pytest.mark.parametrize("source_table", ["spm_unit", "household"]) +def test_source_universe_is_not_inferred_for_generated_future_year( + mixed_source, source_table +): + source = mixed_source.copy() + if source_table == "household": + # The accepted dataset loader flattens entity tables, so a declaration + # stored on this one-SPM-unit-per-household table is also supplied. + source.household[STATUS] = source.spm_unit.pop(STATUS) + original_tables = [frame.copy(deep=True) for frame in source.tables] + simulation = Microsimulation(dataset=source) + assert list(simulation.calculate(STATUS, YEAR)) == ["INCLUDED", "OUTSIDE"] + future_status = simulation.calculate(STATUS, YEAR + 1) + assert list(future_status) == ["UNRESOLVED", "UNRESOLVED"] + with pytest.raises(SPMInputError): + simulation.calculate("spm_unit_spm_threshold", YEAR + 1) + for original, current in zip(original_tables, source.tables): + pd.testing.assert_frame_equal(original, current) + + +@pytest.mark.parametrize("earnings,expected_poor", [(0.0, 1.0), (100_000.0, 0.0)]) +@pytest.mark.parametrize("month", ["01", "07"]) +def test_monthly_poverty_status_preserves_annual_boolean_values_and_missingness( + mixed_source, earnings, expected_poor, month +): + source = mixed_source.copy() + # A single included adult with zero or high earnings makes both ordinary + # and deep-poverty outcomes unambiguous, alongside the outside child. + source.person = source.person.loc[source.person.person_id != 102].copy() + source.marital_unit = source.marital_unit.loc[ + source.marital_unit.marital_unit_id != 102 + ].copy() + source.person["employment_income"] = [earnings, 0.0] + source.spm_unit["hud_hap"] = [0.0, 30_000.0] + simulation = Microsimulation(dataset=source) + for variable in POVERTY_INDICATORS: + annual = simulation.calculate(variable, YEAR, map_to="person") + monthly = simulation.calculate(variable, f"{YEAR}-{month}", map_to="person") + assert annual.iloc[0] == expected_poor, variable + np.testing.assert_array_equal(monthly, annual, err_msg=variable) + assert np.isnan(monthly.iloc[1]), variable + assert monthly.count() == annual.count() == 2 + assert monthly.mean() == annual.mean() == expected_poor + + +def test_all_outside_measurements_have_no_provider_calls_or_poverty_denominator( + mixed_source, monkeypatch +): + source = mixed_source.copy() + source.spm_unit[STATUS] = ["OUTSIDE", "OUTSIDE"] + + def forbidden(*args, **kwargs): + raise AssertionError("An all-outside population must not execute SPM") + + monkeypatch.setattr(PolicyEngineSPMProvider, "calculate_unit", forbidden) + simulation = Microsimulation(dataset=source) + for variable in ALL_MEASUREMENTS: + result = simulation.calculate(variable, YEAR, map_to="person") + assert np.isnan(result).all(), variable + assert result.count() == 0, variable + assert np.isnan(result.mean()), variable + + +@pytest.mark.parametrize("variable", ALL_MEASUREMENTS) +def test_unresolved_measurement_status_fails_closed( + mixed_source, variable, monkeypatch +): + source = mixed_source.copy() + # Make both units calculable so composition/geography errors cannot mask + # a missing check for the unresolved source classification. + source.person["age"] = [40, 8, 40] + source.household["county_fips"] = ["06037", "36061"] + source.spm_unit[STATUS] = ["INCLUDED", "UNRESOLVED"] + + def forbidden(*args, **kwargs): + raise AssertionError("Unresolved measurement scope must fail before SPM") + + monkeypatch.setattr(PolicyEngineSPMProvider, "calculate_unit", forbidden) + with pytest.raises(SPMInputError) as error: + Microsimulation(dataset=source).calculate(variable, YEAR) + assert error.value.code == "SPM_UNIVERSE_REQUIRED" + + +@pytest.mark.parametrize("variable", ALL_MEASUREMENTS) +def test_unreadable_measurement_status_fails_closed( + mixed_source, variable, monkeypatch +): + """An unreadable declaration is not a declaration of exclusion. + + A producer may store this enum as member indices rather than member names, + and core decodes an index outside the enum to ``"unknown"`` instead of + rejecting it (``EnumArray.decode_to_str``). Reading that as anything but a + failure would drop the unit from the measurement with no error, which is + exactly the exclusion-inferred-from-missing-data the scope declaration + exists to prevent. + """ + source = mixed_source.copy() + # Make both units calculable so composition/geography errors cannot mask + # a missing check for the unreadable source classification. + source.person["age"] = [40, 8, 40] + source.household["county_fips"] = ["06037", "36061"] + # One declared unit and one index the enum does not define. + source.spm_unit[STATUS] = np.array([0, 99], dtype=np.int8) + + def forbidden(*args, **kwargs): + raise AssertionError("An unreadable measurement scope must fail before SPM") + + monkeypatch.setattr(PolicyEngineSPMProvider, "calculate_unit", forbidden) + with pytest.raises(SPMInputError) as error: + Microsimulation(dataset=source).calculate(variable, YEAR) + assert error.value.code == "SPM_UNIVERSE_REQUIRED" + + +def test_member_index_declarations_resolve_the_same_scope_as_member_names( + mixed_source, +): + """Both accepted storage encodings of the declaration measure the same units. + + ``Enum.encode`` takes a string column by member name and an integer column + as member indices, so a producer may ship either. Neither encoding may + change which units are measured. + """ + by_name = mixed_source.copy() + by_name.spm_unit[STATUS] = ["INCLUDED", "OUTSIDE"] + by_index = mixed_source.copy() + by_index.spm_unit[STATUS] = np.array([0, 1], dtype=np.int8) + thresholds = [] + for source in (by_name, by_index): + simulation = Microsimulation(dataset=source) + assert list(simulation.calculate(STATUS, YEAR)) == ["INCLUDED", "OUTSIDE"] + thresholds.append( + np.asarray(simulation.calculate("spm_unit_spm_threshold", YEAR)) + ) + assert thresholds[0][0] > 0 + assert np.isnan(thresholds[0][1]) + np.testing.assert_array_equal(thresholds[0], thresholds[1]) + + +def test_dataset_without_source_universe_defaults_to_unresolved(mixed_source): + source = mixed_source.copy() + source.person["age"] = [40, 8, 40] + source.household["county_fips"] = ["06037", "36061"] + source.spm_unit.drop(columns=[STATUS], inplace=True) + simulation = Microsimulation(dataset=source) + with pytest.raises(SPMInputError) as error: + simulation.calculate("spm_unit_spm_threshold", YEAR) + assert error.value.code == "SPM_UNIVERSE_REQUIRED" + + +def test_ordinary_household_situation_keeps_included_default(): + simulation = Simulation( + situation={ + "people": {"person1": {"age": {YEAR: 40}}}, + "households": { + "household1": {"members": ["person1"], "county_fips": {YEAR: "06037"}} + }, + } + ) + assert simulation.calculate("spm_unit_spm_threshold", YEAR)[0] > 0 + + +def test_mixed_calculations_preserve_source_tables_and_membership(mixed_source): + before = [frame.copy(deep=True) for frame in mixed_source.tables] + simulation = Microsimulation(dataset=mixed_source) + simulation.calculate("household_net_income", YEAR) + simulation.calculate("spm_unit_is_in_spm_poverty", YEAR) + for original, current in zip(before, mixed_source.tables): + pd.testing.assert_frame_equal(original, current) + for entity in ("household", "tax_unit", "spm_unit", "family", "marital_unit"): + np.testing.assert_array_equal( + simulation.calculate(f"person_{entity}_id", YEAR), + mixed_source.person[f"person_{entity}_id"], + ) + + +@pytest.mark.parametrize( + "module_name,factory_name", + [ + pytest.param( + "congress.tlaib.boost.boost_middle_class_tax_credit", + "create_boost_middle_class_tax_credit", + id="boost", + ), + pytest.param( + "congress.tlaib.end_child_poverty_act", + "create_end_child_poverty_act", + id="ecpa", + ), + pytest.param( + "congress.tlaib.economic_dignity_for_all_agenda.edaa_end_child_poverty_act", + "create_ecpa_only", + id="edaa", + ), + pytest.param("states.tx.rebate.tx_rebate", "create_tx_rebate", id="tx-rebate"), + ], +) +@pytest.mark.parametrize("abolish_housing", [False, True]) +def test_contributed_benefit_overrides_preserve_universe_separation( + mixed_source, monkeypatch, module_name, factory_name, abolish_housing +): + _capture_provider(monkeypatch) + factory = getattr( + import_module(f"policyengine_us.reforms.{module_name}"), factory_name + ) + reform = ( + factory(), + {"gov.hud.abolition": {f"{YEAR}-01-01.{YEAR}-12-31": abolish_housing}}, + ) + system = CountryTaxBenefitSystem(reform=reform) + + def simulate(source): + return Microsimulation( + dataset=source, + tax_benefit_system=system, + ) + + simulation = simulate(mixed_source) + standalone = simulate(_only_included(mixed_source)) + measured = simulation.calculate("spm_unit_benefits", YEAR) + assert measured[0] == standalone.calculate("spm_unit_benefits", YEAR)[0] + # HUD abolition removes the capped housing term, so an explicit mask must + # still exclude outside resources even without a NaN from that term. + assert np.isnan(measured[1]) + ordinary = simulation.calculate("household_benefits", YEAR) + assert len(ordinary) == 2 + assert np.isfinite(ordinary).all() + source_without_award = mixed_source.copy() + source_without_award.spm_unit["hud_hap"] = [30_000.0, 0.0] + comparison = simulate(source_without_award).calculate("household_benefits", YEAR) + assert ordinary[0] == comparison[0] + expected_component = 0 if abolish_housing else 30_000 + assert ordinary[1] - comparison[1] == pytest.approx(expected_component, abs=0.01) + + +@pytest.mark.parametrize("status", ["INCLUDED", "OUTSIDE", "UNRESOLVED", None]) +@pytest.mark.parametrize("geography_kind", ["county", "national"]) +def test_ordinary_actual_benefits_do_not_require_scope_or_spm_geography( + mixed_source, monkeypatch, status, geography_kind +): + source = mixed_source.copy() + source.household["county_fips"] = ["", ""] + if status is None: + source.spm_unit.drop(columns=[STATUS], inplace=True) + else: + source.spm_unit[STATUS] = status + + def forbidden(*args, **kwargs): + raise AssertionError("Ordinary income must not execute SPM measurement") + + monkeypatch.setattr(PolicyEngineSPMProvider, "calculate_unit", forbidden) + simulation = Microsimulation(dataset=source, spm={"geography_kind": geography_kind}) + # Ordinary benefits may depend on county. Hold that input fixed while + # varying only the SPM scope declaration and provider selection. + reference_source = mixed_source.copy() + reference_source.household["county_fips"] = ["", ""] + reference_source.spm_unit[STATUS] = "INCLUDED" + reference = Microsimulation(dataset=reference_source) + for name in ( + "housing_assistance", + "household_benefits", + "household_net_income", + "cbo_household_means_tested_transfers", + "ca_cpuc_countable_income", + ): + actual = simulation.calculate(name, YEAR) + assert np.isfinite(actual).all(), name + np.testing.assert_array_equal( + actual, reference.calculate(name, YEAR), err_msg=name + ) + np.testing.assert_array_equal( + simulation.calculate("housing_assistance", YEAR), [30_000, 30_000] + ) + assert simulation.spm_provenance()["years"] == {} + + +@pytest.mark.parametrize("route", ["no_assistance", "hud_abolition"]) +@pytest.mark.parametrize( + "variable", + [ + "spm_unit_capped_housing_subsidy", + "spm_unit_benefits", + "spm_unit_net_income", + *DISTRIBUTION_OUTPUTS, + *POVERTY_INDICATORS, + ], +) +def test_unresolved_scope_precedes_resource_early_returns( + mixed_source, monkeypatch, variable, route +): + source = mixed_source.copy() + source.spm_unit[STATUS] = ["INCLUDED", "UNRESOLVED"] + source.person["age"] = [40, 8, 40] + source.household["county_fips"] = ["06037", "36061"] + if route == "no_assistance": + source.spm_unit["hud_hap"] = 0.0 + reform = None + else: + reform = {"gov.hud.abolition": {f"{YEAR}-01-01.{YEAR}-12-31": True}} + + def forbidden(*args, **kwargs): + raise AssertionError("Unresolved scope must fail before provider execution") + + monkeypatch.setattr(PolicyEngineSPMProvider, "calculate_unit", forbidden) + simulation = Microsimulation(dataset=source, reform=reform) + assert np.isfinite(simulation.calculate("household_net_income", YEAR)).all() + with pytest.raises(SPMInputError) as error: + simulation.calculate(variable, YEAR) + assert error.value.code == "SPM_UNIVERSE_REQUIRED" + + +@pytest.mark.parametrize("variable", sorted(REJECTED_DATASET_INPUTS)) +@pytest.mark.parametrize("ingress", ["dataset", "set_input"]) +def test_computed_spm_outputs_cannot_be_supplied_as_inputs( + mixed_source, variable, ingress +): + source = mixed_source.copy() + frame = getattr(source, system.variables[variable].entity.key) + if ingress == "dataset": + frame[variable] = 1.0 + with pytest.raises(ValueError, match="formula-owned SPM output"): + Microsimulation(dataset=source) + else: + simulation = Microsimulation(dataset=source) + with pytest.raises(ValueError, match="formula-owned SPM output"): + simulation.set_input(variable, YEAR, np.ones(len(frame))) + + +def test_explicit_multi_year_source_status_is_preserved_without_carry(mixed_source): + first = mixed_source.copy() + second = mixed_source.copy() + second.time_period = str(YEAR + 1) + second.spm_unit[STATUS] = ["OUTSIDE", "INCLUDED"] + # Make the newly included unit valid independently of its former status. + second.person["age"] = [40, 8, 40] + second.household["county_fips"] = ["06037", "36061"] + source = USMultiYearDataset(datasets=[first, second]) + originals = { + year: [table.copy(deep=True) for table in dataset.tables] + for year, dataset in source.datasets.items() + } + simulation = Microsimulation(dataset=source) + for year, statuses, included_index in ( + (YEAR, ["INCLUDED", "OUTSIDE"], 0), + (YEAR + 1, ["OUTSIDE", "INCLUDED"], 1), + ): + assert list(simulation.calculate(STATUS, year)) == statuses + threshold = simulation.calculate("spm_unit_spm_threshold", year) + assert threshold[included_index] > 0 + assert np.isnan(threshold[1 - included_index]) + assert list(simulation.calculate(STATUS, YEAR + 2)) == ["UNRESOLVED"] * 2 + with pytest.raises(SPMInputError) as error: + simulation.calculate("spm_unit_spm_threshold", YEAR + 2) + assert error.value.code == "SPM_UNIVERSE_REQUIRED" + # Revisiting a stored year must not reuse a later year's scope. + assert list(simulation.calculate(STATUS, YEAR)) == ["INCLUDED", "OUTSIDE"] + for year, dataset in source.datasets.items(): + for before, after in zip(originals[year], dataset.tables): + pd.testing.assert_frame_equal(before, after) + + +@pytest.mark.parametrize("recipient", ["included", "outside"]) +def test_shared_household_allocation_retains_outside_members(recipient, monkeypatch): + calls = _capture_provider(monkeypatch) + situation = { + "people": { + "adult": {"age": {YEAR: 40}}, + "child1": {"age": {YEAR: 12}}, + "child2": {"age": {YEAR: 10}}, + }, + "households": { + "home": { + "members": ["adult", "child1", "child2"], + "county_fips": {YEAR: "06037"}, + } + }, + "spm_units": { + "included": {"members": ["adult"], STATUS: {YEAR: "INCLUDED"}}, + "outside": { + "members": ["child1", "child2"], + STATUS: {YEAR: "OUTSIDE"}, + }, + }, + } + for name, unit in situation["spm_units"].items(): + unit["housing_assistance"] = {YEAR: 9_000 if name == recipient else 0} + unit["hud_ttp"] = {YEAR: 0} + unit["spm_unit_tenure_type"] = {YEAR: "RENTER"} + simulation = Simulation(situation=situation) + expected_awards = [9_000, 0] if recipient == "included" else [0, 9_000] + np.testing.assert_array_equal( + simulation.calculate("housing_assistance", YEAR), expected_awards + ) + np.testing.assert_array_equal( + simulation.calculate("spm_unit_allocated_housing_subsidy", YEAR), [3_000, 6_000] + ) + np.testing.assert_array_equal(simulation.calculate("spm_unit_size", YEAR), [1, 2]) + cap = simulation.calculate("spm_unit_capped_housing_subsidy", YEAR) + housing_portion = simulation.calculate( + "spm_unit_spm_threshold_housing_portion", YEAR + ) + assert cap[0] == min(3_000, housing_portion[0]) + assert np.isnan(cap[1]) + assert calls and all(call["adults"] >= 1 for call in calls) + # Scope masks measurement after member-share allocation; it does not award + # all of the household's assistance to its one included person. + np.testing.assert_array_equal( + simulation.calculate("spm_unit_allocated_housing_subsidy", YEAR), [3_000, 6_000] + ) + + +def _rank_source(include_outside=True): + """Unequal included weights and a dominant, very rich outside record.""" + count = 4 if include_outside else 3 + ids = list(range(1, count + 1)) + groups = ("household", "tax_unit", "spm_unit", "family", "marital_unit") + tables = {entity: pd.DataFrame({f"{entity}_id": ids}) for entity in groups} + tables["household"]["household_weight"] = [1.0, 2.0, 4.0, 1_000_000.0][:count] + tables["household"]["county_fips"] = ["06037"] * count + tables["spm_unit"][STATUS] = ["INCLUDED", "INCLUDED", "INCLUDED", "OUTSIDE"][:count] + return USSingleYearDataset( + person=pd.DataFrame( + { + "person_id": ids, + "age": [40] * count, + "employment_income": [1_000, 10_000, 100_000, 100_000_000][:count], + **{f"person_{entity}_id": ids for entity in groups}, + } + ), + **tables, + time_period=YEAR, + ) + + +def test_deciles_rank_only_included_people_and_preserve_monthly_stock(): + mixed = Microsimulation(dataset=_rank_source()) + standalone = Microsimulation(dataset=_rank_source(include_outside=False)) + actual = mixed.calculate("spm_unit_income_decile", YEAR) + expected = standalone.calculate("spm_unit_income_decile", YEAR) + assert np.issubdtype(actual.dtype, np.floating) + np.testing.assert_array_equal(actual[:3], expected) + assert np.isnan(actual[3]) + assert np.all((expected >= 1) & (expected <= 10)) + np.testing.assert_array_equal(expected, np.floor(expected)) + assert len(np.unique(expected)) > 1, "The ranking fixture must distinguish incomes" + for month in ("01", "07"): + np.testing.assert_array_equal( + mixed.calculate("spm_unit_income_decile", f"{YEAR}-{month}"), actual + ) + + +def test_all_outside_deciles_do_not_call_the_ranker(mixed_source, monkeypatch): + from microdf import MicroSeries + + source = mixed_source.copy() + source.spm_unit[STATUS] = "OUTSIDE" + + def forbidden(*args, **kwargs): + raise AssertionError("An all-outside population has no ranks") + + monkeypatch.setattr(MicroSeries, "decile_rank", forbidden) + result = Microsimulation(dataset=source).calculate("spm_unit_income_decile", YEAR) + assert np.isnan(result).all() + + +@pytest.mark.parametrize("invalid", [np.nan, np.inf, -np.inf]) +@pytest.mark.parametrize("variable", DISTRIBUTION_OUTPUTS) +def test_included_distribution_income_must_be_finite(mixed_source, invalid, variable): + # Formula injection exercises the measurement guard: NaN stored inputs are + # rejected earlier by Core and cannot prove that guard works. + class spm_unit_net_income(Variable): + def formula(spm_unit, period, parameters): + return np.array([invalid, 0.0]) + + class spm_unit_oecd_equiv_net_income(Variable): + def formula(spm_unit, period, parameters): + return np.array([invalid, 0.0]) + + class InvalidIncome(Reform): + def apply(self): + self.update_variable( + spm_unit_net_income + if variable == "spm_unit_oecd_equiv_net_income" + else spm_unit_oecd_equiv_net_income + ) + + simulation = Microsimulation(dataset=mixed_source, reform=InvalidIncome) + with pytest.raises(SPMInputError) as error: + simulation.calculate(variable, YEAR) + assert error.value.code == "SPM_MEASUREMENT_INVALID" + + +@pytest.mark.parametrize("status", ["INCLUDED", "OUTSIDE", "UNRESOLVED"]) +def test_explicit_annual_source_status_can_override_its_fallback(mixed_source, status): + simulation = Microsimulation(dataset=mixed_source) + simulation.set_input(STATUS, YEAR, [status, status]) + assert list(simulation.calculate(STATUS, YEAR)) == [status, status] + + +@pytest.mark.parametrize("problem", ["geography", "composition"]) +def test_explicit_inclusion_preserves_canonical_input_validation(mixed_source, problem): + source = mixed_source.copy() + source.spm_unit[STATUS] = "INCLUDED" + if problem == "geography": + source.person["age"] = [40, 8, 40] + expected_code = "SPM_GEOGRAPHY_REQUIRED" + else: + source.household["county_fips"] = ["06037", "36061"] + expected_code = "SPM_COMPOSITION_REQUIRED" + with pytest.raises(SPMInputError) as error: + Microsimulation(dataset=source).calculate("spm_unit_spm_threshold", YEAR) + assert error.value.code == expected_code + + +@pytest.mark.parametrize( + "module_name,factory_name", + [ + ( + "congress.tlaib.boost.boost_middle_class_tax_credit", + "create_boost_middle_class_tax_credit", + ), + ("congress.tlaib.end_child_poverty_act", "create_end_child_poverty_act"), + ( + "congress.tlaib.economic_dignity_for_all_agenda.edaa_end_child_poverty_act", + "create_ecpa_only", + ), + ("states.tx.rebate.tx_rebate", "create_tx_rebate"), + ], +) +@pytest.mark.parametrize("abolish_housing", [False, True]) +def test_contributed_spm_benefits_refuse_unresolved_scope_before_housing_branch( + mixed_source, monkeypatch, module_name, factory_name, abolish_housing +): + source = mixed_source.copy() + source.spm_unit[STATUS] = ["INCLUDED", "UNRESOLVED"] + source.spm_unit["hud_hap"] = 0.0 + factory = getattr( + import_module(f"policyengine_us.reforms.{module_name}"), factory_name + ) + reform = ( + factory(), + {"gov.hud.abolition": {f"{YEAR}-01-01.{YEAR}-12-31": abolish_housing}}, + ) + + def forbidden(*args, **kwargs): + raise AssertionError("Unresolved scope must fail before provider execution") + + monkeypatch.setattr(PolicyEngineSPMProvider, "calculate_unit", forbidden) + simulation = Microsimulation(dataset=source, reform=reform) + assert np.isfinite(simulation.calculate("household_benefits", YEAR)).all() + for name in ("spm_unit_benefits", "spm_unit_net_income"): + with pytest.raises(SPMInputError) as error: + simulation.calculate(name, YEAR) + assert error.value.code == "SPM_UNIVERSE_REQUIRED" + + +@pytest.mark.parametrize("invalid", [np.nan, np.inf, -np.inf]) +@pytest.mark.parametrize("variable", POVERTY_INDICATORS) +def test_included_poverty_resources_must_be_finite(mixed_source, invalid, variable): + class spm_unit_net_income(Variable): + def formula(spm_unit, period, parameters): + return np.array([invalid, 0.0]) + + class InvalidResources(Reform): + def apply(self): + self.update_variable(spm_unit_net_income) + + simulation = Microsimulation(dataset=mixed_source, reform=InvalidResources) + with pytest.raises(SPMInputError) as error: + simulation.calculate(variable, YEAR) + assert error.value.code == "SPM_MEASUREMENT_INVALID" + + +@pytest.mark.parametrize("invalid", [np.nan, np.inf, -np.inf, 0.0, -1.0]) +@pytest.mark.parametrize("variable", POVERTY_INDICATORS) +def test_included_poverty_thresholds_must_be_positive_and_finite( + mixed_source, invalid, variable +): + class spm_unit_spm_threshold(Variable): + def formula(spm_unit, period, parameters): + return np.array([invalid, np.nan]) + + class InvalidThreshold(Reform): + def apply(self): + self.update_variable(spm_unit_spm_threshold) + + simulation = Microsimulation(dataset=mixed_source, reform=InvalidThreshold) + with pytest.raises(SPMInputError) as error: + simulation.calc(variable, period=YEAR) + assert error.value.code == "SPM_MEASUREMENT_INVALID" + + +def test_outside_role_placeholders_preserve_measurements_and_ordinary_outcomes( + mixed_source, +): + """An outside role may change composition, but cannot change named outcomes.""" + false_source = mixed_source.copy() + false_source.person.loc[false_source.person.person_id == 301, "age"] = 17 + true_source = false_source.copy() + true_source.person.loc[ + true_source.person.person_id == 301, "is_spm_independent_minor_role" + ] = True + false_model = Microsimulation(dataset=false_source) + true_model = Microsimulation(dataset=true_source) + assert false_model.calc("spm_measurement_adults", period=YEAR).iloc[1] == 0 + assert true_model.calc("spm_measurement_adults", period=YEAR).iloc[1] == 1 + for variable in ALL_MEASUREMENTS: + false_result = false_model.calc(variable, period=YEAR, map_to="person") + true_result = true_model.calc(variable, period=YEAR, map_to="person") + np.testing.assert_array_equal(false_result, true_result, err_msg=variable) + assert false_result.isna().tolist() == [False, False, True], variable + assert true_result.isna().tolist() == [False, False, True], variable + assert false_result.count() == true_result.count() == 4, variable + assert false_result.sum() == true_result.sum(), variable + assert false_result.mean() == true_result.mean(), variable + for variable in ORDINARY_OUTCOMES: + false_result = false_model.calc(variable, period=YEAR, map_to="person") + true_result = true_model.calc(variable, period=YEAR, map_to="person") + np.testing.assert_array_equal(false_result, true_result, err_msg=variable) + assert not false_result.isna().any(), variable + assert false_result.count() == true_result.count() == 9, variable + assert false_result.sum() == true_result.sum(), variable + assert false_result.mean() == true_result.mean(), variable + + +def _county_type_collision_source(mixed_source): + source = mixed_source.copy() + source.household["state_code"] = ["NY", "NY"] + # Both spell the same county after string conversion, but only the first + # is a valid source input. The second belongs to an outside unit. + source.household["county_fips"] = ["36061", 36061] + return source + + +@pytest.mark.parametrize("simulation_type", [Simulation, Microsimulation]) +def test_outside_numeric_county_does_not_poison_included_measurements( + mixed_source, simulation_type +): + source = _county_type_collision_source(mixed_source) + simulation = simulation_type(dataset=source) + standalone = simulation_type(dataset=_only_included(source)) + provider = simulation.tax_benefit_system.spm_forecast_provider + original_types = provider._untyped_counties + before = original_types.copy() + assert before[(YEAR, "36061")] == "36061" + + for variable in MEASUREMENT_AMOUNTS[:6]: + actual = simulation.calculate(variable, YEAR) + expected = standalone.calculate(variable, YEAR) + assert np.isfinite(actual[0]), variable + assert actual[0] == expected[0], variable + assert np.isnan(actual[1]), variable + # Selection must use a private typing view, not mutate the original + # receipt to forget the malformed outside input. + assert provider._untyped_counties is original_types + assert provider._untyped_counties == before + + # A selected provider view must still populate the simulation's original + # cache and provenance, rather than strand them on a temporary provider. + assert provider._amount_cache + provenance = simulation.spm_provenance() + assert set(provenance["years"]) == {str(YEAR)} + assert provenance["geographies"] + assert { + receipt["county_assignment"]["county_fips"] + for receipt in provenance["geographies"] + } == {"36061"} + + +@pytest.mark.parametrize("simulation_type", [Simulation, Microsimulation]) +def test_cached_valid_county_amount_does_not_accept_newly_included_numeric_county( + mixed_source, simulation_type +): + source = _county_type_collision_source(mixed_source) + # Identical one-adult compositions ensure the numeric row would hit the + # valid row's exact provider cache key if typing were checked too late. + source.person = source.person.loc[source.person.person_id != 102].copy() + source.person["age"] = [40, 40] + source.marital_unit = source.marital_unit.loc[ + source.marital_unit.marital_unit_id != 102 + ].copy() + source = source.copy() # Rebuild the table tuple after replacing frames. + simulation = simulation_type(dataset=source) + provider = simulation.tax_benefit_system.spm_forecast_provider + before = provider._untyped_counties.copy() + for variable in MEASUREMENT_AMOUNTS[:6]: + values = simulation.calculate(variable, YEAR) + assert np.isfinite(values[0]) and np.isnan(values[1]), variable + assert (YEAR, 1, 0, "renter", "36061") in provider._amount_cache + + simulation.set_input(STATUS, YEAR, ["INCLUDED", "INCLUDED"]) + for variable in MEASUREMENT_AMOUNTS[:6]: + # Core's explicit input API does not invalidate dependent holders. + # Recalculate the output while deliberately retaining the provider memo. + simulation.delete_arrays(variable, YEAR) + with pytest.raises(SPMInputError) as error: + simulation.calculate(variable, YEAR) + assert error.value.code == "SPM_GEOGRAPHY_REQUIRED", variable + assert provider._untyped_counties == before + + +@pytest.mark.parametrize("simulation_type", [Simulation, Microsimulation]) +def test_unassisted_included_numeric_county_does_not_poison_assisted_housing_cap( + mixed_source, simulation_type +): + source = _county_type_collision_source(mixed_source) + source.person["age"] = [40, 8, 40] + source.spm_unit[STATUS] = "INCLUDED" + source.spm_unit["hud_hap"] = [30_000.0, 0.0] + simulation = simulation_type(dataset=source) + standalone = simulation_type(dataset=_only_included(source)) + provider = simulation.tax_benefit_system.spm_forecast_provider + before = provider._untyped_counties.copy() + + cap = simulation.calculate("spm_unit_capped_housing_subsidy", YEAR) + expected = standalone.calculate("spm_unit_capped_housing_subsidy", YEAR) + assert np.isfinite(cap[0]) and cap[0] > 0 + np.testing.assert_array_equal(cap, [expected[0], 0.0]) + assert provider._untyped_counties == before + # Only the cap may ignore this unassisted unit's geography: a threshold + # measurement still selects every included unit and must reject its type. + with pytest.raises(SPMInputError) as error: + simulation.calculate("spm_unit_spm_threshold", YEAR) + assert error.value.code == "SPM_GEOGRAPHY_REQUIRED" + + +@pytest.mark.parametrize("new_status", ["OUTSIDE", "UNRESOLVED"]) +def test_scope_input_mutation_cannot_reuse_cached_measurements( + mixed_source, new_status +): + simulation = Microsimulation(dataset=mixed_source) + outputs = ( + "spm_unit_spm_threshold", + "person_in_poverty", + "spm_unit_income_decile", + ) + for variable in outputs: + cached = simulation.calc(variable, period=YEAR, map_to="person") + assert cached.count() == 4, variable + assert cached.isna().tolist() == [False, False, True], variable + ordinary = simulation.calc("household_net_income", period=YEAR, map_to="person") + simulation.set_input(STATUS, YEAR, [new_status, "OUTSIDE"]) + for variable in outputs: + if new_status == "UNRESOLVED": + with pytest.raises(SPMInputError) as error: + simulation.calc(variable, period=YEAR, map_to="person") + assert error.value.code == "SPM_UNIVERSE_REQUIRED", variable + else: + result = simulation.calc(variable, period=YEAR, map_to="person") + assert result.isna().all(), variable + assert result.count() == 0, variable + np.testing.assert_array_equal( + simulation.calc("household_net_income", period=YEAR, map_to="person"), + ordinary, + ) + + +def test_observed_role_input_mutation_cannot_reuse_cached_composition(mixed_source): + source = mixed_source.copy() + source.person["age"] = [40, 8, 17] + source.person["is_spm_independent_minor_role"] = [False, False, True] + source.household["county_fips"] = ["06037", "36061"] + source.spm_unit[STATUS] = "INCLUDED" + simulation = Microsimulation(dataset=source) + assert simulation.calc("spm_measurement_adults", period=YEAR).tolist() == [1, 1] + outputs = ( + "spm_unit_spm_threshold", + "person_in_poverty", + "spm_unit_income_decile", + ) + for variable in outputs: + assert not simulation.calc(variable, period=YEAR).isna().any(), variable + simulation.set_input("is_spm_independent_minor_role", "eternity", [False] * 3) + assert simulation.calc("spm_measurement_adults", period=YEAR).tolist() == [1, 0] + for variable in outputs: + with pytest.raises(SPMInputError) as error: + simulation.calc(variable, period=YEAR) + assert error.value.code == "SPM_COMPOSITION_REQUIRED", variable + + +def _outcome_snapshot(simulation, variables): + snapshot = {} + for variable in variables: + result = simulation.calc(variable, period=YEAR, map_to="person") + # Detach fixture values, while computing every weighted statistic on + # the original MicroSeries. Later cache mutations cannot rewrite this. + snapshot[variable] = { + "values": result.tolist(), + "count": result.count(), + "sum": result.sum(), + "mean": result.mean(), + } + return snapshot + + +def _assert_outcome_snapshot(simulation, snapshot): + for variable, expected in snapshot.items(): + actual = simulation.calc(variable, period=YEAR, map_to="person") + np.testing.assert_array_equal(actual, expected["values"], err_msg=variable) + assert actual.count() == expected["count"], variable + assert actual.sum() == expected["sum"], variable + assert actual.mean() == expected["mean"], variable + + +@pytest.mark.parametrize("source_input", [STATUS, "is_spm_independent_minor_role"]) +def test_unchanged_source_assignment_preserves_inputs_and_all_outcomes( + mixed_source, source_input +): + original_tables = [table.copy(deep=True) for table in mixed_source.tables] + simulation = Microsimulation(dataset=mixed_source) + outcomes = _outcome_snapshot(simulation, (*ALL_MEASUREMENTS, *ORDINARY_OUTCOMES)) + source_names = ( + STATUS, + "is_spm_independent_minor_role", + "age", + "employment_income", + "county_fips", + "hud_hap", + "hud_ttp", + ) + original_inputs = { + name: simulation.calc(name, period=YEAR).tolist() for name in source_names + } + source_period = YEAR if source_input == STATUS else "eternity" + simulation.set_input(source_input, source_period, original_inputs[source_input]) + for name, expected in original_inputs.items(): + assert simulation.calc(name, period=YEAR).tolist() == expected, name + _assert_outcome_snapshot(simulation, outcomes) + for before, after in zip(original_tables, mixed_source.tables, strict=True): + pd.testing.assert_frame_equal(before, after) + + +def test_branch_scope_mutation_preserves_parent_and_sibling_snapshots(mixed_source): + original_tables = [table.copy(deep=True) for table in mixed_source.tables] + parent = Microsimulation(dataset=mixed_source) + outputs = ( + "spm_unit_spm_threshold", + "person_in_poverty", + "spm_unit_income_decile", + *ORDINARY_OUTCOMES, + ) + snapshot = _outcome_snapshot(parent, outputs) + # Share policy parameters; these tiny branches must not clone the full + # parameter tree merely to change one observed source declaration. + target = parent.get_branch("scope_mutation_target", clone_system=False) + sibling = parent.get_branch("scope_mutation_sibling", clone_system=False) + _assert_outcome_snapshot(target, snapshot) + _assert_outcome_snapshot(sibling, snapshot) + target.set_input(STATUS, YEAR, ["OUTSIDE", "OUTSIDE"]) + assert list(target.calc(STATUS, period=YEAR)) == ["OUTSIDE", "OUTSIDE"] + for variable in outputs[:3]: + result = target.calc(variable, period=YEAR, map_to="person") + assert result.isna().all(), variable + assert result.count() == 0, variable + _assert_outcome_snapshot( + target, {name: snapshot[name] for name in ORDINARY_OUTCOMES} + ) + for untouched in (parent, sibling): + assert list(untouched.calc(STATUS, period=YEAR)) == ["INCLUDED", "OUTSIDE"] + _assert_outcome_snapshot(untouched, snapshot) + for before, after in zip(original_tables, mixed_source.tables, strict=True): + pd.testing.assert_frame_equal(before, after) + + +@pytest.mark.parametrize( + "variable,period,value,expected_calls", + [ + (STATUS, YEAR, ["OUTSIDE", "OUTSIDE"], 1), + ("is_spm_independent_minor_role", "eternity", [False, False, True], 1), + ("employment_income", YEAR, [7_000.0, 0.0, 0.0], 0), + ("county_fips", YEAR, ["36061", "06037"], 0), + ], + ids=["scope", "role", "ordinary-income", "ordinary-county"], +) +def test_full_cache_invalidation_is_limited_to_spm_source_assignments( + mixed_source, monkeypatch, variable, period, value, expected_calls +): + """The new source contract must not widen ordinary set_input semantics.""" + simulation = Microsimulation(dataset=mixed_source) + original = simulation._invalidate_all_caches + calls = [] + + def tracked_invalidation(): + calls.append(True) + return original() + + monkeypatch.setattr(simulation, "_invalidate_all_caches", tracked_invalidation) + simulation.set_input(variable, period, value) + assert len(calls) == expected_calls diff --git a/policyengine_us/tests/core/test_ssi_state_supplement_medicaid_dependency.py b/policyengine_us/tests/core/test_ssi_state_supplement_medicaid_dependency.py index a64c80a17a7..595935c9db2 100644 --- a/policyengine_us/tests/core/test_ssi_state_supplement_medicaid_dependency.py +++ b/policyengine_us/tests/core/test_ssi_state_supplement_medicaid_dependency.py @@ -112,6 +112,11 @@ def test_future_poverty_in_all_states(year): }, ) poor = simulation.calculate("spm_unit_is_in_spm_poverty", year) - assert poor.dtype == np.dtype(bool) + # The canonical indicator is a nullable float now that SPM measurement scope + # is an explicit source decision: NaN marks a record outside the declared + # universe. A household situation is wholly in-universe, so every unit here + # is observed, and none of these states may go missing. + assert np.issubdtype(poor.dtype, np.floating) + assert np.isin(poor, [0.0, 1.0]).all() assert poor.size == len(groups) assert np.isfinite(simulation.calculate("spm_unit_benefits", year)).all() diff --git a/policyengine_us/tests/microsimulation/test_microsim.py b/policyengine_us/tests/microsimulation/test_microsim.py index 7c7470a2198..6f9051b26a0 100644 --- a/policyengine_us/tests/microsimulation/test_microsim.py +++ b/policyengine_us/tests/microsimulation/test_microsim.py @@ -21,19 +21,15 @@ def test_legacy_cps_file_with_stored_measurements_is_rejected_at_load(): def test_legacy_enhanced_cps_lacks_source_backed_spm_independence_roles(): """enhanced_cps_2024 loads, but cannot produce SPM measurements. - Its county column does hold five-digit FIPS codes, so geography is not what - fails. 18 of its 43,134 SPM units are a lone 15-to-17-year-old carrying no - source-backed SPM independence role, so those units classify no measurement - adult and every output that reaches the threshold fails closed over the - file. Only the threshold is asserted here: it raises straight off the - composition, without building the whole resource chain over 43,134 units. - - Resource outputs are no longer among the outputs that reach it. The - capped housing subsidy asks for the SPM housing portion only for units - receiving housing assistance, and none of these 18 units does - which is - asserted below - so household net income and benefits compute over the - file. Computing them here would add two minutes to this suite, so they - are left to the resource tests. + The file lacks an annual source declaration of SPM scope, which must fail + before geography or composition is evaluated for a measurement. Its county + column still holds five-digit FIPS codes. Separately, 18 of its 43,134 SPM + units are lone 15-to-17-year-olds with no source-backed independence role; + those units classify no measurement adult and none receives housing aid. + These primitive checks remain meaningful even though scope is the first + missing prerequisite. Only the threshold is requested here, without building + the resource chain. Ordinary household income and actual benefits remain + independent of the SPM measurement scope. """ import numpy as np from spm_calculator.errors import SPMInputError @@ -54,7 +50,7 @@ def test_legacy_enhanced_cps_lacks_source_backed_spm_independence_roles(): assert not assisted[adults < 1].any() with pytest.raises(SPMInputError) as error: simulation.calculate("spm_unit_spm_threshold", 2024) - assert error.value.code == "SPM_COMPOSITION_REQUIRED" + assert error.value.code == "SPM_UNIVERSE_REQUIRED" def test_county_persists_across_periods(): diff --git a/policyengine_us/tests/unit/test_spm_measurement_boundary.py b/policyengine_us/tests/unit/test_spm_measurement_boundary.py index 2c75172b353..e72e658425e 100644 --- a/policyengine_us/tests/unit/test_spm_measurement_boundary.py +++ b/policyengine_us/tests/unit/test_spm_measurement_boundary.py @@ -26,10 +26,10 @@ def test_spm_caps_have_only_spm_resource_consumers(): }, "spm_unit_oecd_equiv_net_income": {"spm_unit_income_decile"}, "spm_unit_income_decile": set(), - "poverty_gap": {"in_poverty"}, - "deep_poverty_gap": {"in_deep_poverty"}, - "spm_unit_is_in_spm_poverty": set(), - "spm_unit_is_in_deep_spm_poverty": set(), + "poverty_gap": set(), + "deep_poverty_gap": set(), + "spm_unit_is_in_spm_poverty": {"in_poverty"}, + "spm_unit_is_in_deep_spm_poverty": {"in_deep_poverty"}, "in_poverty": {"person_in_poverty"}, "in_deep_poverty": set(), "person_in_poverty": set(), diff --git a/policyengine_us/variables/household/income/household/person_in_poverty.py b/policyengine_us/variables/household/income/household/person_in_poverty.py index 58baf5e2c38..01d111273ea 100644 --- a/policyengine_us/variables/household/income/household/person_in_poverty.py +++ b/policyengine_us/variables/household/income/household/person_in_poverty.py @@ -6,7 +6,8 @@ class person_in_poverty(Variable): documentation = "Whether person is in poverty" entity = Person definition_period = YEAR - value_type = bool + value_type = float + quantity_type = STOCK def formula(person, period, parameters): return person.spm_unit("in_poverty", period) diff --git a/policyengine_us/variables/household/income/spm_unit/in_deep_poverty.py b/policyengine_us/variables/household/income/spm_unit/in_deep_poverty.py index db7ff02c41f..847ee298642 100644 --- a/policyengine_us/variables/household/income/spm_unit/in_deep_poverty.py +++ b/policyengine_us/variables/household/income/spm_unit/in_deep_poverty.py @@ -2,11 +2,12 @@ class in_deep_poverty(Variable): - value_type = bool + value_type = float + quantity_type = STOCK entity = SPMUnit label = "in deep poverty" documentation = "Whether a household is in deep poverty." definition_period = YEAR def formula(spm_unit, period, parameters): - return spm_unit("deep_poverty_gap", period) > 0 + return spm_unit("spm_unit_is_in_deep_spm_poverty", period) diff --git a/policyengine_us/variables/household/income/spm_unit/in_poverty.py b/policyengine_us/variables/household/income/spm_unit/in_poverty.py index 120b371cdbe..9c6e1b42470 100644 --- a/policyengine_us/variables/household/income/spm_unit/in_poverty.py +++ b/policyengine_us/variables/household/income/spm_unit/in_poverty.py @@ -2,11 +2,12 @@ class in_poverty(Variable): - value_type = bool + value_type = float + quantity_type = STOCK entity = SPMUnit label = "in poverty" documentation = "Whether a household is in poverty." definition_period = YEAR def formula(spm_unit, period, parameters): - return spm_unit("poverty_gap", period) > 0 + return spm_unit("spm_unit_is_in_spm_poverty", period) diff --git a/policyengine_us/variables/household/income/spm_unit/spm_unit_benefits.py b/policyengine_us/variables/household/income/spm_unit/spm_unit_benefits.py index 41ead044f46..fc2b50a2a07 100644 --- a/policyengine_us/variables/household/income/spm_unit/spm_unit_benefits.py +++ b/policyengine_us/variables/household/income/spm_unit/spm_unit_benefits.py @@ -1,4 +1,5 @@ from policyengine_us.model_api import * +from policyengine_us.spm import spm_universe_mask class spm_unit_benefits(Variable): @@ -9,6 +10,7 @@ class spm_unit_benefits(Variable): unit = USD def formula(spm_unit, period, parameters): + included = spm_universe_mask(spm_unit, period) BENEFITS = [ "social_security", "ssi", @@ -86,4 +88,4 @@ def formula(spm_unit, period, parameters): BENEFITS.append("assigned_aca_ptc") if not parameters(period).gov.hud.abolition: BENEFITS.append("spm_unit_capped_housing_subsidy") - return add(spm_unit, period, BENEFITS) + return where(included, add(spm_unit, period, BENEFITS), np.nan) diff --git a/policyengine_us/variables/household/income/spm_unit/spm_unit_capped_housing_subsidy.py b/policyengine_us/variables/household/income/spm_unit/spm_unit_capped_housing_subsidy.py index 8e123bf641e..61308bfe033 100644 --- a/policyengine_us/variables/household/income/spm_unit/spm_unit_capped_housing_subsidy.py +++ b/policyengine_us/variables/household/income/spm_unit/spm_unit_capped_housing_subsidy.py @@ -1,5 +1,6 @@ from policyengine_us.model_api import * -from policyengine_us.spm import masked_policyengine_amount +from policyengine_us.spm import masked_policyengine_amount, spm_universe_mask +from spm_calculator.errors import SPMInputError class spm_unit_capped_housing_subsidy(Variable): @@ -11,17 +12,25 @@ class spm_unit_capped_housing_subsidy(Variable): reference = "https://www2.census.gov/programs-surveys/supplemental-poverty-measure/datasets/spm/spm_techdoc.pdf" def formula(spm_unit, period, parameters): + included = spm_universe_mask(spm_unit, period) housing_assistance = spm_unit( "spm_unit_allocated_housing_subsidy", period ).astype("float64") + if not np.isfinite(housing_assistance[included]).all() or np.any( + housing_assistance[included] < 0 + ): + raise SPMInputError( + "SPM_HOUSING_ASSISTANCE_INVALID", + "Included allocated housing assistance must be finite and nonnegative.", + ) # Only a unit allocated housing assistance has anything to cap. Consulting # the canonical housing portion for the other units would demand SPM # geography and composition from resource and benefit consumers that # never use the measurement, so the cap is evaluated for assisted units - # alone; every other unit's capped subsidy is zero by construction. - assisted = housing_assistance > 0 + # alone. Other included units receive zero; outside units remain missing. + assisted = included & (housing_assistance > 0) if not assisted.any(): - return np.zeros_like(housing_assistance) + return np.where(included, 0, np.nan) # Apply the country-owned cap to the unrounded canonical housing amount; # the model stores this final benefit amount with one dtype conversion. housing_portion = masked_policyengine_amount( @@ -31,4 +40,6 @@ def formula(spm_unit, period, parameters): "float64" ) cap = max_(housing_portion - tenant_payment, 0) - return where(assisted, min_(housing_assistance, cap), 0) + return where( + included, where(assisted, min_(housing_assistance, cap), 0), np.nan + ) diff --git a/policyengine_us/variables/household/income/spm_unit/spm_unit_income_decile.py b/policyengine_us/variables/household/income/spm_unit/spm_unit_income_decile.py index f9f2b828bec..80ce96bc79d 100644 --- a/policyengine_us/variables/household/income/spm_unit/spm_unit_income_decile.py +++ b/policyengine_us/variables/household/income/spm_unit/spm_unit_income_decile.py @@ -1,15 +1,30 @@ from policyengine_us.model_api import * from microdf import MicroSeries +from policyengine_us.spm import spm_universe_mask +from spm_calculator.errors import SPMInputError class spm_unit_income_decile(Variable): - value_type = int + value_type = float + quantity_type = STOCK entity = SPMUnit label = "Income decile" documentation = "The income decile of the SPM unit, person-weighted and using OECD-equivalised net income" definition_period = YEAR def formula(spm_unit, period, parameters): + included = spm_universe_mask(spm_unit, period) + result = np.full(included.shape, np.nan) + if not included.any(): + return result income = spm_unit("spm_unit_oecd_equiv_net_income", period) + if not np.isfinite(income[included]).all(): + raise SPMInputError( + "SPM_MEASUREMENT_INVALID", + "Included units require finite equivalised SPM income for ranking.", + ) weights = spm_unit("spm_unit_weight", period) * spm_unit.nb_persons() - return MicroSeries(income, weights=weights).decile_rank() + result[included] = MicroSeries( + income[included], weights=weights[included] + ).decile_rank() + return result diff --git a/policyengine_us/variables/household/income/spm_unit/spm_unit_is_in_deep_spm_poverty.py b/policyengine_us/variables/household/income/spm_unit/spm_unit_is_in_deep_spm_poverty.py index cdba4149ca5..4c93a484b17 100644 --- a/policyengine_us/variables/household/income/spm_unit/spm_unit_is_in_deep_spm_poverty.py +++ b/policyengine_us/variables/household/income/spm_unit/spm_unit_is_in_deep_spm_poverty.py @@ -1,13 +1,16 @@ from policyengine_us.model_api import * +from policyengine_us.spm import nullable_spm_indicator, spm_universe_mask class spm_unit_is_in_deep_spm_poverty(Variable): - value_type = bool + value_type = float + quantity_type = STOCK entity = SPMUnit label = "SPM unit in deep SPM poverty" definition_period = YEAR def formula(spm_unit, period, parameters): + spm_universe_mask(spm_unit, period) income = spm_unit("spm_unit_net_income", period) poverty_threshold = spm_unit("spm_unit_spm_threshold", period) / 2 - return income < poverty_threshold + return nullable_spm_indicator(spm_unit, period, income, poverty_threshold) diff --git a/policyengine_us/variables/household/income/spm_unit/spm_unit_is_in_spm_poverty.py b/policyengine_us/variables/household/income/spm_unit/spm_unit_is_in_spm_poverty.py index c0cc3a3ef68..6143cc9a754 100644 --- a/policyengine_us/variables/household/income/spm_unit/spm_unit_is_in_spm_poverty.py +++ b/policyengine_us/variables/household/income/spm_unit/spm_unit_is_in_spm_poverty.py @@ -1,13 +1,16 @@ from policyengine_us.model_api import * +from policyengine_us.spm import nullable_spm_indicator, spm_universe_mask class spm_unit_is_in_spm_poverty(Variable): - value_type = bool + value_type = float + quantity_type = STOCK entity = SPMUnit label = "SPM unit in SPM poverty" definition_period = YEAR def formula(spm_unit, period, parameters): + spm_universe_mask(spm_unit, period) income = spm_unit("spm_unit_net_income", period) poverty_threshold = spm_unit("spm_unit_spm_threshold", period) - return income < poverty_threshold + return nullable_spm_indicator(spm_unit, period, income, poverty_threshold) diff --git a/policyengine_us/variables/household/income/spm_unit/spm_unit_oecd_equiv_net_income'.py b/policyengine_us/variables/household/income/spm_unit/spm_unit_oecd_equiv_net_income'.py index 1384f741fb5..141338e1653 100644 --- a/policyengine_us/variables/household/income/spm_unit/spm_unit_oecd_equiv_net_income'.py +++ b/policyengine_us/variables/household/income/spm_unit/spm_unit_oecd_equiv_net_income'.py @@ -1,4 +1,6 @@ from policyengine_us.model_api import * +from policyengine_us.spm import spm_universe_mask +from spm_calculator.errors import SPMInputError class spm_unit_oecd_equiv_net_income(Variable): @@ -11,6 +13,12 @@ class spm_unit_oecd_equiv_net_income(Variable): reference = "https://www.oecd.org/economy/growth/OECD-Note-EquivalenceScales.pdf" def formula(spm_unit, period, parameters): + included = spm_universe_mask(spm_unit, period) number_of_people = spm_unit.nb_persons() net_income = spm_unit("spm_unit_net_income", period) - return net_income / (number_of_people**0.5) + if not np.isfinite(net_income[included]).all(): + raise SPMInputError( + "SPM_MEASUREMENT_INVALID", + "Included units require finite SPM resources for equivalised income.", + ) + return where(included, net_income / (number_of_people**0.5), np.nan) diff --git a/policyengine_us/variables/household/income/spm_unit/spm_unit_spm_universe_status.py b/policyengine_us/variables/household/income/spm_unit/spm_unit_spm_universe_status.py new file mode 100644 index 00000000000..9cf5824d2ea --- /dev/null +++ b/policyengine_us/variables/household/income/spm_unit/spm_unit_spm_universe_status.py @@ -0,0 +1,21 @@ +from policyengine_us.model_api import * +from policyengine_us.spm import default_spm_universe_status + + +class SPMUniverseStatus(Enum): + INCLUDED = "Included in the declared SPM measurement universe" + OUTSIDE = "Outside the declared SPM measurement universe" + UNRESOLVED = "Measurement universe has not been resolved" + + +class spm_unit_spm_universe_status(Variable): + value_type = Enum + possible_values = SPMUniverseStatus + default_value = SPMUniverseStatus.UNRESOLVED + entity = SPMUnit + definition_period = YEAR + label = "SPM measurement universe status" + reference = "https://www.census.gov/content/dam/Census/library/working-papers/2020/demo/SEHSD-WP2020-09.pdf#page=6" + + def formula(spm_unit, period, parameters): + return default_spm_universe_status(spm_unit) diff --git a/pyproject.toml b/pyproject.toml index 066396cce76..6f46b671f87 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -19,7 +19,7 @@ classifiers = [ "Topic :: Scientific/Engineering :: Information Analysis", ] dependencies = [ - "microdf-python>=1.0.0", + "microdf-python>=1.3.0", # pandas 3 copy-on-write enables the shallow-copy fast path in dataset # extension and is available on every supported Python version. "pandas>=3.0", diff --git a/uv.lock b/uv.lock index 89ce26364ef..a7d91487c66 100644 --- a/uv.lock +++ b/uv.lock @@ -1201,15 +1201,15 @@ wheels = [ [[package]] name = "microdf-python" -version = "1.2.1" +version = "1.3.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "numpy" }, { name = "pandas" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/e7/96/6f9f37f79f2c6440d91036a7bf8111dd4b983c577a7e96d45bf3ca4171f3/microdf_python-1.2.1.tar.gz", hash = "sha256:d4f58e4e0c21decd0c6d425b115db8acc72751c558f48d2a1c3a6619f168a94a", size = 19641, upload-time = "2026-01-25T13:40:57.147Z" } +sdist = { url = "https://files.pythonhosted.org/packages/a3/1b/a4b6a53c8e6668c190e1928ebf9f8c67d6a4995ff4a4b0e3403ded30d4ab/microdf_python-1.3.0.tar.gz", hash = "sha256:f352fe32c25a4fd6d6188ef6aaca0214b53b29a8a7913aca6af0362dd9c922bd", size = 28141, upload-time = "2026-04-17T23:20:57.018Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/cd/2e/375ab71f8d91b691597247b186a4d7b156d2ed975dfb00450e560beae747/microdf_python-1.2.1-py3-none-any.whl", hash = "sha256:3c3d318a82cba7db0ef5a72e8a73a6072fe0bc7a9cb59b1eac01a26ee8c82e7c", size = 20879, upload-time = "2026-01-25T13:40:55.877Z" }, + { url = "https://files.pythonhosted.org/packages/b9/aa/ce692e0488b5e6c0943f52886cf6a15f5d9a6dee7744b190a3a1d67f5d6e/microdf_python-1.3.0-py3-none-any.whl", hash = "sha256:e17757a198648108eae24f39384e27b5ae8061acf1b43fa56520010dc00a7b55", size = 28908, upload-time = "2026-04-17T23:20:56.051Z" }, ] [[package]] @@ -1758,7 +1758,7 @@ requires-dist = [ { name = "coverage", marker = "extra == 'dev'", specifier = ">=7.9.2" }, { name = "furo", marker = "extra == 'dev'", specifier = ">=2024.8.6" }, { name = "jupyter-book", marker = "extra == 'dev'", specifier = ">=1.0.4.post1" }, - { name = "microdf-python", specifier = ">=1.0.0" }, + { name = "microdf-python", specifier = ">=1.3.0" }, { name = "pandas", specifier = ">=3.0" }, { name = "policyengine-core", specifier = ">=3.30.1" }, { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.9.0" },