From d618eba543f21c0bcb94cc92fa16fe6baf52372d Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 18 Sep 2026 09:11:10 -0400 Subject: [PATCH 1/4] Uprate dollar inputs by national totals per capita Household weights grow with total population. Dollar inputs that follow a national total (IRS SOI series, CBO income projections) grew by the full total as well, so weighted totals counted population growth twice: 0.9 percent too high in 2025, 5.1 percent by 2035 on the default dataset. For each national total a variable uprates by, build a sibling parameter holding the total divided by population and point the variable at it. The dataset-extension path resolves its overrides the same way. CMS per-capita spending, CPI-U and the population series are unchanged. The Trustees 2025 long-run scenario extends the SOI totals by the average wage index; it now also carries population growth so each record's income still follows the average wage. Co-Authored-By: Claude Fable 5.1 --- changelog.d/uprating-per-capita.fixed.md | 1 + docs-quarto/methodology/spm-poverty.qmd | 2 +- policyengine_us/data/economic_assumptions.py | 21 +- policyengine_us/reforms/ssa/trustees_2025.py | 11 +- policyengine_us/system.py | 5 + .../tests/core/test_per_capita_uprating.py | 265 ++++++++++++++++++ .../parameters/test_uprating_extensions.py | 12 +- policyengine_us/tests/test_system_import.py | 2 +- policyengine_us/tools/per_capita_uprating.py | 157 +++++++++++ 9 files changed, 467 insertions(+), 9 deletions(-) create mode 100644 changelog.d/uprating-per-capita.fixed.md create mode 100644 policyengine_us/tests/core/test_per_capita_uprating.py create mode 100644 policyengine_us/tools/per_capita_uprating.py diff --git a/changelog.d/uprating-per-capita.fixed.md b/changelog.d/uprating-per-capita.fixed.md new file mode 100644 index 00000000000..c668252bb38 --- /dev/null +++ b/changelog.d/uprating-per-capita.fixed.md @@ -0,0 +1 @@ +Uprate dollar inputs by national totals per capita, so weighted totals track each national total instead of also growing with the population-uprated weights (a 0.9 percent overshoot in 2025 rising to 5.1 percent by 2035). diff --git a/docs-quarto/methodology/spm-poverty.qmd b/docs-quarto/methodology/spm-poverty.qmd index 30bd3686559..5de931f0750 100644 --- a/docs-quarto/methodology/spm-poverty.qmd +++ b/docs-quarto/methodology/spm-poverty.qmd @@ -50,7 +50,7 @@ The [MOOP decomposition page](moop-decomposition.qmd) records the design and the Population results use the Microcosm US dataset. The current default is Build P: 57,240 Current Population Survey Annual Social and Economic Supplement households with 166,321 people, whose weights are fitted to 5,659 administrative and demographic targets (IRS Statistics of Income by state and by income, Census population estimates by age, sex and state, CMS Medicaid and Marketplace enrollment, USDA SNAP participation and benefits, SSA SSI, TANF, state tax collections and BEA national accounts). Poverty rates are never a target. That is a rule of the calibration, not an accident: a model whose weights were fitted to survey-based poverty rates could not be checked against them. -2025 values are the 2024 survey values carried forward by the model's uprating parameters: each uprated input grows by the ratio of its uprating parameter between the two years. The main income sources (employment, self-employment and partnership income, Social Security, pensions and retirement account distributions, capital gains, dividends, interest, rental and farm income, and unemployment compensation) follow [IRS Statistics of Income series](https://github.com/PolicyEngine/policyengine-us/tree/main/policyengine_us/parameters/calibration/gov/irs/soi). Each series holds IRS totals through its last recorded year and a CBO projection extends it from there, so the growth applied from 2024 to 2025 is CBO's. Non-premium medical spending and the survey-reported premiums that enter SPM resources follow [CMS's projection of per-capita out-of-pocket health spending](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/calibration/gov/hhs/cms/moop_per_capita.yaml). Child support paid and received, the energy subsidy, real estate taxes and household assets follow [CPI-U](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/gov/bls/cpi/cpi_u.yaml). Every other dollar input, including rent and child care expenses, follows [CBO's projection of total adjusted gross income](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/calibration/gov/cbo/income_by_source.yaml), the default that [`default_uprating.py`](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/tools/default_uprating.py) assigns. Household weights follow the model's [total population series](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/calibration/gov/census/populations/total.yaml), which takes its values from CBO's demographic outlook; person and SPM unit weights read the household weight. Inputs without an uprating parameter (ages, take-up indicators, tenure, geography) carry over unchanged, so household composition is held at its 2024 state. The population is projected, not resampled. +2025 values are the 2024 survey values carried forward by the model's uprating parameters: each uprated input grows by the ratio of its uprating series between the two years. Household weights grow with total population, so an input that follows a national total grows by that total divided by population ([`per_capita_uprating.py`](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/tools/per_capita_uprating.py)); its weighted total then tracks the national total. The main income sources (employment, self-employment and partnership income, Social Security, pensions and retirement account distributions, capital gains, dividends, interest, rental and farm income, and unemployment compensation) follow [IRS Statistics of Income series](https://github.com/PolicyEngine/policyengine-us/tree/main/policyengine_us/parameters/calibration/gov/irs/soi). Each series holds IRS totals through its last recorded year and a CBO projection extends it from there, so the growth applied from 2024 to 2025 is CBO's, per capita. Non-premium medical spending and the survey-reported premiums that enter SPM resources follow [CMS's projection of per-capita out-of-pocket health spending](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/calibration/gov/hhs/cms/moop_per_capita.yaml), which is already a per-person series. Child support paid and received, the energy subsidy, real estate taxes and household assets follow [CPI-U](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/gov/bls/cpi/cpi_u.yaml). Every other dollar input, including rent and child care expenses, follows [CBO's projection of total adjusted gross income](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/calibration/gov/cbo/income_by_source.yaml) per capita, the default that [`default_uprating.py`](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/tools/default_uprating.py) assigns. Household weights follow the model's [total population series](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/calibration/gov/census/populations/total.yaml), which takes its values from CBO's demographic outlook; person and SPM unit weights read the household weight. Inputs without an uprating parameter (ages, take-up indicators, tenure, geography) carry over unchanged, so household composition is held at its 2024 state. The population is projected, not resampled. ## What the model does not do diff --git a/policyengine_us/data/economic_assumptions.py b/policyengine_us/data/economic_assumptions.py index 5d9c1a44da4..42bb97d1229 100644 --- a/policyengine_us/data/economic_assumptions.py +++ b/policyengine_us/data/economic_assumptions.py @@ -4,6 +4,10 @@ USSingleYearDataset, USMultiYearDataset, ) +from policyengine_us.tools.per_capita_uprating import ( + is_national_total_path, + per_capita_path, +) # The default end year for dataset extension is derived at runtime from # the CPI-U parameter YAML (gov.bls.cpi.cpi_u). When the CPI-U YAML @@ -174,7 +178,7 @@ def _apply_single_year_uprating(current, previous, system): if uprating_path is None: continue - param = _resolve_parameter(system.parameters, uprating_path) + param = _resolve_uprating_parameter(system.parameters, uprating_path) if param is None: continue @@ -191,6 +195,21 @@ def _apply_single_year_uprating(current, previous, system): current_df[col] = prev_df[col] * factor +def _resolve_uprating_parameter(parameters, path): + """Resolve an uprating path, preferring its per-capita series. + + Weights grow with population, so a column uprated by a national total + uses the total's per-capita sibling (see ``per_capita_uprating``). The + overrides above name national totals and resolve through here. A + parameter tree without the sibling falls back to the path as given. + """ + if is_national_total_path(path): + per_capita = _resolve_parameter(parameters, per_capita_path(path)) + if per_capita is not None: + return per_capita + return _resolve_parameter(parameters, path) + + def _resolve_parameter(parameters, path): """Resolve a dotted parameter path like 'gov.bls.cpi.cpi_u'.""" node = parameters diff --git a/policyengine_us/reforms/ssa/trustees_2025.py b/policyengine_us/reforms/ssa/trustees_2025.py index 0edfe1e77ed..20e7d8be422 100644 --- a/policyengine_us/reforms/ssa/trustees_2025.py +++ b/policyengine_us/reforms/ssa/trustees_2025.py @@ -157,10 +157,16 @@ def apply_trustees_2025_soi_income_projection( benefits use these calibration totals as uprating factors. If they flatten after the CBO projection window, nominal incomes freeze in long-run microsimulations even when NAWI and tax parameters keep growing. + + Inputs uprate by each total divided by population (see + ``policyengine_us.tools.per_capita_uprating``), so each total grows by + average-wage growth times population growth. Each record's income then + follows the average wage and the weights carry population growth. """ nawi = parameters.gov.ssa.nawi soi = parameters.calibration.gov.irs.soi + population = parameters.calibration.gov.census.populations.total for parameter_name in TRUSTEES_2025_SOI_INCOME_UPRATING_PARAMETERS: parameter = getattr(soi, parameter_name) @@ -170,9 +176,12 @@ def apply_trustees_2025_soi_income_projection( wage_growth = float(nawi(f"{year}-01-01")) / float( nawi(f"{year - 1}-01-01") ) + population_growth = float(population(f"{year}-01-01")) / float( + population(f"{year - 1}-01-01") + ) parameter.update( period=f"year:{year}-01-01:1", - value=previous_value * wage_growth, + value=previous_value * wage_growth * population_growth, ) final_value = float(parameter(f"{end_year}-01-01")) diff --git a/policyengine_us/system.py b/policyengine_us/system.py index f7798fd3cd6..74c810c3e03 100644 --- a/policyengine_us/system.py +++ b/policyengine_us/system.py @@ -32,6 +32,7 @@ uprate_parameters, ) from .tools.default_uprating import add_default_uprating +from .tools.per_capita_uprating import add_per_capita_uprating from policyengine_us.data.dataset_schema import ( US_ENTITIES, USSingleYearDataset, @@ -146,6 +147,10 @@ def __init__( self.add_variables(*create_50_state_variables()) + # Last, so reforms to a national total or to the population series + # reach the per-capita series the variables uprate by. + add_per_capita_uprating(self) + def clone(self): return clone_spm_system(self) diff --git a/policyengine_us/tests/core/test_per_capita_uprating.py b/policyengine_us/tests/core/test_per_capita_uprating.py new file mode 100644 index 00000000000..51cf0cd57ad --- /dev/null +++ b/policyengine_us/tests/core/test_per_capita_uprating.py @@ -0,0 +1,265 @@ +"""Dollar inputs uprate by per-capita growth, so weighted totals track the +national totals they follow instead of counting population growth twice.""" + +from functools import lru_cache + +import numpy as np +import pandas as pd +import pytest +from policyengine_core.parameters.operations.get_parameter import get_parameter +from policyengine_core.reforms import Reform + +from policyengine_us import CountryTaxBenefitSystem, Simulation +from policyengine_us.data.dataset_schema import USSingleYearDataset +from policyengine_us.data.economic_assumptions import ( + MICRODATA_UPRATING_OVERRIDES, + _resolve_uprating_parameter, + extend_single_year_dataset, +) +from policyengine_us.reforms.ssa import apply_trustees_2025_economic_assumptions +from policyengine_us.system import system +from policyengine_us.tools.per_capita_uprating import ( + PER_CAPITA_SUFFIX, + POPULATION_PATH, + add_per_capita_uprating, + is_national_total_path, + per_capita_path, +) + +SOI_EMPLOYMENT = "calibration.gov.irs.soi.employment_income" +CBO_AGI = "calibration.gov.cbo.income_by_source.adjusted_gross_income" +CMS_MOOP = "calibration.gov.hhs.cms.moop_per_capita" +CPI_U = "gov.bls.cpi.cpi_u" +YEARS = (2025, 2030, 2035) +BASE_YEAR = 2024 + + +def _value(path, year, parameters=None): + parameters = system.parameters if parameters is None else parameters + return get_parameter(parameters, path)(f"{year}-01-01") + + +def _growth(path, year, parameters=None): + return _value(path, year, parameters) / _value(path, BASE_YEAR, parameters) + + +@pytest.mark.parametrize( + "path, expected", + [ + (SOI_EMPLOYMENT, True), + (CBO_AGI, True), + ("calibration.gov.cbo.unemployment_compensation", True), + (CMS_MOOP, False), + (CPI_U, False), + (POPULATION_PATH, False), + (per_capita_path(SOI_EMPLOYMENT), False), + (None, False), + ], +) +def test_national_total_paths_are_identified(path, expected): + assert is_national_total_path(path) is expected + + +def test_no_variable_uprates_by_a_national_total(): + offenders = { + variable.name: variable.uprating + for variable in system.variables.values() + if is_national_total_path(variable.uprating) + } + assert offenders == {} + + +@pytest.mark.parametrize( + "variable, expected", + [ + ("employment_income_before_lsr", per_capita_path(SOI_EMPLOYMENT)), + ("pre_subsidy_rent", per_capita_path(CBO_AGI)), + ("spm_unit_pre_subsidy_childcare_expenses", per_capita_path(CBO_AGI)), + ("other_medical_expenses", CMS_MOOP), + ("child_support_expense", CPI_U), + ("household_weight", POPULATION_PATH), + ], +) +def test_variables_point_at_the_expected_series(variable, expected): + assert system.variables[variable].uprating == expected + + +def test_every_per_capita_series_is_its_total_over_population(): + paths = { + variable.uprating + for variable in system.variables.values() + if variable.uprating and variable.uprating.endswith(PER_CAPITA_SUFFIX) + } + totals = {path[: -len(PER_CAPITA_SUFFIX)] for path in paths} - { + # CMS publishes this series per capita; it has no total to divide. + CMS_MOOP[: -len(PER_CAPITA_SUFFIX)] + } + assert len(totals) >= 18 + for total in totals: + for year in (2015, 2017, BASE_YEAR, *YEARS, 2050): + expected = _value(total, year) / _value(POPULATION_PATH, year) + assert _value(per_capita_path(total), year) == pytest.approx( + expected, rel=1e-12 + ) + + +def test_microdata_overrides_resolve_to_per_capita_series(): + for column, path in MICRODATA_UPRATING_OVERRIDES.items(): + resolved = _resolve_uprating_parameter(system.parameters, path) + assert resolved is not None, column + if is_national_total_path(path): + assert resolved.name == per_capita_path(path), column + else: + assert resolved.name == path, column + + +def _tiny_dataset(): + person = pd.DataFrame( + { + "person_id": [1, 2, 3, 4], + "person_household_id": [1, 1, 2, 3], + "employment_income_before_lsr": [50_000.0, 0.0, 120_000.0, 30_000.0], + "pre_subsidy_rent": [12_000.0, 0.0, 30_000.0, 9_000.0], + "other_medical_expenses": [500.0, 250.0, 4_000.0, 0.0], + "child_support_expense": [0.0, 0.0, 6_000.0, 0.0], + "age": [40.0, 8.0, 55.0, 70.0], + } + ) + household = pd.DataFrame( + { + "household_id": [1, 2, 3], + "household_weight": [1_000.0, 2_500.0, 400.0], + } + ) + return USSingleYearDataset( + person=person, + household=household, + tax_unit=pd.DataFrame({"tax_unit_id": [1, 2, 3]}), + spm_unit=pd.DataFrame({"spm_unit_id": [1, 2, 3]}), + family=pd.DataFrame({"family_id": [1, 2, 3]}), + marital_unit=pd.DataFrame({"marital_unit_id": [1, 2, 3, 4]}), + time_period=BASE_YEAR, + ) + + +def _weighted_total(dataset, column): + weights = dataset.household.set_index("household_id")["household_weight"] + person_weights = dataset.person["person_household_id"].map(weights) + return float((dataset.person[column] * person_weights).sum()) + + +@lru_cache(maxsize=1) +def _extended(): + return extend_single_year_dataset(_tiny_dataset(), end_year=2035, system=system) + + +@pytest.mark.parametrize("year", YEARS) +def test_extended_dataset_totals_track_national_totals(year): + base, later = _extended().datasets[BASE_YEAR], _extended().datasets[year] + + def growth(column): + return _weighted_total(later, column) / _weighted_total(base, column) + + population = _growth(POPULATION_PATH, year) + assert later.household["household_weight"].sum() / base.household[ + "household_weight" + ].sum() == pytest.approx(population, rel=1e-9) + # National totals: the weighted total follows the total itself. + assert growth("employment_income_before_lsr") == pytest.approx( + _growth(SOI_EMPLOYMENT, year), rel=1e-9 + ) + assert growth("pre_subsidy_rent") == pytest.approx(_growth(CBO_AGI, year), rel=1e-9) + # Per-person rates and price indices: amounts follow the series and the + # weights add population growth. + assert growth("other_medical_expenses") == pytest.approx( + _growth(CMS_MOOP, year) * population, rel=1e-9 + ) + assert growth("child_support_expense") == pytest.approx( + _growth(CPI_U, year) * population, rel=1e-9 + ) + # Inputs without an uprating parameter carry over unchanged. + assert later.person["age"].tolist() == base.person["age"].tolist() + + +def test_simulation_uprating_tracks_national_totals(): + simulation = Simulation( + situation={ + "people": { + "adult": { + "age": {BASE_YEAR: 40}, + "employment_income_before_lsr": {BASE_YEAR: 50_000}, + } + }, + "households": { + "household": { + "members": ["adult"], + "household_weight": {BASE_YEAR: 1_000}, + } + }, + } + ) + for year in YEARS: + income = simulation.calculate("employment_income_before_lsr", year)[0] + weight = simulation.calculate("household_weight", year)[0] + assert income / 50_000 == pytest.approx( + _growth(per_capita_path(SOI_EMPLOYMENT), year), rel=1e-6 + ) + assert (income * weight) / (50_000 * 1_000) == pytest.approx( + _growth(SOI_EMPLOYMENT, year), rel=1e-6 + ) + + +class _LargerPopulation(Reform): + def apply(self): + def modify(parameters): + population = get_parameter(parameters, POPULATION_PATH) + # System init applies a reform twice, so set an absolute value. + population.update( + period="year:2030-01-01:1", + value=_value(POPULATION_PATH, 2030) * 2, + ) + return parameters + + self.modify_parameters(modify) + + +def test_reform_to_population_reaches_per_capita_series(): + reformed = CountryTaxBenefitSystem(reform=_LargerPopulation).parameters + path = per_capita_path(SOI_EMPLOYMENT) + assert _value(path, 2030, reformed) == pytest.approx( + _value(path, 2030) / 2, rel=1e-12 + ) + assert _value(path, 2029, reformed) == pytest.approx(_value(path, 2029), rel=1e-12) + + +def test_rebuilding_is_idempotent(): + rebuilt = CountryTaxBenefitSystem() + before = {v.name: v.uprating for v in rebuilt.variables.values()} + value = _value(per_capita_path(CBO_AGI), 2030, rebuilt.parameters) + add_per_capita_uprating(rebuilt) + assert {v.name: v.uprating for v in rebuilt.variables.values()} == before + assert _value(per_capita_path(CBO_AGI), 2030, rebuilt.parameters) == value + + +class _Trustees2025(Reform): + def apply(self): + def modify(parameters): + apply_trustees_2025_economic_assumptions(parameters) + return parameters + + self.modify_parameters(modify) + + +def test_trustees_long_run_incomes_grow_with_average_wages(): + """The Trustees scenario ages each record's income with the average wage + after the CBO window; population growth belongs to the weights.""" + parameters = CountryTaxBenefitSystem(reform=_Trustees2025).parameters + path = per_capita_path(SOI_EMPLOYMENT) + for year in (2040, 2050): + wage_growth = _value("gov.ssa.nawi", year, parameters) / _value( + "gov.ssa.nawi", year - 1, parameters + ) + per_record_growth = _value(path, year, parameters) / _value( + path, year - 1, parameters + ) + assert per_record_growth == pytest.approx(wage_growth, rel=1e-9) diff --git a/policyengine_us/tests/policy/baseline/parameters/test_uprating_extensions.py b/policyengine_us/tests/policy/baseline/parameters/test_uprating_extensions.py index aab2e2a4a03..e4c5cd4ead2 100644 --- a/policyengine_us/tests/policy/baseline/parameters/test_uprating_extensions.py +++ b/policyengine_us/tests/policy/baseline/parameters/test_uprating_extensions.py @@ -3,6 +3,7 @@ import pytest from policyengine_us.system import system +from policyengine_us.tools.per_capita_uprating import per_capita_path from policyengine_us.parameters.uprating_extensions import ( LONG_RUN_CBO_INCOME_BY_SOURCE_PARAMETERS, round_social_security_amount, @@ -112,9 +113,8 @@ def test_social_security_benefit_inputs_use_gross_benefit_uprater(): "social_security_survivors", "social_security_dependents", ]: - assert ( - system.variables[variable_name].uprating - == "calibration.gov.irs.soi.social_security" + assert system.variables[variable_name].uprating == per_capita_path( + "calibration.gov.irs.soi.social_security" ) @@ -146,8 +146,10 @@ def test_soi_income_upraters_extend_without_trustees_reform(): def test_retirement_distribution_inputs_use_pension_upraters(): """Retirement account components should age with pension income, not AGI.""" - taxable_uprater = "calibration.gov.irs.soi.taxable_pension_income" - tax_exempt_uprater = "calibration.gov.irs.soi.tax_exempt_pension_income" + taxable_uprater = per_capita_path("calibration.gov.irs.soi.taxable_pension_income") + tax_exempt_uprater = per_capita_path( + "calibration.gov.irs.soi.tax_exempt_pension_income" + ) for variable_name in [ "csrs_retirement_pay", diff --git a/policyengine_us/tests/test_system_import.py b/policyengine_us/tests/test_system_import.py index 87804832320..b997c92658b 100644 --- a/policyengine_us/tests/test_system_import.py +++ b/policyengine_us/tests/test_system_import.py @@ -90,6 +90,6 @@ def test_capital_gains_indexation_inputs_have_expected_uprating(): system = CountryTaxBenefitSystem() assert ( system.variables["long_term_capital_gains_basis"].uprating - == "calibration.gov.irs.soi.long_term_capital_gains" + == "calibration.gov.irs.soi.long_term_capital_gains_per_capita" ) assert system.variables["long_term_capital_gains_years_held"].uprating is None diff --git a/policyengine_us/tools/per_capita_uprating.py b/policyengine_us/tools/per_capita_uprating.py new file mode 100644 index 00000000000..60ca01ed85e --- /dev/null +++ b/policyengine_us/tools/per_capita_uprating.py @@ -0,0 +1,157 @@ +"""Uprate dollar inputs by per-capita growth rather than aggregate growth. + +Household weights grow with total population +(``calibration.gov.census.populations.total``). A dollar input that also +grows with a national total, such as a CBO income projection or an IRS +Statistics of Income series, would count population growth twice: once in +the amount and once in the weight. Dividing each national total by +population makes the weighted total of the input track the national total. + +For every national total that a variable uprates by, this module adds a +sibling parameter holding the total divided by population +(``_per_capita``) and points the variable at it. Series that are +already per-person rates or price indices (CMS per-capita spending, CPI-U) +are left alone, as is the population series that uprates the weights. +""" + +from policyengine_core.parameters import Parameter +from policyengine_core.parameters.operations.get_parameter import get_parameter +from policyengine_core.periods import instant + +POPULATION_PATH = "calibration.gov.census.populations.total" +NATIONAL_TOTAL_PREFIX = "calibration.gov." +POPULATION_PREFIX = "calibration.gov.census.populations." +PER_CAPITA_MARKER = "per_capita" +PER_CAPITA_SUFFIX = "_per_capita" + + +def is_national_total_path(path: str | None) -> bool: + """Whether an uprating parameter path names a national total. + + National totals live under ``calibration.gov``. Paths that already say + ``per_capita`` and the population series itself are excluded. + """ + if not path: + return False + return ( + path.startswith(NATIONAL_TOTAL_PREFIX) + and not path.startswith(POPULATION_PREFIX) + and PER_CAPITA_MARKER not in path + ) + + +def per_capita_path(path: str) -> str: + return path + PER_CAPITA_SUFFIX + + +def _per_capita_values(total: Parameter, population: Parameter) -> dict[str, float]: + """Total divided by population at every instant either series changes. + + Instants are limited to the span the total covers. After its last value + the total is flat only because the projection ends, so the per-capita + series is held flat there too and amounts per record stop changing. + """ + total_instants = [entry.instant_str for entry in total.values_list] + first, last = min(total_instants), max(total_instants) + instants = set(total_instants) | { + entry.instant_str + for entry in population.values_list + if first <= entry.instant_str <= last + } + values = {} + for instant_str in sorted(instants): + total_value = total(instant_str) + population_value = population(instant_str) + if total_value is None or not population_value: + continue + values[instant_str] = total_value / population_value + return values + + +def _add_per_capita_parameter(parameters, path: str) -> None: + total = get_parameter(parameters, path) + population = get_parameter(parameters, POPULATION_PATH) + parent_path, _, name = path.rpartition(".") + parent = get_parameter(parameters, parent_path) + sibling_name = name + PER_CAPITA_SUFFIX + label = total.metadata.get("label", name) + sibling = Parameter( + per_capita_path(path), + { + "description": ( + f"{path} divided by {POPULATION_PATH}. PolicyEngine derives " + "this series when it builds the tax-benefit system and uses " + "it to uprate dollar inputs." + ), + "metadata": { + "unit": total.metadata.get("unit", "currency-USD"), + "label": f"{label} per capita", + "economy": False, + "household": False, + }, + "values": _per_capita_values(total, population), + }, + ) + # Rebuilding keeps the series in step with a reform that changed the + # total or the population after an earlier build. + if sibling_name in parent.children: + del parent.children[sibling_name] + parent.add_child(sibling_name, sibling) + + +def add_per_capita_uprating(system) -> None: + """Point every variable that uprates by a national total at its + per-capita sibling, building the sibling parameters as needed. + + Call this after reforms are applied, so a reform that changes a national + total or the population series flows into the per-capita series. + """ + from policyengine_us.data.economic_assumptions import ( + MICRODATA_UPRATING_OVERRIDES, + ) + + paths = { + variable.uprating + for variable in system.variables.values() + if is_national_total_path(variable.uprating) + } + paths |= { + path + for path in MICRODATA_UPRATING_OVERRIDES.values() + if is_national_total_path(path) + } + # A second call finds variables already pointing at per-capita series; + # refresh those series from their totals as well. + paths |= { + variable.uprating[: -len(PER_CAPITA_SUFFIX)] + for variable in system.variables.values() + if variable.uprating + and variable.uprating.startswith(NATIONAL_TOTAL_PREFIX) + and variable.uprating.endswith(PER_CAPITA_SUFFIX) + and _has_parameter( + system.parameters, variable.uprating[: -len(PER_CAPITA_SUFFIX)] + ) + } + for path in sorted(paths): + _add_per_capita_parameter(system.parameters, path) + for variable in system.variables.values(): + if is_national_total_path(variable.uprating): + variable.uprating = per_capita_path(variable.uprating) + + +def _has_parameter(parameters, path: str) -> bool: + node = parameters + for part in path.split("."): + node = getattr(node, "children", {}).get(part) + if node is None: + return False + return True + + +__all__ = [ + "POPULATION_PATH", + "PER_CAPITA_SUFFIX", + "add_per_capita_uprating", + "is_national_total_path", + "per_capita_path", +] From 6021ba55d63f2d393a6bfa05ee53fc380fbc1c0d Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 18 Sep 2026 09:46:17 -0400 Subject: [PATCH 2/4] Address review of per-capita uprating - HHS state median income was projected from 2027 by aggregate AGI while incomes now grow per capita, so the threshold outran incomes by population growth (2.8 percent by 2035) and shifted childcare and energy eligibility. Project it by AGI per capita; build that series before parameter uprating. - Refresh the derived series after every modify_parameters call. Core's Simulation applies the reform again after system init, and a reform applied only then left the series stale. - Dataset extension now raises when a national total has no per-capita sibling, and when a derived series no longer equals its total over population, instead of quietly double counting or ignoring a change. - Classify national totals by family (CBO, IRS SOI) and reject other calibration paths: state totals and head counts need other denominators. - Mark derived series with derived_from; refresh only those, and refuse to overwrite a hand-written series that shares the derived name. - Clear ancestor at-instant caches after adding a derived series. - Docs: default_uprating assigns the total; per_capita_uprating converts it. - Tests for each of the above, the post-2055 boundary, and mock fixtures that carry the sibling. Co-Authored-By: Claude Fable 5.1 --- docs-quarto/methodology/spm-poverty.qmd | 2 +- policyengine_us/data/economic_assumptions.py | 27 ++- .../parameters/gov/hhs/smi/amount.yaml | 3 +- policyengine_us/system.py | 15 +- .../tests/core/test_per_capita_uprating.py | 152 ++++++++++++++ .../test_extend_single_year_dataset.py | 25 ++- policyengine_us/tools/per_capita_uprating.py | 192 +++++++++++++----- 7 files changed, 354 insertions(+), 62 deletions(-) diff --git a/docs-quarto/methodology/spm-poverty.qmd b/docs-quarto/methodology/spm-poverty.qmd index 5de931f0750..cde383b882c 100644 --- a/docs-quarto/methodology/spm-poverty.qmd +++ b/docs-quarto/methodology/spm-poverty.qmd @@ -50,7 +50,7 @@ The [MOOP decomposition page](moop-decomposition.qmd) records the design and the Population results use the Microcosm US dataset. The current default is Build P: 57,240 Current Population Survey Annual Social and Economic Supplement households with 166,321 people, whose weights are fitted to 5,659 administrative and demographic targets (IRS Statistics of Income by state and by income, Census population estimates by age, sex and state, CMS Medicaid and Marketplace enrollment, USDA SNAP participation and benefits, SSA SSI, TANF, state tax collections and BEA national accounts). Poverty rates are never a target. That is a rule of the calibration, not an accident: a model whose weights were fitted to survey-based poverty rates could not be checked against them. -2025 values are the 2024 survey values carried forward by the model's uprating parameters: each uprated input grows by the ratio of its uprating series between the two years. Household weights grow with total population, so an input that follows a national total grows by that total divided by population ([`per_capita_uprating.py`](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/tools/per_capita_uprating.py)); its weighted total then tracks the national total. The main income sources (employment, self-employment and partnership income, Social Security, pensions and retirement account distributions, capital gains, dividends, interest, rental and farm income, and unemployment compensation) follow [IRS Statistics of Income series](https://github.com/PolicyEngine/policyengine-us/tree/main/policyengine_us/parameters/calibration/gov/irs/soi). Each series holds IRS totals through its last recorded year and a CBO projection extends it from there, so the growth applied from 2024 to 2025 is CBO's, per capita. Non-premium medical spending and the survey-reported premiums that enter SPM resources follow [CMS's projection of per-capita out-of-pocket health spending](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/calibration/gov/hhs/cms/moop_per_capita.yaml), which is already a per-person series. Child support paid and received, the energy subsidy, real estate taxes and household assets follow [CPI-U](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/gov/bls/cpi/cpi_u.yaml). Every other dollar input, including rent and child care expenses, follows [CBO's projection of total adjusted gross income](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/calibration/gov/cbo/income_by_source.yaml) per capita, the default that [`default_uprating.py`](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/tools/default_uprating.py) assigns. Household weights follow the model's [total population series](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/calibration/gov/census/populations/total.yaml), which takes its values from CBO's demographic outlook; person and SPM unit weights read the household weight. Inputs without an uprating parameter (ages, take-up indicators, tenure, geography) carry over unchanged, so household composition is held at its 2024 state. The population is projected, not resampled. +2025 values are the 2024 survey values carried forward by the model's uprating parameters: each uprated input grows by the ratio of its uprating series between the two years. Household weights grow with total population, so an input that follows a national total grows by that total divided by population ([`per_capita_uprating.py`](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/tools/per_capita_uprating.py)); its weighted total then tracks the national total. The main income sources (employment, self-employment and partnership income, Social Security, pensions and retirement account distributions, capital gains, dividends, interest, rental and farm income, and unemployment compensation) follow [IRS Statistics of Income series](https://github.com/PolicyEngine/policyengine-us/tree/main/policyengine_us/parameters/calibration/gov/irs/soi). Each series holds IRS totals through its last recorded year and a CBO projection extends it from there, so the growth applied from 2024 to 2025 is CBO's, per capita. Non-premium medical spending and the survey-reported premiums that enter SPM resources follow [CMS's projection of per-capita out-of-pocket health spending](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/calibration/gov/hhs/cms/moop_per_capita.yaml), which is already a per-person series. Child support paid and received, the energy subsidy, real estate taxes and household assets follow [CPI-U](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/gov/bls/cpi/cpi_u.yaml). Every other dollar input, including rent and child care expenses, follows [CBO's projection of total adjusted gross income](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/calibration/gov/cbo/income_by_source.yaml), the default that [`default_uprating.py`](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/tools/default_uprating.py) assigns, per capita like every national total. Household weights follow the model's [total population series](https://github.com/PolicyEngine/policyengine-us/blob/main/policyengine_us/parameters/calibration/gov/census/populations/total.yaml), which takes its values from CBO's demographic outlook; person and SPM unit weights read the household weight. Inputs without an uprating parameter (ages, take-up indicators, tenure, geography) carry over unchanged, so household composition is held at its 2024 state. The population is projected, not resampled. ## What the model does not do diff --git a/policyengine_us/data/economic_assumptions.py b/policyengine_us/data/economic_assumptions.py index 42bb97d1229..c72459f9833 100644 --- a/policyengine_us/data/economic_assumptions.py +++ b/policyengine_us/data/economic_assumptions.py @@ -5,6 +5,8 @@ USMultiYearDataset, ) from policyengine_us.tools.per_capita_uprating import ( + DERIVED_FROM, + check_per_capita_series_is_current, is_national_total_path, per_capita_path, ) @@ -182,6 +184,12 @@ def _apply_single_year_uprating(current, previous, system): if param is None: continue + derived_from = getattr(param, "metadata", {}).get(DERIVED_FROM) + if derived_from: + for period in (previous_period, current_period): + check_per_capita_series_is_current( + system.parameters, derived_from, period + ) prev_val = param(previous_period) curr_val = param(current_period) if prev_val == 0: @@ -196,17 +204,26 @@ def _apply_single_year_uprating(current, previous, system): def _resolve_uprating_parameter(parameters, path): - """Resolve an uprating path, preferring its per-capita series. + """Resolve an uprating path, using the per-capita series for a national + total. Weights grow with population, so a column uprated by a national total uses the total's per-capita sibling (see ``per_capita_uprating``). The - overrides above name national totals and resolve through here. A - parameter tree without the sibling falls back to the path as given. + overrides above name national totals and resolve through here. A tree + that holds the total but not its sibling raises: falling back to the + total would quietly count population growth twice. """ if is_national_total_path(path): + if _resolve_parameter(parameters, path) is None: + return None per_capita = _resolve_parameter(parameters, per_capita_path(path)) - if per_capita is not None: - return per_capita + if per_capita is None: + raise ValueError( + f"{path} is a national total but {per_capita_path(path)} is " + "missing from the parameter tree. Build the system with " + "CountryTaxBenefitSystem, or call add_per_capita_uprating." + ) + return per_capita return _resolve_parameter(parameters, path) diff --git a/policyengine_us/parameters/gov/hhs/smi/amount.yaml b/policyengine_us/parameters/gov/hhs/smi/amount.yaml index 7099bb650f3..481a73e7174 100644 --- a/policyengine_us/parameters/gov/hhs/smi/amount.yaml +++ b/policyengine_us/parameters/gov/hhs/smi/amount.yaml @@ -580,7 +580,8 @@ metadata: unit: currency-USD period: year uprating: - parameter: calibration.gov.cbo.income_by_source.adjusted_gross_income + # Per capita, like the incomes this threshold is compared against. + parameter: calibration.gov.cbo.income_by_source.adjusted_gross_income_per_capita start_instant: 2027-01-01 reference: - title: 60 percent of estimated State Median Income adjusted for family size, by state, FFY 2018 (Attachment 1, LIHEAP-IM-2017-03) diff --git a/policyengine_us/system.py b/policyengine_us/system.py index 74c810c3e03..af9c644d584 100644 --- a/policyengine_us/system.py +++ b/policyengine_us/system.py @@ -32,7 +32,10 @@ uprate_parameters, ) from .tools.default_uprating import add_default_uprating -from .tools.per_capita_uprating import add_per_capita_uprating +from .tools.per_capita_uprating import ( + add_per_capita_parameters_for_parameter_uprating, + add_per_capita_uprating, +) from policyengine_us.data.dataset_schema import ( US_ENTITIES, USSingleYearDataset, @@ -114,6 +117,7 @@ def __init__( ) self.parameters = propagate_parameter_metadata(self.parameters) self.parameters = interpolate_parameters(self.parameters) + add_per_capita_parameters_for_parameter_uprating(self.parameters) self.parameters = uprate_parameters(self.parameters) self.parameters = propagate_parameter_metadata(self.parameters) add_default_uprating(self) @@ -150,6 +154,15 @@ def __init__( # Last, so reforms to a national total or to the population series # reach the per-capita series the variables uprate by. add_per_capita_uprating(self) + self._per_capita_uprating_built = True + + def modify_parameters(self, modifier_function): + result = super().modify_parameters(modifier_function) + # Reforms applied after init (core's Simulation applies the reform + # once more) must reach the derived per-capita series too. + if getattr(self, "_per_capita_uprating_built", False): + add_per_capita_uprating(self) + return result def clone(self): return clone_spm_system(self) diff --git a/policyengine_us/tests/core/test_per_capita_uprating.py b/policyengine_us/tests/core/test_per_capita_uprating.py index 51cf0cd57ad..fdd5798ab52 100644 --- a/policyengine_us/tests/core/test_per_capita_uprating.py +++ b/policyengine_us/tests/core/test_per_capita_uprating.py @@ -18,9 +18,16 @@ ) from policyengine_us.reforms.ssa import apply_trustees_2025_economic_assumptions from policyengine_us.system import system +from policyengine_us.tests.microsimulation.data.fixtures.test_extend_single_year_dataset import ( # noqa: E501 + MockSystem, + MockVariable, + build_mock_parameters, +) from policyengine_us.tools.per_capita_uprating import ( + DERIVED_FROM, PER_CAPITA_SUFFIX, POPULATION_PATH, + _add_per_capita_parameter, add_per_capita_uprating, is_national_total_path, per_capita_path, @@ -53,6 +60,11 @@ def _growth(path, year, parameters=None): (CPI_U, False), (POPULATION_PATH, False), (per_capita_path(SOI_EMPLOYMENT), False), + # A state total and a head count: national population is the wrong + # denominator, so these are not treated as national totals. + ("calibration.gov.aca.spending.state.AL", False), + ("calibration.gov.aca.enrollment.state.AK", False), + ("gov.ssa.nawi", False), (None, False), ], ) @@ -263,3 +275,143 @@ def test_trustees_long_run_incomes_grow_with_average_wages(): path, year - 1, parameters ) assert per_record_growth == pytest.approx(wage_growth, rel=1e-9) + + +def test_trustees_long_run_holds_after_the_population_series_ends(): + """Population is flat after 2055, so totals and per-record incomes both + grow with the average wage alone.""" + parameters = CountryTaxBenefitSystem(reform=_Trustees2025).parameters + path = per_capita_path(SOI_EMPLOYMENT) + assert _value(POPULATION_PATH, 2060, parameters) == _value( + POPULATION_PATH, 2056, parameters + ) + wage_growth = _value("gov.ssa.nawi", 2060, parameters) / _value( + "gov.ssa.nawi", 2059, parameters + ) + for series in (path, SOI_EMPLOYMENT): + growth = _value(series, 2060, parameters) / _value(series, 2059, parameters) + assert growth == pytest.approx(wage_growth, rel=1e-9) + assert _value(path, 2060, parameters) == pytest.approx( + _value(SOI_EMPLOYMENT, 2060, parameters) + / _value(POPULATION_PATH, 2060, parameters), + rel=1e-12, + ) + + +def test_smi_threshold_keeps_pace_with_default_incomes(): + """HHS state median income is projected from 2027; it must grow like the + incomes tested against it, or eligibility drifts with population.""" + for year in (2028, 2030, 2035, 2040): + threshold = _value("gov.hhs.smi.amount.CA", year) / _value( + "gov.hhs.smi.amount.CA", 2027 + ) + income = _value(per_capita_path(CBO_AGI), year) / _value( + per_capita_path(CBO_AGI), 2027 + ) + assert threshold == pytest.approx(income, rel=1e-9) + + +def test_derived_series_are_reachable_through_at_instant_views(): + node = system.parameters("2030-01-01").calibration.gov.irs.soi + assert node.employment_income_per_capita == pytest.approx( + _value(per_capita_path(SOI_EMPLOYMENT), 2030) + ) + + +def test_derived_series_are_marked_and_cms_series_is_untouched(): + derived = get_parameter(system.parameters, per_capita_path(SOI_EMPLOYMENT)) + assert derived.metadata[DERIVED_FROM] == SOI_EMPLOYMENT + cms = get_parameter(system.parameters, CMS_MOOP) + assert DERIVED_FROM not in cms.metadata + assert cms.file_path is not None + + +def test_changes_after_init_reach_the_series_or_fail_loudly(): + fresh = CountryTaxBenefitSystem() + path = per_capita_path(SOI_EMPLOYMENT) + before = _value(path, 2030, fresh.parameters) + + # A reform applied after init, as core's Simulation does, refreshes it. + fresh.apply_reform_set(_LargerPopulation) + assert _value(path, 2030, fresh.parameters) == pytest.approx(before / 2, rel=1e-12) + + # A parameter edited behind the system's back leaves the series stale; + # dataset extension refuses to use it. + total = get_parameter(fresh.parameters, SOI_EMPLOYMENT) + total.update(period="year:2030-01-01:1", value=total("2030-01-01") * 3) + with pytest.raises(ValueError, match="out of date"): + extend_single_year_dataset(_tiny_dataset(), end_year=2030, system=fresh) + add_per_capita_uprating(fresh) + extended = extend_single_year_dataset(_tiny_dataset(), end_year=2030, system=fresh) + assert 2030 in extended.datasets + + +def test_dataset_extension_reads_the_per_capita_sibling(): + total = {"2024-01-01": 100.0, "2025-01-01": 110.0} + sibling = {"2024-01-01": 50.0, "2025-01-01": 52.0} + mock = MockSystem( + variables={ + "employment_income": MockVariable("employment_income", SOI_EMPLOYMENT) + }, + parameters=build_mock_parameters( + {SOI_EMPLOYMENT: total, per_capita_path(SOI_EMPLOYMENT): sibling} + ), + ) + dataset = USSingleYearDataset( + person=pd.DataFrame({"person_id": [1], "employment_income": [1_000.0]}), + household=pd.DataFrame({"household_id": [1]}), + tax_unit=pd.DataFrame({"tax_unit_id": [1]}), + spm_unit=pd.DataFrame({"spm_unit_id": [1]}), + family=pd.DataFrame({"family_id": [1]}), + marital_unit=pd.DataFrame({"marital_unit_id": [1]}), + time_period=BASE_YEAR, + ) + extended = extend_single_year_dataset(dataset, end_year=2025, system=mock) + assert extended.datasets[2025].person["employment_income"][0] == pytest.approx( + 1_000.0 * 52.0 / 50.0 + ) + + +def test_missing_per_capita_sibling_raises(): + parameters = build_mock_parameters( + {SOI_EMPLOYMENT: {"2024-01-01": 100.0}}, per_capita_siblings=False + ) + with pytest.raises(ValueError, match="national total"): + _resolve_uprating_parameter(parameters, SOI_EMPLOYMENT) + + +class _StubVariable: + def __init__(self, uprating): + self.uprating = uprating + + +class _StubSystem: + def __init__(self, uprating): + self.variables = {"stub": _StubVariable(uprating)} + self.parameters = None + + +def test_unclassified_calibration_path_is_rejected(): + with pytest.raises(ValueError, match="national-total family"): + add_per_capita_uprating(_StubSystem("calibration.gov.aca.spending.state.AL")) + + +def test_existing_per_capita_series_is_never_overwritten(): + """A series someone wrote by hand that happens to share the derived name + must stop the build, not be replaced.""" + from policyengine_core.parameters import ParameterNode + + values = {"values": {"2024-01-01": 10.0}} + tree = ParameterNode( + "", + data={ + "calibration": { + "gov": { + "cbo": {"spending": values, "spending_per_capita": values}, + "census": {"populations": {"total": values}}, + } + } + }, + ) + with pytest.raises(ValueError, match="already exists"): + _add_per_capita_parameter(tree, "calibration.gov.cbo.spending") diff --git a/policyengine_us/tests/microsimulation/data/fixtures/test_extend_single_year_dataset.py b/policyengine_us/tests/microsimulation/data/fixtures/test_extend_single_year_dataset.py index 29d0e786128..32f0df8ece9 100644 --- a/policyengine_us/tests/microsimulation/data/fixtures/test_extend_single_year_dataset.py +++ b/policyengine_us/tests/microsimulation/data/fixtures/test_extend_single_year_dataset.py @@ -114,11 +114,32 @@ def __getattr__(self, name): raise AttributeError(name) -def build_mock_parameters(param_specs: dict[str, dict[str, float]]): - """Build a mock parameter tree from {dotted_path: {period: value}}.""" +def build_mock_parameters( + param_specs: dict[str, dict[str, float]], + per_capita_siblings: bool = True, +): + """Build a mock parameter tree from {dotted_path: {period: value}}. + + Dataset extension reads a national total through its per-capita sibling, + which the real system derives. The mock gives each national total a + sibling with the same values, as if population were constant, unless the + spec supplies one or ``per_capita_siblings`` is False. + """ + from policyengine_us.tools.per_capita_uprating import ( + is_national_total_path, + per_capita_path, + ) + root = _ParameterNode() for path, values in param_specs.items(): root._add_path(path, MockParameter(values)) + sibling = per_capita_path(path) + if ( + per_capita_siblings + and is_national_total_path(path) + and sibling not in param_specs + ): + root._add_path(sibling, MockParameter(values)) return root diff --git a/policyengine_us/tools/per_capita_uprating.py b/policyengine_us/tools/per_capita_uprating.py index 60ca01ed85e..d14029ef711 100644 --- a/policyengine_us/tools/per_capita_uprating.py +++ b/policyengine_us/tools/per_capita_uprating.py @@ -12,38 +12,63 @@ (``_per_capita``) and points the variable at it. Series that are already per-person rates or price indices (CMS per-capita spending, CPI-U) are left alone, as is the population series that uprates the weights. + +Parameters can follow a per-capita series too: a parameter whose ``uprating`` +metadata names ``_per_capita`` gets that sibling built before +parameter uprating runs. Income thresholds projected this way keep pace with +the incomes they are compared against. """ +import math + from policyengine_core.parameters import Parameter -from policyengine_core.parameters.operations.get_parameter import get_parameter -from policyengine_core.periods import instant POPULATION_PATH = "calibration.gov.census.populations.total" -NATIONAL_TOTAL_PREFIX = "calibration.gov." -POPULATION_PREFIX = "calibration.gov.census.populations." +# Families of national totals. A path elsewhere under ``calibration`` may be a +# state total or a head count, where national population is the wrong +# denominator, so those are rejected rather than guessed at. +NATIONAL_TOTAL_PREFIXES = ( + "calibration.gov.cbo.", + "calibration.gov.irs.soi.", +) +CALIBRATION_PREFIX = "calibration." PER_CAPITA_MARKER = "per_capita" PER_CAPITA_SUFFIX = "_per_capita" +DERIVED_FROM = "derived_from" def is_national_total_path(path: str | None) -> bool: - """Whether an uprating parameter path names a national total. - - National totals live under ``calibration.gov``. Paths that already say - ``per_capita`` and the population series itself are excluded. - """ + """Whether an uprating parameter path names a national total.""" if not path: return False - return ( - path.startswith(NATIONAL_TOTAL_PREFIX) - and not path.startswith(POPULATION_PREFIX) - and PER_CAPITA_MARKER not in path - ) + return path.startswith(NATIONAL_TOTAL_PREFIXES) and PER_CAPITA_MARKER not in path def per_capita_path(path: str) -> str: return path + PER_CAPITA_SUFFIX +def _is_unclassified_calibration_path(path: str | None) -> bool: + """A ``calibration`` path this module cannot classify: not a national + total, not already per capita, not the population series.""" + if not path or not path.startswith(CALIBRATION_PREFIX): + return False + return not ( + is_national_total_path(path) + or PER_CAPITA_MARKER in path + or path == POPULATION_PATH + ) + + +def _find(parameters, path: str): + node = parameters + for part in path.split("."): + node = getattr(node, "children", {}).get(part) + if node is None: + return None + return node + + def _per_capita_values(total: Parameter, population: Parameter) -> dict[str, float]: """Total divided by population at every instant either series changes. @@ -69,11 +94,23 @@ def _per_capita_values(total: Parameter, population: Parameter) -> dict[str, flo def _add_per_capita_parameter(parameters, path: str) -> None: - total = get_parameter(parameters, path) - population = get_parameter(parameters, POPULATION_PATH) + total = _find(parameters, path) + population = _find(parameters, POPULATION_PATH) + if total is None or population is None: + raise ValueError( + f"Cannot build {per_capita_path(path)}: {path} or {POPULATION_PATH} " + "is missing from the parameter tree." + ) parent_path, _, name = path.rpartition(".") - parent = get_parameter(parameters, parent_path) + parent = _find(parameters, parent_path) sibling_name = name + PER_CAPITA_SUFFIX + existing = parent.children.get(sibling_name) + if existing is not None and existing.metadata.get(DERIVED_FROM) != path: + raise ValueError( + f"{per_capita_path(path)} already exists and was not derived from " + f"{path}. Rename one of them; PolicyEngine derives " + f"{PER_CAPITA_SUFFIX} series itself." + ) label = total.metadata.get("label", name) sibling = Parameter( per_capita_path(path), @@ -88,70 +125,121 @@ def _add_per_capita_parameter(parameters, path: str) -> None: "label": f"{label} per capita", "economy": False, "household": False, + DERIVED_FROM: path, }, "values": _per_capita_values(total, population), }, ) - # Rebuilding keeps the series in step with a reform that changed the - # total or the population after an earlier build. - if sibling_name in parent.children: + if existing is not None: del parent.children[sibling_name] parent.add_child(sibling_name, sibling) + # ``add_child`` leaves at-instant views built earlier without the child. + parent.clear_parent_cache() + + +def _uprating_reference(parameter) -> str | None: + meta = getattr(parameter, "metadata", {}).get("uprating") + if isinstance(meta, dict): + meta = meta.get("parameter") + return meta if isinstance(meta, str) else None + + +def add_per_capita_parameters_for_parameter_uprating(parameters) -> None: + """Build the per-capita series that parameters uprate by. + + Call this before parameter uprating runs. Only totals that are complete + at that point qualify: a total that is itself extended by parameter + uprating would be divided before its projection exists. + """ + references = { + reference + for parameter in parameters.get_descendants() + if (reference := _uprating_reference(parameter)) + and reference.endswith(PER_CAPITA_SUFFIX) + and is_national_total_path(reference[: -len(PER_CAPITA_SUFFIX)]) + and _find(parameters, reference) is None + } + for reference in sorted(references): + total_path = reference[: -len(PER_CAPITA_SUFFIX)] + if _uprating_reference(_find(parameters, total_path)) is not None: + raise ValueError( + f"{reference} is used to uprate a parameter, but {total_path} " + "is itself extended by parameter uprating, so it is incomplete " + "when parameter uprating runs." + ) + _add_per_capita_parameter(parameters, total_path) def add_per_capita_uprating(system) -> None: """Point every variable that uprates by a national total at its - per-capita sibling, building the sibling parameters as needed. + per-capita sibling, building or refreshing the sibling parameters. - Call this after reforms are applied, so a reform that changes a national - total or the population series flows into the per-capita series. + Runs at the end of system init and again after every parameter reform, so + a reform to a national total or to the population series reaches the + series the variables uprate by. """ from policyengine_us.data.economic_assumptions import ( MICRODATA_UPRATING_OVERRIDES, ) - paths = { - variable.uprating - for variable in system.variables.values() - if is_national_total_path(variable.uprating) - } - paths |= { - path - for path in MICRODATA_UPRATING_OVERRIDES.values() - if is_national_total_path(path) - } - # A second call finds variables already pointing at per-capita series; - # refresh those series from their totals as well. - paths |= { - variable.uprating[: -len(PER_CAPITA_SUFFIX)] - for variable in system.variables.values() - if variable.uprating - and variable.uprating.startswith(NATIONAL_TOTAL_PREFIX) - and variable.uprating.endswith(PER_CAPITA_SUFFIX) - and _has_parameter( - system.parameters, variable.uprating[: -len(PER_CAPITA_SUFFIX)] + in_use = [variable.uprating for variable in system.variables.values()] + in_use += list(MICRODATA_UPRATING_OVERRIDES.values()) + unclassified = sorted({p for p in in_use if _is_unclassified_calibration_path(p)}) + if unclassified: + raise ValueError( + f"Uprating paths {unclassified} are under 'calibration' but are not " + f"in a national-total family {NATIONAL_TOTAL_PREFIXES}. Dividing a " + "state total or a head count by national population would be " + "wrong, so add the family explicitly if it is a national total." ) - } - for path in sorted(paths): + + totals = {path for path in in_use if is_national_total_path(path)} + # Series derived earlier, by a previous call or for parameter uprating: + # refresh them from their totals. + for path in in_use: + parameter = _find(system.parameters, path) if path else None + derived_from = getattr(parameter, "metadata", {}).get(DERIVED_FROM) + if derived_from: + totals.add(derived_from) + for path in sorted(totals): _add_per_capita_parameter(system.parameters, path) for variable in system.variables.values(): if is_national_total_path(variable.uprating): variable.uprating = per_capita_path(variable.uprating) -def _has_parameter(parameters, path: str) -> bool: - node = parameters - for part in path.split("."): - node = getattr(node, "children", {}).get(part) - if node is None: - return False - return True +def check_per_capita_series_is_current(parameters, total_path: str, period: str): + """Raise if a derived series no longer equals its total over population. + + A parameter changed without going through ``modify_parameters`` leaves + the derived series behind; uprating with it would silently ignore the + change. + """ + total = _find(parameters, total_path) + population = _find(parameters, POPULATION_PATH) + derived = _find(parameters, per_capita_path(total_path)) + if total is None or population is None or derived is None: + return + total_value, population_value = total(period), population(period) + if total_value is None or not population_value: + return + expected = total_value / population_value + if not math.isclose(derived(period), expected, rel_tol=1e-9): + raise ValueError( + f"{per_capita_path(total_path)} is out of date at {period}: " + f"{derived(period)} against {expected}. A parameter changed " + "outside modify_parameters; call add_per_capita_uprating(system)." + ) __all__ = [ + "DERIVED_FROM", + "NATIONAL_TOTAL_PREFIXES", "POPULATION_PATH", "PER_CAPITA_SUFFIX", + "add_per_capita_parameters_for_parameter_uprating", "add_per_capita_uprating", + "check_per_capita_series_is_current", "is_national_total_path", "per_capita_path", ] From ab96702d7e0584535597240a9f0e6417ce22d5e2 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 18 Sep 2026 10:59:18 -0400 Subject: [PATCH 3/4] Build fewer systems in the per-capita uprating tests The Rest CI job already runs near its 60-minute limit; cache the Trustees build and share one fresh system across the post-init checks (five builds to three). Co-Authored-By: Claude Fable 5.1 --- .../tests/core/test_per_capita_uprating.py | 26 ++++++++++--------- 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/policyengine_us/tests/core/test_per_capita_uprating.py b/policyengine_us/tests/core/test_per_capita_uprating.py index fdd5798ab52..7a501b5e4d2 100644 --- a/policyengine_us/tests/core/test_per_capita_uprating.py +++ b/policyengine_us/tests/core/test_per_capita_uprating.py @@ -244,15 +244,6 @@ def test_reform_to_population_reaches_per_capita_series(): assert _value(path, 2029, reformed) == pytest.approx(_value(path, 2029), rel=1e-12) -def test_rebuilding_is_idempotent(): - rebuilt = CountryTaxBenefitSystem() - before = {v.name: v.uprating for v in rebuilt.variables.values()} - value = _value(per_capita_path(CBO_AGI), 2030, rebuilt.parameters) - add_per_capita_uprating(rebuilt) - assert {v.name: v.uprating for v in rebuilt.variables.values()} == before - assert _value(per_capita_path(CBO_AGI), 2030, rebuilt.parameters) == value - - class _Trustees2025(Reform): def apply(self): def modify(parameters): @@ -262,10 +253,15 @@ def modify(parameters): self.modify_parameters(modify) +@lru_cache(maxsize=1) +def _trustees_parameters(): + return CountryTaxBenefitSystem(reform=_Trustees2025).parameters + + def test_trustees_long_run_incomes_grow_with_average_wages(): """The Trustees scenario ages each record's income with the average wage after the CBO window; population growth belongs to the weights.""" - parameters = CountryTaxBenefitSystem(reform=_Trustees2025).parameters + parameters = _trustees_parameters() path = per_capita_path(SOI_EMPLOYMENT) for year in (2040, 2050): wage_growth = _value("gov.ssa.nawi", year, parameters) / _value( @@ -280,7 +276,7 @@ def test_trustees_long_run_incomes_grow_with_average_wages(): def test_trustees_long_run_holds_after_the_population_series_ends(): """Population is flat after 2055, so totals and per-record incomes both grow with the average wage alone.""" - parameters = CountryTaxBenefitSystem(reform=_Trustees2025).parameters + parameters = _trustees_parameters() path = per_capita_path(SOI_EMPLOYMENT) assert _value(POPULATION_PATH, 2060, parameters) == _value( POPULATION_PATH, 2056, parameters @@ -326,11 +322,17 @@ def test_derived_series_are_marked_and_cms_series_is_untouched(): assert cms.file_path is not None -def test_changes_after_init_reach_the_series_or_fail_loudly(): +def test_rebuilds_reforms_and_stale_edits_after_init(): fresh = CountryTaxBenefitSystem() path = per_capita_path(SOI_EMPLOYMENT) before = _value(path, 2030, fresh.parameters) + # Rebuilding is a no-op. + uprating = {v.name: v.uprating for v in fresh.variables.values()} + add_per_capita_uprating(fresh) + assert {v.name: v.uprating for v in fresh.variables.values()} == uprating + assert _value(path, 2030, fresh.parameters) == before + # A reform applied after init, as core's Simulation does, refreshes it. fresh.apply_reform_set(_LargerPopulation) assert _value(path, 2030, fresh.parameters) == pytest.approx(before / 2, rel=1e-12) From 3e12bd0a5cd8f88a3fb78f32c6b768d4db309b91 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 18 Sep 2026 13:29:05 -0400 Subject: [PATCH 4/4] Reproject the 2031 Washington ECEAP boundary cases The two one-dollar cases pin the 50 percent SMI floor for 2031. State median income is now projected per capita with AGI, so the floor is 69,317.41 instead of 70,405.39: the old value divided by population growth from 2027. Co-Authored-By: Claude Fable 5.1 --- .../wa/dcyf/eceap/wa_eceap_income_eligible.yaml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/policyengine_us/tests/policy/baseline/gov/states/wa/dcyf/eceap/wa_eceap_income_eligible.yaml b/policyengine_us/tests/policy/baseline/gov/states/wa/dcyf/eceap/wa_eceap_income_eligible.yaml index e656f7ef0ac..93635df21df 100644 --- a/policyengine_us/tests/policy/baseline/gov/states/wa/dcyf/eceap/wa_eceap_income_eligible.yaml +++ b/policyengine_us/tests/policy/baseline/gov/states/wa/dcyf/eceap/wa_eceap_income_eligible.yaml @@ -138,8 +138,8 @@ members: [parent, child, sibling] state_code: WA output: - # 50% SMI for WA family of 3 in 2031 (after uprating) = $70,405.39. - # $55,000 < $70,405.39, so income-eligible under the 2030-08-01 50% SMI threshold. + # 50% SMI for WA family of 3 in 2031 (projected per capita with AGI) = $69,317.41. + # $55,000 < $69,317.41, so income-eligible under the 2030-08-01 50% SMI threshold. wa_eceap_income_eligible: [true, true, true] - name: 2031 SMI era, family of 3 at $75,000, above 50% SMI threshold. @@ -167,7 +167,7 @@ members: [parent, child, sibling] state_code: WA output: - # 50% SMI family of 3 in 2031 = $70,405.39; $75,000 > threshold. + # 50% SMI family of 3 in 2031 = $69,317.41; $75,000 > threshold. wa_eceap_income_eligible: [false, false, false] - name: Oregon family, income below threshold, not eligible due to residency. @@ -469,13 +469,13 @@ # 36% SMI family of 3 in 2026 = $42,128.25; $42,129 > threshold. wa_eceap_income_eligible: [false, false, false] -- name: 2031 family of 3 at $73,057, one dollar below 50% SMI floor, eligible. +- name: 2031 family of 3 at $69,316, one dollar below 50% SMI floor, eligible. period: 2031 input: people: parent: age: 30 - employment_income: 70_404 + employment_income: 69_316 child: age: 4 employment_income: 0 @@ -494,16 +494,16 @@ members: [parent, child, sibling] state_code: WA output: - # 50% SMI family of 3 in 2031 = $70,405.39. $70,404 <= threshold. + # 50% SMI family of 3 in 2031 = $69,317.41. $69,316 <= threshold. wa_eceap_income_eligible: [true, true, true] -- name: 2031 family of 3 at $73,058, one dollar above 50% SMI floor, not eligible. +- name: 2031 family of 3 at $69,318, one dollar above 50% SMI floor, not eligible. period: 2031 input: people: parent: age: 30 - employment_income: 70_406 + employment_income: 69_318 child: age: 4 employment_income: 0