Draft: Integrate modeled state unemployment insurance into income flows

[daphnehanse11]

Tracks #8301.

Purpose

This is the draft umbrella PR for wiring modeled state unemployment insurance benefits into unemployment_compensation and downstream income flows.

Keep this PR as a draft until the relevant state rules and policyengine-us-data pipelines are ready. State UI rule implementations can still merge separately when they are calculator-safe; this PR is for the baseline microsimulation integration step.

Why draft

The current PA wiring is conceptually right for household calculations, but it should not enter baseline aggregation until enhanced CPS can populate or impute the required wage-history and unemployment inputs. Otherwise, the integration would be partial: modeled UI would depend on inputs that are not yet available in the microsim data pipeline, and could mix calculator-only logic into baseline income flows prematurely.

Scope

• Add modeled state UI variables to unemployment_compensation only after each state has usable data inputs.
• Coordinate with policyengine-us-data for extraction/imputation of required variables.
• Add microsimulation regressions that protect reported unemployment compensation and validate nonzero modeled UI where expected.
• Use this branch as the collection point for aggregator wiring across states.

Current contents

• PA UC aggregator wiring and an income-flow test. This is kept here as the first candidate integration, but it is not merge-ready until the data pipeline prerequisites are met.

Checklist

• Inventory state UI variables and required inputs for each modeled state.
• Keep state UI rule/calculator PRs separate where they can safely merge without baseline aggregation.
• Add or impute required inputs in policyengine-us-data.
• Bump policyengine-us-data minimum policyengine-us version after any new model variables are released.
• Validate enhanced CPS has nonzero modeled UI for expected state populations.
• Add a microsim regression confirming unemployment_compensation does not drop relative to the reported baseline.
• Add state UI variables to unemployment_compensation.adds once data-backed.
• Re-run targeted UI, household income-flow, TaxSIM, and full CI checks.
• Undraft only after the data pipeline and microsim validation are complete.

Tests so far

These passed for the current PA-only wiring, but they are not sufficient to merge this PR:

• uv run python -m policyengine_core.scripts.policyengine_command test policyengine_us/tests/policy/baseline/gov/states/pa/dli/unemployment_compensation -c policyengine_us
• uv run python -m policyengine_core.scripts.policyengine_command test policyengine_us/tests/policy/contrib/taxsim/taxsim_v11.yaml -c policyengine_us
• uv run python -m policyengine_core.scripts.policyengine_command test policyengine_us/tests/policy/baseline/household/income/spm_unit/spm_unit_benefits.yaml -c policyengine_us
• uv run --extra dev ruff check policyengine_us/variables/gov/states/unemployment_compensation.py
• uv run --extra dev ruff format --check policyengine_us/variables/gov/states/unemployment_compensation.py

[codecov]

Codecov Report

❌ Patch coverage is 86.20690% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.72%. Comparing base (8326a14) to head (f4aa5e9).
⚠️ Report is 572 commits behind head on main.

Files with missing lines	Patch %	Lines
...cyengine_us/variables/contrib/taxsim/taxsim_pui.py	0.00%	1 Missing ⚠️
...icyengine_us/variables/contrib/taxsim/taxsim_ui.py	0.00%	1 Missing ⚠️
...eductions/capped_qualified_tuition_expenses_ald.py	0.00%	1 Missing ⚠️
...ef/ut_homeowner_renter_relief_nontaxable_income.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##              main    #8303       +/-   ##
============================================
- Coverage   100.00%   73.72%   -26.28%     
============================================
  Files           12       15        +3     
  Lines          161      255       +94     
  Branches         0        9        +9     
============================================
+ Hits           161      188       +27     
- Misses           0       64       +64     
- Partials         0        3        +3     

Flag	Coverage Δ
unittests	73.72% <86.20%> (-26.28%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[MaxGhenis]

Converted this to a draft umbrella PR for modeled state UI income-flow integration. The PA UC wiring is a useful first candidate, but we should not merge it into baseline unemployment_compensation until the corresponding policyengine-us-data inputs/imputations and microsim regression are in place. That avoids shipping a partial calculator-only aggregation path before enhanced CPS can support it.

[DTrim99]

@PavelMakarchuk — picking this draft up now that the AL/NY/OK/PA/UT UI programs are all merged. I've rebased onto current main and extended the wiring to every merged state UI program: unemployment_compensation now adds = ["al_ui", "ny_ui", "ok_ui", "pa_uc", "ut_ui"], with a "feeds unemployment compensation income flows" integration case added per state (mirroring Daphne's PA Case 8). State UI suites are 47/47 and the taxable-UC, my_friend_ben, and analytics-coverage partner signature tests are unaffected.

But before this comes out of draft I want your call on two things — you lead US encoding so this is your decision, not something to escalate.

1. Should modeled state UI feed baseline unemployment_compensation at all?

unemployment_compensation is the survey-reported income field (SOI-uprated from the enhanced CPS). Adding the modeled programs to it has an awkward interaction either way:

• Double-count risk: the survey value already reflects the UC these households actually received. Layering modeled *_ui/*_uc on top would count the same benefit twice for anyone the model fires on.
• Or it's inert: because unemployment_compensation is provided as a dataset input, input precedence means the adds formula isn't evaluated in the microsim — so in the baseline economy (web app) the modeled UI doesn't actually flow, even though the unit tests pass (they set the state-UI inputs and leave unemployment_compensation unset, so the formula fires). Classic "green unit test, no microsim effect."

So as written this is either a no-op in the baseline or a double-count — the unit tests can't distinguish them. This is exactly why ny_ui/ok_ui were left deliberately unwired with the "must not double-count with unemployment_compensation" note.

My read: modeled state UI probably shouldn't be additively wired into the baseline survey aggregate — it more naturally belongs behind a "use modeled UI" reform (replace the survey UC with the modeled amount for those states), or requires decomposing the survey UC so the modeled portion isn't double-counted. But I'd rather have your steer than pick unilaterally.

2. Needs a microsim validation before merge

Whatever we choose, this changes a calibrated national aggregate, so it needs a microsim run (not just the scalar tests) confirming national unemployment_compensation doesn't move unexpectedly (neither inflated by double-counting nor collapsed toward the modeled-only value). Happy to run that once we agree on the intended behavior.

Question for you: do you want modeled state UI in baseline income at all, or gated behind a reform? If baseline, how should it reconcile with the survey UC to avoid the double-count? I'll implement whichever you prefer across all five states and keep it draft until then.

Rebase/extension done with Claude Code assistance.

[PavelMakarchuk]

Requesting changes for one confirmed import regression at 926a7f2f6ea5d9dd825ec49b72b722af98ab83b3.

[P1] Resolve the incompatible computation modes in unemployment_compensation. The new adds on line 12 is combined with the existing uprating on line 11. Core rejects that combination during variable registration:

ValueError: Variable "unemployment_compensation" mixes computation modes: adds/subtracts and uprating.

This prevents the country package from importing, before any household calculation or state eligibility check runs. The head-matched smoke-import job installs successfully and then raises this error; the microsimulation job fails with the same exception. I reproduced it locally under Core 3.32.6 and checked the exact base/head variable definitions: the base initializes with uprating, while the head fails. Zero targeted YAML cases could execute.

Please use a compatible design for reported-income uprating and modeled state-benefit aggregation, then rerun package import and the five integration files. Also validate reported-benefit preservation, modeled fallback and future-year uprating against the intended microsimulation dataset; simply removing uprating would not establish that those semantics remain correct.

I am not treating draft status, branch staleness, or presumed loss of reported benefits as defects. The data review found CPS explicitly supplies reported unemployment compensation, and no population loss or double counting was demonstrated. The blocker here is the reproducible import failure.

[DTrim99]

@PavelMakarchuk Resolved the computation-mode conflict via Option A: unemployment_compensation is restored to the uprated reported input (uprating only, no adds), a new modeled_state_unemployment_compensation aggregates the five state UI programs (al_ui, ny_ui, ok_uc, pa_uc, ut_ui), and income flows now route through a new total_unemployment_compensation = where(reported > 0, reported, modeled) fallback, with the 13 direct consumers repointed. I rejected the rename alternatives (making unemployment_compensation itself a fallback formula, or adds over a reported twin) precisely because the enhanced-CPS microdata writes the unemployment_compensation dataset column directly: renaming it would zero out ~all reported UC in future-year microsim until a coordinated policyengine-us-data column remap ships, which is the regression you flagged. Option A keeps the dataset column mapping and the uprating attribute untouched, so future-year aggregates are unchanged (UI wage inputs are absent in microdata, so modeled=0 and total==reported). Package import now registers cleanly (the check_computation_modes ValueError is gone); the five integration files pass along with new reported-preservation, modeled-fallback, and no-double-count cases in each.

[DTrim99]

Superseded by #9558 — rebased onto current main under new ownership (original author has left PolicyEngine). Pavel's mode-conflict blocker is resolved there; review history referenced in the new PR.