Register weighted-microdata rules, weight the builder tests, and check change weights and household counts

[anth-volk]

Fixes #806

Summary

Phase 1 step 1.9: weighted-microdata rules. Stacked on #804.

The executor's entity frames are MicroDataFrames weighted by <entity>_weight, so any column's .sum() is already weighted. #767 fixed budget.households, which summed household_weight through the household frame and published the sum of squared weights, about 10^4 times the count. It escaped because the builder test used plain pd.DataFrames.

This PR:

• registers the rule that weights aren't values;
• guards the shapes a source check can recognise;
• makes the builder tests weight as production does;
• adds two runtime checks.

Five commits, each passing the guards:

1. Builder tests through one weighted factory (test-only). fixtures/weighted_frames.weighted_entity_frame builds every output-builder test's entity frames as MicroDataFrames. It rejects a weight column that is the same on every row.
2. The rules, guards and proofs.
3. Runtime check: warn when change totals are weighted differently on each side.
4. Runtime check: refuse an implausible published household count.
5. The UK local-authority handoff becomes a recorded, ratcheted exception in the module allowed to strip weights.

Rules

Rule	Status	Findings
sim.microdata.weights-not-values: weights, values, labels and counts each allow different operations	doc-only	—
sim.microdata.weights-not-aggregated-as-values: a weight is never the variable of _sum_output_variable, _change_output_variable, Aggregate or ChangeAggregate, nor frame[<weight>] followed by sum, mean, median, quantile, std, count, groupby or cumsum	enforced	0
sim.microdata.weight-stripping-confined: raw values (np.asarray, np.array, .values, .to_numpy(), .tolist(), pd.DataFrame, pd.Series) are read from a weighted frame only in simulation_output_common.py. There, counting entities is always allowed, and every other read is a recorded exception	ratcheted	1 recorded exception
sim.microdata.builder-tests-use-weighted-frames: tests/test_simulation_output_*.py build entity frames only through the shared factory	enforced	0

The guards live in projects/policyengine-simulation-executor/tests/test_microdata_weight_boundaries.py, so the executor leg of the Test job runs them.

How the guards recognise weights and weighted frames:

• A weight is a string or f-string ending in _weight or _weights (including f"{entity}_weight"), a name assigned from one, or a .weights attribute.
• A weighted frame is any of:
  • <simulation>.output_dataset.data.<entity>;
  • getattr(<simulation>.output_dataset.data, ...);
  • _entity_data(...);
  • a MicroDataFrame or MicroSeries built with weights=;
  • anything assigned, indexed or computed from those.
• What they can't see: they don't follow data through function calls or containers. So the whole-frame readers that iterate output_dataset.data.entity_data are outside them: simulation_microdata, the Stage 12 simulation worker and the precompute determinism check.

Proof

Each rule has a bad and a good fixture under rules/fixtures/<rule-id>/, and a test_rule_proof_* test asserting exactly which expressions are flagged.

• Aggregation:
  • Bad fixture: the budget builder before Count households once: sum the weight column unweighted #767, with households=_sum_output_variable(baseline, "household_weight", entity="household").
  • Good fixture: Count households once: sum the weight column unweighted #767's replacement, _count_output_entities(baseline, entity="household", weight="household_weight").
• Stripping: np.asarray(household["household_net_income"]).sum(), an unweighted sample total and the mirror image of Count households once: sum the weight column unweighted #767, against the weighted frame's own .sum().
• Builder tests: the plain pd.DataFrame households the builder test used when Count households once: sum the weight column unweighted #767 escaped, and a hand-built MicroDataFrame, against the shared factory.

Reverting #767's line on this branch now fails the aggregation guard and three builder tests. Before commit 1, only #767's own test could catch it: the other builder tests used plain frames, or weights of 1 and 1, which give the same count either way.

The recorded stripping exception

The UK local-authority breakdown calls policyengine's compute_longwise_uk_geography_impacts with plain household frames, so it can apply this service's own boundary metadata.

Why the strip is deliberate. That function reads income and the baseline's household_weight as arrays, and weights each area by hand. Handing it the weighted frame would weight twice if it ever reduced a column directly.

What commit 5 does:

• The helper. The handoff moves into simulation_output_common.py as _unweighted_household_frame. Its docstring says why the helper exists and what would remove it.
• The constituency check. The constituency breakdown already hands policyengine the simulations, as the US congressional district breakdown does. It only used the plain frame to check for its area-code column, and now reads the weighted frame for that.
• The ratchet. The stripping rule now checks the designated module as well. Counting entities is always allowed there. Every other read must be listed in rules/baselines/sim-microdata-weight-stripping.txt, and that list may only shrink.
  • Its one entry is _unweighted_household_frame.
  • test_weight_stripping_review_ratchets_the_designated_module pins how findings are sorted.
  • Adding a new read to the module fails the guard.

What removes the exception. policyengine's local-authority output needs a way to skip its own area lookup. The breakdown can then hand over the simulations, like the constituency breakdown.

Builder test changes

• _macro_baseline_reform:
  • Its two households now weigh 0.5 and 1.5. The values are chosen so the weighted totals still equal the golden macro result's budget, so CURRENT_SINGLE_YEAR_MACRO_RESULT is unchanged.
  • Unweighted sums would give 120, 40, 24 and 1200 instead of 100, 30, 20 and 1000.
  • Summing the weights through the weighted frame (Count households once: sum the weight column unweighted #767) would give 2.5 households instead of 2.
• test_builder_budgetary_impact_uses_materialized_columns_and_uk_state_tax_zero:
  • It now expects weighted totals, written out in the test as weight * (reform - baseline) sums.
  • Tax 25 (was 15), state tax 8 (was 5), benefits -5 (was -3), budget 30 (was 18), baseline net income 500 (was 300). Households stay at 3.
  • The old values were unweighted sums of plain frames, which production never returned.

Runtime checks

• Weight mismatch (_change_output_variable):
  • reform - baseline keeps the reform's weights, so a change total over differently weighted sides would pass silently.
  • It now logs a warning when the two frames' weights differ in length, or by more than a float32 round trip (relative tolerance 1e-6). The total is still published.
  • Making this an error waits until logs show that no legitimate run produces a mismatch.
• ImplausibleEntityCountError (build_budgetary_impact):
  • The household count must be finite, non-negative, and at most:
    • US: 200 million (about 131 million households, US Census Bureau Current Population Survey 2023);
    • UK: 60 million (about 28 million, ONS Families and households 2023).
  • Otherwise the run fails. GET /jobs/{job_id} returns 500 with status: failed and the redacted Simulation failed (correlation_id=...), and a Stage 12 report is marked failed with the exception's name as its error summary.
  • Country-level bounds can't catch an inflated count in a small region.

Testing

• Suites:
  • executor uv run pytest tests/: 1278 passed, 22 skipped, 3 deselected;
  • entry: 528 passed;
  • contract: 207 passed.
• Guards:
  • python scripts/run_quality_guards.py: rules-registry and ci-scripts passed;
  • python -m scripts.rules guard --base-ref origin/ruff-complexity: passed;
  • python -m scripts.rules render leaves no diff.
• Rules tool: python -m unittest discover -s scripts/rules/tests -t . passes.
• Ruff: ruff check . and ruff format --check (ruff 0.15.22, as CI pins it) are clean, with no new noqa.
• Changelog: changelog_entry.yaml parses with PyYAML, and every entry is a plain string.
• Negative checks:
  • With Count households once: sum the weight column unweighted #767's line reverted, the aggregation guard and three builder tests fail.
  • With a new strip added to simulation_output_common.py, the stripping guard fails.
• No published schema or OpenAPI document changes.

🤖 Generated with Claude Code