Register weighted-microdata rules, with AST guards and #767 proofs

AGENTS.md

@@ -29,6 +29,10 @@ When changing correlation identifiers, telemetry transport, spans, stage
 names, logging, or observability failure handling, read
 `docs/engineering/skills/observability.md`.
 
+When changing code that aggregates simulation microdata (`MicroSeries`,
+`MicroDataFrame` or economy result columns), read
+`docs/engineering/skills/microdata-weights.md`.
+
 When modifying the `Makefile`, read
 `docs/engineering/skills/repository-maintenance.md`.
 
@@ -94,6 +98,11 @@ To add or change a rule, read `docs/engineering/skills/rules-registry.md`.
 - Checked: `api.migration.alembic-v1-qualified`, `api.migration.alembic-v2-qualified`, `api.migration.bundle-supported-by-simulation-api`, `api.migration.contracts-consistent`, `api.migration.cross-database-integration`
 - `api.migration.alembic-autogenerated-only`: Alembic revisions are autogenerated against an explicitly selected v1 or v2 configuration; never hand-written DDL or data operations. (doc-only, not checked: follow it yourself)
 
+### Weighted microdata
+
+- Checked: `api.microdata.unweighted-series-are-labels`, `api.microdata.weight-stripping-confined`, `api.microdata.weights-not-aggregated-as-values`
+- `api.microdata.weights-not-values`: Microdata columns are weights, values, labels or counts: weights are only summed plainly to count entities or passed as weights=, values are aggregated with their own entity's weights, labels only group or filter, and counts derive from weights. (doc-only, not checked: follow it yourself)
+
 ### Code quality and types
 
 - Checked: `api.quality.alembic-revisions-unused-imports`, `api.quality.ci-cd-scripts-ruff-exemptions`, `api.quality.command-line-modules-ruff-exemptions`, `api.quality.command-line-scripts-ruff-exemptions`, `api.quality.environment-variables-load`, `api.quality.mypy-v2`, `api.quality.ruff-format`, `api.quality.ruff-lint`, `api.quality.runtime-evaluated-models-resolve`, `api.quality.tests-pass`, `api.quality.tests-ruff-exemptions`, `api.quality.unittest-ruff-exemptions`

changelog.d/microdata-weight-rules.added.md

@@ -0,0 +1 @@
+Added weighted-microdata rules to the registry, proven on the Simulation API's #767 household count: code in the package never aggregates a weight column as a value, never reads raw values from weighted microdata outside a designated module, and never reduces an unweighted label series. No current code uses microdata, so all three start enforced with no findings. A doc-only rule records that weights, values, labels and counts each allow different operations.

docs/engineering/skills/README.md

@@ -16,6 +16,8 @@ Current skills:
 - `ci-workflows.md`: one line of code per workflow `run:` block, tested
   workflow scripts, and shellcheck.
 - `github-prs.md`: PR workflow and migration PR handoff expectations.
+- `microdata-weights.md`: weights, values, labels and counts in simulation
+  microdata, and the guards that keep weights from being aggregated as values.
 - `migration_contracts.md`: API v2 migration route contracts, route-group
   metadata, generated migration artifacts, and quality guards.
 - `observability.md`: identifier ownership, HTTP and simulation transport,

docs/engineering/skills/microdata-weights.md

@@ -0,0 +1,82 @@
+# Microdata Weights Skill
+
+Read this before adding code that aggregates simulation microdata: anything
+that builds a `MicroSeries` or `MicroDataFrame`, or reads per-household or
+per-person columns from an economy result. No module in the package does
+today: the last one, the unused economy comparison, has been removed. The
+rules below keep new code honest.
+
+## Weights aren't values
+
+A microdata column is one of four kinds, and each allows different operations
+(rule `api.microdata.weights-not-values`):
+
+| Kind | Examples | Allowed | Never |
+| --- | --- | --- | --- |
+| Weight | `household_weight`, `person_weight` | A plain (unweighted) sum, to count entities; use as `weights=` | Aggregation as a value through a weighted series, frame or aggregator |
+| Value | `household_net_income`, `person_in_poverty` | Weighted aggregation, with its own entity's weights | Aggregation after the weights are stripped |
+| Label | `household_income_decile`, `age`, `is_male`, `race` | Grouping and filtering | Summing or averaging |
+| Count | `households`, people per decile | Derived from weights | |
+
+A weighted `MicroSeries` or `MicroDataFrame` weights every `.sum()`, `.mean()`
+and `.count()`. Aggregating a weight column that way weights it by itself and
+returns the sum of squared weights. In the Simulation API (#766, fixed by
+#767), that made `budget.households` about 10^4 times too high on every run,
+and its test missed it because it used plain `pd.DataFrame`s.
+
+policyengine's economy outputs arrive as `baseline` and `reform` dicts, whose
+microdata columns sit beside their weight columns. Code that builds a
+`MicroSeries` from them by hand follows three patterns:
+
+- Values pass `weights=`, and reform values reuse the baseline's weights
+  (`weights=baseline_income.weights`).
+- Labels (income and wealth deciles, `age`, `is_male`, `race`) are built
+  without `weights=` and are only grouping or filter keys.
+- Households are counted with a plain sum of `household_weight`.
+
+## Rules
+
+- `api.microdata.weights-not-aggregated-as-values`: a weight is never the value
+  of a weighted `MicroSeries`, the variable of a weighted aggregator, or
+  `<frame>[<weight>]` followed by a weighted reduction (`sum`, `mean`,
+  `median`, `quantile`, `std`, `count`, `groupby`, `cumsum`).
+- `api.microdata.weight-stripping-confined`: raw values are read from weighted
+  microdata (`np.array`, `np.asarray`, `.values`, `.to_numpy()`, `.tolist()`,
+  `pd.DataFrame(...)`, `pd.Series(...)`) only in a designated stripping module.
+  Summing stripped values gives an unweighted sample total, the mirror image of
+  #767. The Public API has no designated module yet.
+- `api.microdata.unweighted-series-are-labels`: a `MicroSeries` built without
+  `weights=` is never reduced, so it stays a grouping or filter key.
+
+The guards live in `tests/unit/services/test_microdata_weight_boundaries.py`
+and run in `make test`.
+
+## How the guard recognises a weight
+
+The guards read the source, so they recognise names, not types:
+
+- a weight is a string or f-string ending in `_weight` or `_weights` (including
+  `f"{entity}_weight"`), a name assigned from one, or a `.weights` attribute;
+- weighted microdata is a `MicroSeries` or `MicroDataFrame` built with
+  `weights=`, a name assigned from one, anything indexed, filtered or computed
+  from those, and, in a module that imports microdf, a string-keyed read from a
+  `baseline` or `reform` dict;
+- a label is a `MicroSeries` built without `weights=`, or a comparison or
+  filter of one.
+
+They don't follow data through function calls or containers, so a weight passed
+through a helper or a dict escapes them.
+
+## Handling a finding
+
+Each finding needs a decision, recorded in the PR:
+
+- **Fix it.** Count entities with a plain sum of the weights, aggregate values
+  through a weighted series, and reduce the weighted series grouped or filtered
+  by a label rather than the label itself.
+- **Move it.** A deliberate strip (for example, a per-household quantity used
+  only to sort households into buckets) moves into the designated stripping
+  module with a comment saying why, and the guard names that module.
+
+All three guards start with no findings and no baseline, so any finding
+fails the build until it's decided.

docs/generated/rules.md

@@ -8,16 +8,17 @@ Every rule that governs `policyengine-api` is listed here with how it is enforce
 
 | Tier | enforced | ratcheted | advisory | doc-only | Total |
 | --- | ---: | ---: | ---: | ---: | ---: |
-| mechanical | 48 | 0 | 1 | 12 | 61 |
+| mechanical | 51 | 0 | 1 | 13 | 65 |
 | metric | 0 | 0 | 0 | 0 | 0 |
 | judgment | 0 | 0 | 0 | 0 | 0 |
 
-**Blocking:** 48 of 61 rules.
+**Blocking:** 51 of 65 rules.
 
 ## Gaps
 
 ### Not enforced (doc-only)
 
+- `api.microdata.weights-not-values`: Microdata columns are weights, values, labels or counts: weights are only summed plainly to count entities or passed as weights=, values are aggregated with their own entity's weights, labels only group or filter, and counts derive from weights.
 - `api.migration.alembic-autogenerated-only`: Alembic revisions are autogenerated against an explicitly selected v1 or v2 configuration; never hand-written DDL or data operations.
 - `api.observability.id-is-diagnostic-only`: observability_id is used only to query diagnostic records, never for idempotency, cache ownership, authorization, database identity, routing or calculation behavior.
 - `api.observability.registered-stages-only`: Span names come from the stage plans in policyengine_api.observability.stages; route and service code never adds span-name string literals.
@@ -264,6 +265,31 @@ flowchart LR
   e_CI_job_v2_integration_check --> j_CI_pr_yml_v2_integration_check
 ```
 
+## Weighted microdata
+
+| Rule | Statement | Tier | Status | Blocking | Enforced by | CI | Proof |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| `api.microdata.unweighted-series-are-labels` | A MicroSeries built without weights= is a label: it only groups or filters weighted series and is never reduced itself. | mechanical | enforced | yes | pytest `tests/unit/services/test_microdata_weight_boundaries.py` | `pr.yml:test` | yes |
+| `api.microdata.weight-stripping-confined` | Raw values are read from weighted microdata (np.array, np.asarray, .values, .to_numpy(), .tolist(), pd.DataFrame or pd.Series) only in a designated stripping module, and the Public API has none yet. | mechanical | enforced | yes | pytest `tests/unit/services/test_microdata_weight_boundaries.py` | `pr.yml:test` | yes |
+| `api.microdata.weights-not-aggregated-as-values` | A weight column is never aggregated as a value: never the value of a weighted MicroSeries, the variable of a weighted aggregator, or frame[<weight>] followed by a weighted reduction. | mechanical | enforced | yes | pytest `tests/unit/services/test_microdata_weight_boundaries.py` | `pr.yml:test` | yes |
+| `api.microdata.weights-not-values` | Microdata columns are weights, values, labels or counts: weights are only summed plainly to count entities or passed as weights=, values are aggregated with their own entity's weights, labels only group or filter, and counts derive from weights. | mechanical | doc-only | no | nothing | — | — |
+
+```mermaid
+flowchart LR
+  r_api_microdata_unweighted_series_are_labels("api.microdata.unweighted-series-are-labels (enforced)")
+  e_pytest_tests_unit_services_test_microdata_weight_boundaries_py["pytest tests/unit/services/test_microdata_weight_boundaries.py"]
+  j_CI_pr_yml_test["CI pr.yml:test"]
+  r_api_microdata_weight_stripping_confined("api.microdata.weight-stripping-confined (enforced)")
+  r_api_microdata_weights_not_aggregated_as_values("api.microdata.weights-not-aggregated-as-values (enforced)")
+  r_api_microdata_weights_not_values("api.microdata.weights-not-values (doc-only)")
+  gap_unenforced>"not enforced"]
+  r_api_microdata_unweighted_series_are_labels --> e_pytest_tests_unit_services_test_microdata_weight_boundaries_py
+  e_pytest_tests_unit_services_test_microdata_weight_boundaries_py --> j_CI_pr_yml_test
+  r_api_microdata_weight_stripping_confined --> e_pytest_tests_unit_services_test_microdata_weight_boundaries_py
+  r_api_microdata_weights_not_aggregated_as_values --> e_pytest_tests_unit_services_test_microdata_weight_boundaries_py
+  r_api_microdata_weights_not_values -.-> gap_unenforced
+```
+
 ## Code quality and types
 
 | Rule | Statement | Tier | Status | Blocking | Enforced by | CI | Proof |

rules/fixtures/api.microdata.unweighted-series-are-labels/bad_decile.py

@@ -0,0 +1,20 @@
+"""Proof fixture: unweighted label series reduced. The rule must reject it."""
+
+from __future__ import annotations
+
+from microdf import MicroSeries
+
+ADULT_MIN_AGE = 18
+
+
+def average_decile(baseline: dict) -> float:
+    """Average the decile numbers themselves, as if they were incomes."""
+    decile = MicroSeries(baseline["household_income_decile"])
+    return float(decile.mean())
+
+
+def children(baseline: dict) -> float:
+    """Count the sample's child records, ignoring person weights."""
+    age = MicroSeries(baseline["age"])
+    is_child = age < ADULT_MIN_AGE
+    return float(is_child.sum())

rules/fixtures/api.microdata.unweighted-series-are-labels/good_decile.py

@@ -0,0 +1,28 @@
+"""Proof fixture: label series used only to group and filter.
+
+The rule must accept it.
+"""
+
+from __future__ import annotations
+
+from microdf import MicroSeries
+
+ADULT_MIN_AGE = 18
+
+
+def average_income_by_decile(baseline: dict) -> dict:
+    """Average household net income in each income decile, weighted by household."""
+    income = MicroSeries(
+        baseline["household_net_income"], weights=baseline["household_weight"]
+    )
+    decile = MicroSeries(baseline["household_income_decile"])
+    return income.groupby(decile).mean().to_dict()
+
+
+def child_poverty_rate(baseline: dict) -> float:
+    """Share of children in poverty, weighted by person."""
+    in_poverty = MicroSeries(
+        baseline["person_in_poverty"], weights=baseline["person_weight"]
+    )
+    age = MicroSeries(baseline["age"])
+    return float(in_poverty[age < ADULT_MIN_AGE].mean())

rules/fixtures/api.microdata.weight-stripping-confined/bad_net_income.py

@@ -0,0 +1,23 @@
+"""Proof fixture: weights stripped before a total. The rule must reject it.
+
+Summing the raw values of a weighted column gives an unweighted sample total,
+the mirror image of #767.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+from microdf import MicroSeries
+
+
+def baseline_net_income(baseline: dict) -> float:
+    """Total household net income over the sample's records, ignoring weights."""
+    return float(np.array(baseline["household_net_income"]).sum())
+
+
+def reform_net_income(reform: dict) -> float:
+    """Total household net income after dropping the weights it was given."""
+    income = MicroSeries(
+        reform["household_net_income"], weights=reform["household_weight"]
+    )
+    return float(income.values.sum())

rules/fixtures/api.microdata.weight-stripping-confined/good_net_income.py

@@ -0,0 +1,24 @@
+"""Proof fixture: totals that keep their weights. The rule must accept it."""
+
+from __future__ import annotations
+
+from microdf import MicroSeries
+
+
+def baseline_net_income(baseline: dict) -> float:
+    """Total household net income, weighted by household."""
+    income = MicroSeries(
+        baseline["household_net_income"], weights=baseline["household_weight"]
+    )
+    return float(income.sum())
+
+
+def reform_net_income(baseline: dict, reform: dict) -> float:
+    """Total reform household net income, weighted by the baseline's households."""
+    baseline_income = MicroSeries(
+        baseline["household_net_income"], weights=baseline["household_weight"]
+    )
+    income = MicroSeries(
+        reform["household_net_income"], weights=baseline_income.weights
+    )
+    return float(income.sum())

rules/fixtures/api.microdata.weights-not-aggregated-as-values/bad_budget.py

@@ -0,0 +1,37 @@
+"""Proof fixture: a weight aggregated as a value. The rule must reject it.
+
+``build_budgetary_impact`` holds the Simulation API line that #767 fixed. The
+household frame is a MicroDataFrame weighted by ``household_weight``, so
+summing ``household_weight`` through it returns the sum of squared weights.
+``budgetary_impact`` makes the same mistake with a hand-built MicroSeries.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from microdf import MicroSeries
+from policyengine_simulation_executor.simulation_macro_output import BudgetaryImpact
+from policyengine_simulation_executor.simulation_output_common import (
+    _sum_output_variable,
+)
+
+if TYPE_CHECKING:
+    from policyengine.core import Simulation
+
+
+def build_budgetary_impact(baseline: Simulation) -> BudgetaryImpact:
+    """Count households as the Simulation API did before #767."""
+    return BudgetaryImpact(
+        households=_sum_output_variable(
+            baseline, "household_weight", entity="household"
+        ),
+    )
+
+
+def budgetary_impact(baseline: dict) -> dict:
+    """Count households by weighting the household weights by themselves."""
+    households = MicroSeries(
+        baseline["household_weight"], weights=baseline["household_weight"]
+    )
+    return {"households": households.sum()}

rules/fixtures/api.microdata.weights-not-aggregated-as-values/good_budget.py

@@ -0,0 +1,32 @@
+"""Proof fixture: entities counted with a plain sum of their weights.
+
+The rule must accept it. ``build_budgetary_impact`` holds #767's replacement
+line in the Simulation API, and ``budgetary_impact`` counts households with a
+plain sum of ``household_weight``.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from policyengine_simulation_executor.simulation_macro_output import BudgetaryImpact
+from policyengine_simulation_executor.simulation_output_common import (
+    _count_output_entities,
+)
+
+if TYPE_CHECKING:
+    from policyengine.core import Simulation
+
+
+def build_budgetary_impact(baseline: Simulation) -> BudgetaryImpact:
+    """Count households as #767 does: a plain sum of the household weights."""
+    return BudgetaryImpact(
+        households=_count_output_entities(
+            baseline, entity="household", weight="household_weight"
+        ),
+    )
+
+
+def budgetary_impact(baseline: dict) -> dict:
+    """Count households with a plain sum of their weights."""
+    return {"households": sum(baseline["household_weight"])}