Map FRS EMPSTATI 11 to OTHER_INACTIVE and refuse unknown adult codes (uk-data#526 parity)

changelog.d/uk-frs-empstati-other-inactive.fixed.md

@@ -0,0 +1 @@
+The UK `frs_employment` stage maps FRS EMPSTATI code 11 ("Other Inactive" in the UKDS FRS 2024-25 data dictionary) to `employment_status` OTHER_INACTIVE instead of LONG_TERM_DISABLED, matching the incumbent's uk-data#526, so other-inactive adults no longer enter the ESA health-condition and support-group proxies. Codes 1-11 are mapped one data-dictionary label each; people outside `adult.tab` stay CHILD, and an adult with a blank or unknown code now refuses the build instead of defaulting to LONG_TERM_DISABLED.

experiments/uk-frs-empstati-other-inactive/README.md

@@ -0,0 +1,112 @@
+# FRS EMPSTATI 11 → OTHER_INACTIVE: receipts
+
+The UK `frs_employment` stage now maps EMPSTATI codes 1-11 one data-dictionary
+label each (UKDS SN 9563, FRS 2024-25 adult table), so code 11 ("Other
+Inactive") is `OTHER_INACTIVE` rather than `LONG_TERM_DISABLED`. This is the
+same mapping as the incumbent's uk-data#526 (head `1832adeb`; its mapping and accept/refuse logic are unchanged since `d3984002`, and only the refusal message has changed). Everything below
+was run on 2026-10-02 against the pinned `adult.tab`
+(sha256 `4eaea080…658d`). Only aggregates are reported, and every cell here
+covers at least 127 survey households, so none falls under the 10-household
+suppression rule.
+
+## The licensed input
+
+Every one of the 27,714 adults in `adult.tab` carries a code from 1 to 11. No
+code is blank, non-integer or outside that range, and no `person_id` repeats.
+So the new refusal of unknown adult codes does not stop the real build.
+`child.tab` has no EMPSTATI column.
+
+| EMPSTATI | Data-dictionary label | Status | Adults | Grossed (GROSS4) |
+|---:|---|---|---:|---:|
+| 1 | Full-time Employee | FT_EMPLOYED | 10,297 | 22.51m |
+| 2 | Part-time Employee | PT_EMPLOYED | 2,701 | 5.57m |
+| 3 | Full-time Self-Employed | FT_SELF_EMPLOYED | 1,446 | 2.95m |
+| 4 | Part-time Self-Employed | PT_SELF_EMPLOYED | 664 | 1.25m |
+| 5 | Unemployed | UNEMPLOYED | 484 | 1.31m |
+| 6 | Retired | RETIRED | 8,585 | 12.04m |
+| 7 | Student | STUDENT | 383 | 1.34m |
+| 8 | Looking after family/home | CARER | 429 | 1.00m |
+| 9 | Permanently sick/disabled | LONG_TERM_DISABLED | 1,678 | 3.24m |
+| 10 | Temporarily sick/injured | SHORT_TERM_DISABLED | 131 | 0.29m |
+| 11 | Other Inactive | OTHER_INACTIVE (was LONG_TERM_DISABLED) | 916 | 2.05m |
+
+Before this change, `LONG_TERM_DISABLED` held 2,594 adults (codes 9 and 11);
+it now holds the 1,678 with code 9. These are the same counts uk-data#526
+reports for its FRS 2024-25 build.
+
+## Differential against uk-data#526
+
+`differential_vs_uk_data_526.py` loads uk-data's code table and
+`derive_employment_status_from_frs` from the PR head (`1832adeb`) by AST and
+compares them with microcosm's:
+
+```
+code tables identical: 11 codes
+fixed cases agree: 34
+hypothesis cases agree: 2000 examples
+refusal exception types (pandas 3.0.3): {'microcosm': ['ValueError'], 'uk-data': ['ValueError']}
+licensed FRS 2024-25 people: 34966; mismatches: 0
+```
+
+Agreement means the same statuses, or a refusal on both sides. Each side's
+refusal exception type is recorded, and the script asserts both are
+`ValueError`. The earlier run at #526's head `fb026659` found that
+uk-data's message formatted the codes with `sorted(set(codes.astype(str)))`.
+Under pandas 3, `astype(str)` keeps NaN as a float, so a blank adult code
+next to another unknown code raised `TypeError` instead (`ValueError` under
+uk-data's locked pandas 2.3.3). That was reported to the #526 owner and fixed
+in `89c48e07`.
+
+The fixed cases are codes 0-12, -1, 11.5, NaN and 99, each as an adult and as
+a child row. The licensed comparison runs microcosm's stage derivation
+(`derive_frs_employment`, reading `adult.tab` through the sha-pinned reader)
+against uk-data's function fed the way `create_frs` feeds it (child rows filled
+with 0, adult records by `adult.tab` membership), over all 34,966 people.
+
+## What reads `employment_status`
+
+- `frs_legacy_proxies` reads the frame's `employment_status`.
+  `ESA_HEALTH_EMPLOYMENT_STATUSES` is (`LONG_TERM_DISABLED`,
+  `SHORT_TERM_DISABLED`), and the support group needs `LONG_TERM_DISABLED`.
+  Code-11 adults therefore leave both ESA proxies. `legacy_jobseeker_proxy`
+  reads `UNEMPLOYED` only and does not change.
+- In policyengine-uk 2.100.0 (microcosm's lock), no variable formula reads
+  `employment_status`. The labour-supply dynamics read the self-employed and
+  student statuses only, and the three proxies are not engine variables.
+- The instruments that see the column are blind to this change:
+  - The eFRS parity reference (uk-data 1.56.16) records `employment_status` as
+    an unweighted non-empty-string share (`_nonzero_share` in
+    `tools/build_uk_efrs_parity_reference.py`).
+  - The release input-coverage gate counts a row as signal when it differs
+    from the engine default, `UNEMPLOYED` (`_nondefault_signal_mask`).
+  - The input-mass parity skips string columns entirely
+    (`microcosm/build/input_mass.py`). The ESA proxies are not engine
+    variables, so the reference totals leave them out as well.
+- PR CI never compares these categories or proxies with a released uk-data
+  H5. The engine-uk parity-reference tests load only the committed extraction.
+- Until uk-data ships #526, microcosm's code-11 adults and ESA proxies differ
+  from the pinned incumbent, and no committed instrument measures that
+  difference.
+
+## Mutation check
+
+`mutation_check.py` applies each mutation to a temporary copy of
+microcosm-build's sources, placed first on `PYTHONPATH`, and runs the three
+test files (engine-free employment and legacy-proxy tests, and the engine-uk
+enum test). Every mutation must fail them:
+
+```
+baseline (unmutated copy): rc=0
+killed: 11 back to LONG_TERM_DISABLED (rc=1)
+killed: unknown adult codes default to LONG_TERM_DISABLED (rc=1)
+killed: every row treated as an adult (rc=1)
+killed: adult records by code presence, not adult.tab membership (rc=1)
+killed: 9 and 10 swapped (rc=1)
+killed: adult count printed (rc=1)
+killed: children get a non-CHILD status (rc=1)
+killed: code 0 accepted for adults (rc=1)
+killed: non-integer codes truncated (rc=1)
+killed: support group ignores hours worked (rc=1)
+killed: OTHER_INACTIVE added to the ESA health statuses (rc=1)
+11/11 mutations killed
+```

experiments/uk-frs-empstati-other-inactive/differential_vs_uk_data_526.py

@@ -0,0 +1,186 @@
+"""Differential check: microcosm's EMPSTATI mapping against policyengine-uk-data#526.
+
+policyengine-uk-data is not a microcosm dependency and #526 is unreleased, so
+this runs once, by hand, rather than in CI. It loads uk-data's
+``FRS_EMPSTATI_EMPLOYMENT_STATUS`` and ``derive_employment_status_from_frs``
+from a checkout of the PR head (by AST, so the rest of ``frs.py`` and its
+imports are never executed), then checks the two implementations agree:
+
+1. on every code 0-11 and a set of unknown codes, as adult and child rows;
+2. on Hypothesis-generated mixes of adult and child rows, raising together;
+3. on the licensed FRS 2024-25 person set (``adult.tab`` plus ``child.tab``,
+   read through microcosm's sha-pinned reader), element by element.
+
+Only aggregates are printed: per-status counts of people, with cells covering
+fewer than 10 survey households suppressed, and the mismatch count, with 1-9
+suppressed. Usage (UK engine environment)::
+
+    uv run --no-sync python \
+        experiments/uk-frs-empstati-other-inactive/differential_vs_uk_data_526.py \
+        --uk-data-checkout <policyengine-uk-data at #526's head> \
+        --frs-dir <licensed FRS 2024-25 tab directory>
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import math
+import subprocess
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+from hypothesis import given, settings
+from hypothesis import strategies as st
+from policyengine_uk.variables.household.income.employment_status import (
+    EmploymentStatus,
+)
+
+from microcosm.build.country_spec import load_country_spec
+from microcosm.build.uk_runtime.frs_employment import (
+    FRS_EMPSTATI_EMPLOYMENT_STATUS,
+    derive_employment_status_from_frs,
+    derive_frs_employment,
+)
+from microcosm.build.uk_runtime.frs_spine import normalize_ids, read_pinned_tab
+
+UK_DATA_526_HEAD = "1832adeb2296b53fe5a7d7194e770118aa34d1f8"
+UK_DATA_NAMES = ("FRS_EMPSTATI_EMPLOYMENT_STATUS", "derive_employment_status_from_frs")
+
+
+def load_uk_data_mapping(checkout: Path):
+    head = subprocess.run(
+        ["git", "-C", str(checkout), "rev-parse", "HEAD"],
+        check=True,
+        capture_output=True,
+        text=True,
+    ).stdout.strip()
+    if head != UK_DATA_526_HEAD:
+        raise SystemExit(f"{checkout} is at {head}, not #526's {UK_DATA_526_HEAD}.")
+    source = (checkout / "policyengine_uk_data/datasets/frs.py").read_text()
+    tree = ast.parse(source)
+    wanted = [
+        node
+        for node in tree.body
+        if (
+            isinstance(node, ast.Assign)
+            and any(getattr(t, "id", None) in UK_DATA_NAMES for t in node.targets)
+        )
+        or (isinstance(node, ast.FunctionDef) and node.name in UK_DATA_NAMES)
+    ]
+    if len(wanted) != len(UK_DATA_NAMES):
+        raise SystemExit(f"Expected {UK_DATA_NAMES} in uk-data frs.py.")
+    namespace = {"EmploymentStatus": EmploymentStatus, "np": np, "pd": pd}
+    exec(compile(ast.Module(body=wanted, type_ignores=[]), "frs.py", "exec"), namespace)
+    return namespace[UK_DATA_NAMES[0]], namespace[UK_DATA_NAMES[1]]
+
+
+REFUSAL_TYPES: dict[str, set[str]] = {"microcosm": set(), "uk-data": set()}
+
+
+def outcome(function, codes, is_adult, *, side):
+    """Statuses, or "refused" for any exception (its type is recorded).
+
+    Either side refusing stops the build. The refusal's exception type is
+    recorded and checked separately, because message formatting has depended
+    on the pandas version before.
+    """
+    try:
+        return ("ok", list(function(codes, is_adult)))
+    except Exception as error:  # noqa: BLE001 - every exception refuses a build
+        REFUSAL_TYPES[side].add(type(error).__name__)
+        return ("refused", None)
+
+
+def suppressed(statuses: np.ndarray, household_ids: np.ndarray) -> dict[str, object]:
+    """People per status, shown only for cells covering 10 or more households."""
+    cells = pd.DataFrame({"status": statuses, "household": household_ids})
+    grouped = cells.groupby("status")
+    people = grouped.size()
+    households = grouped["household"].nunique()
+    return {
+        str(status): (int(people[status]) if households[status] >= 10 else "suppressed")
+        for status in people.index
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--uk-data-checkout", type=Path, required=True)
+    parser.add_argument("--frs-dir", type=Path, required=True)
+    args = parser.parse_args()
+    uk_data_table, uk_data_derive = load_uk_data_mapping(args.uk_data_checkout)
+
+    assert dict(FRS_EMPSTATI_EMPLOYMENT_STATUS) == uk_data_table
+    print("code tables identical: 11 codes")
+
+    fixed = [*range(0, 13), -1, 11.5, math.nan, 99]
+    for code in fixed:
+        for is_adult in (True, False):
+            ours = outcome(
+                derive_employment_status_from_frs, [code], [is_adult], side="microcosm"
+            )
+            theirs = outcome(uk_data_derive, [code], [is_adult], side="uk-data")
+            assert ours == theirs, (code, is_adult, ours, theirs)
+    print(f"fixed cases agree: {len(fixed) * 2}")
+
+    code_values = st.one_of(
+        st.integers(-5, 15), st.just(math.nan), st.floats(0.5, 11.5)
+    )
+    rows = st.lists(st.tuples(st.booleans(), code_values), max_size=40)
+
+    @settings(max_examples=2_000, deadline=None)
+    @given(rows)
+    def agree(sample):
+        codes = [code for _, code in sample]
+        is_adult = [adult for adult, _ in sample]
+        assert outcome(
+            derive_employment_status_from_frs, codes, is_adult, side="microcosm"
+        ) == outcome(uk_data_derive, codes, is_adult, side="uk-data")
+
+    agree()
+    print("hypothesis cases agree: 2000 examples")
+    print(
+        f"refusal exception types (pandas {pd.__version__}):",
+        {side: sorted(types) for side, types in REFUSAL_TYPES.items()},
+    )
+    # Both sides refuse with ValueError at the pinned head, under pandas 2 or 3
+    # (fb026659's message raised TypeError under pandas 3; 89c48e07 fixed it,
+    # 5f9912df changed only tests, and 1832adeb dropped the count from it).
+    assert REFUSAL_TYPES["microcosm"] <= {"ValueError"}
+    assert REFUSAL_TYPES["uk-data"] <= {"ValueError"}
+
+    stages = {stage.stage: stage for stage in load_country_spec("uk").sources.stages}
+    employment = {a["table"]: a for a in stages["frs_employment"].artifacts}
+    spine = {a["table"]: a for a in stages["frs_spine"].artifacts}
+    adult = normalize_ids(
+        read_pinned_tab(args.frs_dir / "adult.tab", employment["adult"])
+    )
+    child = normalize_ids(
+        read_pinned_tab(
+            args.frs_dir / "child.tab", spine["child"], columns=("sernum", "person")
+        )
+    )
+    person = pd.DataFrame(
+        {"person_id": np.concatenate([adult["person_id"], child["person_id"]])}
+    )
+    ours = derive_frs_employment(person, adult)["employment_status"].to_numpy()
+    # uk-data's person table fills the child table's absent EMPSTATI with 0.
+    uk_data_codes = person["person_id"].map(adult.set_index("person_id")["empstati"])
+    theirs = uk_data_derive(
+        uk_data_codes.fillna(0).to_numpy(),
+        person["person_id"].isin(adult["person_id"]).to_numpy(),
+    )
+    mismatches = int((ours != theirs).sum())
+    shown = mismatches if mismatches == 0 or mismatches >= 10 else "1-9 (suppressed)"
+    print(f"licensed FRS 2024-25 people: {len(person)}; mismatches: {shown}")
+    print(
+        "employment_status people (unweighted; cells under 10 households suppressed):",
+        suppressed(ours, (person["person_id"] // 1000).to_numpy()),
+    )
+    assert mismatches == 0
+
+
+if __name__ == "__main__":
+    main()