Store person_household_id and person_benunit_id as integers

[MaxGhenis]

Fixes #2267

What changes

• person_household_id and person_benunit_id are declared value_type = int, like person_id, benunit_id and household_id. policyengine-core then stores them as int32, which holds every whole number from -2,147,483,648 to 2,147,483,647, instead of float32, which is exact only up to 2**24 (16,777,216).
• Simulation.build_from_ids checks the five identifier columns against the dtype of the variable that stores each one (policyengine_uk/utils/entity_ids.py). An identifier outside the int32 range, or a float that is not a whole number, gives an EntityIdWarning, because core's Holder._to_array casts it with astype and would store a different number without an error. Object arrays of Python integers and floats (a pandas column mixing them) are converted and checked too; other objects, such as text or Decimal values, are left to core's own conversion.
• A note in docs/book/usage/simulations.md on the identifier range, and a changelog fragment.

The issue's reproduction on main (a375230) and on this branch:

	main	this branch
calculate("person_household_id")	[900000000.0, 900000000.0, 900000000.0] (float32)	[900000001, 900000002, 900000003] (int32)
to_input_dataframe(), person_household_id__2025	float32, all 900000000.0	int32, exact
UKSingleYearDataset.from_simulation(...).person	float32, all 900000000.0	int32, exact
Microsimulation(dataset=<policyengine-core h5 file>), people per household	[3, 0, 0]	[1, 1, 1]

The last row is a structural failure on main that the issue did not list. build_from_dataset(Dataset) builds the entities from the file, exports them with UKSingleYearDataset.from_simulation, and rebuilds from that export. The export carried the rounded membership column, so the three one-person households became one household of three and two empty households. That held for float64 and int64 identifiers in the file.

Why int, and what I checked first

The issue offered two options: declare both variables int, or keep float and give callers an exact source. int makes every existing reader exact without new API. The other three identifier variables already use it. The checks the brief asked for:

• Datasets that hold these columns as floats. Core's Holder._to_array casts whatever it is given to the variable's dtype, so a float64 or float32 column of whole numbers loads as exact int32. The property test builds every case from float64 tables as well as int64, and a separate test loads float32 columns. The one cached file that stores identifiers as floats, the policyengine-core layout enhanced_frs_2022_23.h5, holds float64 identifiers that are all whole and at most 45,010,024. As float32, 106,804 of its person_id values and 5,184 of its person_benunit_id values change.

• Whether any identifier exceeds 231 - 1.** I read the identifier columns of every cached UK dataset file (read-only, h5py and HDFStore.select on the ID columns; no simulation). None does. The largest is person_id 967,010,009 in populace_uk_2023.h5. That file is also the issue's example: its 535,080 household identifiers are 76,551 distinct float32 values, and its 618,980 benefit unit identifiers are 544,894. enhanced_frs_2024_25.h5 (fetched today) reaches 130,163,015 (person_id) and 130,170 (household_id).

• Build paths that can exceed it, and why out-of-range identifiers warn instead of raising. uk-data's clone_and_assign offsets clone k by k × 10^(digits of the largest identifier). For the current release (FRS 2024-25, largest identifier 130,163,015) that is 10^9, so a local build at the default of 10 clones would reach about 9.1 billion. The release and PR workflows set PE_UK_DATA_OA_CLONES=1, so published files keep the original identifiers. Microcosm's UK build sizes its clone offsets the same way (id_multiplier_for_values; the only identifier range checks I found in its UK build are against int64 overflow) on identifiers that already reach 967 million. Its policyengine-uk adapter passes frames to UKSingleYearDataset without renumbering them. In such builds person_id, benunit_id and household_id already wrap on main without an error, while the populations, built from the given arrays, stay exact. An error here could break those builds. A warning keeps them running and names the columns. Follow-up issues are listed below.

• Metadata API and downstream readers. policyengine-api's build_variables reports valueType from value_type.__name__, so both variables now report int instead of float (and core's json_type is integer instead of number). Its household calculation serializes an int result with .tolist() instead of _serialize_float. Both variables have default value 0 either way. Readers of these columns that now get exact integers:

  • policyengine.py's UK model joins the tables on them (_year_data_with_weights, _match_records).
  • policyengine-uk-data's derive_all_claimants_over_state_pension_age callers group by sim.calculate("person_benunit_id") (datasets/frs.py, datasets/imputations/frs_only.py). FRS benefit unit identifiers are below 2**24 today, so they were exact there already.
  • policyengine-app-v2 uses the two names only as household-JSON link keys.

  No formula under policyengine_uk/variables reads either variable (git grep).

• Round trips. Simulation.build_from_dataset(Dataset) and UKSingleYearDataset.from_simulation now round-trip identifiers up to 2**31 - 1 exactly. Tests below.

• No core change needed. int32 covers every dataset I found. A 64-bit integer store is policyengine-core#467, which is open and covers all integer variables.

Invariants

For identifiers that are whole numbers from -231 to 231 - 1, held as int64 or float64 in the tables, with each benefit unit and household table sorted by identifier and every group having a member (the loaders already require both: an unsorted group table gives each group another group's inputs on main, #2278, and core's join assigns people to groups by rank; this PR changes neither):

1. calculate, to_input_dataframe() and UKSingleYearDataset.from_simulation return all five identifier columns exactly, as integers, in every year of the dataset.
2. Differential: the stored memberships equal the populations' memberships. person_household_id equals populations["household"].ids[members_entity_id], and likewise for benefit units.
3. The entity structure (ids, members_entity_id) is the one the tables describe.
4. A simulation built from a policyengine-core dataset file, which build_from_dataset rebuilds from its own export, has that structure and those identifiers.
5. For integer and float arrays, and object arrays of Python floats and of integers within 64 bits, check_entity_ids warns exactly when core's cast to the storing dtype would change a value or fail. It was checked against astype itself over int64, int32, uint32, uint64, float64, float32 and such object arrays, including NaN, ±inf, fractions and values at each range edge. Other object arrays (text, Decimal, larger integers, nested arrays) are left to core. Building a simulation from in-range identifiers gives no EntityIdWarning. The test module turns that warning into an error, so every in-range test also checks that.
6. Out-of-range identifiers warn for exactly the columns that hold them. Given in UK dataset tables or a DataFrame, they still give the structure the tables describe. A policyengine-core dataset file is not covered: build_from_dataset rebuilds it from the stored identifiers, which have changed and lost their order, so group inputs can land on other groups there. It still warns, on the first build.

Tests

New: policyengine_uk/tests/test_entity_ids.py (pytest + Hypothesis, 30 tests). The property test draws up to four households of up to three benefit units of up to three people. It draws identifiers from the whole int32 range and clusters around 2**24, 900 million and both int32 limits, and shuffles the people.

Run locally, one process per file, -p no:cacheprovider, a fresh --basetemp deleted afterwards. The other three files ran at 608a859; the last commit changes only object-array handling and the new test file.

File	Result	Peak RSS
test_entity_ids.py	30 passed (at bef1bae)	0.46 GB
test_input_records.py	1 passed	0.35 GB
test_simulation_dataset_sources.py	10 passed	0.35 GB
test_date_of_birth.py	10 passed	0.99 GB

test_subsample.py is not on main (it comes with #2265). Mutation checks (at 608a859, 29 tests): with value_type = float restored, 20 fail. With the identifier check disabled, 16 fail (the warning and check tests). ruff format --check . and ruff check . pass on the whole repo.

Not run locally: the full suite and the docs build. CI runs both.

Interaction with #2265

#2265's test_identifiers_beyond_float32_precision_keep_their_households asserts a precondition, len(set(stored person_household_id)) < len(households): the stored values must not tell its 900-million households apart. This PR makes them exact, so that assertion fails once both PRs are merged. On a local merge of the two branches (this head, 608a859, with #2265's fa7b0f0), that line is the only failure in test_subsample.py and test_entity_ids.py (66 passed, 1 failed). The test's own checks, that subsample keeps whole households, all pass. Whichever PR lands second should change that line to assert the stored values are exact, and update the sentence in policyengine_uk/utils/subsample.py's module docstring that says the model stores those columns as 32-bit floats.

Review

Independent review (Subfleet --task review --tier hard, served by Claude Opus 5.5 with read-only tools) at 608a859: APPROVE, no blocking findings. Its findings and what changed:

• Object arrays skipped the check and core truncates their fractions: now converted and checked (2d465a9).
• Invariant 6 overclaimed for policyengine-core dataset files, whose rebuild reads the changed identifiers: narrowed, and that test now checks only the warning.
• Invariants 2 and 3 assume sorted group tables: stated above; the older bug is Unsorted benefit unit or household tables give each group another group's inputs #2278.
• stacklevel points inside the library: left, since skip_file_prefixes needs Python 3.12 and the package supports 3.11.
• Only the first year's identifier columns are checked: left. extend_single_year_dataset copies them unchanged (its upraters write no identifier column), and the structure comes from the first year. A hand-built UKMultiYearDataset whose later years use different identifiers is not checked.

A second review at 2d465a9 (Subfleet --task review --tier hard, served by GPT-6.1 Sol on a Codex lane): APPROVE, no blocking findings. Changed in bef1bae: nested object cells no longer make the check raise; the docstring and invariant 5 name exactly what is checked; the preconditions above add "every group has a member"; the later-years point above is corrected; the property tests suppress Hypothesis's too_slow health check, which the reviewer's slow sandbox tripped while drawing one input. Not changed: a false warning the object conversion could give for an int64 target (all five identifiers are int32).

Follow-ups

• Keep clone identifiers within int32, after which the warning here can become an error: OA clones number identifiers past the 32-bit range policyengine-uk stores policyengine-uk-data#558 (clone_and_assign) and UK clone stages can number identifiers past the 32-bit range policyengine-uk stores microcosm#1194 (the UK clone stages).
• Unsorted benefit unit or household tables give each group another group's inputs #2278: unsorted benefit unit or household tables give each group another group's inputs (found in review; older than this PR).
• Simulation(dataset=DataFrame) fails when a household or benefit unit has more than one person #2266 (the DataFrame loader) is unchanged. The new frame test uses one-person households for that reason.

axiom: n/a: simulation infrastructure, no policy rule change

🤖 Generated with Claude Code

Reproducer: policyengine_uk/tests/test_entity_ids.py::test_identifier_variables_are_stored_as_integers, policyengine_uk/tests/test_entity_ids.py::test_identifiers_in_range_read_back_exactly, policyengine_uk/tests/test_entity_ids.py::test_core_dataset_file_builds_the_tables_structure, policyengine_uk/tests/test_entity_ids.py::test_issue_reproduction_reads_back_in_every_year, policyengine_uk/tests/test_entity_ids.py::test_issue_reproduction_keeps_three_households, policyengine_uk/tests/test_entity_ids.py::test_core_dataset_file_keeps_households_apart