Repository navigation
OA clones number identifiers past the 32-bit range policyengine-uk stores #558
Description
What happens
clone_and_assign (policyengine_uk_data/calibration/clone_and_assign.py) gives clone k identifiers id + k * id_multiplier, where id_multiplier = 10 ** len(str(max_id)) over household_id, person_id and benunit_id. policyengine-uk stores all five identifier variables as 32-bit integers (person_id, benunit_id and household_id already; person_household_id and person_benunit_id from PolicyEngine/policyengine-uk#2273), which hold whole numbers up to 2,147,483,647.
For the current release, FRS 2024-25, the largest identifier is person_id 130,163,015 (read from the cached enhanced_frs_2024_25.h5, which the release builds with one clone, so it keeps the pre-clone identifiers). That makes id_multiplier 10^9:
| clone k | largest person_id |
|---|---|
| 1 | 1,130,163,015 |
| 2 | 2,130,163,015 |
| 3 | 3,130,163,015 (over 2,147,483,647) |
| 9 | 9,130,163,015 |
create_datasets.py uses PE_UK_DATA_OA_CLONES, default 10 (2 with TESTING=1). push.yaml and pull_request.yaml set it to 1, so published files and PR builds are unaffected today. A local build at the default, or any future release with more than three clones, simulates identifiers past the range. create_constituency_target_matrix and create_target_matrix build a Microsimulation from the cloned dataset.
Effect in policyengine-uk
The entity structure is built from the identifier arrays as given, so memberships stay right. The stored identifier values wrap: sim.calculate("household_id"), to_input_dataframe() and UKSingleYearDataset.from_simulation return different numbers, and anything rebuilt from them may not match. After PolicyEngine/policyengine-uk#2273 the simulation gives an EntityIdWarning naming the columns. Once identifiers stay in range, that warning could become an error.
Suggested fix
Size the offset so that (n_clones - 1) * id_multiplier + max_id stays at or below 2,147,483,647, or raise in clone_and_assign when it would not. A smaller offset needs care: clone_and_assign derives each benefit unit's household as benunit_id // 100, and person_id = household_id * 1000 + n and benunit_id = household_id * 100 + n are the FRS layout.
Found while fixing PolicyEngine/policyengine-uk#2267.
🤖 Generated with Claude Code
Activity