Join each person to the group their ID names

[MaxGhenis]

What this fixes

SimulationBuilder.join_with_persons can put a person in the wrong group. It ranked the distinct group IDs persons belong to (np.unique(..., return_inverse=True)) and used that rank to index the declared IDs in sorted order (np.argsort(group_population.ids)). A group with no members has no rank, so when one sorts before an occupied group, every person after it moves down one group. The audit's witness:

groups ["a", "b"]; one person in "b" with income 100
members_entity_id  [0]        # expected [1]
household sums     [100, 0]   # expected [0, 100]
projection of [10, 20] gives 10   # expected 20

Total income is conserved; it is just in the wrong household. Simulation.build_from_dataset calls this for every group entity. Empty groups are otherwise supported: test_bincount_minlength.py covers an empty group whose ID sorts after every occupied ID, a case where the old ranking happened to work.

The fix

• join_with_persons looks each person's group ID up among all declared IDs, using a new group_positions(group_ids, persons_group_ids, entity_key). An empty group can sit anywhere in the declared order.
• IDs match only when they are equal as values. IDs of one kind (both integer, both float, both string) use a stable argsort plus searchsorted, which compares exactly within a kind. IDs of different kinds (int64 with float64, int64 with uint64) or object arrays are matched one by one through a dict. NumPy would otherwise convert them to float64, where neighbouring large IDs such as 253 + 3 and 253 + 4 become equal.
• It fails loudly instead of guessing:
  • a person whose ID names no declared group raises ValueError ("N person(s) belong to household IDs that were not declared: [...]");
  • so does a declared ID that repeats. A repeated ID made one of its groups permanently empty.
• Flat files (Dataset.FLAT_FILE, which is what Simulation.subsample rebuilds from) relied on the old ranking. They declared groups 0..n-1 but passed the raw membership IDs. build_from_dataset now numbers the distinct membership IDs itself (np.unique(..., return_inverse=True)), which gives the same mapping as before.
• A flat file with an unsuffixed household_id column used to skip that path and leave the previous entity's IDs declared: three persons in two households made three households. It now takes the same path and counts two.
• In a flat file, the membership comes from the person's person_household_id column or, failing that, the household_id column repeated on each member's row. A frame with neither raises the same KeyError as master. A frame with missing (NaN) memberships now raises, instead of np.unique merging everyone without one into a single household.
• Flat-file group values follow membership, not row order. The loader took a group variable's column as already per household whenever its length equalled the household count. In a flat file every column is per person. When the group count was correct, that equality held exactly when every household had one person, and then values went in by row order while households are numbered in ID order. With household_id [9, 7, 8], each person read another person's rent: [800, 900, 700] for [900, 700, 800]. This was already on master for frames with both ID columns. Group variables in a flat file now take each occupied household's first member's value; empty frames pass their empty columns through.
• Default roles (no role column) are sized by the number of persons. This repaired historical master 78a6c503, where the loader allocated one role per group; current master 42556d5d already uses self.persons.count. This round preserves that current-master allocation and its recognition of suffixed entity-role columns such as person_household_role__eternity. Explicit role values and the configured default-role name retain their current-master interpretation. The earlier Validate implicit dataset role capacities and role array shapes #595 overlap note described a historical proposal and is superseded by the current merge-target source: role loading and default allocation.

This review round

This round replaces the successful-zero-selection assertion with rejection checks for nonzero source weight and zero retained weight in both quantization modes, after a plain merge of current master 42556d5dff8cde292b657cd1426b5e1351e4c5c8. Direct empty-frame loading remains covered. The merge preserves person-sized default roles and suffixed entity-role recognition, with a regression for explicit suffixed roles. New Hypothesis properties check explicit-role masks, sums, and projections, and verify that zero-selection rejection preserves the dataset, population counts, memberships, and recorded inputs. The body now distinguishes historical behavior from the current merge target; this round adds no core policy choice.

Current review-round validation

Current head: 41e0a15e8bce502b867fcd074eda493ebe181920.

The exact archived current-master source reproduced the reviewer’s P2 rejection and P3 role results. On assigned head cf2eedaee5a05d31ba94a720ae59e61d13b5a812, the new deterministic regressions produced 3 failed, 21 passed: suffixed explicit roles were ignored, and neither quantization mode rejected the positive-weight zero-selection scenario. After the merge, the revised deterministic file has 23 passed, including both rejection modes, both explicit-role column forms, and both direct-empty-frame forms. The obsolete successful-zero-selection assertion was replaced rather than retained. All six properties pass, with 1,000 passing generated examples plus explicit cases.

Files ran directly in the foreground, one at a time, with PYTHONDONTWRITEBYTECODE=1 UV_CACHE_DIR=../hub-evidence/uv-cache uv run --no-sync pytest -q <file>. The property run also used --hypothesis-show-statistics and a workspace-local HYPOTHESIS_STORAGE_DIRECTORY. The local environment was installed from the frozen runtime lock, with Hypothesis 6.168.5 and Ruff 0.17.0 added for validation.

File under tests/core/	Result
test_join_with_persons_identity.py	23 passed
test_join_with_persons_identity_property.py	6 passed; 1,000 passing generated examples plus explicit cases
test_simulation_builder.py	35 passed
test_bincount_minlength.py	2 passed
test_subsample_entity_weight_totals.py	28 passed
test_subsample_membership_ids.py	8 passed

Current total: 102 passed. Ruff format passed for all four PR Python files (unchanged); ruff check . and git diff --check gh/master HEAD passed. Existing warnings concern IPython syntax, dpath.util deprecation in builder error paths, and experimental memory configuration in disk-input tests. No folder runs, full suite, or microsimulations were run. Neither uv.lock nor generated artifacts have a PR diff against current master.

Documentation review: the public join docstring states ID identity, declared order, empty assignments, and rejection rules; no standalone page or generated artifact needs a local update. The existing Towncrier fragment now describes only changes relative to current master. Impact is medium for the caller-visible loader/join semantics; confidence is high for the focused cases. Sampling, default-role allocation, and suffixed role recognition have no production diff against current master.

Preceding review round (historical head cf2eedaee5)

The preceding review reproduced an empty-frame regression introduced by unconditional first-member extraction. Its repair bypassed first-member extraction when the flat frame had no persons, preserving direct empty datasets. That round also tested successful zero-selection subsampling under the older sampling implementation; after the current-master merge, the nonzero-weight scenario must raise instead. Hypothesis checks from that round cover empty/non-empty loader values and person-sized default-role masks. The public join method documents ID identity and rejection rules, and the changelog fragment is one concise sentence.

Source anchors for this repair: zero-group loader guard and public join contract.

Preceding review-round validation (historical head cf2eedaee5)

The results below are preserved evidence from the preceding round, not validation of 41e0a15e8bce502b867fcd074eda493ebe181920. Before the loader repair, the regression file on reviewed head 0d2ba5af182cc80cc10cc2c6629542e9788c24ea plus the new tests had 3 failed, 17 passed: both empty-frame variants and the zero-selection subsample raised ValueError: zero-size array to reduction operation maximum which has no identity. The property file had 1 failed, 3 passed, with the explicit empty-frame example exposing the same exception.

The historical P3 witness executed an AST-extracted build_from_dataset from historical master 78a6c503ca6623f2735410412b8ca9e93816ec0d, using the preceding branch elsewhere. Both joins are identical for this fully occupied one-household case: memberships [0, 0]. The historical master loader returned has_role = [True], projection [10, 10], and an IndexError for the role-filtered sum; the branch loader returned [True, True], [10, 10], and [3.0], respectively. This isolates the historical loader's role allocation rather than claiming a full master simulation run. Current master 42556d5d already returns the branch results, and this round preserves them.

After that preceding repair, these files ran directly in the foreground, one at a time, using uv run --frozen --with hypothesis --with ruff pytest -q <file> (with a workspace-local UV_CACHE_DIR; the property run also used --hypothesis-show-statistics):

File under tests/core/	Result
test_join_with_persons_identity.py	20 passed
test_join_with_persons_identity_property.py	4 passed; 800 passing generated examples plus explicit empty/role cases
test_simulation_builder.py	35 passed
test_bincount_minlength.py	2 passed

Historical total: 61 passed. That preceding implementation's successful zero-selection subsample emitted a zero-weight division warning; this is no longer the expected contract after merging current master. Builder error-path tests retained existing dpath.util deprecation warnings. Ruff format passed for all four changed Python files; ruff check . and git diff --check passed. Neither uv.lock nor generated artifacts changed.

Preceding documentation review: the public join docstring documents declared order, value equality, empty assignments, and rejection rules; no standalone documentation page or generated artifact needed changing. The existing Towncrier fragment was shortened to one sentence. Impact: medium for the PR's caller-visible semantics; confidence: high for the focused cases. That round ran no country microsimulations, folder-wide tests, or full test suite, as requested by the hub.

Invariants

For any distinct group IDs (integers or strings), declared in any order, with any set of empty groups anywhere, and any assignment of persons to occupied groups:

• each person's members_entity_id points at the group their ID names;
• sum and nb_persons, in declared order, equal a plain-Python dict oracle;
• project gives each person their own group's value.

An undeclared or repeated ID raises. Numeric IDs match by value across int32, int64, uint64 and float64, including values near 231, 253 and 2**62.

The additional Hypothesis properties from the preceding round check:

• for flat frames with suffixed or unsuffixed IDs, explicit memberships or group-ID fallback, and zero or more persons, group values equal the first row's value for each sorted membership ID and project to the corresponding members; an empty frame has zero persons, zero groups, and empty group values and projections;
• when roles default to MEMBER, the role array and has_role mask have one entry per person, including zero persons; role-filtered sums equal independent membership counts and role-filtered projection returns each person's group value.

This round adds properties for the current-master invariants preserved by the merge:

• explicit entity-role columns retain their roles with or without period suffixes; per-person role masks, role-filtered sums, and role-filtered projections match independent role/membership oracles;
• when subsampling rounds to zero selected households but an input-weight source total is positive, both quantization modes raise before replacing the dataset or populations; dataset identity, population counts, membership arrays, and recorded input arrays remain unchanged. Direct empty-frame loading remains a separate invariant.

Earlier author test evidence

The results and historical red/green runs in this section were reported by the earlier PR author against the historical source snapshots then available; they are preserved as prior evidence and were not rerun as a group in this review round. They do not establish current-master behavior. Current results are listed separately in the current review-round validation section.

• New test_join_with_persons_identity.py (13 tests):
  • the audit witness, and an interior empty group;
  • an ARRAYS dataset whose middle household is empty, checked through calculate(..., map_to="person");
  • the flat-file paths: a frame with only a household_id column, group values for one-person households in non-ascending ID order, missing memberships, and default roles covering every person;
  • default roles covering every person in an ARRAYS dataset;
  • IDs of different dtypes;
  • the two errors and the no-person case.
• New test_join_with_persons_identity_property.py: two Hypothesis properties, 300 examples each. One checks the identity invariants against a dict oracle; the other checks numeric IDs across dtype pairs.

Historical before (the earlier author's master snapshot of simulations/; group_positions stubbed with that snapshot's ranking so the module imports; an exact master hash was not recorded in this subsection):

test_join_with_persons_identity.py:           6 failed, 2 passed
  assert [0] == [1]; assert [2, 1, 1] == [0, 1, 1]; dataset: assert [0, 1] == [0, 2]; flat: assert 3 == 2
test_join_with_persons_identity_property.py:  1 failed   (minimal: ids [0, -1], one person in 0, where the empty -1 sorts first)

Round 2 (after the independent review): the new tests on the first head d647f78d:

test_join_with_persons_identity.py:           4 failed, 9 passed
  int64 vs float64 near 2**53: assert [0] == [1]
  int64 vs uint64: ValueError ... not declared: [9007199254740993]
  only household_id: KeyError 'person_household_id';  default roles: assert 1 == 2
test_join_with_persons_identity_property.py:  1 failed, 1 passed   (int64 groups, uint64 memberships near 2**62)

Round 3 (after the second review), the new tests on c3a6459c:

test_join_with_persons_identity.py:  3 failed, 14 passed
  rent by person: assert [800.0, 900.0, 700.0] == [900.0, 700.0, 800.0]   (both frame kinds)
  missing memberships: Regex pattern did not match (no error raised)

After:

test_join_with_persons_identity.py           17 passed
test_join_with_persons_identity_property.py   2 passed
test_simulation_builder.py                   35 passed
test_bincount_minlength.py                    2 passed
test_subsample_baseline_branch.py             2 passed
test_subsample_invalidates_cache.py           2 passed
test_simulations.py                           9 passed
data/test_dataset.py                          3 passed
test_dataset_getattr.py                       2 passed
test_dump_restore.py                          1 passed

The earlier author reported that Ruff format and Ruff check passed, that the listed files ran one at a time, and that the full suite was left to CI because the shared host was saturated.

Where empty groups can come from

The country-loader observations below are retained prior-author context and were not re-audited in this round. The core sampling statement describes the current merge target.

• Simulation.subsample: a rebuilt flat frame has no declared group without members, because groups are numbered from selected persons' membership IDs. Sampling can select zero households when int(household_count * frac) == 0, but current master rejects that selection if a nonzero input-weight total cannot be preserved. This PR preserves that rejection, independently of its direct empty-frame loading guard: rounding, weight guard.
• Household situations (build_from_entities, the household API path): do not call join_with_persons. Their memberships come from add_group_entity. The prior author reported that a situation with households h1 (empty) and h2 (one person) already gave members [1] and sums [0, 100] on the historical master they tested. This PR leaves that builder path unchanged.
• policyengine.py (US model): declares np.unique of the person membership column and first checks that it equals the entity table's IDs (_build_simulation_from_dataset), so no empty groups.
• policyengine-us: declares the dataset's own *_id arrays, so a dataset with an unoccupied ID would hit the bug.
• policyengine-uk (build_from_ids): declares np.unique of the benunit and household tables' IDs, so a dataset with an unoccupied ID would hit the bug.

Prior author evidence on default datasets

The earlier author compared a historical master snapshot's mapping with this PR's on every group entity of the default datasets, using group_positions lifted from the branch and the stored ID arrays. These dataset comparisons have not been rerun in this review round and are not a comparison against current master 42556d5d:

Dataset	Entities	Empty groups	Undeclared memberships	Repeated IDs	Mapping
populace-us-2024-spm-20260909	household, tax_unit, spm_unit, family, marital_unit	0	0	0	identical
enhanced_cps_2024 (policyengine-us DEFAULT_DATASET)	same five	0	0	0	identical
UK enhanced_frs_2023_24 @1.40.3 (UK test default)	benunit, household	0	0	0	identical
UK enhanced_frs_2023_24 and 2024_25 @1.57.4	benunit, household	0	0	0	identical

The earlier author reported identical membership arrays; the flat-file changes apply only to flat files, which none of these datasets is. Identical membership indices support unchanged group association, but do not establish that every API result is unchanged. The default-role mask changed relative to historical master 78a6c503; current master already has person-sized masks, which this round preserves. The earlier author did not run a microsim A/B, and this review round does not run microsimulations.

For flat files (Dataset.from_dataframe, and Simulation.subsample, which rebuilds from one):

• frames whose group count was already correct retain the same normalized membership numbering;
• when the old group count was correct, group-value order changes for one-person groups whose membership IDs do not ascend in row order; the broken unsuffixed group-ID path can additionally change group counts and values for multi-person groups;
• missing-membership checks raise where current master could form groups from missing membership values.

Historical default-role comparison against master 78a6c503: dataset loading supplied one role per group, whereas the branch supplied one per person. For an ARRAYS dataset with two persons assigned to one household and no role column, historical master returned [True] for person.has_role(Household.MEMBER) and raised IndexError for household.sum(np.array([1.0, 2.0]), role=Household.MEMBER); the branch returned [True, True] and [3.0]. Arithmetic: 1 + 2 = 3. Projection of [10] with role=Household.MEMBER already returned [10, 10] historically through broadcasting. Current master 42556d5d already returns [True, True], [3.0], and [10, 10]; this round preserves those results, explicit role interpretation, configured default-role names, and recognition of suffixed entity-role columns.

Behaviour change to note: a caller that declared repeated group IDs, or memberships naming undeclared IDs, used to get a silent mapping and now gets a ValueError. None of the datasets above do either.

Caller-visible core semantics

The comparison here is against the current merge target, master at 42556d5dff8cde292b657cd1426b5e1351e4c5c8. Earlier evidence against 78a6c503ca6623f2735410412b8ca9e93816ec0d is historical and is labeled separately.

• Joining uses each membership's declared group ID, in declared group order. For declared IDs ["a", "b"] and the sole membership ["b"], members_entity_id changes from [0] to [1], sums of person income [100] change from [100, 0] to [0, 100], and projection of group values [10, 20] changes from [10] to [20]. Arithmetic: the sole income 100 belongs to declared position 1. Numeric IDs compare by exact value across dtypes without lossy conversion of large integers. Sources: current-master ranking, declared-order sum, and PR value-identity contract.
• Repeated declared group IDs and memberships naming undeclared groups raise ValueError where current master could silently assign persons. Missing flat-file memberships also raise ValueError where current master could form groups from missing values. These are observable new validation errors; persons are neither dropped nor imputed. Sources: current-master join, PR errors, and missing-membership guard.
• Flat files prefer an explicit person-membership column and otherwise use the group-ID column repeated on person rows. Distinct membership IDs remain sorted and normalized to 0..n-1; group count now comes from that selected source even when an unsuffixed group-ID column is present. The group-ID fallback makes previously broken frames load correctly; a frame missing both sources still raises KeyError. Sources: current-master flat-file loader and PR membership-source selection.
• Every flat-file group-variable column is interpreted as person rows and reduced to each group's first row. With one person per household, IDs [9, 7, 8] normalize to memberships [2, 0, 1]; rents [900, 700, 800] become group values [700, 800, 900] and project back to [900, 700, 800], replacing current master's [800, 900, 700]. Arithmetic: normalized groups 0, 1, 2 name raw IDs 7, 8, 9, whose rents are 700, 800, 900. With multiple persons per group, first-member selection is already current master's convention. Direct empty flat frames still load as zero persons, zero groups, and empty values. Sources: current-master length shortcut and first-member extraction and PR first-member extraction and empty guard.
• The merge preserves current master's person-sized default roles and recognition of suffixed entity-role columns such as person_household_role__eternity; explicit role values and the configured default-role name retain current-master interpretation. For an ARRAYS dataset with two persons in one household and no role column, both current master and this PR return has_role(MEMBER) == [True, True], role-filtered sum [3] for incomes [1, 2], and role-filtered projection [10, 10] for group value [10]. Arithmetic: 1 + 2 = 3. The older [True] mask and failing sum belonged to historical master 78a6c503, whose projection already returned [10, 10] through broadcasting. Sources: current-master role-column recognition/default allocation, historical allocation, role mask, and projection broadcasting.
• The merge preserves current master's sampling-weight contract. For two households with weights [1, 1], subsample(frac=0.1) selects int(2 * 0.1) = 0 households; source total 1 + 1 = 2 cannot be preserved by retained total 0, so both current master and this PR raise ValueError before replacing the dataset or rebuilding populations. This is separate from loading a directly supplied empty frame. Sources: current-master sample rounding, weight-total rejection, and later rebuild.

Person-variable columns pass through unchanged. Non-flat datasets retain declared group IDs and order. Valid memberships whose old ranking already matched their declared IDs keep their association and resulting sums/projections. Situations built through build_from_entities retain their existing path.

The governing sources are immutable engine contracts cited above; no tax statute, regulation, form, or instruction applies to this data-loading repair.

Methodology

These engine-contract choices introduce no new choice this round.

• ID identity: use exact numeric value equality across dtypes; alternatives are dtype-sensitive identity or common-dtype coercion. Relative to current master, correct mappings remain unchanged; empty-group misassignments are corrected. Lossy coercion was an earlier PR defect. Identity contract.
• Invalid memberships: reject duplicate declared IDs, undeclared memberships, and missing flat-file memberships; alternatives are dropping or imputing persons. Relative to current master, previously accepted invalid data raises. Validation contract, missing-value guard.
• Flat files: prefer explicit memberships and first-member group values; alternatives are rejecting conflicts or selecting later/aggregated values. Relative to current master, supported membership precedence and multi-person first-member behavior remain; group-ID fallback, unsuffixed group counts, and one-person value ordering are repaired. Loader contract.

Default roles, suffixed entity-role recognition, and sampling-weight rejection preserve current-master behavior. No tax-policy source applies.

axiom: n/a: engine fix in dataset loading, no tax-policy change

Other audit findings (historical context)

The 2026-10-06 invariants audit of historical master 757147c7 also reported the findings below. This list preserves the earlier author's ownership/status notes and is not a statement of their current status. In particular, #571 and #573 are now included in current master and in this round's plain merge; this PR adds no changes to those mechanisms.

• Stock ADD changing later plain reads: previously assigned to Cache ADD and DIVIDE results only where a plain read returns them #571; now merged in current master.
• Year deletion leaving monthly fast-cache entries: draft Drop every contained period from the fast cache when deleting or setting a period #566.
• Disk deletion of contained periods, and disk branch-name parsing and prefix collisions: draft Delete the periods within a deleted period from disk storage #565 (it inherits Keep branch period reads, dumps, and disk deletion consistent #552).
• Clone aliases bound to the parent: fixed by Bind a clone's calc and df to the clone, and give it its own system's entities #586.
• Serialization and subsample input provenance: previously assigned to Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576 and Rebuild a subsampled simulation from its inputs, not its calculated values #573; Rebuild a subsampled simulation from its inputs, not its calculated values #573 is now merged in current master.
• Restored entity counts: Fix restore_simulation entity counts #548.
• A monthly input leaving a cached annual sum stale (set_input for one period leaves the variable's cached sum or twelfth over an overlapping period in place #579): tracked separately.

The earlier author classified branch staleness after set_input and raw-file restore as documented behaviour. That historical classification is retained as context, not re-established by this review round.

🤖 Generated with Claude Code