Skip to content

Reduce and rank over a population with no persons - #590

Open
MaxGhenis wants to merge 3 commits into
masterfrom
fix-empty-group-reductions
Open

MaxGhenis wants to merge 3 commits into
masterfrom
fix-empty-group-reductions

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

What this fixes

With no persons in a simulation, GroupPopulation.all, max and min raise:

ValueError: zero-size array to reduction operation maximum which has no identity

reduce fills each group with its neutral element, but it first took numpy.max of the persons' positions to size its loop, and that array is empty. members_position (numpy.max(members_entity_id)) and Population.get_rank (numpy.max(positions)) have the same problem. All three now size their loop as 0 when there are no persons, so it runs zero times. They use a length check rather than numpy.max(..., initial=-1), which would raise for an unsigned index array. get_rank reshapes its value matrix to (count, 0) so it stays two-dimensional. sum and nb_persons already returned 0 for this case.

The independent review found that an empty membership given as [], or built by build_default_simulation(count=0) (np.array(range(0))), is float64. NumPy indexes only with integers, so ranks and projections over no persons still raised. The members_entity_id setter now stores an empty membership as an integer array, and build_default_simulation uses np.arange(count), which is integer even for 0.

The audit verifier's witness was one household and no persons. With this PR, all returns [True], any [False], max [-inf], min [inf], sum [0], nb_persons [0], and get_rank returns an empty array.

Invariants

For any number of groups (1 to 6) and any assignment of 0 to 30 persons to them (so groups may be empty and there may be no persons at all), given as an integer array or a plain list:

  • sum, nb_persons, any, all, max and min equal a plain-Python oracle, with 0, False, True, -inf and inf for a group with no members;
  • get_rank gives each person the position of their value among their group's values, ties going to the person listed first.

Tests

  • New test_group_population_empty.py: the witness for every reduction, get_rank with no persons, a plain [] membership, and a default simulation of no one (rank, nth person, project, projector max, all).
  • New test_group_population_reductions_property.py: one Hypothesis property, 300 examples.

Before the fix, on master's policyengine_core/populations:

test_group_population_empty.py:               2 failed   (ValueError: zero-size array to reduction operation maximum which has no identity)
test_group_population_reductions_property.py: 1 failed   (minimal: one group, no persons)

Round 2 (after the independent review), the new tests on the first head eca6a621:

test_group_population_empty.py:                 2 failed, 2 passed (plain [] and count=0 cases)
test_group_population_reductions_property.py:   1 failed   (minimal: one group, no persons, given as a list)

After:

test_group_population_empty.py:                 4 passed
test_group_population_reductions_property.py:   1 passed
test_simulation_builder.py:                    35 passed
test_entities.py:                              23 passed
test_dump_restore.py:                           1 passed
test_bincount_minlength.py:                     2 passed
test_projectors.py:                             6 passed

Ruff format and ruff check pass. I ran these files one at a time and left the full suite to CI, because the shared host is saturated.

Impact

No country output can change. For any non-empty population the length check takes the same numpy.max(x) + 1 as before, and the reshape is a no-op when the matrix already has rows. Only populations with zero persons behave differently, and those used to raise. np.arange(count) gives the same integer values as np.array(range(count)) for every positive count. No microsimulation runs with zero persons, so no A/B run is needed.

axiom: n/a: engine fix in population reductions, no policy change

Other audit findings, handled elsewhere

The 2026-10-06 invariants audit of master 757147c7 also confirmed defects that this PR leaves alone, because other work owns them:

Branch staleness after set_input and raw-file restore are documented behaviour, not defects.

🤖 Generated with Claude Code

MaxGhenis and others added 3 commits October 6, 2026 12:59
GroupPopulation.reduce (behind all, max and min), members_position and
Population.get_rank each took numpy.max of the persons' positions or group
indices to size a loop. With no persons that array is empty, so they raised
"zero-size array to reduction operation maximum which has no identity"
instead of returning the neutral element reduce already fills each group
with. They now start the maximum at -1, so the loop runs zero times; get_rank
keeps its value matrix two-dimensional with no columns.

Adds the audit witness (one household, no persons) and a Hypothesis
property: for 1 to 6 groups and 0 to 30 persons, sum, nb_persons, any, all,
max, min and get_rank equal a plain-Python oracle, empty groups included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
numpy.max(x, initial=-1) raises OverflowError for an unsigned index
array, which master handled for any non-empty population. A length check
gives 0 for no persons and leaves every other case exactly as on master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses the independent review of eca6a62 (one should-fix finding).

An empty list or numpy.array([]) is float64, and so were the IDs and
memberships build_default_simulation(count=0) made (np.array(range(0))).
NumPy indexes only with integers or booleans, so get_rank, project,
value_nth_person and projector reductions over no persons still raised
IndexError or TypeError. The members_entity_id setter now stores an empty
membership as an integer array, and build_default_simulation uses
np.arange(count), which is integer even for 0.

Adds regressions for a plain [] membership and a default simulation of no
one, and lets the Hypothesis property pass memberships as plain lists.
All fail on eca6a62 (the property shrinks to one group, no persons,
given as a list).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant