Skip to content

Return an empty array from chained vector lookups that select no rows - #589

Open
MaxGhenis wants to merge 2 commits into
masterfrom
fix-empty-vector-parameter-lookup
Open

MaxGhenis wants to merge 2 commits into
masterfrom
fix-empty-vector-parameter-lookup

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

What this fixes

A chained vector parameter lookup that selects no rows raises once it reaches numeric leaves. The audit's witness is a tree with one leaf a.b.c = 0 and three empty key arrays:

node = ParameterNode(data={"a": {"b": {"c": {"2020-01-01": 0}}}})("2020-01-01")
none = np.asarray([], dtype=str)
node[none][none][none]
# IndexError: index 0 is out of bounds for size 0  (vectorial_parameter_node_at_instant.py:416)

After the empty selection every numeric child holds zero values. The "scalar per child" path, meant for 0-d and length-1 children, read the first value of each. That path now takes only 0-d and length-1 children. Length-0 children go to the existing N-element path, which stacks the children into a (K+1, N) matrix and returns a length-0 float array.

The independent review found two more cases in the same code, now fixed:

  • Unknown keys after an empty selection. A one-element key broadcasts over zero rows, so node[none][["missing"]] returned an empty array. The unknown-key check only fired when the result held NaN, so nothing raised. It now also fires when any key index is the missing-child sentinel, and the lookup raises ParameterNotFoundError as it does for rows that exist.
  • Nodes (structured levels) after an empty selection. Zero rows took the one-row path. A valid one-element key returned one row of NaN, and node[none][["b"]][["c"]] read [nan] (master too). Zero rows now give zero rows at nodes as at leaves, broadcasting the key the same way: no rows for a one-element key, and a shape-mismatch error for a key of two or more.

Invariants

For any parameter tree of depth 1 to 3 (1 to 3 children per node, numeric leaves) and any 0 to 20 paths through it:

  • Looking the paths up one level at a time with key arrays gives an array of shape (number of paths,). Each element equals the scalar lookup of its path. Zero paths give an empty array.
  • A key that names no child raises ParameterNotFoundError at any level, among valid keys for the selected rows, or alone when no row is selected.

Tests

  • test_fancy_indexing.py::test_empty_fancy_indexing_at_every_level (leaf, node and triple lookups on the existing fixture tree) and ::test_empty_fancy_indexing_on_a_single_path (the audit witness).
  • test_fancy_indexing.py::test_one_key_after_an_empty_selection: one-element keys after an empty selection, at a node and at a leaf.
  • New test_fancy_indexing_property.py: the two properties above, 300 examples each.

Before the fix, on master's policyengine_core/parameters:

test_fancy_indexing.py:            2 failed, 8 passed   (IndexError: index 0 is out of bounds for size 0)
test_fancy_indexing_property.py:   1 failed             (minimal: depth 2, one leaf, no paths)

Round 2 (after the independent review), the new tests on the first head f8426179:

test_fancy_indexing.py:            1 failed, 10 passed   (node[none][["b"]] had 1 row: assert 1 == 0)
test_fancy_indexing_property.py:   1 failed, 1 passed    (DID NOT RAISE ParameterNotFoundError)

After:

test_fancy_indexing.py:                   11 passed
test_fancy_indexing_property.py:           2 passed
test_parameters.py:                       14 passed
test_numpy2_structured_arrays.py:          4 passed
test_vectorial_parameter_node_copy.py:    64 passed

Ruff format and ruff check pass. I ran these files one at a time and left the full suite to CI, because the shared host is saturated.

Impact

No country output can change. The only inputs that take a different path are selections of zero rows, which used to raise or return NaN rows, and unknown keys, which now raise in one more case. Children with one value, which includes every first-level lookup, still take the scalar path. Children with two or more values still take the N-element path. A microsimulation never looks up parameters for zero rows, so no A/B run is needed.

axiom: n/a: engine fix in vector parameter lookup, no policy change

Other audit findings, handled elsewhere

The 2026-10-06 invariants audit of master 757147c7 also confirmed defects that this PR leaves alone, because other work owns them:

Branch staleness after set_input and raw-file restore are documented behaviour, not defects.

🤖 Generated with Claude Code

MaxGhenis and others added 2 commits October 6, 2026 12:57
A chained vector parameter lookup such as node[keys][keys][keys] with empty
key arrays raised IndexError once it reached numeric leaves: after the empty
selection each leaf held zero values, and the scalar-per-child path read the
first one. Leaves with one value still take that path; leaves with zero
values now take the N-element path, which yields an array of length zero.

Adds regressions at every level of the fixture tree and on the audit's
single-path witness, plus a Hypothesis property: for trees of depth 1 to 3
and 0 to 20 paths, the vector lookup returns one value per path, each equal
to the scalar lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses the independent review of f842617 (one should-fix finding),
and a second defect the extended property found.

- After a selection of no rows, a one-element unknown key broadcast over
  those rows and returned an empty array, so the NaN check that raises
  ParameterNotFoundError never fired. The check now also fires when any
  key index is the missing-child sentinel, whatever the result holds.
- At a node (structured level), zero rows took the one-row path: a valid
  one-element key returned one row of NaN, and node[none][["b"]][["c"]]
  read NaN where it should give no rows (on master too). Zero rows now
  give zero rows at nodes as at leaves, broadcasting the key the same way
  (a key of two or more over zero rows raises a shape mismatch).

Adds a regression for both, and a Hypothesis property: a key naming no
child raises ParameterNotFoundError at any level, among valid keys for the
selected rows or alone after selecting none. Both fail on f842617.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant