Skip to content

Fix indexing an empty DynamicTableRegion selection in memory - #1549

Merged
rly merged 5 commits into
devfrom
fix_data_access_hdmf
Jul 30, 2026
Merged

Fix indexing an empty DynamicTableRegion selection in memory#1549
rly merged 5 commits into
devfrom
fix_data_access_hdmf

Conversation

@h-mayorquin

Copy link
Copy Markdown
Contributor

Motivation

Selecting a zero-length region from a DynamicTableRegion raised IndexError: arrays used as indices must be of integer (or boolean) type when the target table's columns held their data as numpy arrays. This happens for an empty slice and, more commonly, for a ragged region row that references no target rows, which is the shape produced when rows are appended to a table whose DynamicTableRegion column has no targets for them. DynamicTableRegion.get takes np.unique of the selected row indices, and np.unique of an empty selection returns a float64 array, which numpy rejects as an index. A column whose data is a list takes a different branch in Data.get, indexing elementwise, and tolerates the float array, which is why the existing TestDynamicTableRegion suite, whose fixture is built from lists, stayed green.

The same selection works for files on disk, because the indices come back from HDF5 already integer-typed, so the container behaves differently in memory than it did after a round trip. I think it should work for both cases. I cast the unique row indices to an integer dtype rather than returning early on an empty selection, because region data are row indices: the cast is a no-op when the selection is non-empty, and it fixes the class of problem (a non-integer index array) rather than the one input that produces it.

I found this downstream in AllenNeuralDynamics/aind-ephys-pipeline#121, reported by @alejoe91 @CodyCBakerPhD, where units from a second probe are appended to an NWB units table that already has an electrodes region column. Those units are written with an empty region, the file writes and reads back correctly, and only inspecting the in-memory NWBFile before writing raised.

How to test the behavior?

import numpy as np
from hdmf.common import DynamicTable, VectorData

target = DynamicTable(
    name="target", description="t",
    columns=[VectorData(name="x", description="x", data=np.array([1, 2, 3]))],
)
table = DynamicTable(name="table", description="t")
table.add_column(name="region", description="r", index=True, table=target)
table.add_row(region=[0])
table.add_row(region=[])   # a row that references no target rows

table["region"][0]   # one-row DataFrame, before and after
table["region"][1]   # IndexError before, empty DataFrame after

Checklist

  • Did you update CHANGELOG.md with your changes?
  • Does the PR clearly describe the problem and the solution?
  • Have you reviewed our Contributing Guide?
  • Does the PR use "Fix #XXX" notation to tell GitHub to close the relevant issue numbered XXX when the PR is merged?

@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.22%. Comparing base (4f7fe44) to head (d4c91ff).

Additional details and impacted files
@@           Coverage Diff           @@
##              dev    #1549   +/-   ##
=======================================
  Coverage   93.22%   93.22%           
=======================================
  Files          41       41           
  Lines       10221    10221           
  Branches     2115     2115           
=======================================
  Hits         9529     9529           
  Misses        415      415           
  Partials      277      277           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rly and others added 4 commits July 29, 2026 23:52
VectorIndex.get inspects arg[0] to detect a boolean mask before the
len(indices) > 0 guard that handles the empty case, so an empty list or
array raised IndexError. This surfaces through
DynamicTableRegion.get when a region row references no target rows and
the target table has a ragged column.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Separate the paragraphs of the DynamicTableRegion.get comment block, and
state what the with_array_columns fixture is rather than contrasting it
with the list-backed fixture.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rly

rly commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Thanks @h-mayorquin ! I added an extra check to cover the case where the DynamicTableRegion points to a ragged column and this error was raised: IndexError: index 0 is out of bounds for axis 0 with size 0

@rly
rly merged commit 98f19e8 into dev Jul 30, 2026
28 checks passed
@rly
rly deleted the fix_data_access_hdmf branch July 30, 2026 04:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants