[apdavison]

Problem

NWBIO fails on most NWB files written by other software, including the file in #1413, where read() raises ValueError: Invalid annotation … StrDataset. I sampled nine small files from the DANDI Archive, covering ecephys, icephys and ophys; eight of them fail on master. Lazy loading fails even for files that read eagerly. Writing renames the Neo objects in place and drops signal and SpikeTrain descriptions (part of #1516).

The generic BaseTestIO checks also never run for NWBIO, because its mode is "one-file", which the test framework doesn't recognise.

Solution

NWBIO now reads these files. NWB objects it doesn't yet support (processing modules, ImageSeries, extension types, data with more than two dimensions) are skipped with a log message instead of raising an error. Values that aren't valid Neo annotations, such as links to other NWB objects, are skipped. String datasets are converted to str.

Behaviour changes that may affect existing code:

• Scaling: offset and channel_conversion are applied when reading, and a conversion that isn't a power of ten is applied to the data. These three fields no longer appear as nwb: annotations.
• Event notes: JSON in an AnnotationSeries description goes into the Event's notes annotation, as it already does for signals, instead of being merged into the top-level annotations.
• Names: writing no longer changes the names of the objects being written. Where a name has to be made unique in the file, the original is restored on reading.
• comments: the comments of a foreign TimeSeries are available as nwb:comments and are kept when writing.
• nwb: annotations on write: annotations the target NWB class doesn't accept are dropped with a warning.
• Fallback to TimeSeries: a signal that can't be written as its original class is written as a plain TimeSeries. One example is an ElectricalSeries, because writing electrodes isn't supported yet.
• Descriptions: signal and SpikeTrain descriptions are written and read back.
• Units without obs_intervals: t_start and t_stop come from the range of all spike times in the table. A unit with several observation intervals keeps them in nwb:obs_intervals.
• Nothing readable: a file with no readable data objects gives a Block that carries only the file metadata, instead of raising IndexError.

Fixes #1413, partly addresses #1516, and covers the obs_intervals item of #1013 and the keywords part of #1717. It supersedes the draft #1502.

Testing

• NWBIO.mode is now "file", so the BaseTestIO compliance checks run. They run against the NeuroConv and tutorial NWB files already in ephy_testing_data.
• New tests build small NWB files with pynwb in a temporary directory. There is one per fix, and they run with NEO_TESTS_NO_NETWORK.
• I checked by hand that the nine DANDI files I sampled can be read (lazy and normal) and written back without errors. I didn't add the files to the test suite. How do people feel about adding DANDI alongside GIN as a source for test files?
• For Invalid annotation error when trying to read .nwb file #1413, the changes allow the reported 4.9 GB file to be read without errors. It gives 542 SpikeTrains and 4 IrregularlySampledSignals, the same counts reported with Handle more complex metadata in NWB files #1502.

Other comments

This PR makes foreign files readable but doesn't yet preserve their structure. Processing modules, extra table columns, electrode tables and icephys tables are still lost. I'm working on a follow-up PR for this.