Remove segment_index from the spike vector(s) #4816

I am reviewing #4809 (optimize SpikeRetriever and RAM usage for extensions) and I want to bring back the topic of simplifying the spike vector (time ordered and unit ordered) data model. A lot of the memory cost on @alejoe91's benchmark is the spike vector itself and of that 1/3 is the segment index and the segment index is unnecessary. This is an edge case (most recordings are not multi-segment) and the field is only a convenience, syntactic sugar. These are the main places where we use it:

1. To find the boundaries of a segment (the sorting already caches them as segment_slices, see to_spike_vector(return_slices=True)):
   

   spikeinterface/src/spikeinterface/core/waveform_tools.py

   Line 357 in 93c612f

   s0, s1 = np.searchsorted(spikes["segment_index"], [segment_index, segment_index + 1])

   
   The same np.searchsorted is repeated three more times in waveform_tools.py and in basesorting.py, zarrextractors.py, sorting_tools.py and remove_excess_spikes.py.

2. To mask the spikes of one unit in one segment (this could be a slice of the segment plus a mask on unit_index):
   

   spikeinterface/src/spikeinterface/core/analyzer_extension_core.py

   Lines 131 to 133 in 93c612f

   	spike_indices_in_seg = np.flatnonzero(
   	(spikes["segment_index"] == segment_index) & (spikes["unit_index"] == unit_index)
   	)

3. As a sort key when building a sorting from peaks (the slices could be computed once at construction instead):
   

   spikeinterface/src/spikeinterface/core/numpyextractors.py

   Line 498 in 93c612f

   order = np.lexsort((spikes["sample_index"], spikes["segment_index"]))

As a side effect this would remove the branching for the mono-segment case, like the special cases that #4809 adds to SpikeRetriever, PeakRetriever and _get_spike_vector_segment_slices.

The cost is a big refactor: using the slices wherever we rely on np.searchsorted (which I also think would be more efficient) and a compatibility path for the sortings that were saved with the field.

What do you think? Is there something I am missing?