ms3_readtracelist_selection with MSF_PNAMERANGE + time selection silently drops overlapping records

[jschaeff]

Disclaimer: I used IA to help me make this bug report understandable

libmseed version: 3.5.4 (also verified on git master v3.5.4-12-g...)
Environment: Linux x86_64, gcc 12.2, glibc 2.36

---

Summary

ms3_readtracelist_selection returns zero matching trace segments when called with both a byte-range end offset (via MSF_PNAMERANGE) and a time-window selection, even though the byte range contains records whose data overlaps the requested time window. Removing either the end offset or the time selection yields the correct result.

---

Test file

https://s3.seisdata.epos-france.fr/mseed/FR/2014/MLS/HHN.D/FR.MLS.00.HHN.D.2014.046

Time index (from mseedindex):

File offset	Record start time (UTC)	Approx clock time
196608	2014-046T05:17:09.616900Z	05:17:09
274432	2014-046T05:59:46.836900Z	05:59:46
471040	2014-046T06:00:17.497000Z	06:00:17
1646592	2014-046T13:19:48.196200Z	13:19:48

Channel is 100 Hz. Between offsets 196608 and 274432 the file is not a single record — there are many miniSEED records packed contiguously within that byte range (the mseedindex entries are merely the first record at each indexed offset). The record boundaries can be enumerated by passing MSF_RECORDLIST.

Query window: 2014-046 05:18:30 – 05:19:00 (30 s, 3000 samples at 100 Hz).
This window falls entirely inside the byte range [196608, 274432), so MSF_PNAMERANGE should not truncate any needed data.

---

Reproducer (C, self-contained)

The code below downloads the test file once via curl (requires network), then runs four scenarios against the same local copy.

/*
 * gcc -o repro repro.c -I/path/to/libmseed -L/path/to/libmseed -lmseed
 * LD_LIBRARY_PATH=/path/to/libmseed ./repro
 */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "libmseed.h"

#define TEST_URL "https://s3.seisdata.epos-france.fr/mseed/FR/2014/MLS/HHN.D/FR.MLS.00.HHN.D.2014.046"
#define TEST_FILE "/tmp/libmseed_bug_test.mseed"

static int fetch_file(void) {
    FILE *fp = fopen(TEST_FILE, "rb");
    if (fp) { fclose(fp); return 0; }
    char cmd[512];
    snprintf(cmd, sizeof(cmd), "curl -s -o %s %s", TEST_FILE, TEST_URL);
    int rc = system(cmd);
    if (rc != 0) {
        fprintf(stderr, "Please download manually:\n  wget -O %s %s\n",
                TEST_FILE, TEST_URL);
    }
    return rc;
}

/* Return the number of trace segments found by ms3_readtracelist_selection,
 * or -1 on error.  rc_out receives the raw return code from the library. */
static int count_segments(const char *filepath,
                          int64_t t_start, int64_t t_end,
                          int *rc_out) {
    MS3TraceList *mstl = NULL;
    MS3Selections *sel = NULL;
    int nseg = 0;

    if (t_start || t_end)
        ms3_addselect(&sel, "FDSN:FR_MLS_00_H_H_N", t_start, t_end, 0);

    int rc = ms3_readtracelist_selection(
        &mstl, filepath, NULL, sel, 0,
        MSF_PNAMERANGE | MSF_RECORDLIST, 0);

    if (rc_out) *rc_out = rc;

    if (rc == MS_NOERROR && mstl && mstl->numtraceids > 0)
        nseg = mstl->traces.next[0]->numsegments;

    mstl3_free(&mstl, 1);
    ms3_freeselections(sel);
    return nseg;
}

int main(void) {
    if (fetch_file() != 0) return 1;

    /* Query window */
    int64_t t_lo = ms_timestr2hptime("2014,046,05,18,30.000000");
    int64_t t_hi = ms_timestr2hptime("2014,046,05,19,00.000000");

    int rc;

    /* ---- Test 1: baseline — no time filter, no end offset ---- */
    int s1 = count_segments(TEST_FILE "@196608", 0, 0, &rc);
    /* rc here is the raw return value from ms3_readtracelist_selection.
     * MS_NOERROR (0)  = success.
     * MS_ENDOFFILE(-1)= no more data (not an error).
     * Other values     = actual errors. */

    /* ---- Test 2: no time filter, WITH end offset ---- */
    int s2 = count_segments(TEST_FILE "@196608:274432", 0, 0, &rc);

    /* ---- Test 3: WITH time filter, no end offset (CONTROL) ---- */
    int s3 = count_segments(TEST_FILE "@196608", t_lo, t_hi, &rc);

    /* ---- Test 4: WITH time filter, WITH end offset (BUG TRIGGER) ---- */
    int s4 = count_segments(TEST_FILE "@196608:274432", t_lo, t_hi, &rc);

    printf("Test                                segments\n");
    printf("────────────────────────────────────────────────\n");
    printf("1. No time, no end offset           %d\n", s1);
    printf("2. No time, with end offset         %d\n", s2);
    printf("3. WITH time, no end offset (ctrl)  %d\n", s3);
    printf("4. WITH time, with end offset (BUG) %d  ← SHOULD BE >0\n", s4);

    /* Additional probes — wider end offsets still trigger the bug */
    int s5 = count_segments(TEST_FILE "@196608:1646592", t_lo, t_hi, &rc);
    int s6 = count_segments(TEST_FILE "@196608:4816896", t_lo, t_hi, &rc);
    printf("5. end=1646592 + time               %d\n", s5);
    printf("6. end=EOF    + time               %d\n", s6);

    remove(TEST_FILE);
    return (s3 > 0 && s4 == 0) ? 1 : 0;
}

Expected output

Test                                segments
────────────────────────────────────────────────
1. No time, no end offset           1
2. No time, with end offset         1
3. WITH time, no end offset (ctrl)  1
4. WITH time, with end offset (BUG) 1  ← SHOULD BE >0
5. end=1646592 + time               1
6. end=EOF    + time                1

Actual output (bug)

Test                                segments
────────────────────────────────────────────────
1. No time, no end offset           1
2. No time, with end offset         1
3. WITH time, no end offset (ctrl)  1
4. WITH time, with end offset (BUG) 0  ← SHOULD BE >0
5. end=1646592 + time               0
6. end=EOF    + time                1

Observations:

• Tests 4 and 5 return zero segments even though data exists within the time window and falls inside the byte range.
• Test 6 works (end=EOF), proving the time selection itself is correct — the bug only manifests when a finite end_offset is combined with a time window.

---

dataselect commands (same bug, external tool)

The dataselect binary (EarthScope/dataselect) uses the same ms3_readtracelist_selection code path:

### Download once
wget https://s3.seisdata.epos-france.fr/mseed/FR/2014/MLS/HHN.D/FR.MLS.00.HHN.D.2014.046

### [A] end offset, NO time filter → 2 segments (OK)
dataselect -Ps +o - FR.MLS.00.HHN.D.2014.046@196608:274432 | msi -tg -

### [B] end offset, WITH time filter → "No SEED data detected" (BUG)
dataselect -Ps \
  -ts 2014,046,05,18,30.000000 -te 2014,046,05,19,00.000000 \
  +o - FR.MLS.00.HHN.D.2014.046@196608:274432 | msi -tg -

### [C] SAME time filter, NO end offset → 1 segment, 3000 samples (OK)
dataselect -Ps \
  -ts 2014,046,05,18,30.000000 -te 2014,046,05,19,00.000000 \
  +o - FR.MLS.00.HHN.D.2014.046@196608 | msi -tg -

[B] is broken; [A] and [C] work.

---

Root-cause hypothesis

The issue lies in the interaction between MSF_PNAMERANGE and time-based filtering inside ms3_readtracelist_selection. I traced the logic as follows (libmseed 3.5.4 source in selection.c and tracelist.c):

1. Pass 1 — header scan: The function reads record headers sequentially, starting at start_offset, using ms3_readmsr. With MSF_PNAMERANGE it stops when the file position reaches end_offset.

2. Time matching: For each header, ms3_matchselection checks rec->starttime <= sel_end && rec->endtime >= sel_start. The record's endtime is computed as starttime + (samplecnt - 1) / samprate.

3. The bug trigger: When a finite end_offset is set and the record starting at start_offset has a starttime before the query window's starttime, the trace list builder inside ms3_readtracelist_selection appears to use a different code path that skips the record entirely instead of trimming it to the overlapping portion. This code path is apparently taken only when both conditions hold — neither condition alone triggers it.

   Specifically, in selection.c:ms_readtraceselection() the PNAMERANGE branch may terminate the record loop prematurely when the last header read before end_offset has a starttime < sel_start, even if that same record's data extends into the window.

4. Why test 6 works: With end_offset = EOF (no end bound) the PNAMERANGE branch is never taken; the reader follows the normal path which correctly handles overlapping records.

A fix would likely involve modifying the PNAMERANGE branch in ms_readtraceselection() (or the equivalent logic in ms3_readtracelist_selection()) to not discard a record until its endtime has been verified against the selection window, rather than checking only starttime.

---

Workaround

For callers that control the byte range: always set end_offset to at least the byte position of the next record boundary after the window of interest, or use end_offset = 0 (file size) when feasible. In the dataselect case, omitting the :end suffix from the path avoids the bug entirely.

---

[chad-earthscope]

Hi @jschaeff

Some observations:
a) Are you sure you're using "libmseed version: 3.5.4" ?
That version of the library does not include ms_timestr2hptime() which is used in the repro.c code so that might have been a different version of libmseed being used.

b) the byte range suffix syntax for a path name is @START-END using a dash (-) and not a colon (:) as in these examples. Edit: dataselect is an exception, see next comment

c) looking at the records in the test file between bytes 196608-274432 there is no data for the selected time range of this command:

dataselect version 4.3.2
❯ dataselect -Ps \
  -ts 2014,046,05,18,30.000000 -te 2014,046,05,19,00.000000 \
  +o - FR.MLS.00.HHN.D.2014.046@196608-274432 | msi -tg -
ERROR: -: No data records read, not SEED?
ERROR: Cannot read -: No miniSEED data detected

Here's the record details:

❯ msi -O FR.MLS.00.HHN.D.2014.046@196608-274432
196608        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3770 samples, 100 Hz, 2014-02-15T05:17:09.616900Z (046)
200704        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3770 samples, 100 Hz, 2014-02-15T05:17:47.316900Z (046)
204800        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3762 samples, 100 Hz, 2014-02-15T05:34:44.616900Z (046)
208896        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3706 samples, 100 Hz, 2014-02-15T05:35:22.236900Z (046)
212992        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3730 samples, 100 Hz, 2014-02-15T05:35:59.296900Z (046)
217088        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3756 samples, 100 Hz, 2014-02-15T05:36:36.596900Z (046)
221184        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3736 samples, 100 Hz, 2014-02-15T05:37:14.156900Z (046)
225280        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3768 samples, 100 Hz, 2014-02-15T05:52:19.616900Z (046)
229376        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3758 samples, 100 Hz, 2014-02-15T05:52:57.296900Z (046)
233472        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3744 samples, 100 Hz, 2014-02-15T05:53:34.876900Z (046)
237568        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3742 samples, 100 Hz, 2014-02-15T05:54:12.316900Z (046)
241664        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3764 samples, 100 Hz, 2014-02-15T05:54:49.736900Z (046)
245760        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3582 samples, 100 Hz, 2014-02-15T05:55:27.376900Z (046)
249856        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3634 samples, 100 Hz, 2014-02-15T05:56:03.196900Z (046)
253952        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3684 samples, 100 Hz, 2014-02-15T05:56:39.536900Z (046)
258048        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3758 samples, 100 Hz, 2014-02-15T05:57:16.376900Z (046)
262144        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3764 samples, 100 Hz, 2014-02-15T05:57:53.956900Z (046)
266240        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3756 samples, 100 Hz, 2014-02-15T05:58:31.596900Z (046)
270336        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3768 samples, 100 Hz, 2014-02-15T05:59:09.156900Z (046)
❯ msi -O FR.MLS.00.HHN.D.2014.046@196608-274432 -tg
       SourceID                      Start sample                End sample           Gap  Hz  Samples
FDSN:FR_MLS_00_H_H_N        2014-02-15T05:17:09.616900Z  2014-02-15T05:18:25.006900Z   ==  100 7540
FDSN:FR_MLS_00_H_H_N        2014-02-15T05:34:44.616900Z  2014-02-15T05:37:51.506900Z  979.6 100 18690
FDSN:FR_MLS_00_H_H_N        2014-02-15T05:52:19.616900Z  2014-02-15T05:59:46.826900Z  868.1 100 44722
Total: 1 trace(s) with 3 segment(s)

The time window of -ts 2014,046,05,18,30.000000 -te 2014,046,05,19,00.000000 is in a data gap within that byte range of the file, i.e. there is no data matching the selection for dataselect.

There is data for that time in the file but it exists at later byte offsets:

5263360       FDSN:FR_MLS_00_H_H_N, 4, 4096, 2600 samples, 100 Hz, 2014-02-15T04:27:35.610600Z (046)
5267456       FDSN:FR_MLS_00_H_H_N, 4, 4096, 1700 samples, 100 Hz, 2014-02-15T04:43:44.610600Z (046)
5271552       FDSN:FR_MLS_00_H_H_N, 4, 4096, 1700 samples, 100 Hz, 2014-02-15T04:59:44.610600Z (046)
5275648       FDSN:FR_MLS_00_H_H_N, 4, 4096, 3660 samples, 100 Hz, 2014-02-15T05:18:25.016900Z (046)
5279744       FDSN:FR_MLS_00_H_H_N, 4, 4096, 1010 samples, 100 Hz, 2014-02-15T05:37:51.516900Z (046)
5283840       FDSN:FR_MLS_00_H_H_N, 4, 4096, 2344 samples, 100 Hz, 2014-02-15T06:16:38.176900Z (046)
5287936       FDSN:FR_MLS_00_H_H_N, 4, 4096, 986 samples, 100 Hz, 2014-02-15T06:35:51.759000Z (046)
5292032       FDSN:FR_MLS_00_H_H_N, 4, 4096, 3410 samples, 100 Hz, 2014-02-15T07:21:28.519000Z (046)
5296128       FDSN:FR_MLS_00_H_H_N, 4, 4096, 2786 samples, 100 Hz, 2014-02-15T07:50:34.761300Z (046)

and this is why the dataselect command without an end byte offset shows it.

For now, I only see expected behavior.

[chad-earthscope]

Correction on the byte range syntax: the dataselect program specifically supports the colon : separator for legacy usage:
https://github.com/EarthScope/dataselect/blob/938b2ce0d56f4adcd09d99d5206407243d24d150/src/dataselect.c#L2991-L3002

This was changed to be consistent with HTTP byte range request syntax. Either syntax works fine with dataselect.

[jschaeff]

This time, I should have triple checked ... sorry for the false reproduction code. Let me describe the thing by myself :)

This is the data segment we are testing on https://s3.seisdata.epos-france.fr/mseed/FR/2014/MLS/HHN.D/FR.MLS.00.HHN.D.2014.046 on this particular segment:

FDSN:FR_MLS_00_H_H_N        2014-02-15T05:17:09.616900Z  2014-02-15T05:19:01.606900Z  1028 100 11200

There is a segment between 5:17 and 5:19

Tests are done with the dataselect binary v4.3.2

Case 1: dataselect with end offset way after the segment

$ dataselect -Ps -ts 2014,046,05,17,00.000000  +o - FR.MLS.00.HHN.D.2014.046@196608-304432 | msi -tg -                  
       SourceID                      Start sample                End sample           Gap  Hz  Samples
FDSN:FR_MLS_00_H_H_N        2014-02-15T05:17:09.616900Z  2014-02-15T05:18:25.006900Z   ==  100 7540
FDSN:FR_MLS_00_H_H_N        2014-02-15T05:34:44.616900Z  2014-02-15T05:37:51.506900Z  979.6 100 18690
FDSN:FR_MLS_00_H_H_N        2014-02-15T05:52:19.616900Z  2014-02-15T06:04:08.826900Z  868.1 100 70922
Total: 1 trace(s) with 3 segment(s)

Result: I miss data reported by msi between 05:18:25 and 05:19:01

Case 2: dataselect without end offset

dataselect -Ps -ts 2014,046,05,17,00.000000  +o - FR.MLS.00.HHN.D.2014.046@196608 | msi -tg -                         
       SourceID                      Start sample                End sample           Gap  Hz  Samples
FDSN:FR_MLS_00_H_H_N        2014-02-15T05:17:09.616900Z  2014-02-15T05:19:01.606900Z   ==  100 11200
...

Here I get the complete trace, until 05:19:01

PS

Sorry about the confusing bug report, I was trying to go deeper in the libmseed level, because our dataselect webservice is now based on the latest libmseed and shows a similar behaviour (when I detetected this weirdness, the webservice was using the dataselect binary).

[chad-earthscope]

$ dataselect -Ps -ts 2014,046,05,17,00.000000 +o - FR.MLS.00.HHN.D.2014.046@196608-304432 | msi -tg -
Result: I miss data reported by msi between 05:18:25 and 05:19:01

Yes, because those data do not all exist in the byte range 196608-304432.

Instead some of the data are later in the file, see the byte offsets for the selected data here:

❯ msi FR.MLS.00.HHN.D.2014.046 -O -ts 2014-02-15T05:17:09.616900Z  -te 2014-02-15T05:19:01.606900Z
196608        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3770 samples, 100 Hz, 2014-02-15T05:17:09.616900Z (046)
200704        FDSN:FR_MLS_00_H_H_N, 4, 4096, 3770 samples, 100 Hz, 2014-02-15T05:17:47.316900Z (046)
5275648       FDSN:FR_MLS_00_H_H_N, 4, 4096, 3660 samples, 100 Hz, 2014-02-15T05:18:25.016900Z (046)

The data are not in time order in the file. The data record you are wanting covering 05:18:25 and 05:19:01 exists at byte offset 5275648.