[fcostaoliveira]

msmarco-disk: add corpus-sweep-or, every passage fetched at least once per epoch (v8)

Why

rare-terms-or-wide fetches only the top 100 of each query's matches. After one epoch, that set fit in Redis Flex's RAM tier (~6.2M of the 10M documents), so the run never reached flash. corpus-sweep-or fetches every passage of the corpus at least once per epoch, in a new seeded order each epoch. Any cache smaller than the corpus keeps missing, on both Redis Flex and OpenSearch.

What

• Cover. For each passage, pick its rarest eligible segment term with df 1–100. This is the rare-term pool with the same text_match / long-run / Redis-escape eligibility, and it reuses rare-terms-or-wide's postings. Ties go to the smaller term in byte order. The chosen terms' postings cover every passage that holds such a term.

• Docid complement. A text-only cover cannot reach the corpus. On the 10M corpus:

  df bound	passages covered
  ≤ 100	37.0%
  ≤ 200	39.4%
  ≤ 2,000	42.1%
  any	42.9%

  An eligible term must have zero ambiguous passages across the whole corpus, and almost no frequent term does. Every other passage is therefore added by its docid, taken from the docid-batch pool (safe, not a Redis tag piece, unique).

• Line format. [[terms], [docids], count] is the OR of one field-scoped term union and one docid tag union:

  • Redis: ((@segment:(t1 | …)) | (@docid:{d1 | …}))
  • OpenSearch: bool.should of the terms' bool.should and a docid terms filter, at most 65 clauses.

• Packing. Epoch e orders the terms and docids by rank_key(seed, "corpus-sweep-or/epoch-<e>/term|docid", item). It then packs them greedily: an item joins the current line while sum(df) + docids ≤ SWEEP_TOP (200) and the line holds at most SWEEP_MAX_TERMS (64) terms. A line's distinct matches are at most that sum, so the page of 200 fetches every match.

• Query settings. top is fixed at 200, fetch is exactly title, segment, url, and only total_hits: exact is accepted.

• File length. The file holds per_family + 65 lines, like the other families. Epochs are reshuffled, so no line repeats.

Constants

• SWEEP_DF = (1, 100): going above 100 adds little coverage (+2.4 points at 200), and a term must fit in a line.
• SWEEP_TOP = 200: twice the other high-pressure pages, which means more documents per query, and room for a df-100 term next to others.
• SWEEP_MAX_TERMS = 64: most terms have df 1, so without a cap a line could hold up to 200 term clauses. On the 10M corpus the cap is never reached (max 51 terms per line).

Numbers (10M corpus, --disk-queries-per-family 1000000, seed 1)

chosen terms	1,860,417 (df sum 7,270,291; df p50 1, mean 3.9)
passages by term / by docid / unreachable	3,704,836 / 6,295,123 / 0 (100% coverage)
epoch	~69,700 lines; 15 epochs, 14.3 of them in the file
expected count	mean 194.6, p50 200, min 74
per line	~27 terms + ~90 docids; every line has both
fetched documents per epoch	13.57M (term-covered passages ~2× each)
file	1,000,065 lines, 4.4 GB
generation (all families)	5.7 min, 5.2 GB peak RSS (v7 was 5.5 GB)

The manifest records these numbers under families.corpus-sweep-or.batch: coverage, terms, epoch_lines, epoch_lines_each, epoch_expected_count, epoch_fetched_passages, epoch_distinct_passages, terms_per_line and docids_per_line.

Compatibility

• GENERATOR is now vdb-msmarco-disk/v8; KEY_DOMAIN is unchanged.
• All thirteen v7 family files are byte-identical. This is checked by the fixture pins and by comparing SHA-256 against 5ba5429 on the real 1M and 10M corpora.
• v7 manifests are refused.
• The duration-mode single-pass identity pin moves with the generator name, as in v6 and v7.

Usage note

A duration-mode window must cover at least one epoch (~69,700 queries at 10M) to sweep the whole corpus. A shorter window keeps cycling through part of it.

Tests

• Unit:
  • rarest_term tie order.
  • pack_sweep greedy bounds and the term cap.
  • Synthetic corpus: every epoch reaches every passage with each unit exactly once, deterministic, seeded.
  • Independent re-derivation of the fixture file: brute-force eligibility, Redis tokens, each passage's rarest term, docids, epochs.
  • Redis / OpenSearch rendering, parameter checks and malformed-line checks.
  • The Redis-token count oracle covers every line.
  • v8 pins.
• A sweep-gap fixture variant (30 passages without a rare term) exercises docid units. The integration disk suite now uploads it and has a corpus-sweep-or point. Its window must hold lines with docids, and the Redis and OpenSearch replies are checked against the oracle.
• Results:
  • cargo fmt --check and cargo clippy --all-targets -- -D warnings: clean.
  • Lib tests: all pass except 2 opensearch_maintenance mock-socket tests. Those also fail on unmodified master on the test machine.
  • integration_redis: 81 pass.
  • integration_opensearch: 46 pass.

Regenerate

prepare-msmarco --dataset msmarco-cohere-1024-10M-cosine --add-disk-queries \
  [--disk-queries-per-family 1000000] [--disk-queries-seed 1]

On an existing v7 directory, the command reports corpus-sweep-or.jsonl as missing and leaves the fresh generation in .msmarco-disk-v1.staging. Replace the old directory with it.

🤖 Generated with Claude Code

[github-actions]

🤖 Automated first-pass review — a human maintainer's review is still required before merge.

What I checked: the cover's correctness and exact counts (rarest_term → sweep_chosen, and the cross-check of postings against covered rows at generate), whether docid units and term postings stay disjoint, pack_sweep's bounds, epoch reshuffling, the Redis and OpenSearch rendering limits (const asserts on clause/terms/result-window), the v7 byte-identity claim, and how the new family goes through the existing probe layout. The count path looks right to me. A docid unit is only taken when batch_covered[i] is false, which is set from the same sweep_choice that decides term coverage, so union_rows(postings).len() + docids is exact. The independent re-derivation test (corpus_sweep_or_is_epochs_of_its_cover_and_docids) actually re-derives the lines instead of calling the production packer for its expected output, apart from pack_sweep itself, which has its own greedy-bound test. Good work there.

Worth fixing before merge

1. The probe reserve takes 65 lines out of epoch 0, so the "every passage at least once per epoch" claim doesn't hold for the first epoch of the timed region. In the Family::CorpusSweepOr branch, the generated order goes through probe_order(&sweep_counts) (disk_queries.rs, the existing fn at ~L918). That puts line 0 (the prime) plus the PROBE_RESERVE highest-count lines, ties broken by index, into lines 0..65. On 10M, p50 count is 200, so the reserve is just the first ~64 count-200 lines, nearly all from epoch 0. Those lines are never timed. Effect: a window at offset 0 of epoch_lines (~69,700) queries, which the README's usage note tells people to use, misses about 65 × ~195 ≈ 12.6k passages, roughly 0.13% of the corpus. Instead it runs 65 lines of epoch 1 again. The test comment at tests/common/mod.rs (the offset 40 note) already mentions this effect for the fixture. Two other things are off as a result: the manifest's epoch_lines / epoch_fetched_passages / epoch_expected_count describe epoch 0 in generated order, reserve lines included, not what a timed window sees; and epoch boundaries in timed-offset space are shifted by up to 65. The effect is small, but the whole point of the family is complete coverage. Either generate the prime and reserve from a separate pre-epoch (or an extra slice) so every timed epoch is whole, or say in the manifest and README that epoch 0 is short by its reserve lines and give the offset where epoch 1 starts in the timed region. I may be missing a reason the reserve must come from epoch 0. If so, a sentence in the rule text would settle it.

2. The manifest prose contradicts the reshuffle. The selection string (diff L858) ends "…plus the docids of the passages none reaches, in one fixed epoch)", and the code comment above sweep_terms (diff L651–653) says the units are "shuffled once … packed into one epoch of lines, which the file repeats". The code, batch.epoch_order, the PR body and the README all say each epoch is a new seeded order with no repeated lines. selection ships in every v8 manifest, so it should be fixed before files get regenerated at 10M/1M scale.

Optional / follow-up

• coverage.unreachable is only recorded. It's 0 on the 10M corpus, but if another corpus has passages with no band term and an unsafe or duplicate docid, the family would quietly stop being a full sweep while still being named one. Consider a hard error, or at least a loud warning, when unreachable > 0, in line with the rest of the repo's "error rather than a quietly wrong number" stance.
• The ~50 KB/query --dump-results budget is labelled as an estimate. Fine as written, but worth measuring on the first real 10M run and updating the README.
• The two opensearch_maintenance mock-socket failures, reported as also failing on unmodified master on the test machine, are probably worth filing separately so they don't get treated as background noise in later PRs.

Net: careful, well-pinned addition. The v7 byte-identity is backed by fixture pins and SHA comparisons on the real corpora, and the count oracle covers every line. Fix the epoch-0 reserve gap (or document it precisely) and the "one fixed epoch" wording before regenerating, and it looks ready.

[github-actions]

Docker Build Validation

Platforms tested:

• linux/amd64 (built and tested)
• linux/arm64 (build validated)

Tests performed:

• --help command
• --describe datasets command
• Integration test (random-100 dataset + Redis)
• Multi-platform build validation (amd64 + arm64)