[fcostaoliveira]

Why

rare-terms-or draws from the df 11–100 band: only 156,060 eligible terms on the 10M corpus, so an epoch is 3,901 lines and the same terms/postings/passages repeat every few thousand queries. In a 10-minute run both Redis Flex and OpenSearch served it entirely from RAM / page cache — not a disk test.

What

• New family rare-terms-or-wide: same shape as rare-terms-or (OR of 40 segment terms, top: 100, fetch title, segment, url, exact totals only, epoch batches under rare-terms-or-wide/epoch-<e>, exact count = distinct passages holding any term), drawn from the rare-term pool: RARE_OR_WIDE_DF = RARE_DF (1–100), same eligibility code (zero ambiguity, no long run, no Redis escape).
• 10M corpus: 3,179,660 eligible terms (mean df ≈ 3.0) → ~79K lines per epoch (~20× longer), ~120 matches/query spread over the corpus. Lines with <100 matches (short pages, the same on both engines) are recorded in batch.lines_below_top and the expected_count stats.
• Postings for the ~3.2M-term pool are kept in one flat array sized from the pass-1 df (~40 MB at 10M).
• GENERATOR → vdb-msmarco-disk/v7; KEY_DOMAIN unchanged. All twelve v6 family files are byte-identical (hash-pinned tests); v6 manifests are refused (they lack the new file) — regenerate with --add-disk-queries and replace the directory.
• The compound single-pass identity pin moves with the generator name, as it did for v6.
• README: family tables, v7 note, expected sizes, --queries ≤ 150,000 for 10-minute duration points (the default 1M timed lines hold six such windows).

Tests

• Pool equals the brute-force df 1–100 eligible set (a mutation to RARE_OR_DF fails it: 151 vs 3151), exact union counts vs the classifier and the RediSearch tokenizer, own epoch partitions, Redis/OpenSearch rendering, param/fetch checks, new file pinned, v5/v6 files unchanged.
• Integration disk run (Redis + OpenSearch) gains a rare-terms-or-wide point — exercised in CI.
• cargo fmt --check, cargo clippy --all-targets -D warnings, and cargo test (lib/bins/harness/cli) pass locally, except the macOS-flaky opensearch_maintenance mock-server tests, which fail on master too.

🤖 Generated with Claude Code

[github-actions]

🤖 Automated first-pass review — a human maintainer's review is still required before merge.

What I checked: the CSR posting build in pass 2, its checks against pass-1 df, the epoch/batch wiring, the conditions() rendering, the manifest fields, the generator bump and identity pin, the hash pins on v5/v6 files, and the new integration lower bound. I found nothing that needs fixing before merge.

Things that already hold up:

• The count is checked, not trusted. disk_queries.rs:1405 refuses a term whose pass-2 postings overflow its pass-1 df slot. disk_queries.rs:1664 then requires wide_fill[i] == df for every term. So pass-1 df is never trusted on its own: an under-fill or over-fill is an error, not a union that comes out silently short.
• Nothing was silently re-keyed. KEY_DOMAIN is unchanged. The twelve v6 files are hash-pinned (v5_family_files_are_unchanged plus the renamed v6_family_files_are_unchanged, with pins.len() + 5 + 1 == ALL.len()). v6 manifests are refused for all four families tested. Bumping to v7 with refusal, rather than adding the file in place, matches what v6 did.
• The pool test guards the right thing. The eligible(1, 100) assertion, together with below_df11 > 0, is what would catch a mutation back to RARE_OR_DF. The PR body reports it going red (151 vs 3151).
• The integration fix reads the right offset. The second commit uses compound.window.offset, which is the configured relative offset (disk_queries.rs:1897 on base). So skip(RESERVED + offset) lines up with the timed window. Using min(count, 100) as a lower bound is the right shape, since the dump check does the exact per-reply comparison.
• The README numbers are consistent. 150,000 / ~79,500 ≈ 1.9 epochs ("about twice"), vs 150,000 / 3,901 ≈ 38 for rare-terms-or. 1M / 150K → six disjoint windows. ~9.5M u32 rows ≈ 38 MB.

Optional / follow-up (not blockers):

1. The fix's main claim isn't measured yet. The reason for this family is that rare-terms-or ran entirely from RAM / page cache on Redis Flex and OpenSearch. The PR shows that for the old family but doesn't show the new one avoids it. On paper it should: ~120 passages/query × 150K queries ≈ 18M hits over ~9.5M posting rows, so the working set nears the whole 10M corpus. Whether it actually spills depends on the deployment's RAM. One live 10-minute rare-terms-or-wide point would turn this from inference into a result: Flex disk reads, or OpenSearch page-cache misses / iostat, next to the same numbers for rare-terms-or. That is the bar the previous family's diagnosis was held to.
2. Results aren't comparable across the two families. Many -wide lines fetch a short page, so stored bytes leaving the engine per query are lower than for rare-terms-or's full 100-hit pages. QPS or latency between the two families therefore isn't like-for-like. Within one family across engines it's fine, since short pages are the same on both. Two small additions would help: the 10M value of batch.lines_below_top (and the expected_count p50) in the PR body or README, and a sentence warning against cross-family comparison.

Net: careful, well-pinned addition that follows the v6 precedent closely, and the pass-1/pass-2 df check is the right guard on a 3.2M-term CSR build. The live disk-pressure check above would confirm the family does what it was added to do.

[github-actions]

Docker Build Validation

Platforms tested:

• linux/amd64 (built and tested)
• linux/arm64 (build validated)

Tests performed:

• --help command
• --describe datasets command
• Integration test (random-100 dataset + Redis)
• Multi-platform build validation (amd64 + arm64)

[github-actions]

Docker Build Validation

Platforms tested:

• linux/amd64 (built and tested)
• linux/arm64 (build validated)

Tests performed:

• --help command
• --describe datasets command
• Integration test (random-100 dataset + Redis)
• Multi-platform build validation (amd64 + arm64)