From ae81b7844e21e8bb5f6db2d18b6c5b64fa816a3f Mon Sep 17 00:00:00 2001 From: fcostaoliveira Date: Thu, 12 Nov 2026 07:39:19 +0000 Subject: [PATCH 1/2] msmarco-disk: add rare-terms-or-wide, OR of 40 terms over the whole rare band (v7) rare-terms-or draws its terms from the df 11-100 band, which holds only 156,060 eligible terms on the 10M corpus: an epoch is 3,901 lines, so the same terms, postings and passages come back every few thousand queries. In a 10-minute run both Redis Flex and OpenSearch ended up serving it entirely from RAM / page cache, so it does not test the disk. rare-terms-or-wide has the same shape (OR of RARE_OR_N = 40 segment terms, top 100, HIGH_PRESSURE_FETCH, exact totals only, epoch batches keyed "rare-terms-or-wide/epoch-", exact count = distinct passages holding any term) but draws from the rare-term pool: df RARE_OR_WIDE_DF = RARE_DF (1-100) with the same eligibility code (zero ambiguous passages, no over-long run, no Redis escape). On the 10M corpus that is 3,179,660 terms (mean df ~3.0): an epoch of ~79K lines (~20x longer), ~120 matches per query spread over the whole corpus. Lines with fewer than 100 matches are expected and recorded (batch.lines_below_top, expected_count stats). Postings for the ~3.2M-term pool are kept in one flat CSR array sized from the pass-1 df (~40 MB at 10M) instead of a Vec per term. GENERATOR is now vdb-msmarco-disk/v7; KEY_DOMAIN is unchanged. All twelve v6 family files are byte-identical (pinned by the v5/v6 hash tests); v6 manifests are refused, since they lack the new file. The duration-mode single-pass identity pin moves with the generator name, as it did for v6. Tests: pool size equals the brute-force df 1-100 eligible set (fails with 151 vs 3151 if the family used RARE_OR_DF), exact union counts against the classifier and the RediSearch tokenizer, own epoch partitions, Redis / OpenSearch rendering, param checks, the new file pinned, and the integration disk run gains a rare-terms-or-wide point. Co-Authored-By: Claude Opus 5.5 --- README.md | 46 +++-- src/bin/prepare_msmarco.rs | 9 +- src/fulltext.rs | 16 +- src/msmarco/disk_queries.rs | 353 ++++++++++++++++++++++++++++++++---- tests/common/mod.rs | 23 ++- 5 files changed, 386 insertions(+), 61 deletions(-) diff --git a/README.md b/README.md index 5569938a..4701ce77 100644 --- a/README.md +++ b/README.md @@ -2620,6 +2620,7 @@ within a run (unless a point opts in to duration mode, below): | `url-batch` | `url` equal to any of 300 URLs, `top: 100`, with `fetch` | 300 random distinct eligible URLs | | `rare-terms-or` | OR of 40 `segment` terms, `top: 100`, with `fetch` | each df 11–100 | | `rare-terms-or-and-category` | that OR AND `category`, `top: 100`, with `fetch` | category of one passage of the union | +| `rare-terms-or-wide` | OR of 40 `segment` terms, `top: 100`, with `fetch` | each df 1–100 (the `rare-term` pool, v7) | Generate the query files once per prepared corpus (deterministic for a seed; a rerun verifies the existing files byte for byte): @@ -2653,14 +2654,17 @@ generator v4 or earlier are refused; regenerate them. Generator v5 only adds (same seed, same SHA-256), so v4 results stay comparable family by family. Generator v6 only adds the high-pressure families (below): the seven v5 family files are byte-identical to v5's, and v5 query files are refused. +Generator v7 only adds `rare-terms-or-wide` (below): the twelve v6 family +files are byte-identical to v6's, and v6 query files are refused. The queries are the same, but a result's compound identity (`compound.sha256`) covers the window record, which names the generator and the manifest SHA-256. So `compare-results` refuses to pair a v5 result with -a v6 one, even for a byte-identical family. Pair them by hand: same family, -offset and seed, and equal `params.fulltext.compound.window.family_file_sha256`. -Rerunning `--add-disk-queries` on a v5 directory reports the first new file -as differing (it is missing) and leaves the fresh generation in the staging -directory: replace the v5 directory with it. +a v6 one (or a v6 result with a v7 one), even for a byte-identical family. +Pair them by hand: same family, offset and seed, and equal +`params.fulltext.compound.window.family_file_sha256`. Rerunning +`--add-disk-queries` on a directory from an earlier generator reports the +first new file as differing (it is missing) and leaves the fresh generation +in the staging directory: replace the old directory with it. A search point selects one family and a window: @@ -2814,6 +2818,7 @@ A search point selects one family and a window: | `url-batch` | `[[u1, …, u300], n]` | `url` any of 300 random URLs | 100 | passages of those URLs | | `rare-terms-or` | `[[t1, …, t40], n]` | OR of 40 `segment` terms, each df 11–100 | 100 | distinct passages holding any term | | `rare-terms-or-and-category` | `[[t1, …, t40], c, n]` | the same OR AND `category` | 100 | distinct passages of the union in `c` | + | `rare-terms-or-wide` (v7) | `[[t1, …, t40], n]` | OR of 40 `segment` terms, each df 1–100 | 100 | distinct passages holding any term | - Redis: `@docid:{d1 | … | dK}` / `@url:{u1 | … | u300}` (one escaped TAG union), `@segment:(t1 | … | t40)` (one field-scoped union of @@ -2832,7 +2837,18 @@ A search point selects one family and a window: pool (docid batches), every `url-eq`-eligible URL (`url-batch`), every term of df 11–100 eligible under the `text_match` rules (zero ambiguous passages, no long run, no Redis escape — the `deep-fetch` pool) for - `rare-terms-or*`. + `rare-terms-or` and `rare-terms-or-and-category`, and every eligible term + of df 1–100 (the `rare-term` pool, same rules) for `rare-terms-or-wide`. + - Why `rare-terms-or-wide` (v7). The df 11–100 pool holds only 156,060 + terms on the 10M corpus, so a `rare-terms-or` epoch is 3,901 lines and + the same terms, postings and passages come back every few thousand + queries: in a 10-minute run both Redis Flex and OpenSearch served it + entirely from RAM / page cache, which is not a disk test. The df 1–100 + pool holds 3,179,660 eligible terms (mean df ≈ 3.0), so a + `rare-terms-or-wide` epoch is ~79,500 lines (~20× longer) and a line's + passages are spread over the whole corpus. A 150,000-query window spans + about two epochs, so each term comes back about twice instead of ~38 + times. - Counts are exact. A docid batch matches K passages. A `url-batch` count is the sum of its URLs' passage counts (distinct URLs, one URL per passage, and the `url-eq` eligibility keeps every URL's count exact on Redis tag @@ -2847,7 +2863,11 @@ A search point selects one family and a window: `URL_BATCH_N` if it is far from that); `rare-terms-or` about 1,100 (≈ 40 × the mean df of the band, ~27; bounded by 440 less overlaps and 4,000); `rare-terms-or-and-category` that union divided by the number of - categories, roughly. + categories, roughly; `rare-terms-or-wide` about 120 (40 × ~3.0, less + overlaps), so many of its lines match fewer than 100 passages and fetch + a short page (the same on both engines). Its manifest records how many + (`families.rare-terms-or-wide.batch.lines_below_top`, with the + `expected_count` distribution and `batch.pool_mean_df`). - The variable-count families keep the probe layout (the 64 highest counts in the reserve) though their `"none"` is refused. The docid batches have constant counts, so their layout is key order. @@ -2858,7 +2878,8 @@ A search point selects one family and a window: `url-batch` by default (URLs are longer than docids: ~7 GB); with MS MARCO docids of ~40 characters a docid file is closer to 4.5 GB. The whole query directory is ~20 GB at 10M. `rare-terms-or*` lines are - ~0.4 KB on disk (1M lines). + ~0.4 KB on disk (1M lines; `rare-terms-or-wide` terms are a little + longer on average). - Client memory. The runner keeps every loaded query three times (its conditions, the compiled expression and the prepared request), measured at about 312 KB per `docid-batch-1000` query, 157 KB per @@ -2868,14 +2889,17 @@ A search point selects one family and a window: these families, so always set `--queries`. For a 10-minute duration-mode point (`compound.duration_s: 600`) use at most 20,000 queries for `docid-batch-1000`, 40,000 for `docid-batch-500` and `url-batch`, and - 150,000 for `rare-terms-or` and `rare-terms-or-and-category` (4.5–6.6 GB - each). With `--dump-results`, each query's page (up to 1,000 ids) is kept + 150,000 for `rare-terms-or`, `rare-terms-or-and-category` and + `rare-terms-or-wide` (4.5–6.6 GB each; the default 1,000,000 timed lines + hold six such disjoint windows). With `--dump-results`, each query's page (up to 1,000 ids) is kept until the run ends and written out: budget a few more GB at top 1,000. - Generator memory. Generation keeps `url-batch` and `rare-terms-or*` lines in memory as 4-byte ids (~0.4 GB and ~0.16 GB each), plus the passage count of every eligible URL (~1 GB at 10M, including the URL strings) and a category and key per passage (12 bytes each): about 2 GB of peak RSS - over v5 at 10M. + over v5 at 10M. v7 adds the `rare-terms-or-wide` lines (~0.16 GB), its + ~3.2M-term pool and index (a few hundred MB at 10M) and its postings in + one flat array (~9.5M row ids, ~40 MB). - Only the count is checked. BM25 differs between the engines (IDF and length normalization), so the top 100 of a 1,000-passage `rare-terms-or` union are not the same documents on both. Tag-only unions are unranked: diff --git a/src/bin/prepare_msmarco.rs b/src/bin/prepare_msmarco.rs index 6a342b06..03a33bc8 100644 --- a/src/bin/prepare_msmarco.rs +++ b/src/bin/prepare_msmarco.rs @@ -203,10 +203,11 @@ struct Args { add_hybrid_queries: bool, /// Write the `msmarco-disk-v1` compound query families (rare terms, term - /// pairs, rare term AND category, url/docid exact match, deep fetch) into - /// `/msmarco-disk-v1/` from the corpus on disk: two passes over - /// `payloads.jsonl`, nothing else read or rewritten (`PREPARED.json` is - /// unchanged). If the directory already exists the families are + /// pairs, rare term AND category, url/docid exact match, deep fetch, and + /// the docid / url / rare-term-OR batches, `rare-terms-or-wide` included) + /// into `/msmarco-disk-v1/` from the corpus on disk: two passes + /// over `payloads.jsonl`, nothing else read or rewritten (`PREPARED.json` + /// is unchanged). If the directory already exists the families are /// regenerated and must be byte-identical (a verification); a difference /// is an error, never a silent replacement. #[arg(long, conflicts_with_all = ["verify", "discover_crc32", "force", "add_derived_fields", "add_hybrid_queries", "add_knn_truth", "from_prepared", "resume_parts"])] diff --git a/src/fulltext.rs b/src/fulltext.rs index 705d359e..1029058a 100644 --- a/src/fulltext.rs +++ b/src/fulltext.rs @@ -5537,8 +5537,9 @@ mod tests { .len(), 100 ); - // The v6 high-pressure families: fixed page with fetch, exact totals - // only, and the request each engine is sent. + // The v6 high-pressure families and v7's rare-terms-or-wide: fixed + // page with fetch, exact totals only, and the request each engine is + // sent. let fields = [ "title".to_string(), "segment".to_string(), @@ -5550,6 +5551,7 @@ mod tests { (dq::Family::UrlBatch, dq::HIGH_PRESSURE_TOP), (dq::Family::RareTermsOr, dq::HIGH_PRESSURE_TOP), (dq::Family::RareTermsOrAndCategory, dq::HIGH_PRESSURE_TOP), + (dq::Family::RareTermsOrWide, dq::HIGH_PRESSURE_TOP), ] { let name = family.name(); assert!(load(name, fetch.clone(), 10, 5).unwrap_err().contains(name)); @@ -5598,7 +5600,7 @@ mod tests { n ); } - dq::Family::RareTermsOr => { + dq::Family::RareTermsOr | dq::Family::RareTermsOrWide => { assert!(redis.starts_with("(@segment:("), "{redis}"); assert_eq!(redis.matches(" | ").count(), n - 1); assert_eq!(body["query"]["bool"]["should"].as_array().unwrap().len(), n); @@ -5635,7 +5637,7 @@ mod tests { // for every family, so `none` loads and the probe has truncated // pages to compare. // (The batch families refuse `none`: docid batches' pages are their - // whole match sets, and every v6 high-pressure family is exact.) + // whole match sets, and every high-pressure family is exact.) for family in dq::Family::ALL .into_iter() .filter(|f| !f.requires_exact_total()) @@ -5700,9 +5702,11 @@ mod tests { /// which names the generator and the manifest SHA-256, so it moves with a /// generator bump even when the family file is byte-identical: v5 /// (901e8c3, before duration mode) pinned - /// `5a7ddc4c5feff1bd8554c8f2c08ffe8546cd73e35a5a6e7717e2cf53aa5f6df7`. + /// `5a7ddc4c5feff1bd8554c8f2c08ffe8546cd73e35a5a6e7717e2cf53aa5f6df7`, + /// v6 (bf947f9) + /// `59742c10771427be1eafb2811b04c25c52693b1669c4152a4e3c8833c5ecca9a`. const SINGLE_PASS_IDENTITY: &str = - "59742c10771427be1eafb2811b04c25c52693b1669c4152a4e3c8833c5ecca9a"; + "ba69fa26b5dcdcdb912ef71a802b13ecd5feb28d9adf58ff7a43221224a5d562"; #[test] fn disk_duration_mode_cycles_the_window_records_it_and_keeps_single_pass_identity() { diff --git a/src/msmarco/disk_queries.rs b/src/msmarco/disk_queries.rs index a1e6c27b..608eeb9b 100644 --- a/src/msmarco/disk_queries.rs +++ b/src/msmarco/disk_queries.rs @@ -21,6 +21,7 @@ //! | `url-batch` | `url` any of [`URL_BATCH_N`] random URLs, `top: 100`, `fetch` | the passages of those URLs | //! | `rare-terms-or` | OR of [`RARE_OR_N`] `segment` terms, each df [`RARE_OR_DF`], `top: 100`, `fetch` | the DISTINCT passages holding any term | //! | `rare-terms-or-and-category` | that OR AND the `category` of one of its passages | ≥1 | +//! | `rare-terms-or-wide` | OR of [`RARE_OR_N`] `segment` terms, each df [`RARE_OR_WIDE_DF`] (the `rare-term` pool), `top: 100`, `fetch` | the DISTINCT passages holding any term | //! //! # Exact by construction //! @@ -72,6 +73,9 @@ //! passage, `url-eq` eligibility); a `rare-terms-or` line is the size of the //! union of its terms' postings, each passage ONCE ([`union_rows`]); the //! category variant counts the union's passages in the chosen category. The +//! v7 family `rare-terms-or-wide` is `rare-terms-or` drawn from the much +//! larger `rare-term` pool ([`RARE_OR_WIDE_DF`]), so its epoch is ~20× longer +//! and its passages spread over the whole corpus with far less reuse. The //! tag-union batches hold `per_family × 100 / items` timed lines //! ([`tag_batch_timed_lines`]) so their files stay near `docid-batch`'s //! size. Parameters are constants checked at compile time against the @@ -132,7 +136,10 @@ pub const MANIFEST: &str = "manifest.json"; /// families `docid-batch-500`, `docid-batch-1000`, `url-batch`, /// `rare-terms-or` and `rare-terms-or-and-category`; the seven v5 family /// files are byte-identical; v5 files are refused (they lack the new files). -pub const GENERATOR: &str = "vdb-msmarco-disk/v6"; +/// v7: adds `rare-terms-or-wide` ([`RARE_OR_WIDE_DF`]); the twelve v6 family +/// files are byte-identical (same rules, same keys); v6 files are refused +/// (they lack `rare-terms-or-wide.jsonl`). +pub const GENERATOR: &str = "vdb-msmarco-disk/v7"; /// Domain of the seeded rank keys ([`rank_key`]). It stays at its first value /// while only the file layout changes, so the same items are selected. pub const KEY_DOMAIN: &str = "vdb-msmarco-disk/v1"; @@ -188,9 +195,21 @@ pub const RARE_OR_N: usize = 40; /// overlaps, at most 4,000); the band is [`DEEP_DF`]'s, which holds more /// than [`MIN_PER_FAMILY`] eligible terms on the 10M corpus. pub const RARE_OR_DF: (u64, u64) = (11, 100); -/// The fixed page (`top`, with `fetch`) of `url-batch`, `rare-terms-or` and -/// `rare-terms-or-and-category`: their match sets are larger, so each query -/// fetches a full 100-document page. +/// Document-frequency band of each `rare-terms-or-wide` term: [`RARE_DF`], +/// the `rare-term` pool (same eligibility rules). [`RARE_OR_DF`] holds only +/// ~156K eligible terms on the 10M corpus, so a `rare-terms-or` epoch is +/// ~3,900 lines and the same terms, postings and passages come back every +/// few thousand queries: in a 10-minute run both Redis Flex and OpenSearch +/// served it entirely from RAM / page cache, which is not a disk test. The +/// full rare band holds ~3.18M eligible terms (mean df ≈ 3), so an epoch is +/// ~79K lines (~20× longer) and a line's ~120 passages are spread over the +/// whole corpus; some lines match fewer than [`HIGH_PRESSURE_TOP`] passages +/// (a short page, the same on both engines; the manifest records how many). +pub const RARE_OR_WIDE_DF: (u64, u64) = RARE_DF; +/// The fixed page (`top`, with `fetch`) of `url-batch` and the +/// `rare-terms-or*` families: their match sets are larger, so each query +/// fetches a full 100-document page (`rare-terms-or-wide`'s often are not: +/// a short page, the same on both engines). pub const HIGH_PRESSURE_TOP: usize = 100; /// The tag values per line whose line budget the tag-union batches share: /// `docid-batch-K` holds `per_family × 100 / K` timed lines and `url-batch` @@ -225,8 +244,8 @@ const _: () = { // Every term of rare-terms-or-and-category plus its category filter. assert!(RARE_OR_N < OPENSEARCH_MAX_CLAUSE_COUNT); }; -/// The fields every v6 high-pressure family fetches, on both engines, in -/// this order: the same stored bytes leave each engine per hit. +/// The fields every high-pressure family (v6 and v7) fetches, on both +/// engines, in this order: the same stored bytes leave each engine per hit. pub const HIGH_PRESSURE_FETCH: [&str; 3] = ["title", "segment", "url"]; /// Longest `url` used as a tag value. pub const MAX_TAG_LEN: usize = 2048; @@ -245,10 +264,11 @@ pub enum Family { UrlBatch, RareTermsOr, RareTermsOrAndCategory, + RareTermsOrWide, } impl Family { - pub const ALL: [Family; 12] = [ + pub const ALL: [Family; 13] = [ Family::RareTerm, Family::TwoRareTerms, Family::RareTermAndCategory, @@ -261,6 +281,7 @@ impl Family { Family::UrlBatch, Family::RareTermsOr, Family::RareTermsOrAndCategory, + Family::RareTermsOrWide, ]; pub fn name(self) -> &'static str { @@ -277,6 +298,7 @@ impl Family { Family::UrlBatch => "url-batch", Family::RareTermsOr => "rare-terms-or", Family::RareTermsOrAndCategory => "rare-terms-or-and-category", + Family::RareTermsOrWide => "rare-terms-or-wide", } } @@ -303,6 +325,7 @@ impl Family { Family::TwoRareTerms => Some(PAIR_DF), Family::DeepFetch => Some(DEEP_DF), Family::RareTermsOr | Family::RareTermsOrAndCategory => Some(RARE_OR_DF), + Family::RareTermsOrWide => Some(RARE_OR_WIDE_DF), Family::UrlEq | Family::DocidEq | Family::DocidBatch @@ -315,9 +338,11 @@ impl Family { /// Fields the family's conditions use (text, then tags). pub fn fields(self) -> &'static [&'static str] { match self { - Family::RareTerm | Family::TwoRareTerms | Family::DeepFetch | Family::RareTermsOr => { - &[TEXT_FIELD] - } + Family::RareTerm + | Family::TwoRareTerms + | Family::DeepFetch + | Family::RareTermsOr + | Family::RareTermsOrWide => &[TEXT_FIELD], Family::RareTermAndCategory | Family::RareTermsOrAndCategory => { &[TEXT_FIELD, "category"] } @@ -337,7 +362,9 @@ impl Family { Family::DocidBatch500 => Some(DOCID_BATCH_500_K), Family::DocidBatch1000 => Some(DOCID_BATCH_1000_K), Family::UrlBatch => Some(URL_BATCH_N), - Family::RareTermsOr | Family::RareTermsOrAndCategory => Some(RARE_OR_N), + Family::RareTermsOr | Family::RareTermsOrAndCategory | Family::RareTermsOrWide => { + Some(RARE_OR_N) + } _ => None, } } @@ -360,21 +387,22 @@ impl Family { Family::DocidBatch | Family::DocidBatch500 | Family::DocidBatch1000 => { self.batch_size() } - Family::UrlBatch | Family::RareTermsOr | Family::RareTermsOrAndCategory => { - Some(HIGH_PRESSURE_TOP) - } + Family::UrlBatch + | Family::RareTermsOr + | Family::RareTermsOrAndCategory + | Family::RareTermsOrWide => Some(HIGH_PRESSURE_TOP), _ => None, } } /// Whether the family refuses `total_hits: "none"`: the docid batches - /// and every v6 high-pressure family. + /// and every high-pressure family (v6, and v7's `rare-terms-or-wide`). pub fn requires_exact_total(self) -> bool { self.batch_size().is_some() } - /// The exact `fulltext.fetch` field list a v6 high-pressure family - /// requires ([`HIGH_PRESSURE_FETCH`]): fetch parity is the point of these + /// The exact `fulltext.fetch` field list a high-pressure family (v6 and + /// v7) requires ([`HIGH_PRESSURE_FETCH`]): fetch parity is the point of these /// families, so another list (fewer, more or other fields) is refused. /// `None` for the v5 families, which only require `fetch` to be set. pub fn fixed_fetch(self) -> Option<&'static [&'static str]> { @@ -463,7 +491,8 @@ impl Family { | Family::DocidBatch500 | Family::DocidBatch1000 | Family::UrlBatch - | Family::RareTermsOr => 2, + | Family::RareTermsOr + | Family::RareTermsOrWide => 2, Family::TwoRareTerms | Family::RareTermAndCategory | Family::RareTermsOrAndCategory => { 3 } @@ -494,7 +523,7 @@ impl Family { let urls = batch_items(self, &a[0], is_safe_tag, "a safe url tag")?; json!({"url": {"match": {"any": urls}}}) } - Family::RareTermsOr | Family::RareTermsOrAndCategory => { + Family::RareTermsOr | Family::RareTermsOrAndCategory | Family::RareTermsOrWide => { let terms = batch_items( self, &a[0], @@ -502,7 +531,7 @@ impl Family { "a text_match term", )?; let any = json!({"or": terms.iter().map(|t| text(t)).collect::>()}); - if self == Family::RareTermsOr { + if self != Family::RareTermsOrAndCategory { any } else { let cat = s(1)?; @@ -1375,6 +1404,23 @@ pub fn generate(dir: &Path, opts: &GenerateOptions, out: &Path) -> Result = wide_pool + .iter() + .enumerate() + .map(|(i, (_, t, _))| (t.as_str(), i as u32)) + .collect(); + let mut wide_start: Vec = Vec::with_capacity(wide_pool.len() + 1); + wide_start.push(0); + for (_, _, df) in &wide_pool { + wide_start.push(wide_start.last().unwrap() + *df as usize); + } + let mut wide_rows: Vec = vec![0; *wide_start.last().unwrap()]; + let mut wide_fill: Vec = vec![0; wide_pool.len()]; // Pair-pool term ids in lexicographic order, so an id-sorted pair is a // term-sorted pair. let mut pair_terms: Vec<(&str, u64)> = @@ -1425,6 +1471,19 @@ pub fn generate(dir: &Path, opts: &GenerateOptions, out: &Path) -> Result= wide_start[i + 1] { + return Err(format!( + "{SUITE}: term {t:?} matched more passages in pass 2 than its df {} in \ + pass 1", + wide_pool[i].2 + )); + } + wide_rows[at] = row32; + wide_fill[i] += 1; + } if let Some(&i) = rac_index.get(t.as_str()) { rac_cats[i].push(s.category.clone()); let k = rank_key(seed, "rare-term-and-category/passage", &s.docid); @@ -1602,6 +1661,17 @@ pub fn generate(dir: &Path, opts: &GenerateOptions, out: &Path) -> Result Vec<&[u32]> { line.iter() .map(|&i| rare_or_postings[i as usize].as_slice()) @@ -1657,6 +1727,33 @@ pub fn generate(dir: &Path, opts: &GenerateOptions, out: &Path) -> Result Vec<&[u32]> { + line.iter() + .map(|&i| &wide_rows[wide_start[i as usize]..wide_start[i as usize + 1]]) + .collect() + }; + let wide_batches = Batches::new( + Family::RareTermsOrWide, + wide_pool.len(), + |i| wide_pool[i as usize].1.as_str(), + seed, + cap, + )?; + let wide_lines: Vec<&[u32]> = wide_batches.lines().collect(); + let wide_counts: Vec = wide_lines + .par_iter() + .map(|l| union_rows(&wide_postings(l)).len() as u64) + .collect(); + let wide_overlap = wide_lines + .iter() + .zip(&wide_counts) + .filter(|(l, n)| l.iter().map(|&i| wide_pool[i as usize].2).sum::() > **n) + .count(); + let wide_below_top = wide_counts + .iter() + .filter(|&&n| n < HIGH_PRESSURE_TOP as u64) + .count(); let mean = |v: &[u64]| v.iter().sum::() as f64 / v.len().max(1) as f64; let url_pool_mean = mean(&url_pool.iter().map(|(_, n)| *n).collect::>()); let rare_or_pool_mean = mean( @@ -1665,20 +1762,26 @@ pub fn generate(dir: &Path, opts: &GenerateOptions, out: &Path) -> Result>(), ); + let wide_pool_mean = mean(&wide_pool.iter().map(|(_, _, df)| *df).collect::>()); let url_redis_max = url_batches .lines() .map(|l| redis_tag_union_len("url", l.iter().map(|&i| url_pool[i as usize].0.as_str()))) .max() .unwrap_or(0); let or_rule = |family: Family, extra: &str| { + let pool_of = match family { + Family::RareTermsOrWide => "rare-term", + _ => "deep-fetch", + }; format!( - "the pool is every eligible segment term of df {RARE_OR_DF:?} (zero ambiguous \ + "the pool is every eligible segment term of df {:?} (zero ambiguous \ passages, not in any over-long run, not named by the Redis escape rule: the \ - deep-fetch eligibility); epoch e ranks the pool by SHA-256(key domain \ + {pool_of} eligibility); epoch e ranks the pool by SHA-256(key domain \ \"vdb-msmarco-disk/v1\", seed, \"{}/epoch-\", term) and cuts it into \ consecutive batches of {RARE_OR_N} (the last len % {RARE_OR_N} unused); a line is \ one batch in key order; its count is the number of DISTINCT passages holding any \ of its terms{extra}", + family.df_band().expect("a text family"), family.name() ) }; @@ -1781,6 +1884,29 @@ pub fn generate(dir: &Path, opts: &GenerateOptions, out: &Path) -> Result = l + .iter() + .map(|&t| wide_pool[t as usize].1.as_str()) + .collect(); + json!([terms, wide_counts[i]]) + }, + &wide_counts, + &mut counts, + ) + .map_err(err)?; + batch = wide_batches.record(wide_pool.len(), or_rule(family, "")); + batch["df_band"] = json!(RARE_OR_WIDE_DF); + batch["pool_mean_df"] = json!(wide_pool_mean); + batch["lines_with_overlapping_terms"] = json!(wide_overlap); + batch["lines_below_top"] = json!(wide_below_top); + batch["opensearch_bool_should_clauses"] = json!(RARE_OR_N); + batch["opensearch_max_clause_count"] = json!(OPENSEARCH_MAX_CLAUSE_COUNT); } else { let l = probe_layout(family, lines.remove(&family).expect("every family"))?; counts.reserve(l.len()); @@ -1824,7 +1950,8 @@ pub fn generate(dir: &Path, opts: &GenerateOptions, out: &Path) -> Result Result Result.batch.rule (rare-terms-or* draw from the deep-fetch eligible terms, re-ranked per epoch under their own family name).", + "selection": "per family: eligible items ranked by SHA-256(key domain \"vdb-msmarco-disk/v1\", seed, family, item); the smallest kept in key order. rare-term-and-category terms are disjoint from rare-term terms; deep-fetch is drawn from its own band independently (overlaps recorded in term_overlap). Terms: zero ambiguous passages, not in any over-long run, not named by the Redis escape rule in any passage. rare-term-and-category: the category of the term's passage with the smallest seeded docid key. two-rare-terms: co-occurring pairs (one passage has both) ranked by a SplitMix64 mix of the two term keys. docid-batch, docid-batch-500, docid-batch-1000, url-batch, rare-terms-or, rare-terms-or-and-category, rare-terms-or-wide: see families..batch.rule (rare-terms-or and rare-terms-or-and-category draw from the deep-fetch eligible terms, rare-terms-or-wide from the rare-term eligible terms, each re-ranked per epoch under its own family name).", "families": families, }); let text = serde_json::to_string_pretty(&manifest).map_err(|e| e.to_string())?; @@ -2325,7 +2453,9 @@ mod tests { } // A passage matches when its Redis tokens hold ANY term // (each passage once). - Family::RareTermsOr | Family::RareTermsOrAndCategory => { + Family::RareTermsOr + | Family::RareTermsOrAndCategory + | Family::RareTermsOrWide => { let terms: Vec<&str> = a[0] .as_array() .unwrap() @@ -2422,6 +2552,16 @@ mod tests { m["families"]["deep-fetch"]["eligible_pool"], eligible(11, 100) ); + // rare-terms-or-wide draws from the whole rare band (the rare-term + // pool), not rare-terms-or's df 11-100. + assert_eq!( + m["families"]["rare-terms-or-wide"]["eligible_pool"], + eligible(1, 100) + ); + assert_eq!( + m["families"]["rare-terms-or"]["eligible_pool"], + eligible(11, 100) + ); for (term, why) in [ ("edge", "ambiguous passage"), ("cafe", "folding"), @@ -2669,15 +2809,25 @@ mod tests { .contains("SHA-256")); std::fs::write(&f, body).unwrap(); assert!(load_window(&dir, Family::UrlEq, 0, Some(1)).is_ok()); - // A v5 manifest (no high-pressure families) or a v4 one is refused - // for every family, as v5 refused v4: regenerate. + // A v6 manifest (no rare-terms-or-wide), a v5 one (no high-pressure + // families) or a v4 one is refused for every family, as v6 refused + // v5: regenerate. let mp = dir.join(DIR).join(MANIFEST); let manifest = std::fs::read_to_string(&mp).unwrap(); - for old in ["vdb-msmarco-disk/v5", "vdb-msmarco-disk/v4"] { + for old in [ + "vdb-msmarco-disk/v6", + "vdb-msmarco-disk/v5", + "vdb-msmarco-disk/v4", + ] { let stale = manifest.replace(GENERATOR, old); assert_ne!(stale, manifest); std::fs::write(&mp, stale).unwrap(); - for family in [Family::UrlEq, Family::DocidBatch, Family::RareTermsOr] { + for family in [ + Family::UrlEq, + Family::DocidBatch, + Family::RareTermsOr, + Family::RareTermsOrWide, + ] { let e = load_window(&dir, family, 0, Some(1)).unwrap_err(); assert!(e.contains(old) && e.contains(GENERATOR), "{e}"); } @@ -2934,10 +3084,11 @@ mod tests { } } - /// The five v6 family files, pinned like the v5 ones: a change to their - /// selection, keys, layout or counts changes these. + /// The five v6 family files are byte-identical under v7: the SHA-256 of + /// each fixture file as generator v6 (bf947f9) wrote it. With + /// `v5_family_files_are_unchanged` this pins all twelve v6 files. #[test] - fn v6_family_files_are_pinned() { + fn v6_family_files_are_unchanged() { let pins = [ ( Family::DocidBatch500, @@ -3021,8 +3172,10 @@ mod tests { "d1bb81f6e6c46922f5c55a9713794c31861172b90789e4bf8fe91cb2483b9288", ), ]; - // Every v5 family is pinned; the rest are the v6 additions. - assert_eq!(pins.len() + 5, Family::ALL.len()); + // Every v5 family is pinned; the rest are the five v6 additions + // (`v6_family_files_are_unchanged`) and the v7 one + // (`v7_family_file_is_pinned`). + assert_eq!(pins.len() + 5 + 1, Family::ALL.len()); let (_tmp, collisions) = fixture(3000); let a = generate(&collisions, &opts(200, 7), &collisions.join(DIR)).unwrap(); let tmp = tempfile::tempdir().unwrap(); @@ -3035,6 +3188,27 @@ mod tests { } } + /// The v7 family file, pinned like the v6 ones: a change to its + /// selection, keys, layout or counts changes these. + #[test] + fn v7_family_file_is_pinned() { + let (_tmp, collisions) = fixture(3000); + let a = generate(&collisions, &opts(200, 7), &collisions.join(DIR)).unwrap(); + let tmp = tempfile::tempdir().unwrap(); + let plain = tmp.path().join("corpus"); + write_fixture_corpus(&plain, 3000, 4).unwrap(); + let b = generate(&plain, &opts(100, 1), &plain.join(DIR)).unwrap(); + let name = Family::RareTermsOrWide.name(); + assert_eq!( + a["families"][name]["sha256"], + "cfed8fc041a870d2f1cd37f1d2272cb89673d58f7f5ee84bff8854432bc7caa0" + ); + assert_eq!( + b["families"][name]["sha256"], + "7d112208e9a332fb94c95e96ea42d0755ac2f5a6304a0848d2c32416128be6f1" + ); + } + #[test] fn docid_pool_excludes_every_copy_of_a_repeated_docid() { let docids = ["b", "a", "c", "a", "d", "c", "a"]; @@ -3153,18 +3327,31 @@ mod tests { assert_eq!(hp["url_batch_n"], URL_BATCH_N); assert_eq!(hp["rare_or_n"], RARE_OR_N); assert_eq!(hp["rare_or_df"], json!(RARE_OR_DF)); + assert_eq!(hp["rare_or_wide_df"], json!(RARE_OR_WIDE_DF)); assert_eq!(hp["top"], HIGH_PRESSURE_TOP); assert_eq!(m["df_bands"]["rare_or"], json!(RARE_OR_DF)); + assert_eq!(m["df_bands"]["rare_or_wide"], json!(RARE_OR_WIDE_DF)); // Rule texts are single-spaced prose (no source indentation). for family in Family::ALL { if let Some(rule) = m["families"][family.name()]["batch"]["rule"].as_str() { assert!(!rule.contains(" "), "{family:?}: {rule}"); } } - // Brute-force postings (classifier per passage) of every term used. + // Brute-force postings (classifier per passage) of every term used + // (rare-terms-or-wide: of the sampled lines checked below; every + // line's count is checked on Redis tokens in + // `generated_counts_match_a_brute_force_oracle`). + let wide_sample = |i: usize| i < 24 || i.is_multiple_of(16); let mut postings: HashMap> = HashMap::new(); - for f in [Family::RareTermsOr, Family::RareTermsOrAndCategory] { - for line in lines(&out, f) { + for f in [ + Family::RareTermsOr, + Family::RareTermsOrAndCategory, + Family::RareTermsOrWide, + ] { + for (i, line) in lines(&out, f).into_iter().enumerate() { + if f == Family::RareTermsOrWide && !wide_sample(i) { + continue; + } for t in line[0].as_array().unwrap() { let t = t.as_str().unwrap(); if !postings.contains_key(t) { @@ -3325,6 +3512,67 @@ mod tests { want ); } + + // rare-terms-or-wide: the same union over the rare-term pool (df + // 1-100): its pool is rare-term's, far larger than rare-terms-or's, + // so its epoch is longer, and it does use terms below df 11. + let f = &m["families"]["rare-terms-or-wide"]; + assert_eq!(f["df_band"], json!(RARE_DF)); + assert_eq!(f["batch"]["df_band"], json!(RARE_DF)); + assert_eq!( + f["eligible_pool"], + m["families"]["rare-term"]["eligible_pool"] + ); + let wide_pool = f["eligible_pool"].as_u64().unwrap() as usize; + assert!( + wide_pool + > 10 * m["families"]["rare-terms-or"]["eligible_pool"] + .as_u64() + .unwrap() as usize + ); + assert_eq!(f["batch"]["batches_per_epoch"], wide_pool / RARE_OR_N); + assert_eq!(f["batch"]["items_unused_per_epoch"], wide_pool % RARE_OR_N); + assert!(f["batch"]["rule"] + .as_str() + .unwrap() + .contains("rare-term eligibility")); + let wide = lines(&out, Family::RareTermsOrWide); + assert_eq!(wide.len(), 200 + RESERVED); + let (mut below_df11, mut overlapping, mut below_top) = (0, 0, 0); + for (i, line) in wide.iter().enumerate() { + let (_, n) = Family::RareTermsOrWide.conditions(line).unwrap(); + if n < HIGH_PRESSURE_TOP as u64 { + below_top += 1; + } + if !wide_sample(i) { + continue; + } + let terms: Vec<&str> = line[0] + .as_array() + .unwrap() + .iter() + .map(|v| v.as_str().unwrap()) + .collect(); + assert_eq!(terms.len(), RARE_OR_N); + let df_sum: u64 = terms.iter().map(|t| df_of(t)).sum(); + assert_eq!(n, union_of(&terms).len() as u64, "{line}"); + assert!(terms.iter().all(|t| (1..=100).contains(&df_of(t)))); + below_df11 += terms.iter().filter(|t| df_of(t) < 11).count(); + if n < df_sum { + overlapping += 1; + } + } + assert!( + below_df11 > 0, + "rare-terms-or-wide must use the df 1-10 terms" + ); + assert!( + overlapping > 0, + "the fixture must exercise overlapping terms" + ); + // Short pages are expected (mean df ~3): recorded, not refused. + assert!(below_top > 0); + assert_eq!(f["batch"]["lines_below_top"], below_top); } #[test] @@ -3379,7 +3627,32 @@ mod tests { dup_terms[1] = dup_terms[0].clone(); let mut bad_term = terms.clone(); bad_term[0] = "the".into(); + // rare-terms-or-wide renders exactly as rare-terms-or. + let (wide, n) = Family::RareTermsOrWide + .conditions(&json!([terms, 37])) + .unwrap(); + assert_eq!(n, 37); + assert_eq!( + wide, + Family::RareTermsOr + .conditions(&json!([terms, 37])) + .unwrap() + .0 + ); + let expr = crate::compound_text::Expr::compile(&wide, &schema).unwrap(); + assert_eq!(expr.redis(), format!("(@segment:({}))", terms.join(" | "))); + let os = expr.opensearch(); + assert_eq!(os["bool"]["minimum_should_match"], 1); + assert_eq!(os["bool"]["should"].as_array().unwrap().len(), RARE_OR_N); + assert_eq!(Family::RareTermsOrWide.fields(), [TEXT_FIELD]); + assert_eq!(Family::RareTermsOrWide.df_band(), Some((1, 100))); + assert_eq!(Family::RareTermsOrWide.batch_size(), Some(RARE_OR_N)); for (family, bad) in [ + (Family::RareTermsOrWide, json!([terms[1..], 5])), + (Family::RareTermsOrWide, json!([dup_terms, 5])), + (Family::RareTermsOrWide, json!([bad_term, 5])), + (Family::RareTermsOrWide, json!([terms, 0])), + (Family::RareTermsOrWide, json!([terms, "cat03", 5])), (Family::RareTermsOr, json!([terms[1..], 5])), (Family::RareTermsOr, json!([dup_terms, 5])), (Family::RareTermsOr, json!([bad_term, 5])), @@ -3397,6 +3670,7 @@ mod tests { Family::UrlBatch, Family::RareTermsOr, Family::RareTermsOrAndCategory, + Family::RareTermsOrWide, ] { assert_eq!(family.fixed_top(), Some(HIGH_PRESSURE_TOP)); assert!(family.check_params(100, true, false).is_ok()); @@ -3405,7 +3679,8 @@ mod tests { let e = family.check_params(100, true, true).unwrap_err(); assert!(e.contains("exact"), "{e}"); } - // Fetch parity: the v6 families fetch exactly title, segment, url. + // Fetch parity: the v6 and v7 families fetch exactly title, segment, + // url. let f = |v: &[&str]| v.iter().map(|s| s.to_string()).collect::>(); for family in Family::ALL { let pinned = family.fixed_fetch().is_some(); @@ -3434,6 +3709,8 @@ mod tests { // The documented parameters, as literals (a retuned constant must // update the docs and the manifest consumers). assert_eq!(RARE_OR_DF, (11, 100)); + assert_eq!(RARE_OR_WIDE_DF, (1, 100)); + assert_eq!(RARE_OR_WIDE_DF, RARE_DF); assert_eq!((RARE_OR_N, URL_BATCH_N, HIGH_PRESSURE_TOP), (40, 300, 100)); assert_eq!((DOCID_BATCH_500_K, DOCID_BATCH_1000_K), (500, 1000)); assert_eq!(Family::DocidBatch500.fixed_top(), Some(500)); diff --git a/tests/common/mod.rs b/tests/common/mod.rs index f20da23a..b7cbaad2 100644 --- a/tests/common/mod.rs +++ b/tests/common/mod.rs @@ -5072,6 +5072,7 @@ pub fn disk_search_params() -> serde_json::Value { point("url-batch", 100, true, "exact", 0), point("rare-terms-or", 100, true, "exact", 0), point("rare-terms-or-and-category", 100, true, "exact", 0), + point("rare-terms-or-wide", 100, true, "exact", 0), ]) } @@ -5138,9 +5139,27 @@ pub fn assert_disk_run(root: &Path, engine: &str, config: &str, dump: &Path) { fetch["hits"].as_u64().unwrap() >= (k * DISK_QUERIES) as u64, "{ft}" ); + } else if family == Family::RareTermsOrWide { + // Its unions are mostly smaller than the page (most terms + // have df 1-3): at least min(count, 100) per window line + // (each reply's page is checked exactly against the dump + // below). + let offset = ft["compound"]["offset"].as_u64().unwrap() as usize; + let want: u64 = fs::read_to_string(ds.join(DIR).join(family.file_name())) + .unwrap() + .lines() + .skip(RESERVED + offset) + .take(DISK_QUERIES) + .map(|l| { + let line: serde_json::Value = serde_json::from_str(l).unwrap(); + family.conditions(&line).unwrap().1.min(100) + }) + .sum(); + assert!(fetch["hits"].as_u64().unwrap() >= want, "{ft}"); } else if family.batch_size().is_some() { - // url-batch / rare-terms-or*: a full 100-hit page each (their - // unions are larger than the page on the fixture). + // url-batch / rare-terms-or, rare-terms-or-and-category: a + // full 100-hit page each (their unions are larger than the + // page on the fixture). assert!( fetch["hits"].as_u64().unwrap() >= (100 * DISK_QUERIES) as u64, "{ft}" From f981b706202df7a8622b34702ab358349a4b92a5 Mon Sep 17 00:00:00 2001 From: fcostaoliveira Date: Thu, 12 Nov 2026 08:11:36 +0000 Subject: [PATCH 2/2] tests: read rare-terms-or-wide window offset from recorded compound.window The result file records the window under compound.window; the configured compound.offset is not echoed back, so the lower-bound fetch check panicked on unwrap in the Redis and OpenSearch disk-suite integration tests. Co-Authored-By: Claude Opus 5.5 --- tests/common/mod.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/common/mod.rs b/tests/common/mod.rs index b7cbaad2..19433605 100644 --- a/tests/common/mod.rs +++ b/tests/common/mod.rs @@ -5144,7 +5144,10 @@ pub fn assert_disk_run(root: &Path, engine: &str, config: &str, dump: &Path) { // have df 1-3): at least min(count, 100) per window line // (each reply's page is checked exactly against the dump // below). - let offset = ft["compound"]["offset"].as_u64().unwrap() as usize; + // The recorded metadata keeps the window under + // `compound.window` (the configured `compound.offset` is + // not echoed back). + let offset = ft["compound"]["window"]["offset"].as_u64().unwrap() as usize; let want: u64 = fs::read_to_string(ds.join(DIR).join(family.file_name())) .unwrap() .lines()