Repository navigation
Conversation
The inline forward index is 45.6 of a 50.0 GiB disk_seismic_sq index over MS MARCO, and it stores each document's vector 12.97 times -- once per posting list that retains it. That duplication is what makes a block one contiguous read, so it cannot simply go; what can go is most of each extra copy. inline_max_nnz=T (build) leaves one copy of each document whole -- the one the DocLocator directory already points at -- and keeps only the T largest codes of every other. Block scores become lower bounds, so search re-scores its `rescore` (query, DiskSeismicSearchParameters, default 200) best candidates against the whole copies and re-ranks. Truncation is off by default, so nothing changes for an existing index; kFormatVersion goes to 2 because a v1 reader would score a truncated copy as if it were whole, while v1 files still load here. A negative rescore and a malformed inline_max_nnz are rejected rather than silently reinterpreted. Reaching a whole copy is a four-deep pointer chase through tables far past any cache, which dominates the second pass, so resolve_docs walks the candidates through each step together in bounded chunks -- 0.68 -> 0.45 us per re-scored document, bit-identical results. Measured on base_full (lambda=6000 beta=400 alpha=0.4, 8-bit, cut=3 k'=50 k=10, single thread, seed=42 on every build): T=64 with rescore=100 is 44.6% off the file at 0.98x the warm latency and -0.03pp recall@10, inside the noise a k-means seed causes on its own; T=48 with rescore=200 is 54.0% off at 1.15x and -0.04pp. The cold first pass and the resident set grow ~25-28%, because the second pass touches scattered blocks the scan does not. benchmarks/DISK_SEISMIC_BENCH.md has the full curve, two ideas that did not work, and prune_sim, which scores a candidate layout for recall without building it. rescore and the new constructors reach Python through the existing SWIG headers; the disk-index contract tests cover both from there. Resolves opensearch-project#59 Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
zirui-song-18
left a comment
There was a problem hiding this comment.
By the way, neural-search can't set inline_max_nnz or rescore yet, right? Later we might need addition PR in the plugin repo.
| // already causes, measured at inline_max_nnz=48; raising it buys hundredths of a | ||
| // point for a third more latency, and at the candidate-pool size it is simply | ||
| // the unpruned search. Lower it to trade recall for speed. | ||
| inline constexpr int kDefaultRescoreDepth = 200; |
There was a problem hiding this comment.
Issue: kDefaultRescoreDepth = 200 is an absolute depth, so with max(rescore, k) the oversampling shrinks as k grows: 20× at k=10, 2× at k=100, and none from k=200 up, where the second pass only re-sorts the k lower-bound leaders.
I rebuilt base_full with your settings and seed=42 (sizes and recall@10 match the table) and scored against exact top-k at cut=3/k'=50:
- k=100: T=48 at the default loses 0.79 pp recall@100 (2373 queries worse, 23 better); the "free" T=64/R=100 loses 3.08 pp.
- k=200: −3.54 pp (T=48) / −1.54 pp (T=64); with k'=200 it's −7.70 / −4.05 pp, so raising k' for a larger k widens the gap.
- ~3–5·k gets back to parity: at k=100, R=500 is −0.05 pp (T=48) and R=300 −0.03 pp (T=64).
The needed depth also tracks the pass-1 pool, not just k: at neural-search's defaults (cut=10, k'=20) pass 1 scores ~480 docs and T=48 is already within 0.1 pp at R=90 (~1.07× T=0 p50), while 200 costs ~1.25×.
k-NN sizes the same pass k-relatively (max(100, ceil(k · oversample_factor)) in RescoreContext.getFirstPassK). Default to max(200, 3·k) (every k=10 row in the table stays the same), or add an oversample field next to rescore? At minimum, could the comment above name the operating point it was tuned at (λ=6000, cut=3, k'=50, k=10) and say the depth has to grow with k and k'? Its "lower it to trade recall for speed" also points at a dominated setting: at matched recall, a slightly larger k' with a small R wins (NEON, T=48 at k'=60/R=75: 0.94× T=0 p50, −0.01 pp, vs 1.08× for k'=50/R=200).
There was a problem hiding this comment.
Done in ede29d1: the default is now kAutoRescoreDepth (0), which resolves to max(200, 3·k); an explicit value is used as given (still floored at k). The header comment names the operating point the constants were tuned at (T=48, λ=6000, cut=3, k'=50) and says the depth has to grow with k and k', and that a wider pool with a smaller depth can win. I didn't add a separate oversample field.
Measured on base_full T=48 at k=100 (cut=3, k'=50, 1 thread, mmap, warm, Xeon 8124M): top-100 overlap with the T=0 index goes 0.9827 → 0.9946 for 1.17× p50. k=10 is unchanged (still depth 200, identical labels).
| // is at least k; beyond the candidate pool it simply re-scores all of it. | ||
| const detail::FullVectorStore store = | ||
| rescore_needed ? full_vector_store() : detail::FullVectorStore{}; | ||
| const int rescore_depth = rescore_needed ? std::max(budget.rescore, k) : 0; |
There was a problem hiding this comment.
Issue: rescore is only checked for < 0, but it sizes the per-query TopKHolder holder(max(depth, k)), whose ctor does vec.reserve(depth) (ranker.h:52): 8 B × depth per query per thread, although the pool can never exceed Σ n_docs of the k' selected blocks (≤ k'·λ). k_prime gets min(k_prime, candidates.size()); rescore gets no clamp.
Results are identical at every depth, but past ~4M (glibc's 32 MiB mmap threshold) every query maps and unmaps the reserve (+13–47 µs/query single-threaded, and the munmaps serialize the threads). Where the 16 GiB request is refused (RAM+swap < 16 GiB under heuristic overcommit, overcommit_memory=2, ulimit -v), the bad_alloc is thrown inside the omp region → std::terminate (reproduced with ulimit -v 8000000: exit 134). The comment at :176 says a depth past the pool "simply re-scores all of it", and a JNI caller's natural "all" is Integer.MAX_VALUE (neural-search already saturates k' that way).
Clamp the depth to max(k, Σ dir_entry(pl, cid)->n_docs) over the selected blocks before building the holder (the directory is in RAM, so it's O(k')), or cap the reserve in TopKHolder? Either keeps "big value = everything" (the tests' 10^6) working, which rejecting above some N would not.
There was a problem hiding this comment.
Done in ede29d1: the depth is clamped to max(k, Σ n_docs of the selected blocks), read from the in-RAM directory, before the holder is sized, so INT32_MAX means "all". RescoreDepthPastThePoolReScoresAll checks INT32_MAX against 10^5.
| resolve_docs(rescore->store, result.second.data(), | ||
| result.second.size(), element_size, &rescore->scratch, | ||
| &rescore->slices); |
There was a problem hiding this comment.
Issue: none of the new throws on this path can reach a caller. block_budget_query runs inside search()'s #pragma omp parallel (disk_seismic_index_base.cpp:185), and an exception leaving an OpenMP region is std::terminate (also at OMP_NUM_THREADS=1), so SWIG's %exception and the JNI catch never see it. The new EXPECT_THROWs call resolve_doc/resolve_docs directly, so they pass while search() on the same bytes aborts.
The mechanism isn't new (view_block already aborts like this on main), but the triggers are: load_mapped_payload borrows the locator table after only a count check, and a truncated index now reads it on every query. On a T=8 file with one locator slot set out of range, read_index succeeds and the first plain unfiltered search dies with terminate called after throwing an instance of 'std::runtime_error' what(): DiskSeismic: doc locator does not resolve to its doc (exit 134). The same bytes in a T=0 file search fine. A block doc_id >= num_vectors, which main returns as a bogus label, now aborts too.
Could load_mapped_payload range-check the locators once (dir_entry(pl, block) non-null, slot < n_docs, remainder row < remainder_.num_vectors(); ~170 ms over 8.8M docs, since the directory is already in RAM)? And, here or as a follow-up for every search loop: catch inside the omp for body, keep the first std::exception_ptr, skip the remaining queries and rethrow after the region.
There was a problem hiding this comment.
Both, in ede29d1:
load_mapped_payloadrange-checks every locator once:dir_entrynon-null andslot < n_docs, or remainder row < the remainder count.- Both OpenMP loops (block-budget and
exact_match_directory) keep the first exception, skip the remaining queries and rethrow after the region (nsparse/utils/omp_exception.h).
Tests: ReadRejectsALocatorOutsideItsBlock, and SearchReportsALocatorNamingAnotherDoc (200-query batch on a swapped locator; with the capture removed it dies with terminate). A block doc_id >= num_vectors now surfaces as an exception too, instead of aborting.
There was a problem hiding this comment.
Thanks, verified: out-of-range slot, list, block and remainder-row locators now fail read_index (T=0 and T=8), and in a truncated file a swapped locator or a block doc_id ≥ n raises from search() on both the block and exact paths, at 1 and 8 threads — the previous head aborted (exit 134) or mis-ranked silently. Through neural-search's JNI it surfaces as java.lang.Exception and the process survives.
One correction to my ~170 ms estimate for the load-time check: that was warm. The 101 MiB locator table isn't copied to RAM; the scan reads it through the MADV_RANDOM mapping, so on a cold cache every 4 KiB page is a synchronous read. base_full (8.84M docs, Graviton3 + EBS), 84e0244 → ede29d1: warm load 0.27 → 0.51 s; with only the table evicted 0.27 → 15.3 s (+25,903 major faults); fully cold 79.4 → 94.5 s. On a plugin-written 4M-doc T=0 segment: 325 → 429 ms warm and +6.8 s with the table uncached, paid inside the first query on each segment. For T=0 and v1 files the scan buys nothing: their only locator reader is the ≤k exact path, where resolve_doc checks each locator it uses and now raises through the capture.
Could you (1) skip the scan when fwd_.max_doc_nnz() == 0 — T=0 then loads exactly like 84e0244, and a corrupt T=0 file still fails closed per query — and (2) for truncated files wrap it in madvise(MADV_SEQUENTIAL) over the table's pages, restoring MADV_RANDOM after (table-cold 15.3 → 1.3 s; a single MADV_WILLNEED over the range prefetches only 256 KiB on 5.10)? All 684 tests pass with both.
Two test gaps, cheap to close: removing the capture in exact_match_directory, or the remainder-row or entry == nullptr branch of validate_doc_locators, leaves every test green. Could SearchReportsALocatorNamingAnotherDoc also search with a SetIDSelector{victim} (the ≤k path a filtered plugin query takes; without that capture it aborts with exit 134), and ReadRejectsALocatorOutsideItsBlock also try {kRemainder, 0, 0} and a block past its list?
There was a problem hiding this comment.
Both, in bb1176d: the scan runs only when max_doc_nnz() > 0, under MADV_SEQUENTIAL (restored after; cold 101 MiB table here: 15.2 s → 0.42 s). The exact path gets a SetIDSelector{victim} search; the read test adds a block past its list and {kRemainder, 0, 0}. All three mutants now fail.
| continue; | ||
| } | ||
| const InlineDirEntry& entry = *scratch->entries[i]; | ||
| if (scratch->locators[i].slot >= entry.n_docs) { |
There was a problem hiding this comment.
Issue: resolve_docs isn't "Same results as calling resolve_doc in a loop, including the throws" (io.h:95-96). Stage 4 only bounds the slot and doc_slice reads only off[], so resolve_doc's doc_ids[slot] == doc_id check (:80-85) is gone from the re-scoring pass: a locator that names another doc's valid slot gets that doc's vector re-scored under the candidate's id with no error, while the exact path on the same file throws. Repro on a seeded T=8 disk_seismic file: with doc 29's locator patched to doc 0's slot in the same block, a query built from doc 0's vector returns [29, 0, 573, …] scored [10.3424, 10.3424, 2.2653, …] (29's true score is 1.1961).
Only a corrupt locator table gets there, but this is the only runtime check that the locators match the blocks, and doc_id[slot] sits in the block prefix stage 3 already prefetches. Pass the expected id into doc_slice and compare after its header-fit check (reading it earlier could run past entry.len on a corrupt n_docs), and add a swapped-locator case next to ResolveRejectsALocatorPastItsBlock (which only tries slot=1000)? It needs the omp-exception fix to surface as an error rather than an abort.
There was a problem hiding this comment.
Done in ede29d1: doc_slice takes the expected id and compares doc_id[slot] after its header-fit check, so resolve_docs keeps resolve_doc's identity check. Added ResolveRejectsALocatorNamingAnotherDoc (both resolve paths) next to the slot=1000 case, plus the index-level swapped-locator test above.
| // truncated inline forward index: a v2 file may hold partial copies that a | ||
| // v1 reader would score as if whole. Reading v1 stays supported (nothing is | ||
| // truncated there), so only new files need the newer build. | ||
| static constexpr uint32_t kFormatVersion = 2; |
There was a problem hiding this comment.
Suggestion: nothing pins "Reading v1 stays supported", and DEVELOPER_GUIDE asks for "a test that reads the old layout" on a bump. With throw_if_version_unsupported made to reject v1 DSEI/DSSQ, all 674 C++ tests stay green. Two cheap layers:
- Write T=0 DSEI/DSSQ,
patch_file_u32the version to 1, mmap-load,expect_same_resultsagainst the unpatched file. That pins the version check and any future version gating. - The v1 files actually in the field come from 2da4627 (neural-search 3.9.0,
forward_index: per_block→idmap,disk_seismic_sq,…) and are page-aligned, which this tree can no longer write at index level (write_indexhardcodeskPacked). A 400-doc × dim-64 fixture built there is 30 KB gzipped (192 page-aligned blocks, 53 remainder docs), and this PR returns exactly 2da4627's top-10 on it (unfiltered, exact-path and filtered). Check it in with 2da4627's results (there's no zlib here, so unpack in CMake or test from python_tests; compare modulo ties, 23/40 exact-path queries tie) and never regenerate it in place?
Also add DSEI/DSSQ toEachIndexTypeWritesItsCurrentFormatVersion, and update inline_forward_index.h:23 ("no cross-version guarantee") to the policy this PR commits to.
There was a problem hiding this comment.
There was a problem hiding this comment.
Thanks: the relabel test pins DSSQ v1 acceptance, and deferring the 2da4627 fixture and the EachIndexTypeWritesItsCurrentFormatVersion entry is fine with an issue (ideally before neural-search moves its submodule). Out of tree, ede29d1 reads 2da4627's page-aligned fixtures and a 200k-doc file its JNI build wrote exactly as 2da4627 does (32/32 checksums, 500/500 top-10 rows).
Two gaps — my earlier suggestion missed the first. A limit=0 build records max_doc_nnz 0, so its relabelled copy can't see the new v1 gate: with the format_version < 2 branch in load_mapped_payload deleted, all 686 C++ and 183 Python tests still pass. Relabelling a limit=4 build as well and expecting the same results at rescore=10 and rescore=INT32_MAX covers it (on a v1 file the depth must be a no-op): that passes here and differs on 20/20 queries with the branch deleted. And there's no DSEI case: with DiskSeismicIndex::mmap_index rejecting v1, every test still passes. Add both?
There was a problem hiding this comment.
There was a problem hiding this comment.
Nit: needs to update here.
There was a problem hiding this comment.
Updated in ede29d1: the comment now says the order matters on a truncated index, because the first copy visited keeps decides whether a doc enters the pool exact or as a lower bound.
| // it cannot pull in a doc the block scores ranked out, but it still replaces | ||
| // k lower bounds with k exact scores. Skipping it would hand back the lower | ||
| // bounds as if they were scores. | ||
| const bool two_phase = rescore != nullptr && rescore->depth >= k; |
There was a problem hiding this comment.
Nit: the comment just above says skipping the second pass "would hand back the lower bounds as if they were scores", and the header says a non-null rescore means a second pass is required, but two_phase is false for depth < k, so a RescorePass left at its default depth = 0 gets exactly those lower bounds back. search() already raises depth to ≥ k, so here the >= k is dead and the std::max a no-op. two_phase = rescore != nullptr with holder(std::max(rescore->depth, k)), so a short depth re-scores k instead?
There was a problem hiding this comment.
Done in ede29d1: two_phase = rescore != nullptr, and the holder is sized max(depth, k), clamped to the pool.
| throw std::invalid_argument( | ||
| "DiskSeismic index: k_prime (block budget) must be positive"); | ||
| } | ||
| // Checked even when the index is not truncated and the depth is unused, so |
There was a problem hiding this comment.
Nit: this runs after the empty-index return (:83) and the exact-match branch (:92), so "fails the same way" here and "rejected at search time" in disk_seismic_index.h:44 only hold on the block-budget path: rescore=-1 raises on an unfiltered query but is accepted silently with an enumerable selector of ≤ k ids or on an empty index (k_prime=0 has the same gap on main). Results are fine since neither value is read there, but a caller bug only shows on some queries. Hoist resolve_cut_and_budget and both checks to the top of search(), and add a ≤k-selector case to RejectsNegativeRescore?
There was a problem hiding this comment.
Done in ede29d1: resolving cut/budget and both checks now happen at the top of search(). RejectsNegativeRescore gained a ≤k-selector case.
| std::vector<float> score_scratch; | ||
| // Per-thread, so the batched whole-vector resolve reuses its buffers | ||
| // across the queries this thread handles. | ||
| detail::RescorePass rescore{.store = store, .depth = rescore_depth}; |
There was a problem hiding this comment.
Nit: with gcc 14 -Wall -Wextra -Wconversion -Wsign-conversion the changed nsparse/ TUs get two new warnings: -Wmissing-field-initializers here (scratch/slices left out of the designated init) and -Wsign-conversion on above += (*scratch)[i] > cut ? 1 : 0; (inline_forward_index_io.cpp:181). Also <stdexcept> is newly included in disk_seismic_search.cpp but nothing there throws. All harmless; .scratch = {}, .slices = {}, ? 1U : 0U, and drop the include?
There was a problem hiding this comment.
Fixed both warnings and dropped the stray <stdexcept> in ede29d1.
| size_t element_size); | ||
|
|
||
| // Scratch for the batched resolve below, owned by the caller and reused across | ||
| // queries so the walk allocates nothing. |
There was a problem hiding this comment.
Nit: neural-search calls search(1, ...) for every query, so this scratch and RescorePass (disk_seismic_index_base.cpp:195) are rebuilt per call and never reused on the plugin path (a T=48 query there makes 8 more allocations than T=0, 3 of them this walk). Harmless, but locators/entries never exceed kChunk=12, so std::array<…, kChunk> locals in resolve_docs would remove DocResolveScratch altogether; or reword to "within one search() call"?
A throw inside the search's OpenMP region was std::terminate, so a corrupt index aborted the process instead of raising. Both search loops now capture the first exception and rethrow it after the region. Locators are range-checked against the directory once at load, and doc_slice checks that the slot holds the doc being resolved, so a locator naming another doc's slot no longer re-scores that doc's vector under the wrong id. rescore defaults to kAutoRescoreDepth (0), which resolves to max(200, 3k): an absolute 200 left no oversampling from k=200 up and lost 0.8-3.5pp recall at k=100-200. A positive depth is still raised to k, and any depth is now capped at the docs the selected blocks hold, so INT32_MAX means "all" without reserving gigabytes per query. Parameter checks run before every early return. On the write side, a NaN weight sorts as -inf instead of breaking the selection's ordering, a limit no copy reaches records max_doc_nnz 0 (the file is then byte-identical to an untruncated one and search skips the second pass), and a v1 file's padding is never read as a limit. inline_max_nnz must parse as a whole decimal integer. The SQ search parameters take keyword arguments in Python again. Measured on base_full (8-bit, cut=3 k'=50, 1 thread, mmap, warm, 3 interleaved rounds, Xeon 8124M): labels are identical to the previous head at k=10 for T=0 and T=48, at ~1% p50 (not the exception capture, which a no-capture control shows is free). At k=100, T=48 now re-scores 300 instead of 200: top-100 overlap with the T=0 index goes 0.9827 -> 0.9946 for 1.17x p50. The load-time locator check adds the 101 MiB locator table to RssFile on an untruncated index (a truncated one already touches it). Tests that could not catch their own regressions are tightened (remainder docs, the below-pool Python case, idmap vs its delegate, the factory key), and new ones cover each fix plus a multi-chunk batch against one-at-a-time search. Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
|
Right, neural-search can't set either yet. |
zirui-song-18
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround. Most of round 1 is in and verified. Some points are left in thread.
| // truncated inline forward index: a v2 file may hold partial copies that a | ||
| // v1 reader would score as if whole. Reading v1 stays supported (nothing is | ||
| // truncated there), so only new files need the newer build. | ||
| static constexpr uint32_t kFormatVersion = 2; |
There was a problem hiding this comment.
Question: with inline_max_nnz unset — which is all neural-search can set today — the payload is still v1 but the header says 2. Seeded main and this-head builds (20k MS MARCO docs, the plugin's build description and read_csr_and_ids path) differ in one byte under cmp -l, the version (DSSQ, DSEI, and the nested header under idmap). 2da4627, the nsparse neural-search 3.9.0 ships and its main still pins, rejects them ("Unsupported DSSQ index format version 2"); relabelled 1, it loads them and returns identical labels and distances on all 9 search paths I ran. Not blocking — forward compat was never promised, and OpenSearch keeps old nodes off new segments — but once the submodule moves, nothing the plugin writes opens in 3.9's build.
Would you stamp 1 when inline_max_nnz_ == 0? That needs a write-side written_format_version() at index_io.cpp:182, since format_version() is also the newest version read_index accepts. It should return 1 only while format_version() == 2, with a test pinning T=0 → 1 and T>0 → 2 (this reader would silently misread a truncated file stamped 1). I tried it on this head: T=0 is byte-identical to main, T=8 is unchanged, and the only test to update is ReadsAFormatVersionOneFile, whose precondition expects T=0 to be stamped 2. If v2 everywhere is intended, could the description say that untruncated files need the newer build too?
There was a problem hiding this comment.
Yes, in bb1176d: write_index stamps IndexIO::written_format_version(), which the disk types make 1 when inline_max_nnz_ == 0 while format_version() == 2. StampsVersionOneUnlessTruncated pins T=0 → 1 and T>0 → 2 for both types, which also covers the EachIndexTypeWritesItsCurrentFormatVersion entry.
An index built without inline_max_nnz writes exactly the v1 payload but was stamped 2, so builds from before truncation (neural-search 3.9's) rejected it. write_index now stamps IndexIO::written_format_version(); the disk types return 1 when nothing is truncated, while 2 is current. A truncated file is still stamped 2, which a v1 reader rejects rather than score its partial copies as whole. The load-time locator scan now runs only on a truncated index, the one whose every query reads the table. An untruncated index reads it only on the <= k exact path, which checks each locator it resolves, so it loads as 84e0244 did and a corrupt locator still fails that search. On a truncated index the scan runs under MADV_SEQUENTIAL and then restores the mapping's hint: a cold sweep of a 101 MiB table on a local NVMe host goes from 15.2 s and 25,856 major faults to 0.42 s and 1. Tests: a truncated build relabelled v1 searches identically at any rescore depth (DSEI and DSSQ); each type's version stamp with and without truncation; the exact path reports a swapped locator instead of aborting; block-past-list and remainder-row locators fail the load. Each fails when the code it covers is removed. DISK_SEISMIC_BENCH.md states the rescore behind its cold and RSS numbers and fixes their units. Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
Description
Resolves #59. Each document is stored about 13 times in the inline forward index. A new build option,
inline_max_nnz=T, keeps one copy whole and only the T largest codes in the others. The query parameterrescore(default max(200, 3·k)) re-scores the top candidates against the whole copies. Off by default. Only truncated files are format version 2.Measured on
base_full, 8-bit, single thread, seeded builds:At T=48 and rescore=300, the cold first pass grows 28.5% and RSS 25%. The full curve is in
DISK_SEISMIC_BENCH.md.Check List
--signoff.