Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -589,7 +589,7 @@ Every serialized index starts with a fixed header (`nsparse::IndexHeader` in `ns

The payload follows immediately, and its layout is the index type's own business. A type parses it in `read_index` (copying), `mmap_index` (borrowed from a file mapping), or both — `DiskSeismicIndex` is mmap-only and its `read_index` just throws, while `IDMapIndex` has no `mmap_index` at all and instead threads `io_flags` down to its delegate.

Versions are numbered **per index type**, not per file: `IndexIO::format_version()` returns the type's own `kFormatVersion`, so revising one type's payload leaves the others' numbering alone. An `IDMapIndex` writes its own header for the id map and then a second, complete header for the delegate it wraps, each with its own version.
Versions are numbered **per index type**, not per file: `IndexIO::format_version()` returns the type's own `kFormatVersion`, so revising one type's payload leaves the others' numbering alone. A file is stamped with `IndexIO::written_format_version()`, which defaults to the same; a type can stamp an older version when that file's payload is still the older layout, so older builds keep reading it (the disk SEISMIC types write 1 when nothing is truncated). An `IDMapIndex` writes its own header for the id map and then a second, complete header for the delegate it wraps, each with its own version.

The same central code rejects a version outside `1..format_version()` before dispatching to either parse path — a file from a newer build fails with a clear error instead of consuming whatever its fields happen to align with.

Expand Down
12 changes: 12 additions & 0 deletions benchmarks/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,18 @@ target_link_libraries(index_size_stats PRIVATE
absl::flat_hash_map
)

# Recall of a narrower inline forward layout, simulated over an index already on
# disk. Not a google-benchmark target: it reports recall for a grid of candidate
# formats, and deliberately says nothing about latency.
add_executable(prune_sim prune_sim.cpp)
target_include_directories(prune_sim PRIVATE ${PROJECT_SOURCE_DIR})
target_link_libraries(prune_sim PRIVATE
${NSPARSE_BENCH_LIB}
OpenMP::OpenMP_CXX
absl::flat_hash_set
absl::flat_hash_map
)

# Peak-RSS build-memory driver for the term-batched build. Not a
# google-benchmark target: that harness reports throughput, and what matters here
# is the high-water mark of a single build, one configuration per process.
Expand Down
107 changes: 107 additions & 0 deletions benchmarks/DISK_SEISMIC_BENCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,113 @@ of the inline forward index and delta coding would take 29.5 → 19.4 GiB, but d
the critical path of scoring a block and measured a 1.5× warm-latency cost, so it was not
taken. 4-bit values would save 7.4 GiB and is lossy. `doc_id[]`/`off[]` are ~1% together.

## Truncating the extra inline copies — `inline_max_nnz`

Packing the blocks removed bytes nothing read. What remains is bytes that *are* read but are
**duplicated**: `index_size_stats` puts the inline forward index at 45.6 of the 50.0 GiB, and
a doc's vector is stored **12.97 times** over — once per posting list that retains it. One
shared copy per doc would take `comps[] + vals[]` from 44.28 GiB to 3.42. That duplication is
not waste, it is the whole design: it makes a block one contiguous read.

`inline_max_nnz=T` splits the difference. One copy of each doc stays whole (the one the
`DocLocator` directory already points at); every other copy keeps only its T largest codes.
Block scores are then **lower bounds**, so search re-scores its `rescore` best candidates
against the whole copies and re-ranks. Two knobs, one at build time and one per query:

| | T=0 (baseline) | T=64 | T=48 | T=32 |
|---|---|---|---|---|
| index file | 49.967 GiB | 27.667 GiB | **22.971 GiB** | 18.246 GiB |
| vs baseline | | −44.6% | **−54.0%** | −63.5% |
| inline forward index | 45.570 GiB | 23.27 GiB | 18.574 GiB | 13.850 GiB |
| stored nnz | 15.85 e9 | 7.78 e9 | 6.20 e9 | 4.51 e9 |
| cluster summaries | 4.297 GiB (8.6%) | 4.297 (15.5%) | 4.297 (18.7%) | 4.297 (23.6%) |
| build wall / peak RSS | 9:01 / 20.9 GiB | 11:20 / 20.9 | 11:12 / 20.9 | 8:36 / 20.9 |

Latency and recall, `base_full` / λ=6000 β=400 α=0.4 / 8-bit / cut=3 k'=50 k=10 / 1 thread,
`seismic-ec2`, one binary per round over all the files, caches dropped before each arm's first
pass. `seed=42` on every build, so recall differences are the format's and not k-means noise:

| arm | rescore | warm p50 | vs baseline | recall@10 | Δ recall |
|---|---|---|---|---|---|
| T=0 | — | 0.2157 ms | 1.00× | 0.926246 | — |
| T=64 | 10 | 0.1698 ms | 0.79× | 0.844728 | −8.2 pp |
| T=64 | 50 | 0.1899 ms | 0.88× | 0.924900 | −0.13 pp |
| T=64 | 100 | **0.2117 ms** | **0.98×** | 0.925989 | −0.03 pp |
| T=64 | 200 | 0.2565 ms | 1.19× | 0.926232 | −0.001 pp |
| T=64 | 300 | 0.2986 ms | 1.38× | 0.926203 | −0.004 pp |
| T=48 | none | 0.1569 ms | 0.73× | 0.784585 | −14.2 pp |
| T=48 | 100 | 0.2033 ms | 0.94× | 0.924484 | −0.18 pp |
| T=48 | 200 | **0.2470 ms** | **1.15×** | 0.925888 | −0.04 pp |
| T=48 | 300 | 0.2877 ms | 1.33× | 0.926089 | −0.02 pp |
| T=48 | 600 | 0.4046 ms | 1.88× | 0.926203 | −0.004 pp |
| T=32 | 100 | 0.1936 ms | 0.90× | 0.915115 | −1.11 pp |
| T=32 | 200 | 0.2359 ms | 1.09× | 0.922908 | −0.33 pp |
| T=32 | 300 | 0.2760 ms | 1.28× | 0.924699 | −0.15 pp |

Using it, from C++ or Python (the SWIG bindings expose the same names):

```python
index = nsparse.index_factory(dim, "disk_seismic_sq,quantizer=8bit|vmin=0|vmax=3|"
"lambda=6000|beta=400|alpha=0.4|inline_max_nnz=64")
# ... add, build, write_index, read_index(path, nsparse.kUseMmap) ...
params = nsparse.DiskSeismicSQSearchParameters(0.0, 3.0, 3, 50, 100) # vmin vmax cut k' rescore
params.rescore = 200 # per query; default nsparse.kAutoRescoreDepth = max(200, 3·k)
```

`inline_max_nnz` is fixed at build time and recorded in the file (as 0 when no copy was long
enough to cut). `rescore` is ignored on an untruncated index; the default, `kAutoRescoreDepth`
(0), resolves to max(200, 3·k), because the depth a query needs grows with k; an explicit value
is raised to k when smaller and capped at the pool the selected blocks hold; a negative one is
rejected.

Two points worth naming, both at this table's operating point only (warm, k=10, cut=3, k'=50,
λ=6000): at a larger k the depth has to grow with it, which is what the automatic default does.
**T=64 with rescore=100 is free** here: 44.6% off the file at 0.98× the
latency and a recall difference smaller than the ±0.1 pp a k-means seed causes on its own.
**T=48 with rescore=200** buys another 9 points of file size for 1.15× latency, still at
recall parity. Below T=48 recall starts to cost real ground, so T=32 is only worth it when disk
is the binding constraint.

Read the "none" row as the cost of the scan alone: with 39% of the nnz left inline, scoring the
candidate blocks is **0.73×** the baseline. Everything above that is the second pass, and it is
not arithmetic — at ~0.45 µs per re-scored doc it is ~1400 cycles to read 420 bytes, because
reaching one whole copy is a four-deep chase (locator → directory → block header → slice)
through tables far past any cache. That is why `rescore` is the latency knob and why
`resolve_docs` walks the candidates through each step together, in bounded chunks, rather than
one doc at a time: doing that took the per-doc cost from 0.68 µs to 0.45 µs (−34%) with
bit-identical results at every depth.

Two honest costs beyond latency, both from one cold run (caches dropped, all 6,980 queries) at
T=48 with `rescore=300`. Both grow with the depth:

- **Cold first pass** 897 s → 1,153 s (+28.5%). The file is half the size, but the second pass
reads whole copies from blocks the scan did not select: ~142 distinct blocks per query beyond
the scan's 50 at `rescore=300` (~93 at 200), so a cold run faults in more distinct pages even
though there are fewer bytes overall.
- **Resident set** 6.04 → 7.56 GiB (+25%), `RssFile` 5.71 → 7.23, for the same reason. At
`rescore=200`, the default at k=10, a page-exact replay gives `RssFile` 5.71 → 6.60 GiB
(+16%). The smaller file does *not* buy a smaller working set here; it buys disk.

What did **not** work, so nobody repeats it:

- **Choosing the kept components per block instead of per doc.** The hypothesis was that a
query selecting a block correlates with that block's summary, so keeping the components with
the largest `code × block-max` should rank better than the doc's own largest codes.
Simulated over the real index it is a wash — T=48 at depth 100 gives 0.9249 against 0.9246,
T=32 0.9172 against 0.9175. The block max is usually the doc's own value for the components
that matter, so the ordering barely moves. More fundamentally: which components a query will
hit is query information, and no static choice can carry it.
- **Bounding the second pass** so most candidates skip it. The dropped codes are all below the
doc's boundary code, which gives `truncated + boundary × query_L1` as an upper bound — two
orders of magnitude looser than the scores it would have to separate. It prunes nothing.

`prune_sim <index.dat> <queries.csr> <truth.txt> <k> <cut> <k'> <T,...> <R,...> [criterion]`
is what made this affordable: it reads an existing index and runs the real block-budget search
over a *simulated* truncated layout, so a whole (T, rescore) grid costs one pass instead of a
build per cell. Its T=0/R=0 row must reproduce the index's own measured recall — that is the
control, and it does (0.926103 against 0.926074 measured). Trust it for recall only; it
recomputes the truncation per query, so its timings mean nothing.

## Caveats

- **`base_small` is a smoke test, not a benchmark** — it fits in cache, so the
Expand Down
137 changes: 137 additions & 0 deletions benchmarks/index_size_stats.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ uint64_t vbyte_len(uint64_t value) {
return bytes;
}

// Retention checkpoints for the per-doc "keep the T largest codes" policy.
constexpr std::array<uint32_t, 6> kTopT = {16, 24, 32, 48, 64, 96};

struct Stats {
uint64_t n_blocks = 0;
// Doc slots, i.e. postings: a doc counts once per block that holds it.
Expand Down Expand Up @@ -98,8 +101,50 @@ struct Stats {
uint64_t blocks_docids_unsorted = 0;
uint64_t vals_nonzero = 0;
uint64_t vals_le15 = 0; // would fit 4 bits as stored

// Lossy levers. The code histogram gives every code-threshold cut at once
// (how many values, and how much of the L1 mass, sit at or below a code);
// kept_nnz/kept_mass give the per-doc top-T cut. Both weight a doc once per
// block that holds it, which is what the stored bytes do.
std::array<uint64_t, 256> code_hist{};
uint64_t mass_total = 0;
std::array<uint64_t, kTopT.size()> kept_nnz{};
std::array<uint64_t, kTopT.size()> kept_mass{};

// Distinct component ids per block, i.e. what a per-block dictionary could
// fold away: the ratio to nnz bounds any transposed/dictionary layout.
uint64_t distinct_comps = 0;
};

// The T largest codes of one doc, by descending-code walk over a 256-bin
// histogram: how many values that keeps (ties at the cut included, so >= T) and
// how much of the doc's L1 mass they carry. O(nnz + 256) per doc, no sort.
void size_doc_topt(const uint8_t* vals, uint32_t len, Stats* stats) {
std::array<uint32_t, 256> hist{};
uint64_t mass = 0;
for (uint32_t j = 0; j < len; ++j) {
hist[vals[j]] += 1;
mass += vals[j];
}
stats->mass_total += mass;
size_t next = 0;
uint64_t kept = 0;
uint64_t kept_mass = 0;
for (int code = 255; code >= 0 && next < kTopT.size(); --code) {
kept += hist[code];
kept_mass += static_cast<uint64_t>(hist[code]) *
static_cast<uint64_t>(code);
// Every checkpoint this bin reached takes the totals as they stand: a
// partial bin cannot be split without ordering the ties, and rounding up
// to the whole bin is what a writer keeping >= T values would store.
while (next < kTopT.size() && (kept >= kTopT[next] || code == 0)) {
stats->kept_nnz[next] += std::min<uint64_t>(kept, len);
stats->kept_mass[next] += kept_mass;
++next;
}
}
}

// Component ids of one doc: how much delta coding would buy, two ways, and
// whether they ascend (which is what makes the deltas small).
void size_doc_comps(const nsparse::term_t* comps, uint32_t len, Stats* stats) {
Expand Down Expand Up @@ -132,6 +177,22 @@ void size_doc_comps(const nsparse::term_t* comps, uint32_t len, Stats* stats) {
stats->c_comps_flagged += flagged;
}

// Distinct component ids in a block, counted with a stamp array indexed by
// component id: O(nnz), and no per-block clearing (the stamp is the block
// ordinal). `seen` must be sized to the term space.
uint64_t count_distinct(const BlockView& view, uint64_t total_nnz,
uint64_t stamp, std::vector<uint64_t>* seen) {
uint64_t distinct = 0;
for (uint64_t j = 0; j < total_nnz; ++j) {
const uint32_t comp = view.comps[j];
if (comp < seen->size() && (*seen)[comp] != stamp) {
(*seen)[comp] = stamp;
++distinct;
}
}
return distinct;
}

void accumulate(const BlockView& view, size_t element_size, uint64_t align,
Stats* stats) {
const uint64_t n_docs = view.n_docs;
Expand Down Expand Up @@ -194,6 +255,10 @@ void accumulate(const BlockView& view, size_t element_size, uint64_t align,
const uint8_t value = view.vals[j];
stats->vals_nonzero += value != 0 ? 1 : 0;
stats->vals_le15 += value <= kFourBitMax ? 1 : 0;
stats->code_hist[value] += 1;
}
for (uint32_t i = 0; i < view.n_docs; ++i) {
size_doc_topt(view.doc_vals(i, 1), view.nnz(i), stats);
}
}
}
Expand Down Expand Up @@ -260,6 +325,10 @@ int main(int argc, char** argv) {

const uint64_t align = fwd.page_size();
Stats stats;
// Stamped by block ordinal (from 1, so the zero-initialized state is "never
// seen"), which is why the per-block distinct count needs no clearing.
std::vector<uint64_t> seen(static_cast<size_t>(dimension) + 1, 0);
uint64_t stamp = 0;
for (uint32_t pl = 0; pl < fwd.num_lists(); ++pl) {
const uint64_t n_blocks = fwd.num_blocks_in_list(pl);
for (uint32_t block = 0; block < n_blocks; ++block) {
Expand All @@ -268,6 +337,8 @@ int main(int argc, char** argv) {
continue;
}
accumulate(view, element_size, align, &stats);
stats.distinct_comps +=
count_distinct(view, view.offsets[view.n_docs], ++stamp, &seen);
}
}
stats.b_dir = stats.n_blocks * sizeof(nsparse::detail::InlineDirEntry);
Expand Down Expand Up @@ -323,7 +394,73 @@ int main(int argc, char** argv) {
saving("vals: 4-bit codes", stats.b_vals, stats.nnz / 2);
}

// Fewer stored values, rather than fewer bits per value: comps + vals are
// ~3 bytes of every nnz, so dropping an nnz saves all three and also drops
// the dot-product term that reads it. Both cuts are lossy -- the sizes here
// say what they would buy, not whether recall survives them.
if (element_size == 1) {
const double all_nnz = static_cast<double>(stats.nnz);
const double all_mass = static_cast<double>(stats.mass_total);
std::printf(
"\n== dropping low-weight components (lossy; comps+vals = %.1f%% "
"of the file) ==\n",
100.0 * static_cast<double>(stats.b_comps + stats.b_vals) /
static_cast<double>(file_bytes));
std::printf(" keep code > threshold:\n");
uint64_t at_or_below = 0;
uint64_t mass_at_or_below = 0;
for (uint32_t code = 0; code < 256; ++code) {
at_or_below += stats.code_hist[code];
mass_at_or_below +=
stats.code_hist[code] * static_cast<uint64_t>(code);
// A few decades of the curve is enough to read its shape.
if (code != 0 && code != 1 && code != 2 && code != 4 &&
code != 8 && code != 16 && code != 32 && code != 64) {
continue;
}
const double kept = all_nnz - static_cast<double>(at_or_below);
std::printf(
" > %3u keep %5.1f%% of nnz, %5.1f%% of L1 mass "
"file ~%7.3f GiB\n",
code, 100.0 * kept / all_nnz,
100.0 * (all_mass - static_cast<double>(mass_at_or_below)) /
all_mass,
gib(file_bytes) -
(gib(stats.b_comps + stats.b_vals) *
(static_cast<double>(at_or_below) / all_nnz)));
}
std::printf(" keep the T largest codes per doc:\n");
for (size_t i = 0; i < kTopT.size(); ++i) {
const double kept = static_cast<double>(stats.kept_nnz[i]);
std::printf(
" T = %3u keep %5.1f%% of nnz, %5.1f%% of L1 mass "
"file ~%7.3f GiB\n",
kTopT[i], 100.0 * kept / all_nnz,
100.0 * static_cast<double>(stats.kept_mass[i]) / all_mass,
gib(file_bytes) -
(gib(stats.b_comps + stats.b_vals) * (1.0 - kept / all_nnz)));
}
// A deduplicated store costs one copy of the corpus; the inline layout
// stores one per block that holds the doc.
std::printf(
" one shared copy per doc instead of %.2f inline copies: "
"%.3f -> %.3f GiB for comps+vals\n",
static_cast<double>(stats.n_docs) /
static_cast<double>(num_vectors),
gib(stats.b_comps + stats.b_vals),
gib(stats.b_comps + stats.b_vals) /
(static_cast<double>(stats.n_docs) /
static_cast<double>(num_vectors)));
}
Comment on lines +445 to +454

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue: (comps+vals) / (n_docs / num_vectors) is exact only if copy count doesn't correlate with doc length, but long docs sit in more posting lists: at T=0 base_full stores 15.85e9 nnz in 12.97 × 8.84M copies, 138 nnz/copy vs 126.8/doc. One copy is the corpus, 1,121,199,371 nnz × 3 B = 3.13 GiB, not the 3.42 at DISK_SEISMIC_BENCH.md:115 (by bytes the inline index holds ~14.1 copies; 12.97 is the count). The T = … rows (L432) have a similar gap: they cut every copy to ≥T and keep no whole copy, so on the seed=42 T=0 file the T=48 row prints ~21.22 GiB where the T=48 build is 22.971 GiB (6.197e9 stored nnz, 39.1%). Tracking first sight with a std::vector<bool>(num_vectors) in the existing (pl, block) loop — full nnz the first time, min(T, nnz) after — reproduces the written nnz exactly (6,197,434,818 / 7,877,959,871 at T=48/64), and the same sum gives the true one-copy size. Could it also print fwd.max_doc_nnz() (nothing in the output says a file is truncated, and on a truncated file the lossy section runs over copies already cut) and skip the T rows ≥ it?


std::printf("\n== diagnostics ==\n");
std::printf(" distinct comps per block: %.1f of %.1f nnz (%.2fx)\n",
static_cast<double>(stats.distinct_comps) /
static_cast<double>(stats.n_blocks),
static_cast<double>(stats.nnz) /
static_cast<double>(stats.n_blocks),
static_cast<double>(stats.nnz) /
static_cast<double>(stats.distinct_comps));
std::printf(" docs with non-ascending comps: %llu / %llu\n",
static_cast<unsigned long long>(stats.docs_comps_unsorted),
static_cast<unsigned long long>(stats.n_docs));
Expand Down
Loading
Loading