You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[FEATURE] Shrink the on-disk size of the DiskSeismic indexes #59
A disk_seismic(_sq).dat runs several times the size of its corpus, much of it carrying no
information. The inline forward index dominates; duplicating each document's vector per block is
the design, not the issue. On top of that:
Every block is padded to a page (kPageAligned, kDefaultPageSize = 4096 — io/inline_forward_index_io.h:82,92) yet holds roughly ten documents at standard lambda/beta, and a query never reads the padding. InlineLayout::kPacked already exists
(inline_forward_index.h:46); the header records the alignment.
[u16 comps[total_nnz]] (:29) is flat, though a document's ids ascend and so compress.
doc_id[], off[] (:113-114) are uint32_t, beyond what a block needs.
Proposal
Default to kPacked: no new encoding, no format bump.
Add byte accounting, so later steps are sized rather than guessed.
Then evaluate separately, each kept only if latency holds: delta-coded ids, narrower off[],
4-bit values.
Acceptance: smaller .dat, recall and latency unchanged.
Activity
chishui commentedon Sep 24, 2026
resolved by #60