feat: report index coverage, segment count and size in SHOW INDEXES - #794
Open
ivscheianu wants to merge 3 commits into
Open
feat: report index coverage, segment count and size in SHOW INDEXES#794ivscheianu wants to merge 3 commits into
ivscheianu wants to merge 3 commits into
Conversation
SHOW INDEXES already told an operator how many rows an index covers and how many it misses, but left the ratio to be worked out by hand -- and said nothing at all about how the index is laid out on disk. Both matter for deciding whether an index needs rebuilding, and both are already in the metadata this command fetches, so exposing them costs no extra driver work. `indexed_percent` is truncated rather than rounded so it can never overstate coverage: a table one row short of fully indexed reads 99.99, not 100. An empty table reports null rather than 100, because an index covering nothing is not up to date. This is the column that matters most in practice -- on the pinned Lance version a partially covered `zonemap` index prunes the fragments it does not cover, so a predicate on the indexed column silently returns fewer rows than the table holds while COUNT(*) over the same table stays correct. `num_segments` and `size_bytes` describe the physical layout behind one logical index. Because queries search every segment, and Lance can only compact fragments covered by an identical set of segments, a high segment count costs both query time and OPTIMIZE's ability to coalesce. Computing `size_bytes` needs the whole segment list, so the grouping now keeps every segment per name instead of dropping to the first one; when any segment predates index file size tracking the total is null, since a partial sum would understate the index rather than admit the number is unknown.
6 tasks
The zonemap partial-coverage bug was fixed by lance-format#781, so the documentation should not warn about it. Also trim inline comments and test Javadocs to match the code, not the PR history.
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The latest cleanup is behavior-preserving: the inferred boxed values and imported test symbols compile under both Scala 2.12 and 2.13. The implementation still derives coverage from existing metadata, preserves one row per logical index, and reports NULL rather than understating unknown size; #781 remains merged, so the documentation correction is also current.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #789.
What
SHOW INDEXESreports indexed and unindexed fragment and row counts, but leaves the operator to compute coverage by hand and says nothing about how the index is laid out or what it costs. This adds three columns:indexed_percentnum_segmentssize_bytesAll three come from metadata the command already fetches, so there is no extra driver cost. Truncating rather than rounding matters: a table one row short of full coverage reads as 99.99, not 100.
Change
indexed_percentis derived from the row counts already read.size_bytesneeds the whole segment list rather than its first element, so the grouping keeps it instead of calling.headimmediately; output is still one row per logical index.num_segmentsreads the statistics blob, falling back tonum_indicesfor older cores that report only that key.The docs note two things an operator needs and would otherwise guess wrong:
Partial coverage is not merely slower. A partially covered
zonemapindex prunes the fragments it does not cover, so a predicate on the indexed column can return fewer rows than the table holds, whileCOUNT(*)over the table stays correct. It comes from the connector's own driver-side pruning, which builds its surviving fragment set purely from the fragments the committed zones name, with no gate on coverage. Reproducible withCREATE INDEXplus an append, so it predates this change, but this column is what makes it visible.btree,bitmapandbloomfilterreturn complete results while partially covered.A vector index is listed here too, but Spark SQL can neither create nor refresh one; it is maintained through the Lance SDK.