Skip to content

feat: report index coverage, segment count and size in SHOW INDEXES - #794

Open
ivscheianu wants to merge 3 commits into
lance-format:mainfrom
ivscheianu:feat/show-indexes-coverage-columns
Open

feat: report index coverage, segment count and size in SHOW INDEXES#794
ivscheianu wants to merge 3 commits into
lance-format:mainfrom
ivscheianu:feat/show-indexes-coverage-columns

Conversation

@ivscheianu

@ivscheianu ivscheianu commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Part of #789.

What

SHOW INDEXES reports indexed and unindexed fragment and row counts, but leaves the operator to compute coverage by hand and says nothing about how the index is laid out or what it costs. This adds three columns:

Column Type Description
indexed_percent double Share of rows covered, truncated to two decimals so it never overstates. Null for an empty table, where coverage is undefined rather than complete.
num_segments long Physical segments backing the logical index.
size_bytes long Total across the segments' files. Null if any segment predates index file size tracking, since a partial sum would understate rather than admit it is unknown.

All three come from metadata the command already fetches, so there is no extra driver cost. Truncating rather than rounding matters: a table one row short of full coverage reads as 99.99, not 100.

Change

indexed_percent is derived from the row counts already read. size_bytes needs the whole segment list rather than its first element, so the grouping keeps it instead of calling .head immediately; output is still one row per logical index. num_segments reads the statistics blob, falling back to num_indices for older cores that report only that key.

The docs note two things an operator needs and would otherwise guess wrong:

  • Partial coverage is not merely slower. A partially covered zonemap index prunes the fragments it does not cover, so a predicate on the indexed column can return fewer rows than the table holds, while COUNT(*) over the table stays correct. It comes from the connector's own driver-side pruning, which builds its surviving fragment set purely from the fragments the committed zones name, with no gate on coverage. Reproducible with CREATE INDEX plus an append, so it predates this change, but this column is what makes it visible. btree, bitmap and bloomfilter return complete results while partially covered.

    Edit: fixed by fix(read): keep fragments the zonemap does not cover #781.

  • A vector index is listed here too, but Spark SQL can neither create nor refresh one; it is maintained through the Lance SDK.

SHOW INDEXES already told an operator how many rows an index covers and how
many it misses, but left the ratio to be worked out by hand -- and said nothing
at all about how the index is laid out on disk. Both matter for deciding
whether an index needs rebuilding, and both are already in the metadata this
command fetches, so exposing them costs no extra driver work.

`indexed_percent` is truncated rather than rounded so it can never overstate
coverage: a table one row short of fully indexed reads 99.99, not 100. An empty
table reports null rather than 100, because an index covering nothing is not up
to date. This is the column that matters most in practice -- on the pinned Lance
version a partially covered `zonemap` index prunes the fragments it does not
cover, so a predicate on the indexed column silently returns fewer rows than the
table holds while COUNT(*) over the same table stays correct.

`num_segments` and `size_bytes` describe the physical layout behind one logical
index. Because queries search every segment, and Lance can only compact
fragments covered by an identical set of segments, a high segment count costs
both query time and OPTIMIZE's ability to coalesce. Computing `size_bytes` needs
the whole segment list, so the grouping now keeps every segment per name instead
of dropping to the first one; when any segment predates index file size tracking
the total is null, since a partial sum would understate the index rather than
admit the number is unknown.
lance-gatekeeper[bot]

This comment was marked as outdated.

The zonemap partial-coverage bug was fixed by lance-format#781, so the documentation
should not warn about it. Also trim inline comments and test Javadocs to
match the code, not the PR history.
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The latest cleanup is behavior-preserving: the inferred boxed values and imported test symbols compile under both Scala 2.12 and 2.13. The implementation still derives coverage from existing metadata, preserves one row per logical index, and reports NULL rather than understating unknown size; #781 remains merged, so the documentation correction is also current.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant