Skip to content

feat: push zonemap ranges into fragment scans - #807

Draft
majin1102 wants to merge 4 commits into
mainfrom
codex/zonemap-fragment-slices
Draft

feat: push zonemap ranges into fragment scans#807
majin1102 wants to merge 4 commits into
mainfrom
codex/zonemap-fragment-slices

Conversation

@majin1102

Copy link
Copy Markdown
Contributor

Summary

  • plan physical row ranges from zonemap statistics on the driver
  • attach merged FragmentSlice ranges to single-fragment Lance splits
  • pass slices through both ordinary columnar scans and pushed COUNT(*) scans
  • conservatively fall back to full-fragment scans for unsupported predicates, incomplete coverage, or invalid metadata
  • preserve snapshot isolation by planning zonemap ranges and fragments from the same opened dataset version
  • disable fragment row-count limit pruning when a split carries physical slices

Validation

  • ./mvnw -Dmaven.repo.local=/tmp/lance-fragment-slice-m2 spotless:check
  • ./mvnw -Dmaven.repo.local=/tmp/lance-fragment-slice-m2 compile -pl lance-spark-3.5_2.13 -am
  • ./mvnw -Dmaven.repo.local=/tmp/lance-fragment-slice-m2 test -pl lance-spark-3.5_2.13 -am
    • 1401 tests, 0 failures, 0 errors
  • focused FragmentSlice planner tests, including multi-fragment and multi-slice planning
    • 11 tests, 0 failures

Dependency / draft status

Depends on lance-format/lance#8951 for the Java/JNI FragmentSlice scanner API.

The branch currently targets lance-core:12.0.0-beta.11, built locally from that PR into an isolated Maven repository. Keep this PR as draft until the Lance change is merged and a public artifact is available; then replace the dependency version in a follow-up commit.

Scope

This PR only implements ZoneMap to FragmentSlice scan pruning. It does not introduce distributed BTree/FTS index planning, index scan tasks, or plan_splits.

@github-actions github-actions Bot added the enhancement New feature or request label Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant