feat: add task commit factory for Apache Gluten writes - #715
Draft
malinjawi wants to merge 1 commit into
Draft
Conversation
Contributor
|
ACTION NEEDED The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
malinjawi
changed the title
yanghua
pushed a commit
that referenced
this pull request
Sep 2, 2026
## What Adds `LanceArrowStreamScanner`, which plans a Lance fragment scan and exports it as an Arrow C Data Interface stream (`ArrowArrayStream`) for native consumers such as Apache Gluten / Velox. ## Why This is the read-side building block for offloading Lance scans to a native engine. Only the `ArrowArrayStream` C-struct address crosses the JVM/native boundary, so the consumer's Arrow build and classloader do not need to match lance-spark's — Gluten builds Arrow 15 to match Spark 3.5 / Velox, while the Lance Java SDK is on Arrow 18. Passing the raw struct address sidesteps that mismatch entirely. ## How - `LanceArrowStreamScanner.export(fragmentId, inputPartition)` returns a `LanceArrowStream` handle exposing `stream()` / `streamAddress()`. - All scan planning (column projection, filter pushdown, limit/offset, row-id / row-address, batch size) is delegated to the existing `LanceFragmentScanner`, so the exported stream yields exactly the same rows in the same order as the Spark columnar reader. - The Lance native core populates the caller-owned stream directly via `LanceScanner#exportArrowStream(long)` (lance-format/lance#7259), so no Arrow data is materialized on the JVM heap on this path. - `LanceArrowStream` owns the exported stream and the scan behind it; closing it releases the native scan (through the stream's release callback) and then the scanner and dataset handles. ## Dependency `exportArrowStream(long)` first ships in lance-core `11.0.0-beta.21`, so this bumps `lance.version` from `11.0.0-beta.10` (isolated in its own commit). ## Testing - New `LanceArrowStreamScannerTest`: exports each fragment of the bundled test table, re-imports it on the JVM (standing in for a native consumer), and asserts the rows match the columnar reader — run under the leak-checking allocator to verify the export/scanner lifecycle releases cleanly. - Existing `LanceFragmentColumnarBatchScannerTest` still passes (regression on the touched `LanceFragmentScanner`). ## Related - Complements #715 (Gluten write path). - Distinct mechanism from #624 / #623 (serializable native scan descriptor for datafusion-comet): this exposes a live Arrow C stream handle rather than a re-plannable descriptor. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LanceBatchWrite.taskCommit(...)for external writers such as Apache GlutenTaskCommitopaqueDiscussion: #709
Testing
mvn -pl lance-spark-base_2.12 testmake install-base SCALA_VERSION=2.13