build: exclude unused arrow-dataset native from shaded bundle - #746
build: exclude unused arrow-dataset native from shaded bundle#746sezruby wants to merge 2 commits into
Conversation
github-actions
Bot
added
the
chore
The arrow-dataset artifact (and its bundled native library) is copied and shaded into every connector/bundle jar but is not used at runtime. Drop it from the dependency-unpack includeArtifactIds across all modules and exclude org.apache.arrow:arrow-dataset from the shade artifactSet in the bundles, so the native library is no longer packaged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
332e86f to
6e658ea
Compare
|
Rebased onto latest |
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
There was a problem hiding this comment.
❌ Gate recommendation: request changes.
The size reduction is worthwhile, but the bundle must retain the Arrow Dataset Java API that lance-core’s scanner links against. Keep those class files in the shaded and copied-JAR assemblies while filtering only the platform JNI binaries; that removes the unused payload without breaking Spark read paths.
lance-gatekeeper
Bot
added
the
K-changes
Excluding the whole org.apache.arrow:arrow-dataset artifact breaks the Spark read path: lance-core's org.lance.ipc.LanceScanner implements org.apache.arrow.dataset.scanner.Scanner, so loading it fails with NoClassDefFoundError once the arrow-dataset classes are gone. Keep arrow-dataset in copy-dependencies across all modules and, in the shaded bundles, strip only the unused arrow_dataset_jni native (~50-77 MB/platform) via a shade filter while retaining the Java API classes. Runtime native interop goes through the Arrow C Data Interface (arrow-c-data), not this native. Add failsafe integration tests on lance-spark-bundle-3.5_2.13 that run against the shaded jar: assert the native is stripped and the API classes are kept, that LanceScanner links, plus a real write/read smoke in a child JVM proving the packaging is functionally correct. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@fangbo thanks for the approval — this is still mergeable on latest The functional checks are green: Could you re-run the integration CI and merge once it is green? I don't have permission to re-trigger the run myself. |
2 participants
What
Excludes the
org.apache.arrow:arrow-datasetartifact (and its bundledarrow_dataset_jninative) from the shaded Spark bundles.The Arrow Dataset native is currently packaged into every shaded bundle, but it is never used at runtime. Runtime data exchange with the native library goes through the Arrow C Data Interface (
org.apache.arrow.c, from the separatearrow-c-dataartifact). Thearrow-datasetnative adds roughly 50-77 MB uncompressed per platform to the published bundle.Measured on the
linux-x86-64artifact, excluding it saved ~22 MB compressed / ~77 MB uncompressed, with no functional change.Changes
<exclude>org.apache.arrow:arrow-dataset</exclude>to themaven-shade-pluginartifactSetexcludes in each bundle module.arrow-datasetfrom themaven-dependency-plugincopy-dependenciesincludeArtifactIdspackaging lists.arrow-datasetdependency declaration itself is left untouched so it remains available on the test classpath.Why safe
org.apache.arrow.datasetanywhere in the codebase is a single test (lance-spark-base_2.12/.../write/LanceBatchWriteTest.java); there are no references in anysrc/main.src/mainusesorg.apache.arrow.c(the C Data Interface) for native interop.arrow-datasetandarrow-c-dataare separate Maven artifacts with separate native libraries, andarrow-c-datahas no dependency onarrow-dataset— so removing the dataset native does not affect the C Data Interface path.arrow-datasetdependency is retained (not removed), so the existing test continues to compile and run.