Repository navigation
[FEATURE REQUEST] Add ADBC (Arrow Database Connectivity) Data Source #54603
Description
Add a native ADBC (Arrow Database Connectivity) data source to Spark, similar in spirit to the existing JDBC data source but built on the Arrow-native ADBC API.
ADBC is a database connectivity API standard under the Apache Arrow project. It provides a vendor-neutral, columnar alternative to JDBC/ODBC specifically designed for analytical workloads. ADBC drivers return result sets as streams of Arrow data rather than row-by-row, which eliminates expensive row-to-columnar conversions. Since spark itself is row-based, the effect is not as dramatic, but still noticeable.
Why (now):
- There are mature native drivers for PostgreSQL, SQLite, DuckDB, Flight SQL, Snowflake, BigQuery, MySQL, SQL Server, Databricks and so on. It's also very easy to install (and locate) them on a system with dbc cli tool.
- There is now good support for invoking ADBC from Java via JNI bindings to the C++ ADBC driver manager (see blog). This makes it practical to integrate ADBC into Spark's JVM-based architecture. Technically drivers can be implemented in java as well, but the quality of java implementations is pretty low, realistically one will almost almost use a native driver.
- ADBC fits well with spark's columnar read support in data source v2. Generating ArrowColumnVectors from adbc is pretty straightforward. It can be a benefit for external spark accelerators like comet and (presumably photon).
I have a proof-of-concept implementation at spark-adbc that demonstrates the basic read path and not so scientific benchmarks vs jdbc. I'm willing to incrementally implement ADBC data source support upstream if there's interest from the community.
Activity
Vishwaspatel2401 commentedon Mar 5, 2026
HI @tokoko I would like to take a look at this
HyukjinKwon commentedon Mar 6, 2026
Spark can take Arrow batches directly. I think this is a good idea to have one but I think this would need an SPIP (https://spark.apache.org/improvement-proposals.html).
tokoko commentedon Mar 6, 2026
@HyukjinKwon sure, I can start working on drafting a document and share it here first before sending out on a mailing list. thanks!!
holdenk commentedon Mar 6, 2026
I think @devin-petersohn already did some of this too, would be good to sync.
HyukjinKwon commentedon Mar 7, 2026
cc @lidavidm and @alinaliBQ FYI
devin-petersohn commentedon Mar 9, 2026
Happy to help/review!
HyukjinKwon commentedon Jun 15, 2026
FYI, I created https://github.com/HyukjinKwon/adbc-driver-spark that is ADBC with Spark Connect (not the datasource).
tokoko commentedon Jun 15, 2026
@HyukjinKwon I think Columnar's also working on something similar.
Another FYI, I have two arrow PRs up that are "soft" prerequisites for the data source. please, do chime in.
lidavidm commentedon Jun 15, 2026
(That would be https://github.com/adbc-drivers/spark. We would be interested in collaborating on an ADBC driver for Apache Spark, or talking about upstreaming it if there's interest!)
holdenk commentedon Sep 24, 2026
I'm also sort of interested in this, I've got an idea on how we could do something fun here to allow for parallel direct reads from execs too.
holdenk commentedon Sep 24, 2026
I agree though we probably need an SPIP (currently those are structured more around JIRA though)
tokoko commentedon Sep 24, 2026
@holdenk hey, I have a draft of the SPIP document in the repo. Should I turn it into a Google doc and we can collaborate there? I suppose a JIRA/formal proposal stuff can wait, but I can start on that as well if you think that's necessary.
I had two "types" of read parallelism in mind:
AdbcStatement.executePartitions(Driver here refers to adbc driver, not the spark driver) - Spark callsexecutePartitionsduringplanInputPartitions, adbc driver hands back opaque partition descriptors, one per Spark task.partitionColumn/lowerBound/upperBound/numPartitionsoptions.There's no reason why they can't be used together as well. Is that along the lines of what you were thinking?
holdenk commentedon Sep 24, 2026
That is! And yes google doc is a good way to collaborate.
tokoko commentedon Sep 24, 2026
trimmed the text down a bit and moved to a Google doc: SPIP: ADBC Data Source.
holdenk commentedon Oct 10, 2026
Thanks @tokoko , by any chance are you at community over code in Glasgow this week?
tokoko commentedon Oct 10, 2026
No, unfortunately I'm not 😔 I'll DM you on LinkedIn if you prefer to discuss anything more interactively.
HyukjinKwon commentedon Oct 11, 2026
Let's discuss here for better visibility :-).
holdenk commentedon Oct 11, 2026
I shared the Google doc with a few of the ADBC folks at Community over Code so hopefully we'll get some comments on the doc.