TDengine Spark Dialect

English | 简体中文

Table of Contents

• 1. Introduction
• 2. Documentation
• 3. Prerequisites
• 4. Building
• 5. Testing
  • 5.1 Test Execution
  • 5.2 Test Case Addition
  • 5.3 Performance Testing
• 6. Packaging
• 7. CI/CD
• 8. Submitting Issues
• 9. Submitting PRs
• 10. References
• 11. License

1. Introduction

tdengine-spark-dialect is the official Apache Spark JDBC dialect for TDengine. It enables Spark's JDBC data source to read from and write to TDengine with the correct data type mappings, so that TDengine-specific types such as NCHAR and JSON work out of the box.

Features:

• Supports the WebSocket connection mode of the TDengine JDBC driver (jdbc:TAOS-WS://), the recommended connection mode for TDengine 3.x.
• Maps TDengine column types to Spark SQL types, e.g. NCHAR → StringType, JSON → StringType.
• Maps Spark SQL types to TDengine column types when Spark creates tables, e.g. StringType → VARCHAR(4096), DateType → TIMESTAMP.
• Quotes identifiers with backticks and provides a TDengine-compatible table-existence check.
• Compatible with Spark's V2 JDBC aggregate and GROUP BY pushdown (COUNT/SUM/AVG/MIN/MAX) when the pushDownAggregate data source option is enabled.

2. Documentation

• To learn how an application introduces the TDengine JDBC driver that this dialect builds upon, please check the Developer Guide.
• For the JDBC driver reference (data types, connection parameters, FAQs), please check the Reference Manual.
• For Spark's JDBC data source options (dbtable, partitionColumn, fetchsize, etc.), please check the Spark SQL Data Sources documentation.
• This quick guide is mainly for developers who like to contribute, build, and test the Spark dialect by themselves. To learn about TDengine, you can visit the official documentation.

Installation

The dialect is available on Maven Central. Declare both the dialect and the TDengine JDBC driver — the driver is not a transitive dependency of the dialect and must be added explicitly:

<dependency>
    <groupId>com.taosdata.spark</groupId>
    <artifactId>tdengine-spark-dialect</artifactId>
    <version>1.0.0</version>
</dependency>
<dependency>
    <groupId>com.taosdata.jdbc</groupId>
    <artifactId>taos-jdbcdriver</artifactId>
    <version>3.9.2</version>
</dependency>

Or hand both artifacts to Spark directly:

spark-submit --packages com.taosdata.spark:tdengine-spark-dialect:1.0.0,com.taosdata.jdbc:taos-jdbcdriver:3.9.2 ...

You can also download the jars from GitHub Releases and put them on the Spark classpath (spark-submit --jars ... or Spark's jars/ directory).

Usage

Spark does not auto-discover JDBC dialects, so the dialect must be registered once on the driver before running JDBC queries:

import com.taosdata.spark.TDengineDialect;
import org.apache.spark.sql.jdbc.JdbcDialects;

JdbcDialects.registerDialect(new TDengineDialect());

Reading from TDengine:

Dataset<Row> df = spark.read()
        .format("jdbc")
        .option("url", "jdbc:TAOS-WS://127.0.0.1:6041/")
        .option("dbtable", "test.meters")
        .option("user", "root")
        .option("password", "taosdata")
        .load();

Writing to TDengine (append into an existing table):

Properties props = new Properties();
props.setProperty("user", "root");
props.setProperty("password", "taosdata");

df.write()
        .mode("append")
        .jdbc("jdbc:TAOS-WS://127.0.0.1:6041/", "test.meters", props);

Known Limitations

• When Spark creates a table (write with append/overwrite to a non-existing table), all fields of the DataFrame schema must be nullable, because TDengine has no NOT NULL constraint syntax. The first column must also be a TIMESTAMP, as required by TDengine.
• The truncate write option is not supported; TDengine has no TRUNCATE TABLE statement. Use overwrite mode instead, which drops and recreates the table.
• Writing Spark ByteType/ShortType values into TINYINT/SMALLINT columns requires TDengine server >= 3.4.1.13. Spark binds both types via setInt, and only the stmt2 bind path used with server >= 3.4.1.13 converts the value; with older servers the driver's legacy row-bind path fails with a ClassCastException. All other types are unaffected.

3. Prerequisites

System Requirements

• JDK >= 8
• Maven >= 3.6
• Apache Spark 3.3.x at runtime (this project builds against Spark 3.3.2 / Scala 2.13)
• TDengine 3.x server
• com.taosdata.jdbc:taos-jdbcdriver on the Spark classpath at runtime

Installing Build Tools

Ubuntu/Debian:

sudo apt-get update && sudo apt-get install -y openjdk-8-jdk maven

CentOS/RHEL:

sudo yum install -y java-1.8.0-openjdk-devel maven

Local Test Environment

• TDengine has been deployed locally. For specific steps, please refer to Deploy TDengine. Please make sure taosd and taosAdapter have been started; the integration tests connect via WebSocket on port 6041.

4. Building

git clone https://github.com/taosdata/tdengine-spark-dialect.git
cd tdengine-spark-dialect
mvn clean package -Dmaven.test.skip=true

Output: target/tdengine-spark-dialect-<version>.jar

5. Testing

5.1 Test Execution

Execute mvn test in the project directory to run the tests.

• Unit tests cover the dialect class itself (URL handling, identifier quoting, type mappings) and need no database.
• Integration tests start a local-mode SparkSession, then read from and write to the local TDengine server through the WebSocket connection (jdbc:TAOS-WS://127.0.0.1:6041). They are skipped automatically when the server is not reachable. To test against a different TDengine endpoint, override the URL with -Dtdengine.ws.url=...:

mvn test -Dtdengine.ws.url=jdbc:TAOS-WS://192.168.1.100:6041/

After running the tests, a result similar to the following will be printed eventually. If all test cases pass, both Failures and Errors will be 0.

[INFO] Results:
[INFO]
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0

The total number of tests run may be slightly lower against TDengine servers older than 3.4.1.13, where the ByteType/ShortType write tests are skipped (see Known Limitations).

# run the full test suite
mvn test

• Make sure taosd and taosAdapter are running before executing the integration tests.
• If you only need to verify packaging, use mvn clean package -Dmaven.test.skip=true from the build step above.

5.2 Test Case Addition

All tests are located in the src/test/java/com/taosdata/spark directory of the project. Unit tests are in TDengineDialectTest; integration tests are in TDengineDialectIntegrationTest (end-to-end read/write) and TDengineGeneratedSqlTest (the SQL Spark generates: table-existence probe, filter pushdown, CREATE/DROP TABLE, INSERT). You can add new test files or add test cases in existing test files.

The test cases use the JUnit 4 framework. For the integration tests, a dedicated database (spark_dialect_test) with a super table of all common TDengine types is created in the @BeforeClass method, and the database is dropped in the @AfterClass method.

5.3 Performance Testing

Performance testing is in progress.

6. Packaging

mvn clean package -Dmaven.test.skip=true
# Output: target/tdengine-spark-dialect-<version>.jar

7. CI/CD

• Build Workflow: runs mvn clean verify on JDK 8 against a TDengine server started from the official Docker image.

8. Submitting Issues

We welcome the submission of GitHub Issue. When submitting, please provide the following information:

• Problem description, whether it always occurs, and it's best to include a detailed call stack.
• Spark dialect version and Spark version.
• JDBC driver version.
• TDengine server version.

9. Submitting PRs

We welcome developers to contribute to this project. When submitting PRs, please follow these steps:

1. Fork this project, refer to (how to fork a repo).
2. Create a new branch from the main branch with a meaningful branch name (git checkout -b my_branch). Do not modify the main branch directly.
3. Modify the code, ensure all unit tests pass, and add new unit tests to verify the changes.
4. Push the changes to the remote branch (git push origin my_branch).
5. Create a Pull Request on GitHub (how to create a pull request).
6. After submitting the PR, you can find your PR through the Pull Request page and check whether the CI for your PR has passed.

10. References

• TDengine Official Website
• TDengine GitHub
• TDengine JDBC Connector GitHub
• Apache Spark JDBC Data Source

11. License

MIT License