Skip to content

Commit 742ab33

Browse files
authored
Merge branch 'main' into fix/inlay-hint-stale-position
2 parents 5bd930f + c0026dc commit 742ab33

69 files changed

Lines changed: 5770 additions & 233 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/scripts/install-prerequisites.sh‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ ENGINE_DEPENDENCIES=""
1717

1818
if [ "$ENGINE" == "spark" ]; then
1919
ENGINE_DEPENDENCIES="default-jdk"
20+
elif [ "$ENGINE" == "db2" ]; then
21+
ENGINE_DEPENDENCIES="libxml2-dev build-essential"
2022
elif [ "$ENGINE" == "fabric" ]; then
2123
echo "Installing Microsoft package repository"
2224

‎.github/scripts/wait-for-db.sh‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,20 @@ risingwave_ready() {
9090
probe_port 4566
9191
}
9292

93+
db2_ready() {
94+
probe_port 50001
95+
96+
echo "Waiting for Db2 to finish initialising (this can take 2-4 minutes)..."
97+
while true; do
98+
if docker exec db2 su - db2inst1 -c "db2 connect to TESTDB" > /dev/null 2>&1; then
99+
echo "Db2 is accepting connections"
100+
break
101+
fi
102+
echo "Db2 not yet ready; sleeping 15s..."
103+
sleep 15
104+
done
105+
}
106+
93107
echo "Waiting for $ENGINE to be ready..."
94108

95109
READINESS_FUNC="${ENGINE}_ready"

‎.github/workflows/db2.yaml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: db2
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
paths:
8+
- 'sqlmesh/core/engine_adapter/db2.py'
9+
- 'tests/core/engine_adapter/test_db2.py'
10+
- 'tests/core/engine_adapter/integration/test_integration_db2.py'
11+
- 'tests/core/engine_adapter/integration/docker/compose.db2.yaml'
12+
- '.github/workflows/db2.yaml'
13+
- 'pyproject.toml'
14+
pull_request:
15+
branches:
16+
- main
17+
paths:
18+
- 'sqlmesh/core/engine_adapter/db2.py'
19+
- 'tests/core/engine_adapter/test_db2.py'
20+
- 'tests/core/engine_adapter/integration/test_integration_db2.py'
21+
- 'tests/core/engine_adapter/integration/docker/compose.db2.yaml'
22+
- '.github/workflows/db2.yaml'
23+
- 'pyproject.toml'
24+
25+
concurrency:
26+
group: db2-${{ github.event.pull_request.number || github.sha }}
27+
cancel-in-progress: true
28+
29+
permissions:
30+
contents: read
31+
32+
jobs:
33+
db2-test:
34+
runs-on: ubuntu-latest
35+
timeout-minutes: 25
36+
env:
37+
PYTEST_XDIST_AUTO_NUM_WORKERS: 2
38+
SQLMESH__DISABLE_ANONYMIZED_ANALYTICS: '1'
39+
UV: '1'
40+
steps:
41+
- uses: actions/checkout@v7
42+
- name: Set up Python
43+
uses: actions/setup-python@v7
44+
with:
45+
python-version: '3.12'
46+
- name: Install uv
47+
uses: astral-sh/setup-uv@v7
48+
- name: Install SQLMesh dev dependencies
49+
run: |
50+
uv venv .venv
51+
source .venv/bin/activate
52+
make install-dev
53+
- name: Install OS-level dependencies
54+
run: ./.github/scripts/install-prerequisites.sh "db2"
55+
- name: Run tests
56+
run: |
57+
source .venv/bin/activate
58+
make db2-test
59+
- name: Upload test results
60+
uses: actions/upload-artifact@v7
61+
if: ${{ !cancelled() }}
62+
with:
63+
name: test-results-docker-db2
64+
path: test-results/
65+
retention-days: 7

‎.github/workflows/pr.yaml‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ jobs:
1717
python: ${{ steps.filter.outputs.python }}
1818
client: ${{ steps.filter.outputs.client }}
1919
vscode: ${{ steps.filter.outputs.vscode }}
20+
lsp: ${{ steps.filter.outputs.lsp }}
2021
ci: ${{ steps.filter.outputs.ci }}
2122
steps:
2223
- uses: actions/checkout@v7
@@ -37,6 +38,8 @@ jobs:
3738
- 'web/client/**'
3839
vscode:
3940
- 'vscode/**'
41+
lsp:
42+
- 'sqlmesh/lsp/**'
4043
ci:
4144
- '.github/**'
4245
- 'Makefile'
@@ -427,10 +430,12 @@ jobs:
427430
- name: Run CI
428431
run: pnpm run ci
429432
test-vscode-e2e:
430-
runs-on:
431-
labels: [ubuntu-2204-8]
432-
# As at 2026-01-12 this job flakes 100% of the time. It needs investigation
433-
if: false
433+
needs: changes
434+
if:
435+
needs.changes.outputs.vscode == 'true' || needs.changes.outputs.lsp ==
436+
'true' || needs.changes.outputs.ci == 'true' || github.ref ==
437+
'refs/heads/main'
438+
runs-on: ubuntu-latest
434439
steps:
435440
- uses: actions/checkout@v7
436441
- uses: actions/setup-node@v7
@@ -453,7 +458,9 @@ jobs:
453458
source .venv/bin/activate
454459
make install-dev
455460
- name: Install code-server
456-
run: curl -fsSL https://code-server.dev/install.sh | sh
461+
# Pinned so that a code-server release cannot change the editor UI the
462+
# tests drive without the change showing up here first.
463+
run: curl -fsSL https://code-server.dev/install.sh | sh -s -- --version 4.137.0
457464
- name: Install Playwright browsers
458465
working-directory: ./vscode/extension
459466
run: pnpm exec playwright install
@@ -558,7 +565,8 @@ jobs:
558565
run: |
559566
source ../../.venv/bin/activate
560567
sed -i 's/target: in_memory/target: postgres/g' profiles.yml
561-
if [[ $(echo -e "${{ matrix.dbt-version }}\n1.5.0" | sort -V | head -n1) == "${{ matrix.dbt-version }}" ]] && [[ "${{ matrix.dbt-version }}" != "1.5.0" ]]; then
568+
if [[ "${{ matrix.dbt-version }}" == "1.3" ]] || \
569+
[[ "${{ matrix.dbt-version }}" == "1.4" ]]; then
562570
echo "DBT version is ${{ matrix.dbt-version }} (< 1.5.0), removing version parameters..."
563571
sed -i -e 's/, version=1) }}/) }}/g' -e 's/, v=1) }}/) }}/g' models/top_waiters.sql
564572
else

‎Makefile‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -132,13 +132,13 @@ engine-up: engine-clickhouse-up engine-mssql-up engine-mysql-up engine-postgres-
132132
engine-down: engine-clickhouse-down engine-mssql-down engine-mysql-down engine-postgres-down engine-spark-down engine-trino-down
133133

134134
fast-test:
135-
pytest -n auto -m "fast and not cicdonly and not isolated" --junitxml=test-results/junit-fast-test.xml && pytest -m "isolated and not slow" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
135+
pytest -n auto -m "fast and not cicdonly and not isolated" --junitxml=test-results/junit-fast-test.xml && pytest -m "isolated and not slow and not fork_isolated" && pytest tests/test_forking.py::test_parallel_load -m "isolated and fork_isolated and not slow" && pytest tests/test_forking.py::test_parallel_load_multi_repo -m "isolated and fork_isolated and not slow" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
136136

137137
slow-test:
138-
pytest -n auto -m "(fast or slow) and not cicdonly and not isolated" && pytest -m "isolated" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
138+
pytest -n auto -m "(fast or slow) and not cicdonly and not isolated" && pytest -m "isolated and not fork_isolated" && pytest tests/test_forking.py::test_parallel_load -m "isolated and fork_isolated" && pytest tests/test_forking.py::test_parallel_load_multi_repo -m "isolated and fork_isolated" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
139139

140140
cicd-test:
141-
pytest -n auto -m "(fast or slow) and not pyspark and not isolated" --junitxml=test-results/junit-cicd.xml && pytest -m "pyspark" && pytest -m "isolated and not pyspark" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
141+
pytest -n auto -m "(fast or slow) and not pyspark and not isolated" --junitxml=test-results/junit-cicd.xml && pytest -m "pyspark" && pytest -m "isolated and not pyspark and not fork_isolated" && pytest tests/test_forking.py::test_parallel_load -m "isolated and not pyspark and fork_isolated" && pytest tests/test_forking.py::test_parallel_load_multi_repo -m "isolated and not pyspark and fork_isolated" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
142142

143143
core-fast-test:
144144
pytest -n auto -m "fast and not web and not github and not dbt and not jupyter"
@@ -222,6 +222,9 @@ risingwave-test: engine-risingwave-up
222222

223223
starrocks-test: engine-starrocks-up
224224
pytest -n auto -m "starrocks" --reruns 3 --junitxml=test-results/junit-starrocks.xml
225+
226+
db2-test: engine-db2-up
227+
pytest -n auto -m "db2" --reruns 3 --junitxml=test-results/junit-db2.xml
225228

226229
#################
227230
# Cloud Engines #
@@ -235,7 +238,10 @@ bigquery-test: guard-BIGQUERY_KEYFILE engine-bigquery-install
235238
pytest -n auto -m "bigquery" --reruns 3 --junitxml=test-results/junit-bigquery.xml
236239

237240
databricks-test: guard-DATABRICKS_CATALOG guard-DATABRICKS_SERVER_HOSTNAME guard-DATABRICKS_HTTP_PATH guard-DATABRICKS_CONNECT_VERSION engine-databricks-install
238-
$(PIP) install 'databricks-connect==${DATABRICKS_CONNECT_VERSION}'
241+
# Connect 15.3.x requires numpy<2 and pyarrow>=4 (no upper bound). Cap PyArrow
242+
# in the same resolve that downgrades NumPy, or PyArrow 25+ is left installed
243+
# and cannot be imported.
244+
$(PIP) install 'databricks-connect==${DATABRICKS_CONNECT_VERSION}' 'pyarrow<25'
239245
pytest -n auto -m "databricks" --reruns 3 --junitxml=test-results/junit-databricks.xml
240246

241247
redshift-test: guard-REDSHIFT_HOST guard-REDSHIFT_USER guard-REDSHIFT_PASSWORD guard-REDSHIFT_DATABASE engine-redshift-install

‎docs/guides/connections.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ default_gateway: local_db
8484
* [BigQuery](../integrations/engines/bigquery.md)
8585
* [ClickHouse](../integrations/engines/clickhouse.md)
8686
* [Databricks](../integrations/engines/databricks.md)
87+
* [Db2](../integrations/engines/db2.md)
8788
* [DuckDB](../integrations/engines/duckdb.md)
8889
* [Fabric](../integrations/engines/fabric.md)
8990
* [MotherDuck](../integrations/engines/motherduck.md)

‎docs/guides/migrations.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,26 @@ Migrations should ideally run when no one will be running plan/apply.
3636
Migrations should not be run in parallel.
3737
Due to these constraints, it is better for a person responsible for managing SQLMesh to manually issue migrations.
3838
Therefore, it is not recommended to issue migrations from CI/CD pipelines.
39+
40+
## Rolling back a migration
41+
42+
When `sqlmesh migrate` needs to change the project metadata, it first copies each state table to a backup table with a `_backup` suffix, unless `--skip-backup` is passed. The `sqlmesh rollback` command restores the metadata from those backups, returning it to the format used by the SQLMesh version that was installed before the migration. If a migration fails partway through, SQLMesh rolls it back automatically from the same backups.
43+
44+
To undo a migration:
45+
46+
1. Run `sqlmesh rollback` with the SQLMesh version that performed the migration still installed.
47+
2. Reinstall the SQLMesh version the project used before the upgrade, for example by reverting the version change in your requirements file.
48+
49+
The second step is required. Rolling back does not change the installed version of SQLMesh, and the newer version will refuse to run against the restored metadata until it is migrated again:
50+
51+
```bash
52+
> sqlmesh plan
53+
Error: SQLMesh (local) is using version '2' which is ahead of '1' (remote). Please run a migration ('sqlmesh migrate' command).
54+
```
55+
56+
Keep the following in mind before rolling back:
57+
58+
- Only the most recent migration can be rolled back. Restoring consumes the backup tables, so running `sqlmesh rollback` a second time fails with `There are no prior migrations to roll back to.`
59+
- The backups are taken at the moment of migration. Any changes made to the project metadata after the migration, such as plans applied with the newer version, are discarded.
60+
- Like `sqlmesh migrate`, rolling back affects all users of the project and should be issued manually by a single user.
61+
- Rollback is not possible if the migration was run with `--skip-backup`. It also does nothing useful after an upgrade that required no migration, such as a patch release, because no new backup was taken. In that case the backup tables, if present, come from an earlier migration.

‎docs/integrations/engines/clickhouse.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ In general, you must be connected to a ClickHouse server to execute commands. By
4949

5050
To associate an object with a cluster, DDL commands that create or modify it must include the text `ON CLUSTER [your cluster name]`.
5151

52-
If you provide a cluster name in your SQLMesh connection configuration, SQLMesh will automatically inject the `ON CLUSTER` statement into the DDL commands for all objects created while executing the project. We provide more information about clusters in SQLMesh [below](#cluster-specification).
52+
If you provide a cluster name in your SQLMesh connection configuration, SQLMesh automatically adds `ON CLUSTER` to object DDL, except for metadata DDL targeting `Replicated` databases, which replicate those changes themselves. We provide more information about clusters in SQLMesh [below](#cluster-specification).
5353

5454
#### ClickHouse Cloud mode
5555

@@ -96,7 +96,9 @@ For example, we could create a table `my_schema.my_table` on cluster `TheCluster
9696

9797
To create SQLMesh objects on a cluster, provide the cluster name to the `cluster` key in the SQLMesh connection definition (see all connection parameters [below](#localbuilt-in-scheduler)).
9898

99-
SQLMesh will automatically inject the `ON CLUSTER` clause and cluster name you provide into all project DDL statements.
99+
SQLMesh automatically adds the `ON CLUSTER` clause and cluster name you provide to object DDL, except for metadata DDL targeting `Replicated` databases. SQLMesh retains `ON CLUSTER` for database creation and removal and for partition operations.
100+
101+
Partition overwrites in multi-shard `Replicated` databases remain unvalidated. ClickHouse can replicate an ALTER containing both `REPLACE PARTITION` and `DROP PARTITION`, which can conflict with `ON CLUSTER`. This behavior predates the metadata DDL policy described above and depends on the ClickHouse version.
100102

101103
## Model definition
102104

‎docs/integrations/engines/db2.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Db2
2+
3+
This page provides information about how to use SQLMesh with [IBM Db2](https://www.ibm.com/products/db2).
4+
5+
!!! info
6+
The Db2 engine adapter is a community contribution. Due to this, only limited community support is available.
7+
8+
This adapter depends on the external `db2-sqlglot-dialect` package, which pins its own SQLGlot version range. It isn't covered by SQLMesh's core CI and may temporarily break or fail to install after SQLMesh upgrades SQLGlot, until the plugin catches up.
9+
10+
## Local/Built-in Scheduler
11+
12+
**Engine Adapter Type**: `db2`
13+
14+
### Installation
15+
16+
```
17+
pip install "sqlmesh[db2]"
18+
```
19+
20+
### Connection options
21+
22+
| Option | Description | Type | Required |
23+
|---------------------|------------------------------------------------------------------------------------------------|:------:|:--------:|
24+
| `type` | Engine type name - must be `db2` | string | Y |
25+
| `host` | The hostname of the Db2 server | string | Y |
26+
| `port` | The port number of the Db2 server. Default: `50000` | int | N |
27+
| `database` | The name of the Db2 database to connect to | string | Y |
28+
| `username` | The username to use for authentication with the Db2 server | string | Y |
29+
| `password` | The password to use for authentication with the Db2 server | string | Y |
30+
| `db2_schema` | Sets `CURRENTSCHEMA` on the connection. Controls the default schema for unqualified references. Typically set to the same value as `username`. | string | Y |
31+
| `ssl` | Enable TLS/SSL encryption. Default: `false` | bool | N |
32+
| `connect_timeout` | The number of seconds to wait for the connection to the server. Default: `30` | int | N |
33+
| `concurrent_tasks` | Maximum number of tasks to run concurrently. Default: `4` | int | N |
34+
35+
## Important Notes
36+
37+
**State connection:** Db2 is **not supported** as a SQLMesh `state_connection`. Use DuckDB (recommended) or another supported engine for SQLMesh state storage:
38+
39+
```yaml linenums="1"
40+
gateways:
41+
db2:
42+
connection:
43+
type: db2
44+
host: localhost
45+
port: 50000
46+
database: TESTDB
47+
username: db2inst1
48+
password: your_password
49+
db2_schema: db2inst1
50+
state_connection:
51+
type: duckdb
52+
database: ./state/sqlmesh_state.db
53+
54+
default_gateway: db2
55+
56+
model_defaults:
57+
dialect: db2
58+
```
59+
60+
**Table naming:** Db2 rejects table names that start with an underscore (`_`). SQLMesh's default physical table naming convention can generate names beginning with `_`. To avoid this, set `physical_table_naming_convention` to `hash_md5` in your project config:
61+
62+
```yaml
63+
physical_table_naming_convention: hash_md5
64+
```
65+
66+
## Limitations
67+
68+
- **SCD Type 2 unsupported**: SCD Type 2 models are currently not supported on Db2.
69+
- **Single catalog only**: Db2 operates in single-catalog mode; cross-catalog queries are not supported.
70+
- **No inline column comments**: Column-level comments cannot be set inline during table creation.
71+
- **No atomic table replacement**: Db2 does not support `CREATE OR REPLACE TABLE`, so full model refreshes are not atomic. There is a brief window during which the table may be empty or partially populated.
72+
- **Identifier length**: Maximum identifier length is 128 characters.
73+
- **No `SELECT ... FOR UPDATE`**: Db2 does not support `SELECT ... FOR UPDATE` in the same way as OLTP databases; SQLMesh removes this clause when executing queries.
74+
75+
## Resources
76+
77+
- [IBM Db2 Documentation](https://www.ibm.com/docs/en/db2)
78+
- [IBM Db2 SQL Reference](https://www.ibm.com/docs/en/db2/11.5?topic=db2-sql)

‎docs/integrations/engines/starrocks.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
# StarRocks
22

3+
!!! info
4+
The StarRocks engine adapter is a community contribution. Due to this, only limited community support is available.
5+
36
## Overview
47

58
[StarRocks](https://www.starrocks.io/) is a next-generation sub-second MPP OLAP database designed for real-time analytics. It provides high concurrency, low latency, and supports both batch and stream processing.

0 commit comments

Comments
 (0)