Skip to content

Commit 11ed15d

Browse files
authored
Merge branch 'main' into selective-snapshot-hydration
2 parents 427b2f8 + 9d8302e commit 11ed15d

93 files changed

Lines changed: 7736 additions & 918 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.devcontainer/devcontainer.json‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,7 @@
1111
"postCreateCommand": "bash .devcontainer/post-create-command.sh",
1212
"customizations": {
1313
"vscode": {
14-
"extensions": [
15-
"ms-python.python",
16-
"ms-python.vscode-pylance"
17-
]
14+
"extensions": ["ms-python.python", "ms-python.vscode-pylance"]
1815
}
1916
},
2017
"remoteUser": "vscode"

‎.github/scripts/install-prerequisites.sh‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ ENGINE_DEPENDENCIES=""
1717

1818
if [ "$ENGINE" == "spark" ]; then
1919
ENGINE_DEPENDENCIES="default-jdk"
20+
elif [ "$ENGINE" == "db2" ]; then
21+
ENGINE_DEPENDENCIES="libxml2-dev build-essential"
2022
elif [ "$ENGINE" == "fabric" ]; then
2123
echo "Installing Microsoft package repository"
2224

‎.github/scripts/wait-for-db.sh‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,20 @@ risingwave_ready() {
9090
probe_port 4566
9191
}
9292

93+
db2_ready() {
94+
probe_port 50001
95+
96+
echo "Waiting for Db2 to finish initialising (this can take 2-4 minutes)..."
97+
while true; do
98+
if docker exec db2 su - db2inst1 -c "db2 connect to TESTDB" > /dev/null 2>&1; then
99+
echo "Db2 is accepting connections"
100+
break
101+
fi
102+
echo "Db2 not yet ready; sleeping 15s..."
103+
sleep 15
104+
done
105+
}
106+
93107
echo "Waiting for $ENGINE to be ready..."
94108

95109
READINESS_FUNC="${ENGINE}_ready"

‎.github/workflows/db2.yaml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: db2
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
paths:
8+
- 'sqlmesh/core/engine_adapter/db2.py'
9+
- 'tests/core/engine_adapter/test_db2.py'
10+
- 'tests/core/engine_adapter/integration/test_integration_db2.py'
11+
- 'tests/core/engine_adapter/integration/docker/compose.db2.yaml'
12+
- '.github/workflows/db2.yaml'
13+
- 'pyproject.toml'
14+
pull_request:
15+
branches:
16+
- main
17+
paths:
18+
- 'sqlmesh/core/engine_adapter/db2.py'
19+
- 'tests/core/engine_adapter/test_db2.py'
20+
- 'tests/core/engine_adapter/integration/test_integration_db2.py'
21+
- 'tests/core/engine_adapter/integration/docker/compose.db2.yaml'
22+
- '.github/workflows/db2.yaml'
23+
- 'pyproject.toml'
24+
25+
concurrency:
26+
group: db2-${{ github.event.pull_request.number || github.sha }}
27+
cancel-in-progress: true
28+
29+
permissions:
30+
contents: read
31+
32+
jobs:
33+
db2-test:
34+
runs-on: ubuntu-latest
35+
timeout-minutes: 25
36+
env:
37+
PYTEST_XDIST_AUTO_NUM_WORKERS: 2
38+
SQLMESH__DISABLE_ANONYMIZED_ANALYTICS: '1'
39+
UV: '1'
40+
steps:
41+
- uses: actions/checkout@v7
42+
- name: Set up Python
43+
uses: actions/setup-python@v7
44+
with:
45+
python-version: '3.12'
46+
- name: Install uv
47+
uses: astral-sh/setup-uv@v7
48+
- name: Install SQLMesh dev dependencies
49+
run: |
50+
uv venv .venv
51+
source .venv/bin/activate
52+
make install-dev
53+
- name: Install OS-level dependencies
54+
run: ./.github/scripts/install-prerequisites.sh "db2"
55+
- name: Run tests
56+
run: |
57+
source .venv/bin/activate
58+
make db2-test
59+
- name: Upload test results
60+
uses: actions/upload-artifact@v7
61+
if: ${{ !cancelled() }}
62+
with:
63+
name: test-results-docker-db2
64+
path: test-results/
65+
retention-days: 7

‎.github/workflows/pr.yaml‎

Lines changed: 53 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ jobs:
1616
outputs:
1717
python: ${{ steps.filter.outputs.python }}
1818
client: ${{ steps.filter.outputs.client }}
19+
vscode: ${{ steps.filter.outputs.vscode }}
20+
lsp: ${{ steps.filter.outputs.lsp }}
1921
ci: ${{ steps.filter.outputs.ci }}
2022
steps:
2123
- uses: actions/checkout@v7
@@ -34,6 +36,10 @@ jobs:
3436
- 'pyproject.toml'
3537
client:
3638
- 'web/client/**'
39+
vscode:
40+
- 'vscode/**'
41+
lsp:
42+
- 'sqlmesh/lsp/**'
3743
ci:
3844
- '.github/**'
3945
- 'Makefile'
@@ -188,9 +194,10 @@ jobs:
188194

189195
ui-style:
190196
needs: [changes]
191-
if: false
192-
# needs.changes.outputs.client == 'true' || needs.changes.outputs.ci ==
193-
# 'true' || github.ref == 'refs/heads/main'
197+
if:
198+
needs.changes.outputs.client == 'true' || needs.changes.outputs.vscode ==
199+
'true' || needs.changes.outputs.ci == 'true' || github.ref ==
200+
'refs/heads/main'
194201
runs-on: ubuntu-latest
195202
steps:
196203
- uses: actions/checkout@v7
@@ -252,7 +259,17 @@ jobs:
252259
fail-fast: false
253260
matrix:
254261
engine:
255-
[duckdb, postgres, mysql, mssql, trino, spark, clickhouse, risingwave, starrocks]
262+
[
263+
duckdb,
264+
postgres,
265+
mysql,
266+
mssql,
267+
trino,
268+
spark,
269+
clickhouse,
270+
risingwave,
271+
starrocks,
272+
]
256273
env:
257274
PYTEST_XDIST_AUTO_NUM_WORKERS: 2
258275
SQLMESH__DISABLE_ANONYMIZED_ANALYTICS: '1'
@@ -393,10 +410,13 @@ jobs:
393410
retention-days: 7
394411

395412
test-vscode:
413+
needs: changes
414+
if:
415+
needs.changes.outputs.vscode == 'true' || needs.changes.outputs.ci ==
416+
'true' || github.ref == 'refs/heads/main'
396417
env:
397418
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: 1
398419
runs-on: ubuntu-latest
399-
if: false
400420
steps:
401421
- uses: actions/checkout@v7
402422
- uses: actions/setup-node@v7
@@ -410,10 +430,12 @@ jobs:
410430
- name: Run CI
411431
run: pnpm run ci
412432
test-vscode-e2e:
413-
runs-on:
414-
labels: [ubuntu-2204-8]
415-
# As at 2026-01-12 this job flakes 100% of the time. It needs investigation
416-
if: false
433+
needs: changes
434+
if:
435+
needs.changes.outputs.vscode == 'true' || needs.changes.outputs.lsp ==
436+
'true' || needs.changes.outputs.ci == 'true' || github.ref ==
437+
'refs/heads/main'
438+
runs-on: ubuntu-latest
417439
steps:
418440
- uses: actions/checkout@v7
419441
- uses: actions/setup-node@v7
@@ -436,7 +458,9 @@ jobs:
436458
source .venv/bin/activate
437459
make install-dev
438460
- name: Install code-server
439-
run: curl -fsSL https://code-server.dev/install.sh | sh
461+
# Pinned so that a code-server release cannot change the editor UI the
462+
# tests drive without the change showing up here first.
463+
run: curl -fsSL https://code-server.dev/install.sh | sh -s -- --version 4.137.0
440464
- name: Install Playwright browsers
441465
working-directory: ./vscode/extension
442466
run: pnpm exec playwright install
@@ -453,11 +477,27 @@ jobs:
453477
path: vscode/extension/playwright-report/
454478
retention-days: 30
455479
test-dbt-versions:
480+
needs: changes
481+
if:
482+
needs.changes.outputs.python == 'true' || needs.changes.outputs.ci ==
483+
'true' || github.ref == 'refs/heads/main'
456484
runs-on: ubuntu-latest
457485
strategy:
458486
fail-fast: false
459487
matrix:
460-
dbt-version: ['1.3', '1.4', '1.5', '1.6', '1.7', '1.8', '1.9', '1.10', '1.11', '1.12']
488+
dbt-version:
489+
[
490+
'1.3',
491+
'1.4',
492+
'1.5',
493+
'1.6',
494+
'1.7',
495+
'1.8',
496+
'1.9',
497+
'1.10',
498+
'1.11',
499+
'1.12',
500+
]
461501
steps:
462502
- uses: actions/checkout@v7
463503
- name: Set up Python
@@ -525,7 +565,8 @@ jobs:
525565
run: |
526566
source ../../.venv/bin/activate
527567
sed -i 's/target: in_memory/target: postgres/g' profiles.yml
528-
if [[ $(echo -e "${{ matrix.dbt-version }}\n1.5.0" | sort -V | head -n1) == "${{ matrix.dbt-version }}" ]] && [[ "${{ matrix.dbt-version }}" != "1.5.0" ]]; then
568+
if [[ "${{ matrix.dbt-version }}" == "1.3" ]] || \
569+
[[ "${{ matrix.dbt-version }}" == "1.4" ]]; then
529570
echo "DBT version is ${{ matrix.dbt-version }} (< 1.5.0), removing version parameters..."
530571
sed -i -e 's/, version=1) }}/) }}/g' -e 's/, v=1) }}/) }}/g' models/top_waiters.sql
531572
else

‎.prettierignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ docs
3030
examples
3131
posts
3232
.circleci
33+
.github/
3334
README.md
3435
mkdocs.yml
3536
.readthedocs.yaml

‎CONTRIBUTING.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,10 +65,12 @@ See [docs/development.md](docs/development.md) for full setup instructions. Key
6565
python -m venv .venv
6666
source .venv/bin/activate
6767
make install-dev
68-
make style # Run before submitting
68+
make style # Run before submitting
6969
make fast-test # Quick test suite
7070
```
7171

72+
Optionally, `make install-pre-commit` installs git hooks so ruff and mypy run on `git commit`. Hooks do not replace `make style`: they run on staged files, while CI runs `make style` across the tree.
73+
7274
## Coding Standards
7375

7476
- Run `make style` before submitting a pull request

‎Makefile‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -132,13 +132,13 @@ engine-up: engine-clickhouse-up engine-mssql-up engine-mysql-up engine-postgres-
132132
engine-down: engine-clickhouse-down engine-mssql-down engine-mysql-down engine-postgres-down engine-spark-down engine-trino-down
133133

134134
fast-test:
135-
pytest -n auto -m "fast and not cicdonly and not isolated" --junitxml=test-results/junit-fast-test.xml && pytest -m "isolated and not slow" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
135+
pytest -n auto -m "fast and not cicdonly and not isolated" --junitxml=test-results/junit-fast-test.xml && pytest -m "isolated and not slow and not fork_isolated" && pytest tests/test_forking.py::test_parallel_load -m "isolated and fork_isolated and not slow" && pytest tests/test_forking.py::test_parallel_load_multi_repo -m "isolated and fork_isolated and not slow" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
136136

137137
slow-test:
138-
pytest -n auto -m "(fast or slow) and not cicdonly and not isolated" && pytest -m "isolated" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
138+
pytest -n auto -m "(fast or slow) and not cicdonly and not isolated" && pytest -m "isolated and not fork_isolated" && pytest tests/test_forking.py::test_parallel_load -m "isolated and fork_isolated" && pytest tests/test_forking.py::test_parallel_load_multi_repo -m "isolated and fork_isolated" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
139139

140140
cicd-test:
141-
pytest -n auto -m "(fast or slow) and not pyspark and not isolated" --junitxml=test-results/junit-cicd.xml && pytest -m "pyspark" && pytest -m "isolated and not pyspark" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
141+
pytest -n auto -m "(fast or slow) and not pyspark and not isolated" --junitxml=test-results/junit-cicd.xml && pytest -m "pyspark" && pytest -m "isolated and not pyspark and not fork_isolated" && pytest tests/test_forking.py::test_parallel_load -m "isolated and not pyspark and fork_isolated" && pytest tests/test_forking.py::test_parallel_load_multi_repo -m "isolated and not pyspark and fork_isolated" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
142142

143143
core-fast-test:
144144
pytest -n auto -m "fast and not web and not github and not dbt and not jupyter"
@@ -222,6 +222,9 @@ risingwave-test: engine-risingwave-up
222222

223223
starrocks-test: engine-starrocks-up
224224
pytest -n auto -m "starrocks" --reruns 3 --junitxml=test-results/junit-starrocks.xml
225+
226+
db2-test: engine-db2-up
227+
pytest -n auto -m "db2" --reruns 3 --junitxml=test-results/junit-db2.xml
225228

226229
#################
227230
# Cloud Engines #
@@ -235,7 +238,10 @@ bigquery-test: guard-BIGQUERY_KEYFILE engine-bigquery-install
235238
pytest -n auto -m "bigquery" --reruns 3 --junitxml=test-results/junit-bigquery.xml
236239

237240
databricks-test: guard-DATABRICKS_CATALOG guard-DATABRICKS_SERVER_HOSTNAME guard-DATABRICKS_HTTP_PATH guard-DATABRICKS_CONNECT_VERSION engine-databricks-install
238-
$(PIP) install 'databricks-connect==${DATABRICKS_CONNECT_VERSION}'
241+
# Connect 15.3.x requires numpy<2 and pyarrow>=4 (no upper bound). Cap PyArrow
242+
# in the same resolve that downgrades NumPy, or PyArrow 25+ is left installed
243+
# and cannot be imported.
244+
$(PIP) install 'databricks-connect==${DATABRICKS_CONNECT_VERSION}' 'pyarrow<25'
239245
pytest -n auto -m "databricks" --reruns 3 --junitxml=test-results/junit-databricks.xml
240246

241247
redshift-test: guard-REDSHIFT_HOST guard-REDSHIFT_USER guard-REDSHIFT_PASSWORD guard-REDSHIFT_DATABASE engine-redshift-install

‎docs/concepts/tests.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -463,6 +463,37 @@ You can also run tests that match a pattern or substring using a glob pathname e
463463
$ sqlmesh test tests/test_*
464464
```
465465

466+
Passing the path of a model file runs the tests for that model, which is useful for commit hooks and other tools that work with changed files rather than test names:
467+
468+
```
469+
$ sqlmesh test models/full_model.sql
470+
```
471+
472+
Model files and test files can be mixed, and the results are unioned. A test selected by more than one argument still runs only once, so the following runs each of `full_model`'s tests a single time even though both arguments cover them:
473+
474+
```
475+
$ sqlmesh test models/full_model.sql tests/test_full_model.yaml
476+
```
477+
478+
An argument that is neither a known model file nor a known test file is an error, so a mistyped or stale path fails instead of quietly running no tests. A model that simply has no tests is not an error.
479+
480+
You can pass `--local` to run tests without loading state from the configured state connection:
481+
482+
``` bash
483+
$ sqlmesh test --local
484+
```
485+
486+
This keeps offline runs and commit hooks from opening a connection to the state backend.
487+
488+
In multi-repository setups, or when running tests for only a subset of projects, models that exist only in remote state are not loaded under `--local`. Unlike [`sqlmesh lint --local`](../guides/linter.md), which reports additional errors in that situation, a test whose model is missing is **skipped with a warning and the run still succeeds**:
489+
490+
```
491+
[WARNING] Model '"memory"."bronze"."a"' was not found at tests/test_a.yaml
492+
.**Successfully Ran `1` Tests Against `duckdb`**
493+
```
494+
495+
So a passing exit code alone does not mean every test you expected actually ran. Watch the output for these warnings, and keep in mind that a hook using `--local` will not fail on them.
496+
466497
### Testing using notebooks
467498

468499
You can execute tests on demand using the `%run_test` notebook magic as follows:

‎docs/development.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ Once you have activated your virtual environment, you can install the dependenci
4242
make install-dev
4343
```
4444

45-
Optionally, you can use pre-commit to automatically run linters/formatters:
45+
Optionally, `make install-pre-commit` installs git hooks so ruff and mypy run on `git commit`. Hooks do not replace `make style`: they run on staged files, while CI runs `make style` across the tree.
4646

4747
```bash
4848
make install-pre-commit

0 commit comments

Comments
 (0)