Skip to content

test: assert ClickHouse overwrite SQL - #6149

Open
mday-io wants to merge 1 commit into
SQLMesh:mainfrom
mday-io:fix/clickhouse-overwrite-assertions-upstream
Open

mday-io wants to merge 1 commit into
SQLMesh:mainfrom
mday-io:fix/clickhouse-overwrite-assertions-upstream

Conversation

@mday-io

@mday-io mday-io commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Description

Five ClickHouse overwrite tests compared generated SQL with expected statements without asserting the result, allowing incorrect replacement SQL to pass. Restore the assertions across full, time-range, key-based and partition-based overwrites, give staging and affected-partition tables distinct test names, and correct existing DISTINCT and cleanup expectations. Compare unordered partition actions without depending on Python hash order.

Three temporary mutations passed the original tests and failed the repaired assertions: a wrong exchange target, an inverted retained-row predicate, and a skipped incremental partition replacement.

Test Plan

  • All 68 ClickHouse unit cases passed.
  • The five repaired tests passed with PYTHONHASHSEED=1 and 42.
  • All three mutations were rejected at the intended SQL assertions.
  • make style passed after generating the worktree's ignored version module with setuptools-scm.
  • make fast-test passed: 2,895 tests, 13 skipped across all required batches; 4 xdist workers.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

Signed-off-by: Michael Day <mdaytn@gmail.com>
@mday-io
mday-io marked this pull request as ready for review October 16, 2026 21:08

@StuffbyYuki StuffbyYuki left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An optional:

Instead of the prefix/sort/split checks in the tests, perhaps consider making the generated ALTER deterministic at the source. In sqlmesh/core/engine_adapter/clickhouse.py, _build_alter_partition_exp loops over partitions_to_replace and partitions_to_drop, which are Python sets, so the action order can change between runs. Sorting them fixes that:

for partition in sorted(partitions_to_replace):
    ...
for partition in sorted(partitions_to_drop):
    ...

This only changes the order of actions within the single ALTER statement. It doesn't change which partitions are replaced or dropped, or how many statements are executed. As a side benefit, the SQL in logs and ClickHouse query history becomes stable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants