Build SQL dialect auto linter / add in github workflows - #263
Open
keenzarate213 wants to merge 30 commits into
Open
Build SQL dialect auto linter / add in github workflows#263keenzarate213 wants to merge 30 commits into
keenzarate213 wants to merge 30 commits into
Conversation
get latest main into feature branch
…erytime we push code in the branch
keenzarate213
marked this pull request as ready for review
August 3, 2026 15:44
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description & motivation
Adds a new CI check (
sql_dialect_linter) that catches Databricks/Snowflake SQL incompatibilities in the PR.Note
This is the
edu_whversion of the same CI tool in edu_edfi_source here: edanalytics/edu_edfi_source#206For each dialect, the workflow:
edu_whpackage against a dummy dbt profile and dummy macro overrides for anything that would require a database connection during compile (e.g.is_incremental,get_column_values,union_relations).bld_ef3__student_programs,bld_ef3__student_indicators,bld_ef3__student_assessments_long_results, andcfg_assessment_scores.tpdm_warehouseandfinance_warehouse, which are disabled by default, so they get checked linted as well.Also added a check for known Databricks incompatible function/type usage that sqlfluff can't catch since it only validates syntax, doesnt detect a given function or type actually exists. Currently checks for
try_to_date()and castingas time, both valid syntax everywhere but with no Databricks equivalent. I added a list directly insql_dialect_linter.sh(
databricks_incompatible_patterns) so new ones are a one-line addition (we could also maybe store in a yaml?)Versions (sqlfluff, dbt-core, dbt-databricks, dbt-snowflake) are pinned in
.github/workflows/requirements/sql-linter.txt.Also had to add
noqa: PRS(see documntation here) workaround (scoped to a single known macro) for an unsupported sqlfluff parsing gap around Databricks'variant_explodesyntax,and also allow for caching of virtualenv and
dbt_packagesin workflow so dependencies don't getreinstalled/re-downloaded on every push in the branch.
Breaking changes introduced by this PR:
None, this is a new CI/tooling, no changes to warehouse models or logic.
PR Merge Priority:
Changes to existing files:
New files created:
.github/scripts/sql_dialect_linter.sh: compiles + lints edu_wh against a dummyDatabricks/Snowflake profile for the given dialect argument.
.github/workflows/sql_dialect_linter.yml: runs the above on PRs for both dialects..github/workflows/requirements/sql-linter.txt: pinned tool versionsTests and QC done:
see most recent run here: https://github.com/edanalytics/edu_wh/actions/runs/30828507203