Skip to content

Commit b0ce951

Browse files
committed
feat(cli): support project paths and options from environment variables
Signed-off-by: Vamsi-klu <nrvamsi13@gmail.com>
1 parent c0026dc commit b0ce951

7 files changed

Lines changed: 475 additions & 9 deletions

File tree

‎docs/integrations/github.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,21 @@ jobs:
5959
6060
Next checkout the [Core Bot Behavior Configuration Guide](#core-bot-behavior-configuration-guide) to see how to configure the bot's core behavior. Then checkout [Bot Configuration](#bot-configuration) to see how to configure the bot's behavior in general. Finally, checkout [Custom Workflow Configuration](#custom-workflow-configuration) to see the full set of customizations available to the bot.
6161
62+
### Project paths
63+
64+
The bot shares the `SQLMESH_PATHS` environment variable with the `sqlmesh` CLI. You can use it instead of `-p` in a workflow step:
65+
66+
```yaml
67+
- name: Run CI/CD Bot
68+
env:
69+
SQLMESH_PATHS: ${{ github.workspace }}
70+
run: sqlmesh_cicd github --token ${{ secrets.GITHUB_TOKEN }} run-all
71+
```
72+
73+
An explicit `-p` or `--paths` option replaces the entire environment path list. An unset or empty `SQLMESH_PATHS` uses the existing default project path. Export the variable in the workflow environment before invoking the bot; a project's `.env` file cannot locate the project that contains it.
74+
75+
Spaces and Unicode characters remain part of each path. For multiple projects, separate paths with `:` on POSIX runners or `;` on Windows runners. Empty path-list segments refer to the current directory. Use repeated `-p` options for directory names that contain the platform's path separator. See the [CLI reference](../reference/cli.md#project-paths) for examples. Automatic command-option variables such as `SQLMESH_PLAN_START` apply to the main `sqlmesh` CLI; the bot shares the explicit `SQLMESH_PATHS` binding.
76+
6277
## Core Bot Behavior Configuration Guide
6378

6479
There are two fundamental ways the bot can be configured: synchronized or desynchronized production code and data.

‎docs/reference/cli.md‎

Lines changed: 60 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ Usage: sqlmesh [OPTIONS] COMMAND [ARGS]...
77
88
Options:
99
--version Show the version and exit.
10-
-p, --paths TEXT Path(s) to the SQLMesh config/project.
10+
-p, --paths PATH Path(s) to the SQLMesh config/project.
1111
--config TEXT Name of the config object. Only applicable to
1212
configuration defined using Python script.
1313
--gateway TEXT The name of the gateway.
@@ -52,6 +52,65 @@ Commands:
5252
lint Run the linter for the target model(s).
5353
```
5454

55+
## Environment variables
56+
57+
CLI options accept defaults from environment variables. Use `SQLMESH_` followed by the uppercase command and option names, replacing hyphens with underscores. Include every command name for nested commands.
58+
59+
| Environment variable | CLI option |
60+
| --- | --- |
61+
| `SQLMESH_PATHS` | `sqlmesh --paths` |
62+
| `SQLMESH_CONFIG` | `sqlmesh --config` |
63+
| `SQLMESH_PLAN_START` | `sqlmesh plan --start` |
64+
| `SQLMESH_TABLE_DIFF_LIMIT` | `sqlmesh table_diff --limit` |
65+
| `SQLMESH_STATE_EXPORT_OUTPUT_FILE` | `sqlmesh state export --output-file` |
66+
| `SQLMESH_TEST_LOCAL` | `sqlmesh test --local` |
67+
| `SQLMESH_LINT_LOCAL` | `sqlmesh lint --local` |
68+
69+
Options with an explicit environment variable retain their names: `SQLMESH_GATEWAY`, `SQLMESH_IGNORE_WARNINGS`, and `SQLMESH_DOTENV_PATH`. Environment variables do not supply positional arguments, such as the environment in `sqlmesh plan dev`.
70+
71+
An explicit CLI option takes precedence over its environment variable. For example, `sqlmesh plan --start 2025-02-01` uses that date even if `SQLMESH_PLAN_START` specifies another date. Values use the same type validation as the corresponding CLI option.
72+
73+
### Project paths
74+
75+
Set `SQLMESH_PATHS` to select a project when running SQLMesh from another directory:
76+
77+
```bash
78+
export SQLMESH_PATHS="/work/analytics project"
79+
sqlmesh info
80+
```
81+
82+
Spaces and Unicode characters remain part of each path. For multiple projects, separate paths with `:` on POSIX or `;` on Windows. For example, on POSIX:
83+
84+
```bash
85+
export SQLMESH_PATHS="/work/analytics project:/work/finance α"
86+
sqlmesh info
87+
```
88+
89+
In PowerShell, a single project path can be set with:
90+
91+
```powershell
92+
$env:SQLMESH_PATHS = 'C:\work\analytics project'
93+
sqlmesh info
94+
```
95+
96+
Windows path lists use semicolons, preserving drive-letter colons. Shell quotes delimit the assignment and are not part of the value.
97+
98+
Explicit `-p` or `--paths` options replace the entire environment path list. Repeat the option to select multiple projects:
99+
100+
```bash
101+
sqlmesh -p "/work/another project" -p "/work/finance α" info
102+
```
103+
104+
An unset or empty `SQLMESH_PATHS` uses the existing default project path. Relative paths resolve from the invocation directory. Avoid leading, trailing, or repeated path separators: empty segments are retained and refer to the current directory. If a directory name contains the platform's path separator, pass it with an explicit `-p` option.
105+
106+
### Dotenv timing and precedence
107+
108+
Root options, including project paths, are parsed before SQLMesh loads project `.env` files or the file selected by `--dotenv`. Export `SQLMESH_PATHS` before invoking SQLMesh. A project `.env` file cannot select the project that SQLMesh must find to load that file. Other root options, such as `SQLMESH_CONFIG` and `SQLMESH_DOTENV_PATH`, also need to be present before invocation.
109+
110+
For commands that load project configuration, subcommand options are parsed after dotenv loading. Variables such as `SQLMESH_PLAN_START`, `SQLMESH_TEST_LOCAL`, and `SQLMESH_LINT_LOCAL` can therefore come from the selected project's `.env` file. SQLMesh loads dotenv values with replacement enabled, so they can override values already in the process environment. Explicit CLI options still take precedence.
111+
112+
CLI option variables are separate from the `SQLMESH__...` variables that configure project settings. See the [configuration reference](./configuration.md#environment-variables).
113+
55114
## audit
56115

57116
```

‎docs/reference/configuration.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,14 @@ This page lists SQLMesh configuration options and their parameters. Learn more a
44

55
Configuration options for model definitions are listed in the [model configuration reference page](./model_configuration.md).
66

7+
## Environment variables
8+
9+
Project settings use environment variables beginning with `SQLMESH__`, with double underscores separating configuration fields. For example, `SQLMESH__MODEL_DEFAULTS__DIALECT=snowflake` sets the default model dialect, and `SQLMESH__DEFAULT_GATEWAY=production` sets the project's default gateway. These variables override values in configuration files. See the [configuration guide](../guides/configuration.md#environment-variables) for details.
10+
11+
CLI option defaults use a single underscore after `SQLMESH`, such as `SQLMESH_PATHS`, `SQLMESH_CONFIG`, or `SQLMESH_PLAN_START`. An explicit CLI option takes precedence over its environment variable. For example, `SQLMESH_GATEWAY` selects the gateway for a command, while `SQLMESH__DEFAULT_GATEWAY` configures the project's default gateway.
12+
13+
Export root CLI option variables before invoking SQLMesh. Root options are parsed before project `.env` files or a custom `--dotenv` file are loaded, so `SQLMESH_PATHS` inside a project `.env` cannot locate that project. Subcommand option variables can come from the loaded dotenv file, whose values replace existing process environment values. See the [CLI environment-variable reference](./cli.md#environment-variables) for naming, path lists, and precedence.
14+
715
## Root configurations
816

917
A SQLMesh project configuration consists of root level parameters within which other parameters are defined.

‎sqlmesh/cli/main.py‎

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,11 @@ def _sqlmesh_version() -> str:
6666
return "0.0.0"
6767

6868

69-
@click.group(cls=_SQLMeshGroup, no_args_is_help=True)
69+
@click.group(
70+
cls=_SQLMeshGroup,
71+
no_args_is_help=True,
72+
context_settings={"auto_envvar_prefix": "SQLMESH"},
73+
)
7074
@click.version_option(version=_sqlmesh_version(), message="%(version)s")
7175
@opt.paths
7276
@opt.config
@@ -131,13 +135,6 @@ def cli(
131135
load = True
132136
# Local-only gating must hold for any number of --paths, so it stays outside the block below.
133137
load_state = ctx.invoked_subcommand not in LOCAL_ONLY_COMMANDS
134-
# The parent callback constructs Context before Click invokes the subcommand, so inspect its
135-
# parsed args here.
136-
if (
137-
ctx.invoked_subcommand in OPTIONAL_LOCAL_COMMANDS
138-
and "--local" in ctx.meta["subcommand_args"]
139-
):
140-
load_state = False
141138

142139
if len(paths) == 1:
143140
path = os.path.abspath(paths[0])
@@ -152,6 +149,19 @@ def cli(
152149
load = False
153150

154151
configs = load_configs(config, Context.CONFIG_TYPE, paths, dotenv_path=dotenv)
152+
# The subcommand is parsed after this callback. Resolve its local flag after dotenv loading
153+
# so Context avoids state access for the same effective value the subcommand will receive.
154+
if ctx.invoked_subcommand and ctx.invoked_subcommand in OPTIONAL_LOCAL_COMMANDS:
155+
if "--local" in ctx.meta["subcommand_args"]:
156+
load_state = False
157+
else:
158+
command = t.cast(click.Group, ctx.command).get_command(ctx, ctx.invoked_subcommand)
159+
assert command is not None
160+
with click.Context(command, info_name=ctx.invoked_subcommand, parent=ctx) as child:
161+
local = next(param for param in command.params if param.name == "local")
162+
value = local.value_from_envvar(child)
163+
if value is not None:
164+
load_state = not local.type_cast_value(child, value)
155165
log_limit = list(configs.values())[0].log_limit
156166

157167
remove_excess_logs(log_file_dir, log_limit)

‎sqlmesh/cli/options.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,10 @@
88
paths = click.option(
99
"-p",
1010
"--paths",
11+
type=click.Path(readable=False),
1112
multiple=True,
1213
default=[os.getcwd()],
14+
envvar="SQLMESH_PATHS",
1315
help="Path(s) to the SQLMesh config/project.",
1416
)
1517

‎tests/cli/test_cli.py‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
from click.testing import CliRunner
1313
from sqlglot import __version__ as SQLGLOT_VERSION
1414
from sqlmesh import RuntimeEnv
15+
from sqlmesh.cli import main as cli_main
1516
from sqlmesh._version import __version__ as SQLMESH_VERSION
1617
from sqlmesh.cli.project_init import ProjectTemplate, init_example_project
1718
from sqlmesh.cli.main import cli
@@ -2582,6 +2583,187 @@ def _setup_local_only_project(tmp_path, mocker):
25822583
return _patch_state_access(mocker)
25832584

25842585

2586+
@pytest.fixture
2587+
def cli_env(monkeypatch):
2588+
for name in (
2589+
"SQLMESH_PATHS",
2590+
"SQLMESH_CONFIG",
2591+
"SQLMESH_TEST_LOCAL",
2592+
"SQLMESH_LINT_LOCAL",
2593+
"SQLMESH_DOTENV_PATH",
2594+
"SQLMESH__MODEL_DEFAULTS__DIALECT",
2595+
):
2596+
# Register an undo even when absent, since dotenv can add the variable directly.
2597+
monkeypatch.setenv(name, "")
2598+
monkeypatch.delenv(name)
2599+
2600+
2601+
@pytest.mark.parametrize("name", ["project", "analytics project 分析"])
2602+
@pytest.mark.parametrize("relative", [False, True])
2603+
def test_paths_env_selects_project_from_another_directory(
2604+
runner, tmp_path, monkeypatch, cli_env, name, relative
2605+
):
2606+
project = tmp_path / name
2607+
_create_local_only_project(project, "env_project")
2608+
elsewhere = tmp_path / "elsewhere"
2609+
elsewhere.mkdir()
2610+
monkeypatch.chdir(elsewhere)
2611+
monkeypatch.setenv("SQLMESH_PATHS", os.path.relpath(project) if relative else str(project))
2612+
result = runner.invoke(cli, ["--log-file-dir", str(tmp_path), "info"])
2613+
assert result.exit_code == 0, result.output
2614+
assert "Models: 1" in result.output
2615+
2616+
2617+
@pytest.mark.parametrize("command", ["test", "lint"])
2618+
@pytest.mark.parametrize("value", ["true", "1", "false", "0", "", "invalid"])
2619+
@pytest.mark.parametrize("explicit", [False, True])
2620+
def test_local_env_state_gating(
2621+
runner, tmp_path, mocker, monkeypatch, cli_env, command, value, explicit
2622+
):
2623+
state = _setup_local_only_project(tmp_path, mocker)
2624+
init = mocker.spy(Context, "__init__")
2625+
monkeypatch.setenv(f"SQLMESH_{command.upper()}_LOCAL", value)
2626+
args = ["--paths", str(tmp_path), command]
2627+
if explicit:
2628+
args.append("--local")
2629+
result = runner.invoke(cli, args)
2630+
if value == "invalid" and not explicit:
2631+
assert result.exit_code == 2, result.output
2632+
assert "valid boolean" in result.output
2633+
init.assert_not_called()
2634+
state.assert_not_called()
2635+
elif explicit or value in ("true", "1"):
2636+
assert result.exit_code == 0, result.output
2637+
assert init.call_args.kwargs["load_state"] is False
2638+
state.assert_not_called()
2639+
else:
2640+
assert init.call_args.kwargs["load_state"] is True
2641+
state.assert_called()
2642+
2643+
2644+
@pytest.mark.parametrize("command", ["test", "lint"])
2645+
@pytest.mark.parametrize("custom_dotenv", [False, True])
2646+
@pytest.mark.parametrize("value", ["true", "false"])
2647+
def test_local_env_from_dotenv(
2648+
runner, tmp_path, mocker, monkeypatch, cli_env, command, custom_dotenv, value
2649+
):
2650+
state = _setup_local_only_project(tmp_path, mocker)
2651+
init = mocker.spy(Context, "__init__")
2652+
variable = f"SQLMESH_{command.upper()}_LOCAL"
2653+
# The existing config loader lets dotenv override the process environment.
2654+
monkeypatch.setenv(variable, "false" if value == "true" else "true")
2655+
env_file = tmp_path / ("custom.env" if custom_dotenv else ".env")
2656+
env_file.write_text(f"{variable}={value}\n", encoding="utf-8")
2657+
args = ["--paths", str(tmp_path)]
2658+
if custom_dotenv:
2659+
args.extend(["--dotenv", str(env_file)])
2660+
result = runner.invoke(cli, [*args, command])
2661+
assert init.call_args.kwargs["load_state"] is (value == "false")
2662+
if value == "true":
2663+
assert result.exit_code == 0, result.output
2664+
state.assert_not_called()
2665+
else:
2666+
state.assert_called()
2667+
2668+
2669+
@pytest.mark.parametrize("command", ["test", "lint", "format"])
2670+
def test_paths_env_multiple_projects_local(runner, tmp_path, mocker, monkeypatch, cli_env, command):
2671+
projects = (tmp_path / "first project", tmp_path / "second 分析")
2672+
for index, project in enumerate(projects):
2673+
_create_local_only_project(project, f"project_{index}")
2674+
state = _patch_state_access(mocker)
2675+
monkeypatch.setenv("SQLMESH_PATHS", os.pathsep.join(str(project) for project in projects))
2676+
if command != "format":
2677+
monkeypatch.setenv(f"SQLMESH_{command.upper()}_LOCAL", "true")
2678+
result = runner.invoke(cli, [command])
2679+
assert result.exit_code == 0, result.output
2680+
state.assert_not_called()
2681+
2682+
2683+
def test_paths_env_init_new_destination(runner, tmp_path, monkeypatch, cli_env):
2684+
project = tmp_path / "new project"
2685+
project.mkdir()
2686+
monkeypatch.setenv("SQLMESH_PATHS", str(project))
2687+
result = runner.invoke(cli, ["init", "duckdb", "--template", "empty"])
2688+
assert result.exit_code == 0, result.output
2689+
assert (project / "config.yaml").exists()
2690+
2691+
2692+
@pytest.mark.parametrize("invalid_config", [False, True])
2693+
def test_paths_env_project_errors(runner, tmp_path, monkeypatch, cli_env, invalid_config):
2694+
project = tmp_path / "project"
2695+
if invalid_config:
2696+
project.mkdir()
2697+
(project / "config.yaml").write_text("not a config\n", encoding="utf-8")
2698+
monkeypatch.setenv("SQLMESH_PATHS", str(project))
2699+
result = runner.invoke(cli, ["info"])
2700+
assert result.exit_code == 1, result.output
2701+
assert (
2702+
"Invalid YAML configuration" if invalid_config else "project config could not be found"
2703+
) in result.output
2704+
2705+
2706+
def test_paths_env_in_dotenv_does_not_change_root_paths(runner, tmp_path, monkeypatch, cli_env):
2707+
project = tmp_path / "project"
2708+
alternate = tmp_path / "alternate"
2709+
_create_local_only_project(project, "selected_project")
2710+
_create_local_only_project(alternate, "alternate_project")
2711+
(alternate / "models" / "second.sql").write_text(
2712+
"MODEL(name alternate_project.second); SELECT 2 AS col", encoding="utf-8"
2713+
)
2714+
(project / ".env").write_text(f"SQLMESH_PATHS='{alternate}'\n", encoding="utf-8")
2715+
monkeypatch.setenv("SQLMESH_PATHS", str(project))
2716+
result = runner.invoke(cli, ["info"])
2717+
assert result.exit_code == 0, result.output
2718+
assert "Models: 1" in result.output
2719+
2720+
2721+
def test_paths_env_only_in_custom_dotenv_does_not_select_project(
2722+
runner, tmp_path, monkeypatch, mocker, cli_env
2723+
):
2724+
project = tmp_path / "project"
2725+
_create_local_only_project(project, "env_project")
2726+
env_file = tmp_path / "custom.env"
2727+
env_file.write_text(f"SQLMESH_PATHS='{project}'\n", encoding="utf-8")
2728+
load_configs = mocker.spy(cli_main, "load_configs")
2729+
runner.invoke(cli, ["--dotenv", str(env_file), "info"])
2730+
paths = next(param for param in cli.params if param.name == "paths")
2731+
assert load_configs.call_args.args[2] == tuple(paths.default)
2732+
2733+
2734+
@pytest.mark.parametrize("empty_segment", [False, True])
2735+
def test_paths_env_preserves_loader_expansion(
2736+
runner, tmp_path, monkeypatch, mocker, cli_env, empty_segment
2737+
):
2738+
projects = [tmp_path / "projects" / "first", tmp_path / "projects" / "second"]
2739+
for index, project in enumerate(projects):
2740+
_create_local_only_project(project, f"env_project_{index}")
2741+
if empty_segment:
2742+
invocation = tmp_path / "invocation"
2743+
_create_local_only_project(invocation, "invocation")
2744+
monkeypatch.chdir(invocation)
2745+
env_value = os.pathsep.join([str(projects[0]), "", str(projects[1])])
2746+
expected = {projects[0], invocation, projects[1]}
2747+
else:
2748+
env_value = str(tmp_path / "projects" / "*")
2749+
expected = set(projects)
2750+
monkeypatch.setenv("SQLMESH_PATHS", env_value)
2751+
init = mocker.spy(Context, "__init__")
2752+
result = runner.invoke(cli, ["format"])
2753+
assert result.exit_code == 0, result.output
2754+
assert set(init.call_args.kwargs["config"]) == expected
2755+
2756+
2757+
def test_cli_env_separate_from_project_config(runner, tmp_path, monkeypatch, mocker, cli_env):
2758+
_create_local_only_project(tmp_path, "env_project")
2759+
monkeypatch.setenv("SQLMESH_PATHS", str(tmp_path))
2760+
monkeypatch.setenv("SQLMESH__MODEL_DEFAULTS__DIALECT", "spark")
2761+
init = mocker.spy(Context, "__init__")
2762+
result = runner.invoke(cli, ["info"])
2763+
assert result.exit_code == 0, result.output
2764+
assert next(iter(init.call_args.kwargs["config"].values())).model_defaults.dialect == "spark"
2765+
2766+
25852767
def test_format_runs_without_state(runner: CliRunner, tmp_path: Path, mocker):
25862768
mock = _setup_local_only_project(tmp_path, mocker)
25872769
result = runner.invoke(cli, ["--paths", str(tmp_path), "format"])

0 commit comments

Comments
 (0)