Feature Request: sqlmesh plan performs full project loading for every non-prod environment plan #5906

Summary

Every sqlmesh plan <env> where <env> is not prod loads, parses, and processes the entire project — all models, macros, audits, tests, and snapshots — regardless of how many models actually changed. Dev workflows, where engineers iterate on a small subset of models, pay a cost that scales with total project size rather than with the size of the change.

Where this happens

1. Full filesystem load is unconditional

GenericContext.__init__ always calls self.load():
sqlmesh/core/context.py ~L473:

if load:
    self.load()

load() parses every model, macro, audit, test, and metric across all project paths, then runs update_model_schemas() on the full DAG and model.validate_definition() on every model (context.py ~L629–728). There's no way to load only the changed subgraph.

2. Full prod state read during load()

Before plan() is even called, all prod snapshots are fetched from the state backend to populate remote-only models:
sqlmesh/core/context.py ~L687–699:

prod = self.state_reader.get_environment(c.PROD)
if prod:
    for snapshot in self.state_reader.get_snapshots(prod.snapshots).values():
        ...

3. Linting and tests run on all models

lint_models() and _run_plan_tests() run against the full model set before the plan is built, with no scoping to changed models:
sqlmesh/core/context.py ~L1579–1582.

4. Full snapshot creation with a state round-trip

_snapshots() creates snapshots for every model/standalone audit, then bulk-fetches all of them from state:
sqlmesh/core/context.py ~L2814–2842:

nodes = {**(models_override or self._models), **self._standalone_audits}
snapshots = self._nodes_to_snapshots(nodes)
stored_snapshots = self.state_reader.get_snapshots(snapshots.values())

Even with models_override (e.g. via --select-model), it still contains every model — selected ones from local state, the rest sourced from remote.

5. Full context diff against remote state

ContextDiff.create() fetches the entire target environment and bulk-reads all local + all modified remote snapshots:
sqlmesh/core/context_diff.py ~L180–182:

stored = state_reader.get_snapshots(
    [*snapshots.values(), *modified_snapshot_name_to_snapshot_info.values()]
)

6. Full DAG construction and traversal

PlanBuilder._build_dag() iterates every snapshot in the context diff, not just the changed subgraph:
sqlmesh/core/plan/builder.py ~L363–367:

for s_id, context_snapshot in self._context_diff.snapshots.items():
    dag.add(s_id, context_snapshot.parents)

This full DAG is then traversed by _build_directly_and_indirectly_modified, _build_restatements, _build_models_to_backfill (dag.subdag(...) only runs after the full DAG exists), and earliest_interval_start (iterates all snapshots).

7. Dev pays more than prod for DeployabilityIndex

Prod uses the constant-time DeployabilityIndex.all_deployable(); dev iterates every snapshot via DeployabilityIndex.create():
sqlmesh/core/plan/builder.py ~L298–306:

deployability_index = (
    DeployabilityIndex.create(
        self._context_diff.snapshots.values(),  # ALL snapshots
        start=self._start,
        start_override_per_model=self._start_override_per_model,
    )
    if self._is_dev
    else DeployabilityIndex.all_deployable()
)

Dev — the most frequent workflow — is disproportionately penalized.

What dev plans actually need

Only the changed models, their upstream dependencies, and their downstream dependents matter. Everything else in the project is irrelevant, but the current implementation has no concept of this subgraph.

The one scoping that does exist: in dev without --include-unmodified, backfill_models is limited to modified/added model names (context.py ~L1659–1667). This only limits what gets backfilled — it does not reduce the work done to produce the plan.

Impact

• Large projects: sqlmesh plan dev after a one-model change still parses every model, validates every definition, creates every snapshot, and issues multiple full-project state reads. Cost scales with project size, not change size.
• Dev iteration is disproportionately penalized: dev is the most frequent workflow, yet pays more per-plan than prod due to DeployabilityIndex.create iterating all snapshots.
• State backend load: multiple full get_snapshots() calls per plan (during load() for prod state, in _snapshots(), and in ContextDiff.create()) — expensive for networked/DB-backed state stores.
• Linting/tests unscoped: run over the full project even when only one model changed.

Suggested direction

For non-prod environments, determine the changed subgraph first, then scope all subsequent work to it:

1. Change detection before full load — identify changed files (e.g. via mtimes or git diff) to determine the minimal subgraph to load.
2. Lazy/partial context loading — let the Loader load only a specified subgraph, representing unrelated models as lightweight stubs sourced from remote state.
3. Scoped snapshot creation — _snapshots() should only build snapshots for the changed subgraph; unrelated models reuse their stored fingerprints from state.
4. Scoped context diff — ContextDiff.create should operate on a subgraph snapshot dict and fetch only the relevant remote snapshots.
5. Scoped DAG — PlanBuilder._build_dag() should build a DAG scoped to the changed subgraph in dev, not always the full project DAG.
6. Scoped linting/tests — lint_models() and _run_plan_tests() should be scoped to the changed subgraph for dev plans.
7. Scoped DeployabilityIndex — scope DeployabilityIndex.create in dev to the modified subgraph to close the gap with prod's all_deployable().

References

• Context init/unconditional load: sqlmesh/core/context.py ~L473–474
• load() full parse + schema update: sqlmesh/core/context.py ~L629–753
• Prod state read during load(): sqlmesh/core/context.py ~L687–699
• plan_builder() linting/tests/snapshots: sqlmesh/core/context.py ~L1564–1667
• _snapshots() full creation + state read: sqlmesh/core/context.py ~L2814–2842
• ContextDiff.create() full state diff: sqlmesh/core/context_diff.py ~L92–230
• PlanBuilder.build() full DAG construction: sqlmesh/core/plan/builder.py ~L280–367
• DeployabilityIndex dev vs prod branching: sqlmesh/core/plan/builder.py ~L298–306