Fix CI: re-enable and fix test-vscode-e2e

Why this exists

The VSCode extension has Playwright end-to-end tests that drive a real editor (via code-server, which is VS Code in the browser). They cover completions, diagnostics, lineage, go-to-definition, format, etc.

That CI job is test-vscode-e2e in .github/workflows/pr.yaml. It has been turned off since 2026-01-12 (if: false) because it was failing on every PR and blocking merges. The unit/lint job (test-vscode) is already back on; this issue is only the e2e job.

Called out as follow-up on #6004.

This is not the same as test-vscode. That one runs lint + TypeScript compile + a handful of Vitest unit tests. E2E is the suite that actually opens the editor.

Goal

Get the e2e tests passing locally.
Turn the GitHub Actions job back on (with a path filter, so Python-only PRs are not blocked).
Keep it green.

How to turn the job on

In .github/workflows/pr.yaml, find test-vscode-e2e. It currently looks like this:

  test-vscode-e2e:
    runs-on:
      labels: [ubuntu-2204-8]
    # As at 2026-01-12 this job flakes 100% of the time. It needs investigation
    if: false

Change it to match test-vscode (path filter + only run when VSCode or CI config changed):

  test-vscode-e2e:
    needs: changes
    if:
      needs.changes.outputs.vscode == 'true' || needs.changes.outputs.ci ==
      'true' || github.ref == 'refs/heads/main'
    runs-on:
      labels: [ubuntu-2204-8]

Leave the rest of the steps as they are until you know they need changing. The vscode path filter already exists in the changes job (vscode/**).

How to run it locally

You need Node 22+, pnpm 10+, a Python 3.12 venv at the repo root named .venv, and code-server on your PATH. The tests look for .venv/bin/python and spawn code-server.

From the repo root:

# 1. Python env the extension will use
python3.12 -m venv .venv
source .venv/bin/activate
make install-dev

# 2. JS deps
pnpm install

# 3. code-server (VS Code in the browser). Pin a version if you can;
#    CI currently installs whatever "latest" is.
# macOS:
brew install code-server
# or:
# curl -fsSL https://code-server.dev/install.sh | sh

# 4. Playwright's Chromium browser
pnpm --prefix vscode/extension exec playwright install

# 5. Package the extension and run the e2e suite
cd vscode/extension
pnpm run test:e2e

That last command builds the VSCode webview, packs a .vsix, then runs Playwright. It takes a while.

Useful variants:

cd vscode/extension

# Watch the browser (easier to see what is stuck)
pnpm run test:e2e:headed

# Playwright UI
pnpm run test:e2e:ui

# After a vsix already exists, run one file without re-packing
pnpm exec playwright test tests/completions.spec.ts

When tests fail, open the HTML report (screenshots + video of the failure):

open vscode/extension/playwright-report/index.html

On CI the same report is uploaded as the playwright-report artifact (kept 30 days).

What these tests actually do

Very short version:

pnpm run vscode:package builds the React webview and packs sqlmesh-<version>.vsix.
Setup installs that vsix into a temp code-server.
Each test copies examples/sushi into a temp folder, opens it in code-server, and clicks the UI like a user.
Almost every test waits for the status text Loaded SQLMesh Context before doing anything else (waitForLoadedSQLMesh in vscode/extension/tests/utils.ts).

If that text never appears, most of the suite will time out. Start there.

Things to look into (most likely first)

Work these in order. You do not need to be a JS expert; the report will name the failing spec.

1. Unpinned `code-server` (best first guess)

CI does this every run:

- name: Install code-server
  run: curl -fsSL https://code-server.dev/install.sh | sh

That is latest, not a version. The tests click VS Code UI by visible text (models, Explorer, Loaded SQLMesh Context). If a code-server/VS Code update changes the UI, every test can fail the same way.

Try: pin a version that the tests were written against, install that same version locally, and re-run.

2. Selector / timeout failures

Playwright config (vscode/extension/playwright.config.ts):

60 second timeout per test
2 workers in CI, 4 locally
retries already set to 2 in CI

If the report is full of waitForSelector timeouts, it is usually:

SQLMesh never finished loading (Python/LSP problem → no Loaded SQLMesh Context)
or the VS Code UI changed (code-server problem)

The tests hardcode the interpreter as <repo>/.venv/bin/python. If you did not create that venv, loading will fail.

3. Known-flaky specs (skip vs fix)

Some files already document CI-only failures. Do not treat those as a mystery:

tests/lineage.spec.ts — comment: works locally when debugging, not on CI
tests/bad_setup.spec.ts — typing a filename is flaky
tests/completions.spec.ts — macro completions skipped as flaky
tests/tcloud.spec.ts — one case skipped; sign-in window not usable by Playwright

Getting the common path green (open sushi → loaded context → completions/diagnostics/go-to-definition) is more important than unskipping these.

4. Shared code-server + 2 CI workers

Each Playwright worker starts its own code-server on a random port and they share one extensions directory. That can cause races. If pinning code-server is not enough, try workers: 1 in CI (there is history of doing that in this repo).

5. Self-hosted runner

The job uses runs-on: labels: [ubuntu-2204-8] (bigger machine). GitHub ubuntu-latest may be too small; that is why it was moved. If the job never starts, the runner label is the problem, not the tests.

Background (so you do not have to rediscover it)

Disabled in Chore: update databricks and snowflake auth in integration tests #5652 (2026-01-12) with the comment that it failed 100% of the time.
Last known green CI run found: 2025-11-05. From ~2025-11-20 through disable, sampled runs were all red.
Failed jobs ran ~15 minutes and uploaded a ~32 MB Playwright report, so the suite was executing, not dying at install.
Actions logs and that report have expired (HTTP 410). We do not have the failing spec names. You have to reproduce.

Done when

pnpm run test:e2e passes locally (or remaining failures are documented test.skips with a reason)
test-vscode-e2e is re-enabled in pr.yaml with the vscode / ci path filter (not if: false)
The job is green on that PR
code-server is pinned (or another root cause is written down so the next bump does not silently break CI again)

I'd like to take this one if it's free — unassigned with no linked PR right now.

Since the Actions logs and the Playwright report have both expired, I'll start with reproduction rather than changing pr.yaml, and work your list in order: pin code-server (CI currently curls install.sh, so it's whatever is latest), then check whether Loaded SQLMesh Context ever appears at all, since the issue rightly points out that most of the suite hangs off that one wait.

One thing worth setting expectations on. If this turns out to be code-server/VS Code UI drift, the fix is a pin plus whatever selectors moved, and I'll write the root cause down so the next bump doesn't silently break it again. If instead the context never loads under code-server — an LSP or interpreter-discovery problem rather than a UI one — that's a different animal, and I'd rather report back with the reproduction than quietly widen the scope of this issue.

Either way I'll come back with what actually fails before touching the workflow.

@cmgoffena13

@cmgoffena13 — reproduced locally. First fix is up as #6076. Findings below, including two places where the issue's starting assumptions turn out not to hold.

Setup: macOS, Python 3.12.11 venv at the repo root, make install-dev, pnpm install, Playwright Chromium, vsix packaged. code-server installed as two pinned standalone builds rather than the CI one-liner, so the version could be A/B'd: 4.107.1 (the newest release that existed when the job was disabled — if: false landed in 529ed005, authored 2026-01-11 23:42 UTC, and 4.108.0 didn't ship until 19:24 UTC the next day) and 4.137.0 (current).

Full suite, 4.137.0, CI-like at --workers=2 --retries=0: 26 failed, 1 skipped, 50 passed in 28 minutes.

`Loaded SQLMesh Context` is not the problem

The issue suggests starting there on the grounds that most of the suite hangs off that wait. It appears fine: 50 tests pass, including hints, both macro-completion cases marked flaky, 5/6 find_references, all 5 broken_project, format, go_to_definition and external_models. SQLMesh is loading.

Pinning code-server would not have fixed it

All 4 render.spec.ts tests fail in 15-25s on every run, with a strict mode violation:

locator('text=sushi.customers (rendered)') resolved to 2 elements:
  1) <a class="label-name">sushi.customers (rendered)</a>
  2) <span class="monaco-highlighted-label">sushi.customers (rendered)</span>
     aka getByLabel('Enable current file context').locator('a')

code-server now bundles Copilot/Chat (lib/vscode/node_modules/@github/copilot-*), and the chat context chip echoes the active editor's name, so a bare text= selector matches twice.

This fails byte-identically on 4.107.1 and 4.137.0. The drift was already present at the pin date. Pinning is still worth doing for reproducibility, but it is not the fix, and that is consistent with "failed 100% of the time" rather than flaking. Corroborating: no commit touched vscode/ between 2025-08-27 and 2026-03-25, spanning the disable — so the breakage came from outside the repo.

#6076 fixes this one by matching the editor tab by role. Verified green on both versions.

What I could not explain

A cluster of ~11 tests times out at 60s on Loaded SQLMesh Context: configuration.spec.ts ×3, python_env.spec.ts ×4, tcloud.spec.ts ×3, venv_naming.spec.ts. The pattern I observed is that every one of them builds its own venv in tempDir via uv venv, and the tests that point at the repo .venv all pass. The extension does activate — its editor-title buttons are in the page snapshot — but no SQLMesh status item ever appears, and the created venv is functional (sqlmesh_lsp present, imports fine). I did not find the mechanism and would rather say so than guess. This is the main thing standing between here and a green suite.

Also genuinely flaky across runs: quickfix, stop:52, tcloud:42, diagnostics:10. And a real teardown race — ENOTEMPTY: directory not empty, rmdir .../.venv/lib/python3.11/site-packages/tenacity — where the tempDir fixture removes the directory while uv or the LSP is still writing into it.

A separate bug found on the way

vscode/extension/src/utilities/exec.ts builds a shell string by joining args unquoted:

const fullCmd = `${command} ${args.join(' ')}`

Any path containing a space breaks it. It surfaced here because my checkout lives under a directory with a space, giving Received: 127 and /bin/sh: /Users/mac/Documents/BrainStorm: No such file or directory in diagnostics.spec.ts. Those two tests pass in a space-free checkout, so it is not the CI failure — but it is a real bug for anyone whose workspace path has a space in it. Happy to raise it separately if useful.

Recommendation

Not a pin-and-fix, and I would leave the job off for now. Three things are between here and turning it on: the render fix (done, #6076), the own-venv cluster (unresolved), and the teardown race. Your hypothesis 4 is also still untested — I ran at --workers=2 and did not isolate whether workers: 1 changes the picture.

Deviations worth weighing before trusting any of this for CI: macOS x86_64 rather than Linux ubuntu-2204-8, and uv selected Python 3.11 locally (no python3.11 on PATH, so it downloaded one) where CI's setup-python would give 3.12.

One last thing: ubuntu-2204-8 is the only occurrence of that label in the entire workflow directory — every other job runs on ubuntu-latest. Worth confirming that runner pool still exists before the job is re-enabled, or it will sit queued rather than fail.

Follow-up on the reproduction: the remaining cluster is a timeout, not a functional failure. The eleven tests that were still failing after the render.spec.ts fix all pass unchanged if the per-test budget is raised.

Decisive A/B on the same commit, same code-server 4.137.0, same --workers=2 --retries=0:

per-test timeout	result
60s (`playwright.config.ts` default)	all 11 fail on `waitForSelector('text=Loaded SQLMesh Context')`
240s (`--timeout=240000`, nothing else changed)	14 passed, 1 skipped, 0 failed (14.4 min)

No source change, no settle delay, no version pin — only the budget. configuration.spec.ts (3), python_env.spec.ts (4), venv_naming.spec.ts (1) and tcloud.spec.ts (3) all pass.

Where the time goes, measured on one such test with a warm uv cache:

uv venv done:            0.2s
uv pip install done:    13.6s
page open:              17.6s
lineage view opened:    27.6s
LOADED SQLMESH CONTEXT: 97.9s   <- ~70s of it waiting on the LSP

~98s needed against a 60s budget.

What separates these eleven from the rest: each builds its own virtualenv and installs sqlmesh[...,lsp] into it, then waits for the language server to start against a cold interpreter. tcloud.spec.ts:24 does two uv pip install calls. Tests that point at the long-lived repo .venv finish in ~30s and pass — confirmed by bisection, since pointing a test at a pre-built uv venv made it pass in 49s. Note bad_setup.spec.ts also builds venvs and passes, because it installs without the lsp extra and never waits for the server.

CI is worse, not better: cold uv cache, two workers on one runner. retries: 2 cannot rescue a systematic overrun, it just spends the time three times.

The remedy is your call

Three options, and I would argue for the third:

Raise the global timeout in playwright.config.ts.
Add test.slow() to the eleven that build their own venv.
Build the test virtualenv once per worker in a fixture, rather than once per test. That removes 14–30s from eleven tests and addresses the cause rather than tolerating it.

1 and 2 are smaller but leave a suite that spends most of its wall-clock reinstalling the same package. Happy to implement whichever you prefer.

Caveats on all of the above

macOS x86_64 rather than Linux ubuntu-2204-8, and uv selected Python 3.11 locally (no python3.11 on PATH, so it downloaded one) where CI's setup-python would give 3.12. The absolute numbers will differ on the runner; the ratio and the conclusion should not.

Still open before the job can go back on

the teardown race — ENOTEMPTY: directory not empty, rmdir .../site-packages/tenacity, where the tempDir fixture removes the directory while uv or the LSP is still writing into it. I have a small retry-with-backoff fix for this ready to put up.
your hypothesis 4 is untested — I ran at --workers=2 throughout and did not isolate whether workers: 1 changes anything.
ubuntu-2204-8 is the only occurrence of that label in the whole workflow directory; every other job is ubuntu-latest. Worth confirming that runner pool still exists, or the job will queue rather than fail.

@tripleaceme

@tripleaceme -- I agree on approach 3 being the best option. Once that stuff is cached it should be a lot faster.

Navigation Menu

Repository navigation

Fix CI: re-enable and fix `test-vscode-e2e` #6071

Why this exists

Goal

How to turn the job on

How to run it locally

What these tests actually do

Things to look into (most likely first)

1. Unpinned `code-server` (best first guess)

2. Selector / timeout failures

3. Known-flaky specs (skip vs fix)

4. Shared code-server + 2 CI workers

5. Self-hosted runner

Background (so you do not have to rediscover it)

Done when

`Loaded SQLMesh Context` is not the problem

Pinning code-server would not have fixed it

What I could not explain

A separate bug found on the way

Recommendation

The remedy is your call

Caveats on all of the above

Still open before the job can go back on

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Fix CI: re-enable and fix test-vscode-e2e #6071

Description

Why this exists

Goal

How to turn the job on

How to run it locally

What these tests actually do

Things to look into (most likely first)

1. Unpinned code-server (best first guess)

2. Selector / timeout failures

3. Known-flaky specs (skip vs fix)

4. Shared code-server + 2 CI workers

5. Self-hosted runner

Background (so you do not have to rediscover it)

Done when

Activity

tripleaceme commented on Sep 17, 2026

tripleaceme commented on Sep 18, 2026

Loaded SQLMesh Context is not the problem

Pinning code-server would not have fixed it

What I could not explain

A separate bug found on the way

Recommendation

tripleaceme commented on Sep 24, 2026

The remedy is your call

Caveats on all of the above

Still open before the job can go back on

cmgoffena13 commented on Sep 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Fix CI: re-enable and fix `test-vscode-e2e` #6071

1. Unpinned `code-server` (best first guess)

`Loaded SQLMesh Context` is not the problem