Run smoke tests against a built quarto (binary mode)

.claude/rules/llm-docs-maintenance.md

@@ -63,3 +63,5 @@ So a relevant deep-dive actually gets read when a matching task comes up, each `
 Granularity rule: the pointer inherits its host rule's `paths:`, so it must live in a rule scoped **no broader than the doc's topic** — otherwise it loads on unrelated work (a reveal.js pointer does not belong in a rule scoped to all of `src/format/**`). If the best existing rule is too generic, create a small dedicated rule scoped to that doc's source instead.
 
 **When adding a new `llm-docs/*.md`:** add its discovery pointer to a matching path-scoped rule, or create a narrowly-scoped rule for it.
+
+**When renaming an `##`/`###` heading:** discovery pointers may quote that heading text to route readers to a specific section (e.g. `built-version-ci.md` → `"Design decisions"`). Grep `.claude/rules/` for the old heading text and update any matching pointer — a stale quote fails loudly (grep/search finds nothing) but still needs fixing.

.claude/rules/testing/built-version-ci.md

@@ -0,0 +1,26 @@
+---
+paths:
+  - .github/workflows/test-smokes.yml
+  - .github/workflows/test-smokes-built.yml
+  - .github/workflows/test-ff-matrix.yml
+  - .github/workflows/create-release.yml
+  - .github/actions/build-dist-tarball/**
+  - .github/actions/merge-extension-tests/**
+---
+
+# Built-Version CI Workflows
+
+`llm-docs/built-version-testing-architecture.md` is the deep dive. Read only what the edit needs:
+
+- OS lists, bucket globs, concurrency-group suffixes, `runners:` inputs → "Built-mode test legs" (scheduler layout).
+- Adding/reordering a leg, changing `quarto-install` resolution or artifact plumbing → "Flow diagrams" + "Built-mode test legs".
+- Removing, inverting, or "simplifying" an existing mechanism → "Design decisions" (D1-D11) first — each records why the obvious alternative was rejected.
+
+Trivial one-line YAML edits: the invariants below plus the doc's "Document map" intro paragraph are enough.
+
+Invariants when editing:
+
+- Playwright and ff-matrix legs in `test-smokes-built.yml` must check both the source mode and `github.event.inputs.buckets == ''`.
+- `test-ff-matrix.yml` owns the ff-matrix bucket glob.
+- Scheduler jobs in `test-smokes-built.yml` set per-leg OS scope through their `runners:` inputs.
+- Keep the per-call suffix in `test-ff-matrix.yml`'s concurrency group so sibling calls cannot cancel one another.

.claude/rules/testing/overview.md

@@ -1,11 +1,12 @@
 ---
 paths:
-  - "tests/**"
+  - tests/**
 ---
 
 # Test Infrastructure
 
-Quarto's test suite lives in `tests/`. For comprehensive documentation, see `tests/README.md`.
+Quarto's test suite lives in `tests/`.
+For comprehensive documentation, see `tests/README.md`.
 
 ## Running Tests
 
@@ -28,18 +29,23 @@ QUARTO_TESTS_NO_CONFIG="true" ./run-tests.sh test.ts    # Linux/macOS
 $env:QUARTO_TESTS_NO_CONFIG=$true; .\run-tests.ps1      # Windows
 ```
 
+**Binary mode:** set `QUARTO_TEST_BIN` to an installed Quarto outside the checkout.
+With no test arguments, binary mode runs `smoke/`; pass Playwright or feature-format targets explicitly.
+See `tests/README.md` and `llm-docs/built-version-testing-architecture.md`.
+
 ## Test Types
 
-| Type | Location | File Pattern | Details |
-|------|----------|--------------|---------|
-| Unit | `tests/unit/` | `*.test.ts` | `.claude/rules/testing/typescript-tests.md` |
-| Smoke | `tests/smoke/` | `*.test.ts` | `.claude/rules/testing/typescript-tests.md` |
-| Smoke-all | `tests/docs/smoke-all/` | `*.qmd` | `.claude/rules/testing/smoke-all-tests.md` |
-| Playwright | `tests/integration/playwright/` | `*.spec.ts` | `.claude/rules/testing/playwright-tests.md` |
+| Type       | Location                        | File Pattern | Details                                     |
+| ---------- | ------------------------------- | ------------ | ------------------------------------------- |
+| Unit       | `tests/unit/`                   | `*.test.ts`  | `.claude/rules/testing/typescript-tests.md` |
+| Smoke      | `tests/smoke/`                  | `*.test.ts`  | `.claude/rules/testing/typescript-tests.md` |
+| Smoke-all  | `tests/docs/smoke-all/`         | `*.qmd`      | `.claude/rules/testing/smoke-all-tests.md`  |
+| Playwright | `tests/integration/playwright/` | `*.spec.ts`  | `.claude/rules/testing/playwright-tests.md` |
 
 ## Dependencies
 
-Tests require R, Python, and Julia. Run configuration script to set up:
+Tests require R, Python, and Julia.
+Run configuration script to set up:
 
 ```bash
 # Linux/macOS
@@ -56,12 +62,13 @@ Managed via:
 
 ## Core Files
 
-| File | Purpose |
-|------|---------|
-| `test.ts` | Test infrastructure (`testQuartoCmd`, `unitTest`) |
-| `verify.ts` | Verification functions |
-| `utils.ts` | Path utilities (`docs()`, `outputForInput()`) |
-| `README.md` | Comprehensive documentation |
+| File            | Purpose                                                                                   |
+| --------------- | ----------------------------------------------------------------------------------------- |
+| `test.ts`       | Test infrastructure (`testQuartoCmd`, `unitTest`)                                         |
+| `quarto-cmd.ts` | Quarto invocation dispatch (`runQuarto`; in-process dev vs `QUARTO_TEST_BIN` binary mode) |
+| `verify.ts`     | Verification functions                                                                    |
+| `utils.ts`      | Path utilities (`docs()`, `outputForInput()`)                                             |
+| `README.md`     | Comprehensive documentation                                                               |
 
 ## Debugging
 

.claude/rules/testing/playwright-tests.md

@@ -1,12 +1,13 @@
 ---
 paths:
-  - "tests/integration/playwright/**/*.spec.ts"
-  - "tests/integration/playwright/**/*.ts"
+  - tests/integration/playwright/**/*.spec.ts
+  - tests/integration/playwright/**/*.ts
 ---
 
 # Playwright Tests
 
-Browser-based tests for interactive features. Tests live in `tests/integration/playwright/tests/`.
+Browser-based tests for interactive features.
+Tests live in `tests/integration/playwright/tests/`.
 
 ## Local Development Workflow
 
@@ -44,6 +45,10 @@ The wrapper (`playwright-tests.test.ts`):
 3. Runs `npx playwright test`
 4. Cleans up rendered output
 
+The suite runs in dev and binary-mode CI.
+Render spawns must use `quartoSpawnEnvOptions()` so a built Quarto cannot inherit dev-tree paths.
+Browser assertions are ignored on Windows CI, so built-version CI has no Windows Playwright leg.
+
 ## Test Structure
 
 Tests use `@playwright/test` framework:
@@ -63,7 +68,8 @@ test("Feature description", async ({ page }) => {
 
 ### Parameterized Tests
 
-When testing the same behavior across multiple formats or configurations, use `test.describe` with a test cases array instead of separate spec files. See `html-math-katex.spec.ts` and `axe-accessibility.spec.ts` for examples.
+When testing the same behavior across multiple formats or configurations, use `test.describe` with a test cases array instead of separate spec files.
+See `html-math-katex.spec.ts` and `axe-accessibility.spec.ts` for examples.
 
 ```typescript
 const testCases = [
@@ -81,11 +87,13 @@ test.describe('Feature across formats', () => {
 });
 ```
 
-**When to use:** Same assertion logic applied to multiple formats, output modes, or configurations. Reduces file count and centralizes shared helpers.
+**When to use:** Same assertion logic applied to multiple formats, output modes, or configurations.
+Reduces file count and centralizes shared helpers.
 
 ### Expected Failures
 
-Use `test.fail()` to document known failures. Playwright inverts the result: the test passes if it fails, and flags if it unexpectedly passes (signaling the fix landed).
+Use `test.fail()` to document known failures.
+Playwright inverts the result: the test passes if it fails, and flags if it unexpectedly passes (signaling the fix landed).
 
 ```typescript
 test('Feature that is known broken', async ({ page }) => {

.claude/rules/testing/test-anti-patterns.md

@@ -1,14 +1,25 @@
 ---
 paths:
-  - "tests/**/*.ts"
-  - "tests/**/*.test.ts"
+  - tests/**/*.ts
+  - tests/**/*.test.ts
 ---
 
 # Test Anti-Patterns
 
+## Don't: Rely on `execute()` Throwing to Fail a Test
+
+The harness converts errors from `TestDescriptor.execute` into log records.
+Put assertions in `verify`, and include a log-reading verifier such as `noErrors` when using `runQuarto(..., { throwOnFailure: false })`.
+
+## Don't: Import `src/quarto.ts` in Tests
+
+Importing `src/quarto.ts` bypasses binary mode.
+Use `testQuartoCmd()`/`runQuarto()` and resolve direct subprocess spawns with `quartoDevCmd()`.
+
 ## Don't: Modify Environment Variables
 
-`Deno.env.set()` modifies process-global state. Deno runs test files in parallel by default, so other tests can see modified values.
+`Deno.env.set()` modifies process-global state.
+Deno runs test files in parallel by default, so other tests can see modified values.
 
 **Details:** `llm-docs/testing-patterns.md` → "Environment Variable Testing Pitfalls"
 
@@ -20,6 +31,9 @@ Never create `Project.toml`, `.venv/`, or `renv.lock` in test fixture directorie
 
 ## Don't: `Deno.chdir()` inside the test body
 
-`Deno.chdir()` mutates process-global cwd, so a test that changes it can leak into other tests in the same process. The harness already changes and restores the working directory: return the directory from `TestContext.cwd`, create fixtures in `setup`, clean up in `teardown` (examples: `tests/unit/dotenv-config.test.ts`, `tests/smoke/use/template.test.ts`). For a temp directory you don't need to run *from*, use `withTempDir` (`tests/utils.ts`). A test that only needs a *relative* input can pass a path relative to the current cwd without changing it.
+`Deno.chdir()` mutates process-global cwd, so a test that changes it can leak into other tests in the same process.
+The harness already changes and restores the working directory: return the directory from `TestContext.cwd`, create fixtures in `setup`, clean up in `teardown` (examples: `tests/unit/dotenv-config.test.ts`, `tests/smoke/use/template.test.ts`).
+For a temp directory you don't need to run *from*, use `withTempDir` (`tests/utils.ts`).
+A test that only needs a *relative* input can pass a path relative to the current cwd without changing it.
 
 **Details:** `llm-docs/testing-patterns.md` → "Working-Directory-Sensitive Tests"

.claude/rules/testing/typescript-tests.md

@@ -1,12 +1,13 @@
 ---
 paths:
-  - "tests/smoke/**/*.test.ts"
-  - "tests/unit/**/*.test.ts"
+  - tests/smoke/**/*.test.ts
+  - tests/unit/**/*.test.ts
 ---
 
 # TypeScript Tests
 
-TypeScript-based tests using Deno. Smoke tests render documents; unit tests verify isolated functionality.
+TypeScript-based tests using Deno.
+Smoke tests render documents; unit tests verify isolated functionality.
 
 ## Running Tests
 
@@ -22,17 +23,22 @@ TypeScript-based tests using Deno. Smoke tests render documents; unit tests veri
 
 ## Core Infrastructure
 
-Core test files (`test.ts`, `verify.ts`, `utils.ts`) are described in `.claude/rules/testing/overview.md` § Core Files.
+Core test files (`test.ts`, `quarto-cmd.ts`, `verify.ts`, `utils.ts`) are described in `.claude/rules/testing/overview.md` § Core Files.
+
+### Binary mode compatibility
+
+Smoke tests must work with either the dev sources or `QUARTO_TEST_BIN`:
+
+- Never `import { quarto } from "../../src/quarto.ts"` in tests — invoke quarto via `testQuartoCmd()`/`runQuarto()`.
+- For subprocesses, use `quartoDevCmd()` or `quartoDevBinCmd()` and pass `quartoSpawnEnvOptions()`.
+- `TestContext.requiresDevQuarto: true` ignores a test in binary mode — rare escape hatch for tests exercising quarto internals in-process.
 
 ### Search for an existing verifier before writing one
 
-`verify.ts` already covers many output shapes — including parsed-content
-verifiers, not just raw-text regex (e.g. `ensureIpynbCellMatches` JSON-parses a
-notebook and matches against joined cell source; `ensureHtmlElements` /
-`ensureHtmlSelectorSatisfies` parse the DOM). Before adding a new `Verify`,
-grep `verify.ts` for the format or assertion you need. Reuse or extend the
-existing helper rather than hand-rolling a near-duplicate. Same applies to
-mock-context and fixture helpers in `tests/unit/**` and `tests/utils.ts`.
+`verify.ts` already covers many output shapes — including parsed-content verifiers, not just raw-text regex (e.g. `ensureIpynbCellMatches` JSON-parses a notebook and matches against joined cell source; `ensureHtmlElements` / `ensureHtmlSelectorSatisfies` parse the DOM).
+Before adding a new `Verify`, grep `verify.ts` for the format or assertion you need.
+Reuse or extend the existing helper rather than hand-rolling a near-duplicate.
+Same applies to mock-context and fixture helpers in `tests/unit/**` and `tests/utils.ts`.
 
 ## Smoke Tests (`tests/smoke/`)
 
@@ -112,11 +118,14 @@ const markdownWithContent = asMappedString("# Title\nSome content");
 
 **Mock Contexts:**
 
-Several subsystems use context interfaces passed to functions. For unit tests, create `createMock*()` helpers with no-op stubs. Key pattern: async callbacks (like `withSpinner`) should just `await op()` so errors propagate normally. Check existing test files for helpers before writing new ones.
-
-| Context | Interface | Existing helpers |
-|---------|-----------|-----------------|
-| `ProjectContext` | `src/project/types.ts` | `tests/unit/project/utils.ts` → `createMockProjectContext()` |
-| `InstallContext` | `src/tools/types.ts` | `tests/unit/tools/chrome-headless-shell.test.ts` → `createMockContext()` |
-| `Format` | `src/config/types.ts` | `tests/unit/format-utils.ts` → `createMockFormat()` (wraps the real `createFormat()` from `src/format/formats-shared.ts` — no hand-rolled cast) |
-| `TempContext` | `src/core/temp-types.ts` | No mock needed — use the real `createTempContext()` from `src/core/temp.ts` directly (see `tests/unit/sass-cache.test.ts`, `tests/unit/ral/safe-remove-dir.test.ts`) |
+Several subsystems use context interfaces passed to functions.
+For unit tests, create `createMock*()` helpers with no-op stubs.
+Key pattern: async callbacks (like `withSpinner`) should just `await op()` so errors propagate normally.
+Check existing test files for helpers before writing new ones.
+
+| Context          | Interface                | Existing helpers                                                                                                                                                     |
+| ---------------- | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ProjectContext` | `src/project/types.ts`   | `tests/unit/project/utils.ts` → `createMockProjectContext()`                                                                                                         |
+| `InstallContext` | `src/tools/types.ts`     | `tests/unit/tools/chrome-headless-shell.test.ts` → `createMockContext()`                                                                                             |
+| `Format`         | `src/config/types.ts`    | `tests/unit/format-utils.ts` → `createMockFormat()` (wraps the real `createFormat()` from `src/format/formats-shared.ts` — no hand-rolled cast)                      |
+| `TempContext`    | `src/core/temp-types.ts` | No mock needed — use the real `createTempContext()` from `src/core/temp.ts` directly (see `tests/unit/sass-cache.test.ts`, `tests/unit/ral/safe-remove-dir.test.ts`) |

.github/actions/build-dist-tarball/action.yml

@@ -0,0 +1,60 @@
+name: "Build Quarto dist tarball"
+description: |
+  Builds a Quarto distribution tarball from the current checkout:
+  configure.sh, quarto-bld prepare-dist --set-version, then a tar.gz of
+  package/pkg-working (top-level directory quarto-<version>) uploaded as a
+  workflow artifact. Shared by create-release.yml (make-tarball /
+  make-arm64-tarball) and test-smokes-built.yml.
+inputs:
+  version:
+    description: "Version to stamp into the distribution (--set-version)"
+    required: true
+  arch:
+    description: "Target architecture: amd64 or arm64"
+    required: false
+    default: "amd64"
+  tarball-name:
+    description: "File name of the produced .tar.gz"
+    required: true
+  artifact-name:
+    description: "Workflow artifact name to upload the tarball as"
+    required: true
+  retention-days:
+    description: "Artifact retention in days (empty = repo default; upload-artifact ignores an empty value)"
+    required: false
+    default: ""
+runs:
+  using: "composite"
+  steps:
+    - name: Configure
+      shell: bash
+      run: ./configure.sh
+
+    - name: Prepare Distribution
+      shell: bash
+      run: |
+        pushd package/src/
+        if [ "${{ inputs.arch }}" = "arm64" ]; then
+          ./quarto-bld prepare-dist --set-version ${{ inputs.version }} --arch aarch64 --log-level info
+        else
+          ./quarto-bld prepare-dist --set-version ${{ inputs.version }} --log-level info
+        fi
+        popd
+
+    - name: Make Tarball
+      shell: bash
+      run: |
+        pushd package/
+        mv pkg-working quarto-${{ inputs.version }}
+        tar --owner=root --group=root -czf "${{ inputs.tarball-name }}" quarto-${{ inputs.version }}
+        mv quarto-${{ inputs.version }} pkg-working
+        popd
+
+    - name: Upload Artifact
+      uses: actions/upload-artifact@v7
+      with:
+        name: ${{ inputs.artifact-name }}
+        path: ./package/${{ inputs.tarball-name }}
+        retention-days: ${{ inputs.retention-days }}
+        # the tarball is already gzipped - re-compressing wastes CPU
+        compression-level: 0

.github/actions/merge-extension-tests/action.yml

@@ -11,6 +11,12 @@ runs:
     - name: Merge julia-engine tests
       shell: bash
       run: |
+        # The upstream tests do not sanitize the environment passed to quarto.
+        # Skip them until PumasAI/quarto-julia-engine strips the dev-tree vars.
+        if [[ -n "$QUARTO_TEST_BIN" ]]; then
+          echo "::notice title=julia-engine tests skipped in binary mode::subtree tests do not sanitize their quarto spawn env yet (see llm-docs/built-version-testing-architecture.md)"
+          exit 0
+        fi
         SUBTREE=src/resources/extension-subtrees/julia-engine/tests
         cp -r "$SUBTREE/docs/julia-engine" tests/docs/julia-engine
         cp -r "$SUBTREE/smoke/julia-engine" tests/smoke/julia-engine