feat: add setup-mocks, a non-isolated suite with mocking setup files

README.md

@@ -2,7 +2,7 @@
 
 > The apps, generators, and bench runner in this repository were generated with [Claude Fable 5](https://www.anthropic.com/news/claude-fable-5-mythos-5) and reviewed by the Vitest team. The results below were measured by hand on the hardware stated in each section.
 
-Generated reference apps for measuring Vitest performance. Each app stands for one kind of project: a tiny utility package, a library, barrel-file graphs, DOM component suites, a dependency-heavy service, a 1300-module monolith, and a long-running DOM suite. The bench runner measures the options that change run time on each of them: `pool`, `environment` (jsdom, happy-dom, and headless Chromium via browser mode), `isolate`, `fsModuleCache`, `maxWorkers`, and cold vs warm caches.
+Generated reference apps for measuring Vitest performance. Each app stands for one kind of project: a tiny utility package, a library, barrel-file graphs, DOM component suites, a dependency-heavy service, a 1300-module monolith (and a variant with 800 test files), a long-running DOM suite, a large non-isolated suite with mocking setup files, and a non-isolated suite whose later files reuse graphs other workers loaded. The bench runner measures the options that change run time on each of them: `pool`, `environment` (jsdom, happy-dom, and headless Chromium via browser mode), `isolate`, `fsModuleCache`, `maxWorkers`, and cold vs warm caches.
 
 ## Usage
 
@@ -244,6 +244,12 @@ A large monorepo: about 1280 modules with 12-deep import chains, import cycles,
 | **threads** | **false** | **true** | **default** | 2.48s | 2.45s | ~0 | 1.90s | 1.98s | +5% |
 | forks | false | false | 50% | — | — | — | 3.36s | 3.17s | −6% |
 
+### enterprise-monolith-xl
+
+enterprise-monolith's graph with 800 test files instead of 150, from the same generator, all in the node environment. That is about 70 files per worker on 12 cores instead of about 14, so under `isolate: false` whatever a reused worker pays per file over a large shared graph is paid five times as often. The original's jsdom files are left out: a worker isn't reused across environments, so mixing them in replaces workers every few files. The default matrix runs only `isolate: false`.
+
+Not measured for 4.1.10 vs 5.0.0 yet.
+
 ### long-haul
 
 A long-running DOM suite: 80 jsdom test files through 2 workers, each file holding a 15MB module-level dataset and rendering tables over it. Node pools rebuild the environment and import the external dependencies again for each of a worker's 40 files. vm pool workers pay once, reuse compiled scripts across contexts, and get recycled by the pinned 512MB `vmMemoryLimit` several times per run. This is the app where the vm pools win by a wide margin. Memory retention across files is out of scope here.
@@ -257,6 +263,18 @@ A long-running DOM suite: 80 jsdom test files through 2 workers, each file holdi
 | forks | happy-dom | — | — | — | 11.38s | 10.89s | −4% |
 | **vmForks** | **happy-dom** | — | — | — | 5.43s | 4.06s | −25% |
 
+### setup-mocks
+
+A large suite meant to run with `isolate: false`: a shared tree of 1500 modules that every test file evaluates through its root, 800 test files that each also import one module of their own, and 6 setup files that each `vi.mock` a service the tests assert on. A reused worker re-runs the setup files before each of its files, so whatever Vitest does per file or per run request (resolving setup files and mock paths, asking the main process for warm modules) is paid hundreds of times per worker, and each round trip to the main process waits behind the other workers' calls. The 25% and 50% `maxWorkers` rows show how that cost scales with the number of workers; with `maxWorkers: 1` all files go to one worker as a single run request, so only the per-file cost remains, with no other worker waiting on the main process. `isolate: true` takes about a minute here and only runs in the full matrix. Modelled on [vjeux/vitest-warm-rpc-repro](https://github.com/vjeux/vitest-warm-rpc-repro).
+
+Not measured for 4.1.10 vs 5.0.0 yet.
+
+### late-graphs
+
+A non-isolated suite where every worker's first test file is trivial and later files reuse large graphs that files on other workers loaded earlier: 4 trivial files that sort first (the app sorts files by name and fixes `maxWorkers` to 4), then 24 test files sharing 12 chains of 250 modules, so the second dozen reuses the chains the first dozen loaded. Anything a worker fetches or decides once, from its first run request, says nothing about the modules its later files need, so a per-worker shortcut shows up here as one round trip per module.
+
+Not measured for 4.1.10 vs 5.0.0 yet.
+
 ### cpu-bound
 
 30 test files that burn real CPU (hashing, sieving, matrix multiplication) on an 8-module graph. The tests dominate, so only scheduling (`maxWorkers`, pool choice) changes anything.

apps/enterprise-monolith-xl/generate.mjs

@@ -0,0 +1,13 @@
+// enterprise-monolith-xl — enterprise-monolith's graph with 800 test files
+// instead of 150, all in the node environment.
+//
+// The same ~1280-module graph, so the difference is how many files each
+// worker runs over it: about 70 per worker on 12 cores. The original's jsdom
+// files are left out because a worker can't be reused across environments,
+// which would replace workers every few files.
+// Under isolate:false, anything a reused worker pays per file over a large
+// shared graph (such as a warm-module snapshot of the file's graph) grows with
+// that count, which the smaller app keeps too low to see.
+import { generate } from '../enterprise-monolith/generate.mjs'
+
+generate(import.meta.url, { name: 'enterprise-monolith-xl', tests: 800, jsdom: false })

apps/enterprise-monolith-xl/package.json

@@ -0,0 +1,20 @@
+{
+  "name": "@bench-app/enterprise-monolith-xl",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "generate": "node generate.mjs",
+    "test": "vitest run"
+  },
+  "dependencies": {
+    "dayjs": "1.11.21",
+    "zod": "4.4.3"
+  },
+  "devDependencies": {
+    "@vitest/coverage-istanbul": "4.1.10",
+    "@vitest/coverage-v8": "4.1.10",
+    "jsdom": "29.1.1",
+    "vite": "8.1.4",
+    "vitest": "4.1.10"
+  }
+}

apps/enterprise-monolith-xl/vitest.config.ts

@@ -0,0 +1,16 @@
+import { fileURLToPath } from 'node:url'
+import { defineConfig } from 'vitest/config'
+import { benchTest } from '../../tools/config/bench-config.js'
+
+export default defineConfig({
+  resolve: {
+    alias: {
+      '@util': fileURLToPath(new URL('./src/util', import.meta.url)),
+      '@core': fileURLToPath(new URL('./src/core', import.meta.url)),
+      '@svc': fileURLToPath(new URL('./src/svc', import.meta.url)),
+    },
+  },
+  test: benchTest({
+    environment: 'node',
+  }),
+})

apps/enterprise-monolith/generate.mjs

@@ -14,6 +14,7 @@
 // Dimensions stressed: module graph scale (transform + per-file RPC fetch
 // waterfall), fs module cache cold/warm, isolate cost at scale, alias
 // resolution, mixed environments.
+import { pathToFileURL } from 'node:url'
 import { createApp, tsModule } from '../../tools/generator/helpers.mjs'
 
 const UTIL = 400
@@ -22,10 +23,12 @@ const SVC = 250
 const FEAT = 250
 const CHAIN = 60
 const GROUPS = 10
-const TESTS = 150
 const CYCLES = [20, 70, 120, 170, 220]
 
-const app = createApp(import.meta.url)
+// the -xl variant reuses this graph with more test files; the body stays
+// unindented so the template literals keep their output byte-for-byte
+export function generate(importMetaUrl, { name, tests: TESTS, jsdom = true }) {
+const app = createApp(importMetaUrl)
 
 for (let i = 0; i < UTIL; i++)
   app.write(`src/util/u${i}.ts`, tsModule(`u${i}`, i))
@@ -137,7 +140,7 @@ ${tsModule(`d${i}`, i)}`)
 for (let t = 0; t < TESTS; t++) {
   const feats = [(t * 3) % FEAT, (t * 3 + 11) % FEAT, (t * 3 + 23) % FEAT]
   const viaBarrel = t % 3 === 0
-  const dom = t % 10 === 7
+  const dom = jsdom && t % 10 === 7
   const imports = feats.map((f, idx) => viaBarrel && idx === 0
     ? `import { f${f}Render } from '../src/feat/group-${Math.floor(f / perGroup)}'`
     : `import { f${f}Render } from '../src/feat/f${f}'`)
@@ -184,4 +187,8 @@ ${extraTests.join('\n')}
 `)
 }
 
-app.report('enterprise-monolith', `${UTIL + CORE + SVC + FEAT + CHAIN + GROUPS} src modules + ${GROUPS} json, ${TESTS} test files (15 jsdom via pragma)`)
+app.report(name, `${UTIL + CORE + SVC + FEAT + CHAIN + GROUPS} src modules + ${GROUPS} json, ${TESTS} test files (${jsdom ? Math.ceil((TESTS - 7) / 10) : 0} jsdom via pragma)`)
+}
+
+if (import.meta.url === pathToFileURL(process.argv[1]).href)
+  generate(import.meta.url, { name: 'enterprise-monolith', tests: 150 })

apps/late-graphs/generate.mjs

@@ -0,0 +1,59 @@
+// late-graphs — reused workers whose later files need modules other workers
+// already loaded.
+//
+// A non-isolated suite where every worker's first test file is trivial, and
+// the files after it import large graphs that a file on some other worker
+// loaded earlier. Whatever a worker learns from its first run request (a
+// warm-module snapshot, for example) says nothing about the modules its later
+// files need, so anything decided once per worker instead of once per file
+// shows up here as one round trip per module.
+//
+//   - trivial: TRIVIAL files that sort first (the config sorts files by
+//     name and fixes maxWorkers to TRIVIAL), one per worker
+//   - chains: CHAINS chains of CHAIN modules; a chain is discovered one
+//     import at a time, like a deep dependency
+//   - heavy: HEAVY files, file h importing chain h % CHAINS, so the second
+//     half reuses the chains the first half loaded
+import { createApp, tsModule } from '../../tools/generator/helpers.mjs'
+
+const TRIVIAL = 4
+const CHAINS = 12
+const HEAVY = 24
+const CHAIN = 250
+
+const app = createApp(import.meta.url)
+
+for (let t = 0; t < TRIVIAL; t++) {
+  app.write(`tests/a-trivial-${t}.test.ts`, `import { expect, it } from 'vitest'
+
+it('is trivial', () => {
+  expect(${t}).toBe(${t})
+})
+`)
+}
+
+for (let c = 0; c < CHAINS; c++) {
+  for (let i = 0; i < CHAIN; i++) {
+    const next = i + 1 < CHAIN
+      ? `import { c${c}m${i + 1}Walk } from './m${i + 1}'\n`
+      : ''
+    const walk = `
+export function c${c}m${i}Walk(): number {
+  return 1${i + 1 < CHAIN ? ` + c${c}m${i + 1}Walk()` : ''}
+}`
+    app.write(`src/c${c}/m${i}.ts`, `${next}${tsModule(`c${c}m${i}`, i, walk)}`)
+  }
+}
+
+for (let h = 0; h < HEAVY; h++) {
+  const c = h % CHAINS
+  app.write(`tests/b-heavy-${String(h).padStart(2, '0')}.test.ts`, `import { expect, it } from 'vitest'
+import { c${c}m0Walk } from '../src/c${c}/m0'
+
+it('walks chain ${c}', () => {
+  expect(c${c}m0Walk()).toBe(${CHAIN})
+})
+`)
+}
+
+app.report('late-graphs', `${TRIVIAL} trivial files first, ${HEAVY} heavy files sharing ${CHAINS} chains of ${CHAIN} modules`)

apps/late-graphs/package.json

@@ -0,0 +1,15 @@
+{
+  "name": "@bench-app/late-graphs",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "generate": "node generate.mjs",
+    "test": "vitest run"
+  },
+  "devDependencies": {
+    "@vitest/coverage-istanbul": "4.1.10",
+    "@vitest/coverage-v8": "4.1.10",
+    "vite": "8.1.4",
+    "vitest": "4.1.10"
+  }
+}

apps/late-graphs/vitest.config.ts

@@ -0,0 +1,19 @@
+import type { TestSpecification } from 'vitest/node'
+import { BaseSequencer } from 'vitest/node'
+import { defineConfig } from 'vitest/config'
+import { benchTest } from '../../tools/config/bench-config.js'
+
+// files run in name order, so every worker starts with a trivial file
+class NameOrderSequencer extends BaseSequencer {
+  async sort(files: TestSpecification[]) {
+    return [...files].sort((a, b) => a.moduleId.localeCompare(b.moduleId))
+  }
+}
+
+export default defineConfig({
+  test: benchTest({
+    environment: 'node',
+    maxWorkers: 4,
+    sequence: { sequencer: NameOrderSequencer },
+  }),
+})

apps/setup-mocks/generate.mjs

@@ -0,0 +1,85 @@
+// setup-mocks — a large non-isolated suite whose setup files mock modules.
+//
+// The shape of the 6,600-file suite reported in vitest-dev/vitest#11752:
+// `isolate: false`, a wide shared module graph, many test files, and a
+// handful of setup files that register `vi.mock` calls for every test file. A
+// reused worker re-runs the setup files before each of its files, so whatever
+// the runtime does per file or per run request (resolving setup files and
+// their mock paths, asking the main process for warm module snapshots, ...)
+// is paid hundreds of times per worker, and every round trip to the main
+// process queues behind the other workers' calls. No other app has mocking
+// setup files, so none of them can see this cost.
+//
+//   - core: a binary tree of CORE modules; every test file evaluates all of
+//     it through the root, like a monorepo's shared packages
+//   - features: one module per test file that no other test file imports,
+//     so every file also loads something new to its worker
+//   - services: SETUPS modules, each mocked by one setup file; tests assert
+//     the mocked values so the mocks are load-bearing
+//
+// Modelled on https://github.com/vjeux/vitest-warm-rpc-repro, scaled down
+// so the app stays affordable in the default matrix.
+import { createApp, tsModule } from '../../tools/generator/helpers.mjs'
+
+const CORE = 1500
+const SETUPS = 6
+const TESTS = 800
+
+const app = createApp(import.meta.url)
+
+for (let i = 0; i < CORE; i++) {
+  const children = [2 * i + 1, 2 * i + 2].filter(c => c < CORE)
+  const imports = children.map(c => `import { m${c}Walk } from './m${c}'`).join('\n')
+  const walk = `
+export function m${i}Walk(): number {
+  return 1${children.map(c => ` + m${c}Walk()`).join('')}
+}`
+  app.write(`src/core/m${i}.ts`, `${imports}${imports ? '\n' : ''}${tsModule(`m${i}`, i, walk)}`)
+}
+
+for (let s = 0; s < SETUPS; s++) {
+  app.write(`src/services/s${s}.ts`, `export function s${s}Fetch(): string {
+  return 'real-s${s}'
+}
+`)
+  app.write(`tests/setup/setup${s}.ts`, `import { vi } from 'vitest'
+
+vi.mock('../../src/services/s${s}', () => ({
+  s${s}Fetch: () => 'mocked-s${s}',
+}))
+`)
+}
+
+for (let t = 0; t < TESTS; t++) {
+  // each feature leans on a different deep node of the shared tree
+  const node = CORE - 1 - (t % (CORE >> 1))
+  app.write(`src/features/f${t}.ts`, `import { m${node}Compute, m${node}Default, m${node}Format } from '../core/m${node}'
+
+export function f${t}Run(): string {
+  return m${node}Format(m${node}Compute({ ...m${node}Default, id: ${t} }))
+}
+`)
+
+  const s = t % SETUPS
+  app.write(`tests/t${t}.test.ts`, `import { describe, expect, it } from 'vitest'
+import { m0Walk } from '../src/core/m0'
+import { f${t}Run } from '../src/features/f${t}'
+import { s${s}Fetch } from '../src/services/s${s}'
+
+describe('t${t}', () => {
+  it('evaluates the shared tree', () => {
+    expect(m0Walk()).toBe(${CORE})
+  })
+
+  it('runs its own feature', () => {
+    expect(f${t}Run()).toMatch(/^ok:/)
+  })
+
+  it('sees the mock from the setup file', () => {
+    expect(s${s}Fetch()).toBe('mocked-s${s}')
+  })
+})
+`)
+}
+
+app.report('setup-mocks', `${CORE} core modules, ${TESTS} test files, ${SETUPS} mocking setup files`)

apps/setup-mocks/package.json

@@ -0,0 +1,15 @@
+{
+  "name": "@bench-app/setup-mocks",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "generate": "node generate.mjs",
+    "test": "vitest run"
+  },
+  "devDependencies": {
+    "@vitest/coverage-istanbul": "4.1.10",
+    "@vitest/coverage-v8": "4.1.10",
+    "vite": "8.1.4",
+    "vitest": "4.1.10"
+  }
+}

apps/setup-mocks/vitest.config.ts

@@ -0,0 +1,9 @@
+import { defineConfig } from 'vitest/config'
+import { benchTest } from '../../tools/config/bench-config.js'
+
+export default defineConfig({
+  test: benchTest({
+    environment: 'node',
+    setupFiles: Array.from({ length: 6 }, (_, s) => `./tests/setup/setup${s}.ts`),
+  }),
+})