From 37acc78053929a1bc55e08365af779e8d59d7d50 Mon Sep 17 00:00:00 2001 From: Diogo Esteves Date: Thu, 29 Oct 2026 10:49:30 +0000 Subject: [PATCH 1/3] docs: CLI reference with a drift test, stack matrix, measured performance, custom-rules FAQ docs/cli.md lists every command and option of the CLI, under the commands --help gives them to, with check's exit codes. Before it, --quiet-without-design-system was in no doc, and --baseline, --ignore-baseline, --include-tests, --components, --docs and --no-watch were each named once, in passing. test/cli-docs.test.ts holds the page to --help and --help to the parser (src/cli.ts), both ways, checks that it names every config file name (onsystem.config.mts too), and that every GitHub Action input has its row and default in docs/required-check.md. The site renders the page at /docs/cli. The quickstart gets a "What it works with" table: UI code, styling, variants, components, tokens, monorepos, agents, CI and runtime, with what is not supported yet. docs/how-it-works.md said only that onsystem "takes milliseconds". Its new "Performance and footprint" section has a table written by scripts/perf.mjs: check on two corpus apps (per run and per file), the Claude Code hook on one edit with an install and through npx, the MCP server's handshake, first call and tool calls, and the package size, install size and provenance, with the machine named. test/perf-docs.test.ts holds the table to what the script writes and to the corpus commits it measured. The FAQ's "takes milliseconds" points to the table too. The FAQ answers "Can I add our own rules?": the rules are a fixed set, a raw next to a Link is already a finding, a team-only rule is an ESLint rule, and CONTRIBUTING.md has the steps for a built-in one. --- docs/cli.md | 95 ++++++++ docs/how-it-works.md | 22 +- docs/quickstart.md | 18 +- scripts/perf.mjs | 416 ++++++++++++++++++++++++++++++++++ site/app/docs/cli/page.tsx | 15 ++ site/lib/data.ts | 1 + site/lib/docs.ts | 8 + site/lib/faq.ts | 6 +- site/lib/llms.ts | 1 + site/lib/site.ts | 1 + site/scripts/generate.mjs | 2 + site/test/docs-agree.test.ts | 2 +- site/test/docs-shell.test.ts | 12 +- test/cli-docs.test.ts | 249 ++++++++++++++++++++ test/custom-rules-faq.test.ts | 65 ++++++ test/perf-docs.test.ts | 114 ++++++++++ 16 files changed, 1022 insertions(+), 5 deletions(-) create mode 100644 docs/cli.md create mode 100644 scripts/perf.mjs create mode 100644 site/app/docs/cli/page.tsx create mode 100644 test/cli-docs.test.ts create mode 100644 test/custom-rules-faq.test.ts create mode 100644 test/perf-docs.test.ts diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 00000000..1dd6c1b4 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,95 @@ +# CLI reference + +`onsystem` is one command. With no command it is the MCP server an agent starts; `check`, `inspect`, `export` and `init` are for the terminal and CI. With nothing installed, run it through npx; installed as a dev dependency, which pins the version for the team and CI, `npx onsystem` runs that install: + +```sh pm +npx -y onsystem check . +npx onsystem check . +``` + +It needs Node.js 20.19 or later. The options are the ones `onsystem --help` prints, listed under the commands that take them, and a test keeps this page and the help in step. + +| Command | What it does | +| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | +| `onsystem`, `onsystem serve` | Starts the MCP server on stdio, for an agent ([set up your agent](clients.md)) | +| `onsystem check ` | Lints files, folders or globs with the rules `check_ui` runs, and exits 1 on errors ([CI and baselines](ci.md)) | +| `onsystem inspect` | Prints what it read: where it found the design system, the components with their props, variants and parts, and the tokens by category | +| `onsystem export --format ` | Prints the tokens it found as a DTCG 2025.10 file (`dtcg`) or a DESIGN.md (`design-md`) ([export](tokens.md#export)) | +| `onsystem init` | Writes `onsystem.config.json` from what it found, checks the project once and offers to record the findings as a baseline | +| `onsystem help` | Prints the help, as `--help` does | + +At a monorepo root with no design system of its own, every command works per project: each file is checked against the design system of the nearest folder with a config, a `components.json` or a `package.json`, and `inspect` lists every project ([monorepo roots](configuration.md#monorepo-roots)). + +## Options for every command + +| Option | What it does | +| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `--root ` | The project root. By default, the folder of the file `--config` names, or the working directory | +| `--config ` | The config file. By default, the first of `onsystem.config.json`, `onsystem.config.ts`, `onsystem.config.mts`, `onsystem.config.js` and `onsystem.config.mjs` in the root; the TypeScript ones need Node.js 22.18 or later ([config file](configuration.md#config-file)) | +| `--components ` | Component sources, repeatable. By default, what [zero config](configuration.md#zero-config) finds | +| `--tokens ` | A token file, repeatable: a DTCG `.json`, a stylesheet or a `DESIGN.md` ([where tokens come from](tokens.md#where-tokens-come-from)) | +| `--docs ` | Component docs in Markdown or MDX, repeatable | +| `--no-cache` | Ignores the cache of what it read from the design system (in `node_modules/.cache/onsystem`, or the system temp folder without a `node_modules`), and reads it all again | +| `-h`, `--help` | Prints the help | +| `-v`, `--version` | Prints the version | + +## `serve` + +`onsystem` or `onsystem serve` starts the MCP server on stdio. An agent starts it from its MCP config; typed in a terminal, it says that it is waiting for an agent, and Ctrl+C quits. It answers the handshake right away and reads the design system in the background, then reads it again when the component, token or docs files, the config or the tsconfig change ([how it works](how-it-works.md)). Started at a monorepo root, it serves every project ([how the server finds your project](clients.md#how-the-server-finds-your-project)). + +| Option | What it does | +| ------------ | ------------------------------------------------------- | +| `--no-watch` | Does not read the design system again when files change | + +## `check` + +`onsystem check ` lints the files, folders and globs it is given with the rules `check_ui` runs ([rules](rules.md)). A folder stands for its `.tsx` and `.jsx` files; folders and globs leave out the design system's own component files, tests and stories, what the config's `exclude` adds and what git ignores, while a file named outright is always checked ([CI and baselines](ci.md)). + +It exits with: + +- `0` when there are no errors, and no more warnings than `--max-warnings`. +- `1` when there are errors, or more warnings than `--max-warnings`. +- `2` when it could not check: an unknown option, a broken config, paths that match no file, a file it could not read, or, with `--require-design-system`, no design system or nothing to check. + +| Option | What it does | +| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `--format ` | `pretty` (the default), `json`, `github` (annotations on the pull request) or `sarif` ([code scanning](ci.md#code-scanning-and-other-sarif-tools)) | +| `--max-warnings ` | Exits 1 when there are more than `n` warnings | +| `--update-baseline` | Records the current findings as accepted in the baseline, and exits 0 ([adopting it in an existing codebase](ci.md#adopting-it-in-an-existing-codebase)) | +| `--prune-baseline` | Drops the baseline entries that no longer occur, and accepts nothing new | +| `--baseline ` | The baseline file. By default `onsystem.baseline.json` in the root, used whenever it exists | +| `--ignore-baseline` | Reports every finding, in the baseline or not | +| `--include-design-system` | Also lints the design system's own component files, which are skipped by default (`"includeDesignSystem": true` in the config does the same) | +| `--include-tests` | Also lints tests and stories (`*.test.tsx`, `*.spec.tsx`, `*.stories.tsx`) in folders and globs | +| `--skip-excluded` | Leaves out the files named outright that the config's `exclude` covers, as folders and globs leave them out. The Claude Code hook passes it | +| `--quiet-without-design-system` | Prints nothing and exits 0 when the project has no design-system components and no config (tokens alone are not a design system), for a hook installed across many projects | +| `--require-design-system` | Exits 2 when no components or no color tokens are found, or when the folders and globs it is given leave no file to check, so CI cannot pass by checking nothing | + +`check` exits 2 when it is given `--quiet-without-design-system` with `--require-design-system`, or `--prune-baseline` with `--update-baseline` or `--ignore-baseline`. Output is in color in a terminal, unless `NO_COLOR` is set. The [GitHub Action](required-check.md) sets `ONSYSTEM_CHANGED_LINES` to a file of the lines a pull request changed; unset it if it is left over in a shell where you run `check` yourself. + +## `inspect` + +`onsystem inspect` prints what it read from the design system: the root and config, where it found the design system (the `detected` line), the components with their props, variants and parts, the tokens by category, and what it could not read. At a monorepo root it lists every project. When it found nothing, see [troubleshooting](troubleshooting.md#inspect-found-nothing). + +| Option | What it does | +| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `--explain` | Adds every candidate zero config looked at and why it was taken or not, the config it resolved to, the tokens that did not resolve and the components whose props did not | +| `--format ` | `json` prints what it read as JSON, for scripts | + +## `export` + +`onsystem export --format dtcg` or `--format design-md` prints the tokens it found as a DTCG 2025.10 file or a DESIGN.md, which other tools read ([export](tokens.md#export)). At a monorepo root, pass `--root` with one project. + +| Option | What it does | +| ------------------- | ------------------------------------------- | +| `--format ` | `dtcg` or `design-md`, which `export` needs | +| `--out ` | Writes to this file instead of stdout | + +## `init` + +`onsystem init` writes `onsystem.config.json` from what zero config found, or asks where the components, tokens and import path are when it found nothing. Then it checks the project once and offers to record the findings as a baseline, and in a terminal it offers to add a short section on onsystem to `AGENTS.md` or `CLAUDE.md`. It never replaces an existing config without `--force`. At a monorepo root it writes a config with only `$schema`, and in a design-system package it also declares the config in `package.json` and writes `ONSYSTEM.md` for the apps' teams ([config file](configuration.md#config-file), [design-system packages](design-system-packages.md)). + +| Option | What it does | +| ------------- | ----------------------------------------------- | +| `-y`, `--yes` | Writes without asking, and records the baseline | +| `--force` | Replaces an existing config file | diff --git a/docs/how-it-works.md b/docs/how-it-works.md index 91f406e7..ef1fca0f 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -21,6 +21,26 @@ Run as is, with no config, on public apps: the counts the [corpus](../corpus) sn | [midday](https://github.com/midday-ai/midday) `5158731`, `apps/dashboard` | workspace package `@midday/ui` → 79 components | 1,340 in `src/`, including 98 icon-only buttons without an accessible name. Adopted with a [baseline](ci.md#adopting-it-in-an-existing-codebase) | | [shadcn/ui](https://github.com/shadcn-ui/ui) website `0132174` | custom `ui` alias → 68 components | 124 in `app/` and `components/`, mostly raw colors (105) | +## Performance and footprint + +Reading the design system is the slow part: one TypeScript program over the component files. `check` and the server cache what they read and read it again only when those files change, so checking a file after that is syntax work. The hook runs one `check` per edit to a UI file. Without onsystem installed in the project, it runs `check` through npx, which starts npm first: a dev dependency ([pin the version](rollout.md#pin-the-version)) takes that out of every edit. + +Measured with onsystem 0.11.0 on Apple M1 Max (10 cores, 32 GB), macOS 27.0.1, Node.js 24.18.0, on 2026-10-29, with `node scripts/perf.mjs`: each time is the median of 5 runs. The corpus checkouts have no `node_modules`, so the types of the libraries their components wrap are not read; the demo has its dependencies installed. + +| Measured | Result | +| -------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | +| `check` on [vercel/chatbot](https://github.com/vercel/chatbot) `c2f8235` (`app` and `components`: 61 files) | 0.51 s, or 0.80 s with `--no-cache`. One file: 347 ms, then 2.6 ms for each other file | +| `check` on [midday-ai/midday](https://github.com/midday-ai/midday) `5158731` (`apps/dashboard/src`: 569 files) | 1.35 s, or 1.68 s with `--no-cache`. One file: 465 ms, then 1.6 ms for each other file | +| The Claude Code hook on one edit (the demo's `app/settings/danger-zone.tsx`), with onsystem installed in the project | 364 ms | +| The same edit through npx, as the plugin runs it in a project without an install (`onsystem@0.11.0` in npm's cache) | 1.14 s | +| The MCP server on [midday-ai/midday](https://github.com/midday-ai/midday) `5158731` (`apps/dashboard`, 79 components): `initialize` answered | 422 ms after it starts | +| Its first tool call, which waits for the design system to be read (`--no-cache`) | 0.80 s after it starts | +| `check_ui` on the app's largest file (1,349 lines) | 13 ms | +| `search_components`, `get_component` (`Button`) | 0.5 ms, 0.4 ms | +| The package (`onsystem@0.11.0` on npm) | 0.9 MB unpacked, 16 files, 8 dependencies | +| Installed with npm into an empty project | 15 packages, 42.2 MB on disk (24.3 MB of it TypeScript, which reads the components) | +| Provenance | Signed npm provenance (slsa.dev/provenance/v1): npm shows the workflow run and commit each release was built from | + ## How it compares These tools work at different layers, and several combine well: @@ -37,7 +57,7 @@ These tools work at different layers, and several combine well: ## Design decisions -**Static analysis, not another model.** The agent is already the LLM; what it lacks is ground truth. Everything here is deterministic, takes milliseconds, makes no network requests, needs no API key, and can be unit-tested rule by rule. The cost: it cannot judge intent, such as whether a `Dialog` was the right call. That stays with the agent and the reviewer. +**Static analysis, not another model.** The agent is already the LLM; what it lacks is ground truth. Everything here is deterministic, checks a file in milliseconds once the design system is read ([measured](#performance-and-footprint)), makes no network requests, needs no API key, and can be unit-tested rule by rule. The cost: it cannot judge intent, such as whether a `Dialog` was the right call. That stays with the agent and the reviewer. **The TypeScript checker over `react-docgen-typescript`.** react-docgen-typescript wraps the same API but hides the AST, and `cva()` parsing, composition and element inference all need it. One program serves all four. The runtime dependency is TypeScript 6, the last release with the JavaScript compiler API; TypeScript 7 (the Go port) does not expose a stable one yet. diff --git a/docs/quickstart.md b/docs/quickstart.md index cef8e3ff..308b61c4 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -58,6 +58,22 @@ npx onsystem check . --update-baseline # writes onsystem.baseline.json: commit - run: npx onsystem check . --format github --require-design-system ``` -`check .` lints every `.tsx` and `.jsx` file under the folder, whatever the layout, and leaves out the design system's own components, tests and stories, and what git ignores. [CI and baselines](ci.md) has the whole workflow. +`check .` lints every `.tsx` and `.jsx` file under the folder, whatever the layout, and leaves out the design system's own components, tests and stories, and what git ignores. [CI and baselines](ci.md) has the whole workflow, and the [CLI reference](cli.md) every option. + +## What it works with + +| | Supported | Not supported yet | +| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | +| UI code | React, in `.tsx` and `.jsx` files, and `.js` files with JSX that a glob names | Vue, Svelte, Angular | +| Styling | Tailwind CSS v3 and v4 classes, `style` objects and color attributes | CSS-in-JS (styled-components, Emotion), CSS Modules, plain stylesheets | +| Variants | `cva()`, `tv()` (tailwind-variants), and props typed as unions of string literals | — | +| Components | Your own, such as shadcn/ui's or ones built on Radix, React Aria, Base UI or another primitives library, in the app or in a workspace package; a design-system package installed with its type declarations ([design-system packages](design-system-packages.md)); MUI (`@mui/material`) | Chakra UI, Mantine, HeroUI, Radix Themes, Ant Design, MUI's other packages | +| Tokens | CSS custom properties (Tailwind v4's `@theme`, shadcn/ui's `:root` variables), a Tailwind v3 `tailwind.config`, DTCG 2025.10 `.tokens.json` files, a `DESIGN.md` ([tokens](tokens.md)) | Values computed in code | +| Monorepos | pnpm, npm, Yarn and Bun workspaces, Lerna, Nx and Turborepo ([monorepo roots](configuration.md#monorepo-roots)) | — | +| Agents | The Claude Code plugin (hook, MCP server and skill), its hook in Codex CLI, and the MCP server in Cursor, VS Code and [other clients](clients.md) | — | +| CI and editors | `onsystem check` in any CI with Node.js, the [GitHub Action](required-check.md), and the [ESLint plugin](eslint.md) for ESLint 8.57, 9 and 10 with a flat config | — | +| Runtime | Node.js 20.19 or later (22.18 or later for a TypeScript config file), on macOS, Linux and Windows | — | + +Zero config finds the design system in the common layouts ([how](configuration.md#zero-config)); the others take a config file with a `components` glob. [How it works](how-it-works.md#limits) lists what it does not check yet. diff --git a/scripts/perf.mjs b/scripts/perf.mjs new file mode 100644 index 00000000..40529ac2 --- /dev/null +++ b/scripts/perf.mjs @@ -0,0 +1,416 @@ +// How long onsystem takes and what it installs: the "Performance and footprint" table in +// docs/how-it-works.md. +// +// node scripts/perf.mjs measure, and print the table +// node scripts/perf.mjs --write measure, and write the table into docs/how-it-works.md +// +// Run `pnpm build` first, and `pnpm corpus fetch --only chatbot --only midday` (same --dir or +// CORPUS_DIR as here) for the two apps it checks. It measures on this machine, so the numbers +// change with it: the table names the machine, and each time is the median of --runs runs. +// +// - `check` with the built CLI on two corpus apps, as the corpus checks them: every file, with +// the design system read again (--no-cache) and from the cache, then one file, which gives +// the time per file after the first. The checkouts have no node_modules (nothing in them is +// installed), so the types of the libraries their components wrap are not read. +// - The Claude Code hook (plugins/onsystem/hooks/check-ui.mjs) on one edit of the demo, which +// has its dependencies installed: with the built CLI as the project's install, and through +// npx with onsystem@ from npm, as the plugin runs it in a project without an install. +// npx gets an npm cache of its own, filled by a first run that is not counted. +// - The MCP server (the built CLI) on midday's dashboard, over stdio: the `initialize` answer, +// the first tool call, which waits for the design system to be read, then check_ui on the +// app's largest file, search_components and get_component. +// - The package as npm has it (onsystem@): its unpacked size and dependencies, an +// install with npm into an empty folder (--ignore-scripts), and its provenance. +import { spawnSync } from 'node:child_process'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { parseArgs } from 'node:util'; + +import { Client } from '@modelcontextprotocol/sdk/client/index.js'; +import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js'; +import * as prettier from 'prettier'; + +const ROOT = path.resolve(import.meta.dirname, '..'); +const CLI = path.join(ROOT, 'dist', 'cli.js'); +const HOOK = path.join(ROOT, 'plugins', 'onsystem', 'hooks', 'check-ui.mjs'); +const DEMO = path.join(ROOT, 'examples', 'shadcn-demo'); +const DEMO_FILE = 'app/settings/danger-zone.tsx'; +const DOC = path.join(ROOT, 'docs', 'how-it-works.md'); +/** The section the table is in: from its "Measured with" line to the next heading. */ +const SECTION = '\n## Performance and footprint\n'; +const MEASURED = '\nMeasured with onsystem '; +/** The corpus runs it checks, by name in corpus/repos.json. */ +const APPS = ['chatbot', 'midday-dashboard']; +/** The run the MCP server is measured on. */ +const SERVER_APP = 'midday-dashboard'; + +const { values } = parseArgs({ + options: { + write: { type: 'boolean', default: false }, + runs: { type: 'string', default: '5' }, + dir: { type: 'string' }, + }, +}); +const RUNS = Number(values.runs); +const corpusDir = path.resolve( + values.dir ?? process.env.CORPUS_DIR ?? path.join(os.tmpdir(), 'onsystem-corpus'), +); +const { version } = JSON.parse(fs.readFileSync(path.join(ROOT, 'package.json'), 'utf8')); +/** A folder for this run's npm cache, install and hook markers, removed at the end. */ +let scratch; + +try { + if (!Number.isInteger(RUNS) || RUNS < 1) fail('--runs expects a whole number of 1 or more.'); + if (!fs.existsSync(CLI)) fail('dist/cli.js is missing: run `pnpm build` first.'); + const apps = corpusApps(); + scratch = fs.mkdtempSync(path.join(os.tmpdir(), 'onsystem-perf-')); + const machine = describeMachine(); + console.error(`Measuring onsystem ${version} on ${machine}, ${RUNS} runs each.`); + const checks = apps.map(measureCheck); + const hook = measureHook(); + const server = await measureServer(checks.find((c) => c.name === SERVER_APP)); + const pkg = measurePackage(); + const table = render({ machine, checks, hook, server, pkg }); + console.log(table); + if (values.write) { + const doc = fs.readFileSync(DOC, 'utf8'); + const section = doc.indexOf(SECTION); + const start = section === -1 ? -1 : doc.indexOf(MEASURED, section); + const end = doc.indexOf('\n## ', start + 1); + if (start === -1 || end === -1 || doc.indexOf('\n## ', section + 1) < start) { + fail('docs/how-it-works.md has no "Measured with onsystem" paragraph under its heading.'); + } + // Prettier lays the table out, as `pnpm format` would. + const next = `${doc.slice(0, start + 1)}${table}${doc.slice(end)}`; + const options = { ...(await prettier.resolveConfig(DOC)), filepath: DOC }; + fs.writeFileSync(DOC, await prettier.format(next, options)); + console.error('Wrote the table into docs/how-it-works.md.'); + } +} catch (error) { + if (!(error instanceof Failure)) throw error; + console.error(error.message); + process.exitCode = 1; +} finally { + if (scratch) fs.rmSync(scratch, { recursive: true, force: true }); +} + +/** The corpus runs it checks, each with its folder in the corpus checkouts. */ +function corpusApps() { + const repos = JSON.parse(fs.readFileSync(path.join(ROOT, 'corpus', 'repos.json'), 'utf8')).repos; + const runs = repos.flatMap((repo) => repo.runs.map((run) => ({ ...run, repo }))); + return APPS.map((name) => { + const run = runs.find((r) => r.name === name); + if (!run) fail(`corpus/repos.json has no run named ${name}.`); + const cwd = path.join(corpusDir, run.repo.name, run.cwd); + if (!fs.existsSync(cwd)) { + fail( + `${cwd} is missing: run \`pnpm corpus fetch --only ${run.repo.name}\` with the same --dir.`, + ); + } + return { ...run, folder: run.cwd, cwd }; + }); +} + +// ─── Measuring ───────────────────────────────────────────────────────────────── + +/** `check` on every file the run checks, read again and from the cache, and on one file. */ +function measureCheck(app) { + const args = ['check', ...app.check, ...(app.args ?? []), '--format', 'json']; + const all = run(process.execPath, [CLI, ...args], { cwd: app.cwd, codes: [0, 1] }); + const files = JSON.parse(all.stdout).map((r) => r.file); + const one = files[0]; + const cold = median(times(() => cli([...args, '--no-cache'], app.cwd))); + // The first run above wrote the cache: in node_modules/.cache/onsystem, or in the system temp + // folder for a checkout without node_modules. + const warm = median(times(() => cli(args, app.cwd))); + const single = median(times(() => cli(['check', one, '--format', 'json'], app.cwd))); + const largest = files + .map((file) => ({ file, size: fs.statSync(path.resolve(app.cwd, file)).size })) + .sort((a, b) => b.size - a.size || a.file.localeCompare(b.file))[0].file; + return { + name: app.name, + repo: app.repo, + cwd: app.cwd, + folder: app.folder, + paths: app.check, + files: files.length, + cold, + warm, + single, + perFile: (warm - single) / (files.length - 1), + largest, + }; +} + +/** One edit through the hook: with the project's install, then through npx. */ +function measureHook() { + const file = path.join(DEMO, DEMO_FILE); + const payload = JSON.stringify({ + session_id: 'perf', + hook_event_name: 'PostToolUse', + tool_name: 'Write', + tool_input: { file_path: file, content: fs.readFileSync(file, 'utf8') }, + cwd: DEMO, + }); + const tmp = path.join(scratch, 'hook-tmp'); + fs.mkdirSync(tmp); + const env = (extra) => ({ ...process.env, TMPDIR: tmp, TEMP: tmp, TMP: tmp, ...extra }); + const edit = (extra) => + run(process.execPath, [HOOK], { cwd: DEMO, input: payload, env: env(extra), codes: [2] }); + const installed = median(times(() => edit({ ONSYSTEM_BIN: CLI }))); + // npx installs onsystem@ from npm into a cache of its own; the first run fills it. + const npxEnv = { + npm_config_cache: path.join(scratch, 'npm-cache'), + npm_config_ignore_scripts: 'true', + npm_config_update_notifier: 'false', + }; + const npx = run(process.execPath, [HOOK], { + cwd: DEMO, + input: payload, + env: env(npxEnv), + codes: [0, 2], + }); + if (npx.status !== 2) + fail(`the hook could not run onsystem@${version} through npx:\n${npx.stderr}`); + const throughNpx = median(times(() => edit(npxEnv))); + return { installed, npx: throughNpx, file: DEMO_FILE }; +} + +/** The MCP server on one app, over stdio, with check_ui on the largest file its check covers. */ +async function measureServer(app) { + const started = performance.now(); + const transport = new StdioClientTransport({ + command: process.execPath, + args: [CLI, '--root', app.cwd, '--no-watch', '--no-cache'], + stderr: 'pipe', + }); + const client = new Client({ name: 'onsystem-perf', version: '1.0.0' }); + try { + await client.connect(transport); + const initialize = performance.now() - started; + const list = await client.callTool({ name: 'list_components', arguments: {} }); + const firstCall = performance.now() - started; + const components = list.structuredContent?.components ?? []; + const call = async (name, args) => { + const samples = []; + for (let i = 0; i < Math.max(RUNS, 10); i++) { + const t = performance.now(); + const result = await client.callTool({ name, arguments: args }); + samples.push(performance.now() - t); + if (result.isError) fail(`${name} failed: ${JSON.stringify(result.content)}`); + } + return median(samples); + }; + const lines = fs.readFileSync(path.resolve(app.cwd, app.largest), 'utf8').split('\n').length; + const checkUi = await call('check_ui', { path: app.largest }); + const search = await call('search_components', { query: 'confirm a destructive action' }); + const component = components.some((c) => c.name === 'Button') ? 'Button' : components[0]?.name; + const getComponent = await call('get_component', { name: component }); + return { + name: app.name, + repo: app.repo, + folder: app.folder, + components: components.length, + initialize, + firstCall, + checkUi, + lines, + search, + getComponent, + component, + }; + } finally { + await client.close(); + } +} + +/** onsystem@ as npm has it, and installed with npm into an empty folder. */ +function measurePackage() { + const spec = `onsystem@${version}`; + const npm = (args, cwd) => + run(process.execPath, [npmScript(), ...args], { + cwd, + env: { + ...process.env, + npm_config_cache: path.join(scratch, 'npm-cache'), + npm_config_update_notifier: 'false', + }, + codes: [0], + }); + const view = JSON.parse(npm(['view', spec, 'dist', 'dependencies', '--json'], scratch).stdout); + const project = path.join(scratch, 'install'); + fs.mkdirSync(project); + fs.writeFileSync(path.join(project, 'package.json'), '{ "private": true }\n'); + npm(['install', spec, '--ignore-scripts', '--no-audit', '--no-fund'], project); + const lock = JSON.parse(fs.readFileSync(path.join(project, 'package-lock.json'), 'utf8')); + const packages = Object.keys(lock.packages).filter((key) => key.startsWith('node_modules/')); + const modules = path.join(project, 'node_modules'); + const provenance = view.dist?.attestations?.provenance?.predicateType; + return { + spec, + unpacked: view.dist?.unpackedSize, + fileCount: view.dist?.fileCount, + dependencies: Object.keys(view.dependencies ?? {}).length, + packages: packages.length, + size: folderSize(modules), + typescript: folderSize(path.join(modules, 'typescript')), + provenance, + }; +} + +// ─── Output ──────────────────────────────────────────────────────────────────── + +function render({ machine, checks, hook, server, pkg }) { + const date = new Date().toISOString().slice(0, 10); + const app = (c) => + `[${c.repo.url.replace('https://github.com/', '')}](${c.repo.url}) \`${c.repo.sha.slice(0, 7)}\``; + const where = (c) => (c.folder === '.' ? '' : `\`${c.folder}\`, `); + const rows = [ + ...checks.map((c) => [ + `\`check\` on ${app(c)} (${c.paths.map((p) => `\`${path.posix.join(c.folder, p)}\``).join(' and ')}: ${c.files} files)`, + `${seconds(c.warm)}, or ${seconds(c.cold)} with \`--no-cache\`. One file: ${ms(c.single)}, then ${ms(c.perFile)} for each other file`, + ]), + [ + `The Claude Code hook on one edit (the demo's \`${hook.file}\`), with onsystem installed in the project`, + ms(hook.installed), + ], + [ + `The same edit through npx, as the plugin runs it in a project without an install (\`${pkg.spec}\` in npm's cache)`, + ms(hook.npx), + ], + [ + `The MCP server on ${app(server)} (${where(server)}${server.components} components): \`initialize\` answered`, + `${ms(server.initialize)} after it starts`, + ], + [ + 'Its first tool call, which waits for the design system to be read (`--no-cache`)', + `${seconds(server.firstCall)} after it starts`, + ], + [ + `\`check_ui\` on the app's largest file (${server.lines.toLocaleString('en-US')} lines)`, + ms(server.checkUi), + ], + [ + `\`search_components\`, \`get_component\` (\`${server.component}\`)`, + `${ms(server.search)}, ${ms(server.getComponent)}`, + ], + [ + `The package (\`${pkg.spec}\` on npm)`, + `${megabytes(pkg.unpacked)} unpacked, ${pkg.fileCount} files, ${pkg.dependencies} dependencies`, + ], + [ + 'Installed with npm into an empty project', + `${pkg.packages} packages, ${megabytes(pkg.size)} on disk (${megabytes(pkg.typescript)} of it TypeScript, which reads the components)`, + ], + [ + 'Provenance', + pkg.provenance + ? `Signed npm provenance (${pkg.provenance.replace(/^https:\/\//, '')}): npm shows the workflow run and commit each release was built from` + : 'None on npm', + ], + ]; + return [ + `Measured with onsystem ${version} on ${machine}, on ${date}, with \`node scripts/perf.mjs\`: each time is the median of ${RUNS} ${RUNS === 1 ? 'run' : 'runs'}. The corpus checkouts have no \`node_modules\`, so the types of the libraries their components wrap are not read; the demo has its dependencies installed.`, + '', + table(['Measured', 'Result'], rows), + '', + ].join('\n'); +} + +function table(header, rows) { + const line = (cells) => `| ${cells.join(' | ')} |`; + return [line(header), line(header.map(() => '---')), ...rows.map(line)].join('\n'); +} + +function ms(value) { + if (value >= 1000) return seconds(value); + return value < 10 ? `${value.toFixed(1)} ms` : `${Math.round(value)} ms`; +} + +function seconds(value) { + return `${(value / 1000).toFixed(2)} s`; +} + +function megabytes(bytes) { + return `${(bytes / 1e6).toFixed(1)} MB`; +} + +// ─── Helpers ─────────────────────────────────────────────────────────────────── + +function describeMachine() { + const cpus = os.cpus(); + const model = (cpus[0]?.model ?? 'unknown CPU').trim(); + const memory = Math.round(os.totalmem() / 2 ** 30); + const system = + process.platform === 'darwin' + ? `macOS ${spawnSync('sw_vers', ['-productVersion'], { encoding: 'utf8' }).stdout.trim()}` + : `${os.type()} ${os.release()}`; + return `${model} (${cpus.length} cores, ${memory} GB), ${system}, Node.js ${process.versions.node}`; +} + +function cli(args, cwd) { + run(process.execPath, [CLI, ...args], { cwd, codes: [0, 1] }); +} + +/** The wall time of `fn`, one untimed run first, then RUNS timed ones. */ +function times(fn) { + fn(); + const samples = []; + for (let i = 0; i < RUNS; i++) { + const t = performance.now(); + fn(); + samples.push(performance.now() - t); + } + return samples; +} + +function median(samples) { + const sorted = [...samples].sort((a, b) => a - b); + const mid = Math.floor(sorted.length / 2); + return sorted.length % 2 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2; +} + +function run(command, args, { cwd, input, env, codes }) { + const result = spawnSync(command, args, { + cwd, + input, + env: env ?? process.env, + encoding: 'utf8', + maxBuffer: 256 * 2 ** 20, + shell: false, + }); + if (result.error) fail(`${command} ${args.join(' ')}: ${result.error.message}`); + if (!codes.includes(result.status)) { + fail(`${path.basename(command)} ${args.join(' ')} exited ${result.status}\n${result.stderr}`); + } + return result; +} + +/** npm's CLI script, run with this Node, as the hook runs npx: no shell, on Windows too. */ +function npmScript() { + const dir = path.dirname(process.execPath); + const script = [ + path.join(dir, 'node_modules', 'npm', 'bin', 'npm-cli.js'), + path.join(dir, '..', 'lib', 'node_modules', 'npm', 'bin', 'npm-cli.js'), + ].find((f) => fs.existsSync(f)); + if (!script) fail('npm was not found next to this Node.js.'); + return script; +} + +function folderSize(dir) { + let total = 0; + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const file = path.join(dir, entry.name); + if (entry.isDirectory()) total += folderSize(file); + else if (entry.isFile()) total += fs.statSync(file).size; + } + return total; +} + +/** An error that ends the run with its message alone. */ +class Failure extends Error {} + +function fail(message) { + throw new Failure(message); +} diff --git a/site/app/docs/cli/page.tsx b/site/app/docs/cli/page.tsx new file mode 100644 index 00000000..be9895e5 --- /dev/null +++ b/site/app/docs/cli/page.tsx @@ -0,0 +1,15 @@ +import type { Metadata } from 'next'; + +import { DocPage, docsMetadata } from '@/components/doc-page'; +import { MarkdownBlocks } from '@/components/markdown-blocks'; +import { docs } from '@/lib/data'; + +export const metadata: Metadata = docsMetadata('/docs/cli'); + +export default function CliPage() { + return ( + + + + ); +} diff --git a/site/lib/data.ts b/site/lib/data.ts index 91ecdf07..952db472 100644 --- a/site/lib/data.ts +++ b/site/lib/data.ts @@ -252,6 +252,7 @@ export interface Docs { eslint: string; rules: string; tools: string; + cli: string; troubleshooting: string; howItWorks: string; migrating: string; diff --git a/site/lib/docs.ts b/site/lib/docs.ts index 407c8af2..56562960 100644 --- a/site/lib/docs.ts +++ b/site/lib/docs.ts @@ -117,6 +117,14 @@ const PAGES = [ title: 'Tools', description: 'The five MCP tools, the resources and the prompt, with real responses.', }, + { + href: '/docs/cli', + group: 'Reference', + source: 'docs/cli.md', + title: 'CLI reference', + description: + 'Every command and option of the onsystem CLI: serve, check and its exit codes, inspect, export and init.', + }, { href: '/docs/troubleshooting', group: 'Help', diff --git a/site/lib/faq.ts b/site/lib/faq.ts index 50776360..79dd3768 100644 --- a/site/lib/faq.ts +++ b/site/lib/faq.ts @@ -38,7 +38,7 @@ export const faqEntries = (corpus: FaqCorpus): [question: string, answer: string ], [ 'Is there a model behind it?', - 'No. The agent is already the model; what it lacks is ground truth. Everything here is deterministic static analysis that takes milliseconds and can be tested rule by rule. The trade-off: it cannot judge intent, such as whether a `Dialog` was the right call. That stays with the agent and the reviewer.', + 'No. The agent is already the model; what it lacks is ground truth. Everything here is deterministic static analysis, which checks a file in milliseconds once it has read the design system ([measured](/docs/how-it-works#performance-and-footprint)) and can be tested rule by rule. The trade-off: it cannot judge intent, such as whether a `Dialog` was the right call. That stays with the agent and the reviewer.', ], [ 'Does it work with agents other than Claude Code?', @@ -73,6 +73,10 @@ export const faqEntries = (corpus: FaqCorpus): [question: string, answer: string 'Is the Claude Code plugin safe to install everywhere?', 'Yes. The hook stays quiet in projects without a design system, only reports on what Claude just changed, honours a baseline, and never blocks an edit because it could not run. Code installed from other shadcn registries is different: the hook checks it as your own code until you list its folder in `components`, and then leaves it alone, as it does `components/ui` ([Code from shadcn registries](/docs/configuration#code-from-shadcn-registries)).', ], + [ + 'Can I add our own rules?', + "Not to onsystem itself yet. Its rules are a fixed set: the config sets each one to `error`, `warn` or `off` and rejects a rule id it does not know, and the hook, `check_ui`, `onsystem check` and the GitHub Action run only these. Two things cover most requests. A native element your design system wraps is already a finding: with a `Link` component, a raw `` is reported by `prefer-design-system-component`, and the config's `elements` maps an element to the component that replaces it. For a rule only your team needs, write an ESLint rule: it runs next to onsystem's [ESLint plugin](/docs/eslint) in the editor and in CI, though not in the hook or `check_ui`. A rule most design systems would want is welcome as a pull request: [Adding a lint rule](https://github.com/dgesteves/onsystem/blob/main/CONTRIBUTING.md#adding-a-lint-rule) lists the steps.", + ], [ 'Does it work with Vue, Svelte or Angular?', 'Not yet: React only (`.tsx` and `.jsx`). The other [limits](/docs/how-it-works#limits) are listed with how it works.', diff --git a/site/lib/llms.ts b/site/lib/llms.ts index d609d098..bdc661cf 100644 --- a/site/lib/llms.ts +++ b/site/lib/llms.ts @@ -216,6 +216,7 @@ export function llmsFullTxt(): string { ].join('\n\n'), ], ['Tools', '/docs/tools', docs.tools], + ['CLI reference', '/docs/cli', docs.cli], ['Troubleshooting', '/docs/troubleshooting', docs.troubleshooting], ['FAQ', '/docs/faq', faqMarkdown(corpus)], ['How it works', '/docs/how-it-works', docs.howItWorks], diff --git a/site/lib/site.ts b/site/lib/site.ts index 1c42ff25..e9f2051a 100644 --- a/site/lib/site.ts +++ b/site/lib/site.ts @@ -91,6 +91,7 @@ export const DOCS_FILES: Record = { 'tokens.md': '/docs/tokens', 'eslint.md': '/docs/eslint', 'tools.md': '/docs/tools', + 'cli.md': '/docs/cli', 'troubleshooting.md': '/docs/troubleshooting', 'how-it-works.md': '/docs/how-it-works', 'migrating.md': '/docs/migrating', diff --git a/site/scripts/generate.mjs b/site/scripts/generate.mjs index 0c898e68..cc9b24e5 100644 --- a/site/scripts/generate.mjs +++ b/site/scripts/generate.mjs @@ -891,6 +891,7 @@ const docs = { eslint: doc('eslint.md'), rules: doc('rules.md'), tools: doc('tools.md'), + cli: doc('cli.md'), troubleshooting: doc('troubleshooting.md'), howItWorks: doc('how-it-works.md'), migrating: doc('migrating.md'), @@ -1103,6 +1104,7 @@ const ASK_PAGES = { 'tokens.md': '/docs/tokens', 'eslint.md': '/docs/eslint', 'tools.md': '/docs/tools', + 'cli.md': '/docs/cli', 'troubleshooting.md': '/docs/troubleshooting', 'how-it-works.md': '/docs/how-it-works', 'migrating.md': '/docs/migrating', diff --git a/site/test/docs-agree.test.ts b/site/test/docs-agree.test.ts index 12085321..d3d89b2e 100644 --- a/site/test/docs-agree.test.ts +++ b/site/test/docs-agree.test.ts @@ -103,7 +103,7 @@ describe('llms-full.txt', () => { expect(hashed).toEqual([]); expect(lines.filter((line) => line.startsWith('# '))).toEqual(['# onsystem']); // Cut at its `## ` lines, the Tools section runs to the end of the page. - const tools = full.slice(full.indexOf('\n## Tools\n'), full.indexOf('\n## Troubleshooting\n')); + const tools = full.slice(full.indexOf('\n## Tools\n'), full.indexOf('\n## CLI reference\n')); expect(tools.slice(1).match(/\n## /g)).toBeNull(); expect(tools).toContain('\n # Badge\n'); expect(tools).toContain('> check_ui {'); diff --git a/site/test/docs-shell.test.ts b/site/test/docs-shell.test.ts index 99a2665a..3b322aad 100644 --- a/site/test/docs-shell.test.ts +++ b/site/test/docs-shell.test.ts @@ -56,7 +56,17 @@ describe('the docs sidebar', () => { ], ['Gate PRs', ['CI and baselines', 'GitHub Action']], ['Integrations', ['ESLint plugin']], - ['Reference', ['Configuration', 'Design-system packages', 'Design tokens', 'Rules', 'Tools']], + [ + 'Reference', + [ + 'Configuration', + 'Design-system packages', + 'Design tokens', + 'Rules', + 'Tools', + 'CLI reference', + ], + ], ['Help', ['Troubleshooting', 'FAQ', 'How it works', 'Migrating']], ]); expect(DOCS_GROUPS).toEqual(DOCS_NAV.map((g) => g.group)); diff --git a/test/cli-docs.test.ts b/test/cli-docs.test.ts new file mode 100644 index 00000000..108704f6 --- /dev/null +++ b/test/cli-docs.test.ts @@ -0,0 +1,249 @@ +import fs from 'node:fs'; +import path from 'node:path'; + +import ts from 'typescript'; +import { beforeAll, describe, expect, it } from 'vitest'; + +import { main, type Io } from '../src/cli.js'; +import { CONFIG_FILES } from '../src/config.js'; +import { DEMO_ROOT } from './helpers.js'; +import { section } from './markdown.js'; + +// docs/cli.md is the CLI reference, and the options it lists are the ones `--help` prints and the +// parser accepts, under the commands `--help` gives them to, with the same argument names. A +// flag that ships without a row there, or a row left behind by a flag that is gone, fails here. +// Before this page, `--quiet-without-design-system` was in no doc at all, and `--baseline`, +// `--ignore-baseline`, `--include-tests`, `--components`, `--docs` and `--no-watch` were each +// named in one place, in passing. The GitHub Action's inputs are held to their table in +// docs/required-check.md the same way. + +const root = path.resolve(import.meta.dirname, '..'); +const read = (file: string) => + fs.readFileSync(path.join(root, file), 'utf8').replace(/\r\n/g, '\n'); + +const COMMANDS = ['serve', 'check', 'inspect', 'export', 'init'] as const; +/** The section of docs/cli.md for options that do not name a command in `--help`. */ +const SHARED = 'Options for every command'; +/** + * Options the CLI takes for a command whose `--help` text does not name it: `inspect --format + * json`. The test below runs each one, so the row stays true. + */ +const BEYOND_HELP: Record = { '`inspect`': ['--format'] }; + +interface Option { + /** The flags, such as `-y` and `--yes`. */ + flags: string[]; + /** The argument it takes, such as ``, or '' for none. */ + arg: string; + /** The commands `--help` gives it to (`check:` in its text); none for every command. */ + commands: string[]; +} + +/** What `onsystem --help` prints. */ +async function helpText(): Promise { + const out: string[] = []; + const io: Io = { cwd: root, color: false, stdout: (t) => out.push(t), stderr: () => {} }; + expect(await main(['--help'], io)).toBe(0); + return out.join('\n'); +} + +/** The options `--help` lists under Options, each with the text that follows it. */ +function helpOptions(help: string): Option[] { + const start = help.indexOf('\nOptions\n'); + const end = help.indexOf('\nExamples\n'); + expect(start).toBeGreaterThan(-1); + expect(end).toBeGreaterThan(start); + const options: Option[] = []; + let current: (Option & { text: string }) | undefined; + const done = () => { + if (!current) return; + const named = [...current.text.matchAll(/\b(serve|check|inspect|export|init):/g)]; + options.push({ + flags: current.flags, + arg: current.arg, + commands: unique(named.map((m) => m[1] ?? '')), + }); + }; + for (const line of help.slice(start + '\nOptions\n'.length, end).split('\n')) { + const entry = /^ {2}((?:-\w, )?--[\w-]+)(?: (<[^>]+>))?(.*)$/.exec(line); + if (entry) { + done(); + current = { + flags: (entry[1] ?? '').split(', '), + arg: entry[2] ?? '', + commands: [], + text: entry[3] ?? '', + }; + } else if (current) { + current.text += ` ${line.trim()}`; + } + } + done(); + return options; +} + +/** The flags the parser in src/cli.ts accepts: `--no-` for a boolean that defaults to true. */ +function parserFlags(): string[] { + const file = path.join(root, 'src/cli.ts'); + const source = ts.createSourceFile(file, read('src/cli.ts'), ts.ScriptTarget.Latest, true); + let options: ts.ObjectLiteralExpression | undefined; + const visit = (node: ts.Node) => { + if ( + ts.isCallExpression(node) && + ts.isIdentifier(node.expression) && + node.expression.text === 'parseArgs' + ) { + const arg = node.arguments[0]; + const prop = + arg && ts.isObjectLiteralExpression(arg) + ? arg.properties.find((p) => p.name?.getText(source) === 'options') + : undefined; + if (prop && ts.isPropertyAssignment(prop) && ts.isObjectLiteralExpression(prop.initializer)) { + options = prop.initializer; + } + } + ts.forEachChild(node, visit); + }; + visit(source); + if (!options) throw new Error('src/cli.ts: no parseArgs({ options }) found'); + const flags: string[] = []; + for (const prop of options.properties) { + if (!ts.isPropertyAssignment(prop) || !ts.isObjectLiteralExpression(prop.initializer)) continue; + const name = prop.name.getText(source).replace(/^'|'$/g, ''); + const value = (key: string) => { + const p = (prop.initializer as ts.ObjectLiteralExpression).properties.find( + (q) => q.name?.getText(source) === key, + ); + return p && ts.isPropertyAssignment(p) ? p.initializer.getText(source) : undefined; + }; + const short = value('short'); + if (short) flags.push(`-${short.replace(/^'|'$/g, '')}`); + flags.push( + value('type') === "'boolean'" && value('default') === 'true' ? `--no-${name}` : `--${name}`, + ); + } + return flags; +} + +/** The option rows of a docs/cli.md section's tables: `| \`-y\`, \`--yes\` | …`. */ +function docRows(markdown: string): { flags: string[]; arg: string }[] { + return [...markdown.matchAll(/^\| ((?:`-[^`]+`(?:, )?)+) +\|/gm)].map((m) => { + const cells = [...(m[1] ?? '').matchAll(/`([^`]+)`/g)].map((c) => c[1] ?? ''); + const args = unique(cells.map((cell) => /^-\S+ (<[^>]+>)$/.exec(cell)?.[1] ?? '')); + return { + flags: cells.map((cell) => cell.split(' ')[0] ?? ''), + arg: args.find(Boolean) ?? '', + }; + }); +} + +function unique(items: T[]): T[] { + return [...new Set(items)]; +} + +const key = (flags: string[]) => [...flags].sort().join(', '); + +describe('the CLI reference (docs/cli.md)', () => { + const cli = read('docs/cli.md'); + let options: Option[] = []; + let help = ''; + + beforeAll(async () => { + help = await helpText(); + options = helpOptions(help); + }); + + it('reads the options from --help', () => { + // The parse above found every entry: the first and last, one with a short flag, and one whose + // text names two commands. + expect(options[0]?.flags).toEqual(['--root']); + expect(options.at(-1)?.flags).toEqual(['-v', '--version']); + expect(options.find((o) => o.flags.includes('--yes'))).toEqual({ + flags: ['-y', '--yes'], + arg: '', + commands: ['init'], + }); + expect(options.find((o) => o.flags.includes('--format'))?.commands).toEqual([ + 'check', + 'export', + ]); + }); + + it('--help lists every option the parser accepts, and no other', () => { + expect(options.flatMap((o) => o.flags).sort()).toEqual(parserFlags().sort()); + }); + + it('has a section for every command --help lists, and for the options they share', () => { + const usage = help.slice(help.indexOf('\nUsage\n'), help.indexOf('\nOptions\n')); + const listed = [...usage.matchAll(/^ {2}onsystem (?:\[(\w+)\]|(\w+))/gm)].map( + (m) => m[1] ?? m[2] ?? '', + ); + expect(listed).toEqual([...COMMANDS, 'help']); + for (const command of COMMANDS) expect(() => section(cli, `\`${command}\``)).not.toThrow(); + expect(() => section(cli, SHARED)).not.toThrow(); + // `help` is in the commands table at the top. + expect(cli).toMatch(/^\| `onsystem help` +\|/m); + }); + + it('lists each option under every command --help gives it to, with its argument', () => { + for (const option of options) { + const sections = option.commands.length + ? option.commands.map((command) => `\`${command}\``) + : [SHARED]; + for (const heading of sections) { + const rows = docRows(section(cli, heading)); + const row = rows.find((r) => key(r.flags) === key(option.flags)); + expect(row, `${option.flags.join(', ')} under "${heading}"`).toBeDefined(); + expect(row?.arg, `${option.flags.join(', ')} under "${heading}"`).toBe(option.arg); + } + } + }); + + it('lists no option that --help does not give to that command', () => { + for (const heading of [...COMMANDS.map((c) => `\`${c}\``), SHARED]) { + for (const row of docRows(section(cli, heading))) { + const option = options.find((o) => key(o.flags) === key(row.flags)); + expect(option, `${row.flags.join(', ')} under "${heading}"`).toBeDefined(); + const expected = option?.commands.length + ? option.commands.map((c) => `\`${c}\``) + : [SHARED]; + if (row.flags.some((flag) => BEYOND_HELP[heading]?.includes(flag))) continue; + expect(expected, `${row.flags.join(', ')} under "${heading}"`).toContain(heading); + } + } + }); + + it('runs inspect --format json, which --help does not list', async () => { + const out: string[] = []; + const io: Io = { cwd: DEMO_ROOT, color: false, stdout: (t) => out.push(t), stderr: () => {} }; + expect(await main(['inspect', '--format', 'json', '--no-cache'], io)).toBe(0); + const model = JSON.parse(out.join('\n')) as { components: { name: string }[] }; + expect(model.components.map((c) => c.name)).toContain('Button'); + expect(section(cli, '`inspect`')).toMatch(/^\| `--format ` +\| `json` /m); + }); + + it('names every config file name the CLI looks for', () => { + for (const name of CONFIG_FILES) expect(cli).toContain(`\`${name}\``); + }); + + it('is linked from the quickstart', () => { + expect(read('docs/quickstart.md')).toContain('](cli.md)'); + }); +}); + +describe('the GitHub Action inputs (docs/required-check.md)', () => { + it('lists every input of action.yml with its default', () => { + const yml = read('action.yml'); + const block = yml.slice(yml.indexOf('\ninputs:'), yml.indexOf('\noutputs:')); + const inputs = [ + ...block.matchAll(/^ {2}([a-z-]+):\n {4}description:[^\n]*\n(?: {4}default: (.*)\n)?/gm), + ].map((m) => ({ name: m[1] ?? '', default: (m[2] ?? '').replace(/^'(.*)'$/, '$1') })); + expect(inputs.map((i) => i.name)).toContain('only-changed-lines'); + const doc = read('docs/required-check.md'); + for (const input of inputs) { + const row = new RegExp(`^\\| \`${input.name}\` +\\| ([^|]*?) +\\|`, 'm').exec(doc); + expect(row, input.name).not.toBeNull(); + expect(row?.[1], input.name).toBe(input.default ? `\`${input.default}\`` : ''); + } + }); +}); diff --git a/test/custom-rules-faq.test.ts b/test/custom-rules-faq.test.ts new file mode 100644 index 00000000..a4060581 --- /dev/null +++ b/test/custom-rules-faq.test.ts @@ -0,0 +1,65 @@ +import fs from 'node:fs'; +import path from 'node:path'; + +import { describe, expect, it } from 'vitest'; + +import { loadConfig, RULE_IDS } from '../src/config.js'; +import { fixture, load, TSCONFIG } from './helpers.js'; + +// "Can I add our own rules?" is the FAQ's answer to the question users ask once they see the +// rules: what the config does with a rule id it does not know, what already covers the commonest +// ask (no raw where the design system has a Link), and where a new built-in rule starts. + +const root = path.resolve(import.meta.dirname, '..'); +const read = (file: string) => + fs.readFileSync(path.join(root, file), 'utf8').replace(/\r\n/g, '\n'); + +/** The FAQ answer to `question`, as written in site/lib/faq.ts. */ +function answer(question: string): string { + const faq = read('site/lib/faq.ts'); + const start = faq.indexOf(`'${question}',`); + expect(start, question).toBeGreaterThan(-1); + return faq.slice(start, faq.indexOf('\n ],', start)); +} + +describe('the FAQ on custom rules (site/lib/faq.ts)', () => { + const text = answer('Can I add our own rules?'); + + it('says the config rejects a rule id it does not know', async () => { + expect(text).toContain('rejects a rule id it does not know'); + const dir = fixture({ + 'onsystem.config.json': JSON.stringify({ rules: { 'no-raw-anchor': 'error' } }), + }); + await expect(loadConfig({ root: dir })).rejects.toThrow(/Unknown rule "no-raw-anchor"/); + }); + + it('names the rule that reports a raw where the design system has a Link', async () => { + const rule = 'prefer-design-system-component'; + expect(RULE_IDS).toContain(rule); + expect(text).toContain(`\`${rule}\``); + const system = await load( + fixture( + { + 'tsconfig.json': TSCONFIG, + 'onsystem.config.json': '{ "tokens": [] }', + 'components/ui/link.tsx': `import * as React from "react" +export function Link(props: React.ComponentProps<"a">) { + return +}`, + }, + { nodeModules: true }, + ), + ); + const { diagnostics } = system.check(`Docs`, 'page.tsx'); + expect(diagnostics.map((d) => d.ruleId)).toEqual([rule]); + expect(diagnostics[0]?.message).toContain('Use '); + }); + + it('links the steps for adding a built-in rule in CONTRIBUTING.md', () => { + expect(read('CONTRIBUTING.md')).toContain('\n## Adding a lint rule\n'); + expect(text).toContain( + '(https://github.com/dgesteves/onsystem/blob/main/CONTRIBUTING.md#adding-a-lint-rule)', + ); + expect(text).toContain('[ESLint plugin](/docs/eslint)'); + }); +}); diff --git a/test/perf-docs.test.ts b/test/perf-docs.test.ts new file mode 100644 index 00000000..599724e2 --- /dev/null +++ b/test/perf-docs.test.ts @@ -0,0 +1,114 @@ +import fs from 'node:fs'; +import path from 'node:path'; + +import { describe, expect, it } from 'vitest'; + +import { section } from './markdown.js'; + +// docs/how-it-works.md said only that onsystem "takes milliseconds", and the FAQ the same. The +// "Performance and footprint" table has the measured times and the install size instead, written +// by scripts/perf.mjs from runs on a named machine. These tests hold the table to what the +// script writes and to the corpus commits it measured, so a hand edit or a corpus bump shows. + +const root = path.resolve(import.meta.dirname, '..'); +const read = (file: string) => + fs.readFileSync(path.join(root, file), 'utf8').replace(/\r\n/g, '\n'); +const doc = read('docs/how-it-works.md'); + +interface Repos { + repos: { name: string; url: string; sha: string; runs: { name: string }[] }[]; +} + +/** What perf.mjs writes, from the "Measured with" line on, and its rows as [measured, result]. */ +function perfBlock(): { text: string; rows: [string, string][] } { + const perf = section(doc, 'Performance and footprint'); + const start = perf.indexOf('\nMeasured with onsystem '); + expect(start).toBeGreaterThan(-1); + const text = perf.slice(start); + const rows = text + .split('\n') + .filter((line) => line.startsWith('| ') && !/^\| -+ \|/.test(line)) + .slice(1) + .map((line) => { + const [measured = '', result = ''] = line + .slice(2, -2) + .split(' | ') + .map((cell) => cell.trim()); + return [measured, result] as [string, string]; + }); + return { text, rows }; +} + +describe('Performance and footprint (docs/how-it-works.md)', () => { + it('says where, when and with what the numbers were measured', () => { + const { text } = perfBlock(); + expect(text).toMatch( + /^\s*Measured with onsystem \d+\.\d+\.\d+ on .+ \(\d+ cores, \d+ GB\), .+, Node\.js \d+\.\d+\.\d+, on \d{4}-\d{2}-\d{2}, with `node scripts\/perf\.mjs`: each time is the median of \d+ runs\./, + ); + expect(fs.existsSync(path.join(root, 'scripts/perf.mjs'))).toBe(true); + }); + + it('has the rows perf.mjs writes, each with a measured result', () => { + const { rows } = perfBlock(); + const measured = rows.map(([m]) => m); + expect(measured.filter((m) => m.startsWith('`check` on '))).toHaveLength(2); + expect( + measured.filter((m) => m.includes('Claude Code hook') || m.includes('npx')), + ).toHaveLength(2); + expect(measured.some((m) => m.startsWith('The MCP server on '))).toBe(true); + expect(measured.some((m) => m.startsWith('`check_ui` on '))).toBe(true); + expect(measured.some((m) => m.startsWith('`search_components`'))).toBe(true); + expect(measured.some((m) => m.startsWith('The package (`onsystem@'))).toBe(true); + expect(measured).toContain('Installed with npm into an empty project'); + expect(measured).toContain('Provenance'); + for (const [m, result] of rows) { + if (m === 'Provenance') continue; + expect(result, m).toMatch(/\d/); + } + }); + + it('names the corpus commits it measured, as corpus/repos.json pins them', () => { + // A new pin means new code to check: run scripts/perf.mjs --write again. + const { repos } = JSON.parse(read('corpus/repos.json')) as Repos; + const { rows } = perfBlock(); + const apps = rows.filter(([m]) => /^(`check` on |The MCP server on )/.test(m)); + expect(apps).toHaveLength(3); + for (const [m] of apps) { + const link = /\[[^\]]+\]\((https:\/\/github\.com\/[^)]+)\) `([0-9a-f]{7})`/.exec(m); + expect(link, m).not.toBeNull(); + const repo = repos.find((r) => r.url === link?.[1]); + expect(repo, m).toBeDefined(); + expect(repo?.sha.slice(0, 7), m).toBe(link?.[2]); + } + }); + + it('measures the hook on a file the demo has', () => { + const { rows } = perfBlock(); + const hook = rows.find(([m]) => m.startsWith('The Claude Code hook'))?.[0] ?? ''; + const file = /the demo's `([^`]+)`/.exec(hook)?.[1]; + expect(file).toBeDefined(); + expect(fs.existsSync(path.join(root, 'examples/shadcn-demo', file ?? ''))).toBe(true); + }); + + it('says the package has provenance only while releases publish it', () => { + const { rows } = perfBlock(); + const provenance = rows.find(([m]) => m === 'Provenance')?.[1] ?? ''; + const { publishConfig } = JSON.parse(read('package.json')) as { + publishConfig?: { provenance?: boolean }; + }; + if (provenance.startsWith('Signed')) expect(publishConfig?.provenance).toBe(true); + }); +}); + +describe('the speed claims', () => { + it('quote the table rather than a bare "takes milliseconds"', () => { + const faq = read('site/lib/faq.ts'); + for (const [file, text, link] of [ + ['docs/how-it-works.md', doc, '](#performance-and-footprint)'], + ['site/lib/faq.ts', faq, '](/docs/how-it-works#performance-and-footprint)'], + ] as const) { + expect(text, file).not.toMatch(/takes milliseconds/); + expect(text, file).toContain(link); + } + }); +}); From 7b6be9048459740b0c8bc92db632452161264079 Mon Sep 17 00:00:00 2001 From: Diogo Esteves Date: Thu, 29 Oct 2026 11:23:31 +0000 Subject: [PATCH 2/3] docs: say which files check takes when named, and which agent setups are tested - cli.md: a file named outright is checked unless it is one of the design system's own component files or --skip-excluded leaves it out; the exit-2 list names no paths, an unknown --format, a non-integer --max-warnings, a missing --baseline file and --prune-baseline with no baseline. - quickstart.md: the Agents row calls tested only the Tier 1 setups and links the support tiers for Codex CLI, Cursor, VS Code and the rest. - FAQ: no "yet" on custom rules, since no rule API is planned; zero config is said to read a flat src/, src/components/, components/ or app/components/ of primitives wrappers. - configuration.md lists onsystem.config.mts. - perf.mjs prints the load average as it starts, and warns when the load is high or a run from the cache is slower than one without it; the table says that reading the design system takes longer with dependencies installed. - Tests: action.yml inputs are read with the YAML parser and held to one table row each; the perf line accepts "1 run"; the quickstart, FAQ, configuration.md and perf caveat are checked. --- docs/cli.md | 4 +-- docs/configuration.md | 2 +- docs/how-it-works.md | 2 +- docs/quickstart.md | 22 +++++++-------- scripts/perf.mjs | 48 ++++++++++++++++++++++++++++++--- site/lib/faq.ts | 4 +-- test/cli-docs.test.ts | 30 ++++++++++++++++----- test/custom-rules-faq.test.ts | 22 ++++++++++++++- test/perf-docs.test.ts | 10 ++++++- test/support-tiers-docs.test.ts | 10 +++++++ 10 files changed, 125 insertions(+), 29 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 1dd6c1b4..4029a09a 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -43,13 +43,13 @@ At a monorepo root with no design system of its own, every command works per pro ## `check` -`onsystem check ` lints the files, folders and globs it is given with the rules `check_ui` runs ([rules](rules.md)). A folder stands for its `.tsx` and `.jsx` files; folders and globs leave out the design system's own component files, tests and stories, what the config's `exclude` adds and what git ignores, while a file named outright is always checked ([CI and baselines](ci.md)). +`onsystem check ` lints the files, folders and globs it is given with the rules `check_ui` runs ([rules](rules.md)). A folder stands for its `.tsx` and `.jsx` files; folders and globs leave out the design system's own component files, tests and stories, what the config's `exclude` adds and what git ignores. A file named outright is checked too, unless it is one of the design system's own component files (`--include-design-system` checks those) or `--skip-excluded` leaves it out ([CI and baselines](ci.md)). It exits with: - `0` when there are no errors, and no more warnings than `--max-warnings`. - `1` when there are errors, or more warnings than `--max-warnings`. -- `2` when it could not check: an unknown option, a broken config, paths that match no file, a file it could not read, or, with `--require-design-system`, no design system or nothing to check. +- `2` when it could not check: an unknown option, no paths given, an unknown `--format`, a `--max-warnings` that is not an integer, a broken config, paths that match no file, a file it could not read, a `--baseline` file that does not exist (unless `--update-baseline` writes it), `--prune-baseline` with no baseline to prune, or, with `--require-design-system`, no design system or nothing to check. | Option | What it does | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/docs/configuration.md b/docs/configuration.md index 1fd0ff5b..a02a9181 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -39,7 +39,7 @@ The root stays one project, as before, when it has a design system of its own (a ## Config file -`onsystem.config.json` (or `.mjs`, `.js`, or `.ts` on Node.js 22.18 or later) in the project root. Every field is optional; the [JSON Schema](../schema.json) gives editor completion. `npx onsystem init` writes one: the folders zero config found, or the default `components/ui` folders when they hold the components, spelled out (it leaves `components` to zero config when a package's exports name more than a few files or import paths a config cannot spell, and `components` or `tokens` when they come from a package installed in `node_modules`, whose folder and files can change at each upgrade; with such tokens, `components` as well when a config that names them would read other stylesheets), or your answers to where the components, tokens and import path are when it found nothing. It never replaces an existing config without `--force`. At a monorepo root it writes a config with only `$schema`, for rules all projects share. In a design-system package it also declares the config in `package.json`, so every app that depends on the package reads it, and writes `ONSYSTEM.md` for those apps' teams ([design-system packages](design-system-packages.md)); in an app that takes on such a config, it writes none of its own unless `--force` is passed, and then one with only `$schema`, which builds on the package's. +`onsystem.config.json` (or `.mjs`, `.js`, or `.ts` and `.mts` on Node.js 22.18 or later) in the project root. Every field is optional; the [JSON Schema](../schema.json) gives editor completion. `npx onsystem init` writes one: the folders zero config found, or the default `components/ui` folders when they hold the components, spelled out (it leaves `components` to zero config when a package's exports name more than a few files or import paths a config cannot spell, and `components` or `tokens` when they come from a package installed in `node_modules`, whose folder and files can change at each upgrade; with such tokens, `components` as well when a config that names them would read other stylesheets), or your answers to where the components, tokens and import path are when it found nothing. It never replaces an existing config without `--force`. At a monorepo root it writes a config with only `$schema`, for rules all projects share. In a design-system package it also declares the config in `package.json`, so every app that depends on the package reads it, and writes `ONSYSTEM.md` for those apps' teams ([design-system packages](design-system-packages.md)); in an app that takes on such a config, it writes none of its own unless `--force` is passed, and then one with only `$schema`, which builds on the package's. ```json title="onsystem.config.json" { diff --git a/docs/how-it-works.md b/docs/how-it-works.md index ef1fca0f..6add16af 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -25,7 +25,7 @@ Run as is, with no config, on public apps: the counts the [corpus](../corpus) sn Reading the design system is the slow part: one TypeScript program over the component files. `check` and the server cache what they read and read it again only when those files change, so checking a file after that is syntax work. The hook runs one `check` per edit to a UI file. Without onsystem installed in the project, it runs `check` through npx, which starts npm first: a dev dependency ([pin the version](rollout.md#pin-the-version)) takes that out of every edit. -Measured with onsystem 0.11.0 on Apple M1 Max (10 cores, 32 GB), macOS 27.0.1, Node.js 24.18.0, on 2026-10-29, with `node scripts/perf.mjs`: each time is the median of 5 runs. The corpus checkouts have no `node_modules`, so the types of the libraries their components wrap are not read; the demo has its dependencies installed. +Measured with onsystem 0.11.0 on Apple M1 Max (10 cores, 32 GB), macOS 27.0.1, Node.js 24.18.0, on 2026-10-29, with `node scripts/perf.mjs`: each time is the median of 5 runs. The corpus checkouts have no `node_modules`, so the types of the libraries their components wrap are not read: in a project with its dependencies installed, reading the design system takes longer. The demo, where the hook is measured, has its dependencies installed. | Measured | Result | | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | diff --git a/docs/quickstart.md b/docs/quickstart.md index 308b61c4..3a863d11 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -64,16 +64,16 @@ npx onsystem check . --update-baseline # writes onsystem.baseline.json: commit ## What it works with -| | Supported | Not supported yet | -| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | -| UI code | React, in `.tsx` and `.jsx` files, and `.js` files with JSX that a glob names | Vue, Svelte, Angular | -| Styling | Tailwind CSS v3 and v4 classes, `style` objects and color attributes | CSS-in-JS (styled-components, Emotion), CSS Modules, plain stylesheets | -| Variants | `cva()`, `tv()` (tailwind-variants), and props typed as unions of string literals | — | -| Components | Your own, such as shadcn/ui's or ones built on Radix, React Aria, Base UI or another primitives library, in the app or in a workspace package; a design-system package installed with its type declarations ([design-system packages](design-system-packages.md)); MUI (`@mui/material`) | Chakra UI, Mantine, HeroUI, Radix Themes, Ant Design, MUI's other packages | -| Tokens | CSS custom properties (Tailwind v4's `@theme`, shadcn/ui's `:root` variables), a Tailwind v3 `tailwind.config`, DTCG 2025.10 `.tokens.json` files, a `DESIGN.md` ([tokens](tokens.md)) | Values computed in code | -| Monorepos | pnpm, npm, Yarn and Bun workspaces, Lerna, Nx and Turborepo ([monorepo roots](configuration.md#monorepo-roots)) | — | -| Agents | The Claude Code plugin (hook, MCP server and skill), its hook in Codex CLI, and the MCP server in Cursor, VS Code and [other clients](clients.md) | — | -| CI and editors | `onsystem check` in any CI with Node.js, the [GitHub Action](required-check.md), and the [ESLint plugin](eslint.md) for ESLint 8.57, 9 and 10 with a flat config | — | -| Runtime | Node.js 20.19 or later (22.18 or later for a TypeScript config file), on macOS, Linux and Windows | — | +| | Supported | Not supported yet | +| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------- | +| UI code | React, in `.tsx` and `.jsx` files, and `.js` files with JSX that a glob names | Vue, Svelte, Angular | +| Styling | Tailwind CSS v3 and v4 classes, `style` objects and color attributes | CSS-in-JS (styled-components, Emotion), CSS Modules, plain stylesheets | +| Variants | `cva()`, `tv()` (tailwind-variants), and props typed as unions of string literals | — | +| Components | Your own, such as shadcn/ui's or ones built on Radix, React Aria, Base UI or another primitives library, in the app or in a workspace package; a design-system package installed with its type declarations ([design-system packages](design-system-packages.md)); MUI (`@mui/material`) | Chakra UI, Mantine, HeroUI, Radix Themes, Ant Design, MUI's other packages | +| Tokens | CSS custom properties (Tailwind v4's `@theme`, shadcn/ui's `:root` variables), a Tailwind v3 `tailwind.config`, DTCG 2025.10 `.tokens.json` files, a `DESIGN.md` ([tokens](tokens.md)) | Values computed in code | +| Monorepos | pnpm, npm, Yarn and Bun workspaces, Lerna, Nx and Turborepo ([monorepo roots](configuration.md#monorepo-roots)) | — | +| Agents | Tested in the client: the Claude Code plugin (hook, MCP server and skill), the MCP server in Claude Code and Gemini CLI, and the skill through `npx skills add`. Set up from each client's docs, not yet run in it: the hook in Codex CLI, and the MCP server in Cursor, VS Code and [other clients](clients.md) ([support tiers](clients.md#support-tiers)) | — | +| CI and editors | `onsystem check` in any CI with Node.js, the [GitHub Action](required-check.md), and the [ESLint plugin](eslint.md) for ESLint 8.57, 9 and 10 with a flat config | — | +| Runtime | Node.js 20.19 or later (22.18 or later for a TypeScript config file), on macOS, Linux and Windows | — | Zero config finds the design system in the common layouts ([how](configuration.md#zero-config)); the others take a config file with a `components` glob. [How it works](how-it-works.md#limits) lists what it does not check yet. diff --git a/scripts/perf.mjs b/scripts/perf.mjs index 40529ac2..44885025 100644 --- a/scripts/perf.mjs +++ b/scripts/perf.mjs @@ -7,11 +7,14 @@ // Run `pnpm build` first, and `pnpm corpus fetch --only chatbot --only midday` (same --dir or // CORPUS_DIR as here) for the two apps it checks. It measures on this machine, so the numbers // change with it: the table names the machine, and each time is the median of --runs runs. +// Other work on the machine slows the runs: it prints the load average as it starts, and warns +// when the load is high or a run from the cache is slower than the run without it. // // - `check` with the built CLI on two corpus apps, as the corpus checks them: every file, with // the design system read again (--no-cache) and from the cache, then one file, which gives // the time per file after the first. The checkouts have no node_modules (nothing in them is -// installed), so the types of the libraries their components wrap are not read. +// installed), so the types of the libraries their components wrap are not read: reading the +// design system takes longer in a project with its dependencies installed. // - The Claude Code hook (plugins/onsystem/hooks/check-ui.mjs) on one edit of the demo, which // has its dependencies installed: with the built CLI as the project's install, and through // npx with onsystem@ from npm, as the plugin runs it in a project without an install. @@ -59,6 +62,11 @@ const corpusDir = path.resolve( const { version } = JSON.parse(fs.readFileSync(path.join(ROOT, 'package.json'), 'utf8')); /** A folder for this run's npm cache, install and hook markers, removed at the end. */ let scratch; +const cores = os.cpus().length; +/** A 1-minute load average above this, half the cores, means other work competed for the CPU. */ +const BUSY = cores / 2; +/** The highest 1-minute load average seen after a batch of timed runs. */ +let peakLoad = 0; try { if (!Number.isInteger(RUNS) || RUNS < 1) fail('--runs expects a whole number of 1 or more.'); @@ -66,11 +74,18 @@ try { const apps = corpusApps(); scratch = fs.mkdtempSync(path.join(os.tmpdir(), 'onsystem-perf-')); const machine = describeMachine(); - console.error(`Measuring onsystem ${version} on ${machine}, ${RUNS} runs each.`); + console.error( + `Measuring onsystem ${version} on ${machine}, ${RUNS} ${RUNS === 1 ? 'run' : 'runs'} each${loadAverage()}.`, + ); const checks = apps.map(measureCheck); const hook = measureHook(); const server = await measureServer(checks.find((c) => c.name === SERVER_APP)); const pkg = measurePackage(); + if (peakLoad > BUSY) { + warn( + `the 1-minute load average reached ${peakLoad.toFixed(2)} on ${cores} cores while it measured: other work may have slowed these times. Run it again when the machine is idle.`, + ); + } const table = render({ machine, checks, hook, server, pkg }); console.log(table); if (values.write) { @@ -124,6 +139,11 @@ function measureCheck(app) { // The first run above wrote the cache: in node_modules/.cache/onsystem, or in the system temp // folder for a checkout without node_modules. const warm = median(times(() => cli(args, app.cwd))); + if (warm > cold) { + warn( + `on ${app.name}, \`check\` from the cache (${seconds(warm)}) was slower than with --no-cache (${seconds(cold)}): other work on the machine may have slowed it. Run it again when the machine is idle.`, + ); + } const single = median(times(() => cli(['check', one, '--format', 'json'], app.cwd))); const largest = files .map((file) => ({ file, size: fs.statSync(path.resolve(app.cwd, file)).size })) @@ -200,6 +220,7 @@ async function measureServer(app) { samples.push(performance.now() - t); if (result.isError) fail(`${name} failed: ${JSON.stringify(result.content)}`); } + sampleLoad(); return median(samples); }; const lines = fs.readFileSync(path.resolve(app.cwd, app.largest), 'utf8').split('\n').length; @@ -311,7 +332,7 @@ function render({ machine, checks, hook, server, pkg }) { ], ]; return [ - `Measured with onsystem ${version} on ${machine}, on ${date}, with \`node scripts/perf.mjs\`: each time is the median of ${RUNS} ${RUNS === 1 ? 'run' : 'runs'}. The corpus checkouts have no \`node_modules\`, so the types of the libraries their components wrap are not read; the demo has its dependencies installed.`, + `Measured with onsystem ${version} on ${machine}, on ${date}, with \`node scripts/perf.mjs\`: each time is the median of ${RUNS} ${RUNS === 1 ? 'run' : 'runs'}. The corpus checkouts have no \`node_modules\`, so the types of the libraries their components wrap are not read: in a project with its dependencies installed, reading the design system takes longer. The demo, where the hook is measured, has its dependencies installed.`, '', table(['Measured', 'Result'], rows), '', @@ -362,9 +383,30 @@ function times(fn) { fn(); samples.push(performance.now() - t); } + sampleLoad(); return samples; } +/** `, load average 1.52 1.80 2.01` (1, 5 and 15 minutes), or '' on Windows, which has none. */ +function loadAverage() { + if (process.platform === 'win32') return ''; + sampleLoad(); + return `, load average ${os + .loadavg() + .map((n) => n.toFixed(2)) + .join(' ')}`; +} + +/** Keeps the highest 1-minute load average, read after each batch of timed runs. */ +function sampleLoad() { + peakLoad = Math.max(peakLoad, os.loadavg()[0]); +} + +/** A warning on stderr, which the table on stdout and the doc it writes do not carry. */ +function warn(text) { + console.error(`Warning: ${text}`); +} + function median(samples) { const sorted = [...samples].sort((a, b) => a - b); const mid = Math.floor(sorted.length / 2); diff --git a/site/lib/faq.ts b/site/lib/faq.ts index 79dd3768..a32b10b4 100644 --- a/site/lib/faq.ts +++ b/site/lib/faq.ts @@ -63,7 +63,7 @@ export const faqEntries = (corpus: FaqCorpus): [question: string, answer: string ], [ 'My components are not in components/ui. Do I need a config?', - 'Not if the app has a shadcn/ui `components.json`, imports a workspace package named like a design system (`@acme/ui`), is the design-system package itself, or keeps its components in a flat `src/` that wraps React Aria, Radix or another primitives library: [zero config](/docs/configuration#zero-config) finds those. Otherwise a config file with a `components` glob is enough. `npx -y onsystem inspect` prints what was found, and [troubleshooting](/docs/troubleshooting) covers what to do when it finds nothing.', + 'Not if the app has a shadcn/ui `components.json`, imports a workspace package named like a design system (`@acme/ui`), is the design-system package itself, or keeps components that wrap React Aria, Radix or another primitives library in a flat `src/`, `src/components/`, `components/` or `app/components/`: [zero config](/docs/configuration#zero-config) finds those. Otherwise a config file with a `components` glob is enough. `npx -y onsystem inspect` prints what was found, and [troubleshooting](/docs/troubleshooting) covers what to do when it finds nothing.', ], [ "Will check fail on my design system's own components?", @@ -75,7 +75,7 @@ export const faqEntries = (corpus: FaqCorpus): [question: string, answer: string ], [ 'Can I add our own rules?', - "Not to onsystem itself yet. Its rules are a fixed set: the config sets each one to `error`, `warn` or `off` and rejects a rule id it does not know, and the hook, `check_ui`, `onsystem check` and the GitHub Action run only these. Two things cover most requests. A native element your design system wraps is already a finding: with a `Link` component, a raw `` is reported by `prefer-design-system-component`, and the config's `elements` maps an element to the component that replaces it. For a rule only your team needs, write an ESLint rule: it runs next to onsystem's [ESLint plugin](/docs/eslint) in the editor and in CI, though not in the hook or `check_ui`. A rule most design systems would want is welcome as a pull request: [Adding a lint rule](https://github.com/dgesteves/onsystem/blob/main/CONTRIBUTING.md#adding-a-lint-rule) lists the steps.", + "Not to onsystem itself. Its rules are a fixed set: the config sets each one to `error`, `warn` or `off` and rejects a rule id it does not know, and the hook, `check_ui`, `onsystem check` and the GitHub Action run only these. Two things cover most requests. A native element your design system wraps is already a finding: with a `Link` component, a raw `` is reported by `prefer-design-system-component`, and the config's `elements` maps an element to the component that replaces it. For a rule only your team needs, write an ESLint rule: it runs next to onsystem's [ESLint plugin](/docs/eslint) in the editor and in CI, though not in the hook or `check_ui`. A rule most design systems would want is welcome as a pull request: [Adding a lint rule](https://github.com/dgesteves/onsystem/blob/main/CONTRIBUTING.md#adding-a-lint-rule) lists the steps.", ], [ 'Does it work with Vue, Svelte or Angular?', diff --git a/test/cli-docs.test.ts b/test/cli-docs.test.ts index 108704f6..45bb3179 100644 --- a/test/cli-docs.test.ts +++ b/test/cli-docs.test.ts @@ -3,6 +3,7 @@ import path from 'node:path'; import ts from 'typescript'; import { beforeAll, describe, expect, it } from 'vitest'; +import { parse } from 'yaml'; import { main, type Io } from '../src/cli.js'; import { CONFIG_FILES } from '../src/config.js'; @@ -222,8 +223,14 @@ describe('the CLI reference (docs/cli.md)', () => { expect(section(cli, '`inspect`')).toMatch(/^\| `--format ` +\| `json` /m); }); - it('names every config file name the CLI looks for', () => { + it('names every config file name the CLI looks for, as docs/configuration.md does', () => { for (const name of CONFIG_FILES) expect(cli).toContain(`\`${name}\``); + // `onsystem.config.json` (or `.mjs`, `.js`, or `.ts` and `.mts` …) opens its Config file section. + const configFile = section(read('docs/configuration.md'), 'Config file').split('\n\n')[1]; + for (const name of CONFIG_FILES) { + const ext = name.replace(/^onsystem\.config/, ''); + expect(configFile, name).toMatch(new RegExp(`\`(?:onsystem\\.config)?\\${ext}\``)); + } }); it('is linked from the quickstart', () => { @@ -232,12 +239,16 @@ describe('the CLI reference (docs/cli.md)', () => { }); describe('the GitHub Action inputs (docs/required-check.md)', () => { - it('lists every input of action.yml with its default', () => { - const yml = read('action.yml'); - const block = yml.slice(yml.indexOf('\ninputs:'), yml.indexOf('\noutputs:')); - const inputs = [ - ...block.matchAll(/^ {2}([a-z-]+):\n {4}description:[^\n]*\n(?: {4}default: (.*)\n)?/gm), - ].map((m) => ({ name: m[1] ?? '', default: (m[2] ?? '').replace(/^'(.*)'$/, '$1') })); + it('lists every input of action.yml with its default, and no other', () => { + // Read with a YAML parser, so an input with a multi-line description, or with `required` + // before `default`, is not skipped. + const action = parse(read('action.yml')) as { + inputs: Record; + }; + const inputs = Object.entries(action.inputs).map(([name, input]) => ({ + name, + default: input.default === undefined ? '' : String(input.default), + })); expect(inputs.map((i) => i.name)).toContain('only-changed-lines'); const doc = read('docs/required-check.md'); for (const input of inputs) { @@ -245,5 +256,10 @@ describe('the GitHub Action inputs (docs/required-check.md)', () => { expect(row, input.name).not.toBeNull(); expect(row?.[1], input.name).toBe(input.default ? `\`${input.default}\`` : ''); } + // One row per input: none left behind by an input that is gone. + const rows = [...section(doc, 'The workflow').matchAll(/^\| `([a-z-]+)` +\|/gm)].map( + (m) => m[1] ?? '', + ); + expect(rows.sort()).toEqual(inputs.map((i) => i.name).sort()); }); }); diff --git a/test/custom-rules-faq.test.ts b/test/custom-rules-faq.test.ts index a4060581..570de617 100644 --- a/test/custom-rules-faq.test.ts +++ b/test/custom-rules-faq.test.ts @@ -5,10 +5,13 @@ import { describe, expect, it } from 'vitest'; import { loadConfig, RULE_IDS } from '../src/config.js'; import { fixture, load, TSCONFIG } from './helpers.js'; +import { section } from './markdown.js'; // "Can I add our own rules?" is the FAQ's answer to the question users ask once they see the // rules: what the config does with a rule id it does not know, what already covers the commonest -// ask (no raw where the design system has a Link), and where a new built-in rule starts. +// ask (no raw where the design system has a Link), and where a new built-in rule starts. No +// plugin API for rules is planned, so the answer promises none. The answer on components outside +// components/ui names the folders zero config reads, as docs/configuration.md does. const root = path.resolve(import.meta.dirname, '..'); const read = (file: string) => @@ -25,6 +28,11 @@ function answer(question: string): string { describe('the FAQ on custom rules (site/lib/faq.ts)', () => { const text = answer('Can I add our own rules?'); + it('promises no way to add rules to onsystem itself', () => { + expect(text).toContain('Not to onsystem itself. Its rules are a fixed set'); + expect(text).not.toMatch(/\byet\b/); + }); + it('says the config rejects a rule id it does not know', async () => { expect(text).toContain('rejects a rule id it does not know'); const dir = fixture({ @@ -63,3 +71,15 @@ export function Link(props: React.ComponentProps<"a">) { expect(text).toContain('[ESLint plugin](/docs/eslint)'); }); }); + +describe('the FAQ on components outside components/ui (site/lib/faq.ts)', () => { + it('names the flat folders of primitives wrappers zero config reads', () => { + const text = answer('My components are not in components/ui. Do I need a config?'); + const zeroConfig = section(read('docs/configuration.md'), 'Zero config'); + for (const folder of ['src/', 'src/components/', 'components/', 'app/components/']) { + expect(text, folder).toContain(`\`${folder}\``); + expect(zeroConfig, folder).toContain(`\`${folder}\``); + } + expect(text).toContain('[zero config](/docs/configuration#zero-config)'); + }); +}); diff --git a/test/perf-docs.test.ts b/test/perf-docs.test.ts index 599724e2..fdbb7806 100644 --- a/test/perf-docs.test.ts +++ b/test/perf-docs.test.ts @@ -43,11 +43,19 @@ describe('Performance and footprint (docs/how-it-works.md)', () => { it('says where, when and with what the numbers were measured', () => { const { text } = perfBlock(); expect(text).toMatch( - /^\s*Measured with onsystem \d+\.\d+\.\d+ on .+ \(\d+ cores, \d+ GB\), .+, Node\.js \d+\.\d+\.\d+, on \d{4}-\d{2}-\d{2}, with `node scripts\/perf\.mjs`: each time is the median of \d+ runs\./, + /^\s*Measured with onsystem \d+\.\d+\.\d+ on .+ \(\d+ cores, \d+ GB\), .+, Node\.js \d+\.\d+\.\d+, on \d{4}-\d{2}-\d{2}, with `node scripts\/perf\.mjs`: each time is the median of \d+ runs?\./, ); expect(fs.existsSync(path.join(root, 'scripts/perf.mjs'))).toBe(true); }); + it('says which way the missing node_modules cut, in the words perf.mjs writes', () => { + const { text } = perfBlock(); + const caveat = /runs?\. (The corpus checkouts [^\n]*)/.exec(text)?.[1] ?? ''; + expect(caveat).toContain('reading the design system takes longer'); + // perf.mjs writes it in a template literal, with its backticks escaped. + expect(read('scripts/perf.mjs')).toContain(caveat.replace(/`/g, '\\`')); + }); + it('has the rows perf.mjs writes, each with a measured result', () => { const { rows } = perfBlock(); const measured = rows.map(([m]) => m); diff --git a/test/support-tiers-docs.test.ts b/test/support-tiers-docs.test.ts index 556a5d54..e6589aee 100644 --- a/test/support-tiers-docs.test.ts +++ b/test/support-tiers-docs.test.ts @@ -109,6 +109,16 @@ describe('the support tiers in docs/clients.md', () => { } }); + it('the quickstart calls tested only the Tier 1 setups, and links the tiers', () => { + const row = /^\| Agents +\| (.*?) +\|/m.exec(read('docs/quickstart.md'))?.[1] ?? ''; + expect(row).toContain('](clients.md#support-tiers)'); + const tested = row.slice(0, row.indexOf('not yet run in it')); + expect(tested).toMatch(/^Tested in the client: /); + for (const { setup } of tiers().filter((r) => r.tier === '2')) { + expect(tested, setup).not.toContain(setup.replace(/ \(.*\)$/, '')); + } + }); + it('is linked from the top of the page, before any setup', () => { const intro = clients.slice(0, clients.indexOf('\n## ')); expect(intro).toContain('](#support-tiers)'); From b833a8cba37c8ae4a726f3d0667a7047072444b7 Mon Sep 17 00:00:00 2001 From: Diogo Esteves Date: Thu, 29 Oct 2026 11:33:46 +0000 Subject: [PATCH 3/3] test(perf-docs): escape backslashes as well as backticks when matching perf.mjs's template literal --- test/perf-docs.test.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/perf-docs.test.ts b/test/perf-docs.test.ts index fdbb7806..dffbb370 100644 --- a/test/perf-docs.test.ts +++ b/test/perf-docs.test.ts @@ -52,8 +52,8 @@ describe('Performance and footprint (docs/how-it-works.md)', () => { const { text } = perfBlock(); const caveat = /runs?\. (The corpus checkouts [^\n]*)/.exec(text)?.[1] ?? ''; expect(caveat).toContain('reading the design system takes longer'); - // perf.mjs writes it in a template literal, with its backticks escaped. - expect(read('scripts/perf.mjs')).toContain(caveat.replace(/`/g, '\\`')); + // perf.mjs writes it in a template literal, with its backslashes and backticks escaped. + expect(read('scripts/perf.mjs')).toContain(caveat.replace(/[\\`]/g, '\\$&')); }); it('has the rows perf.mjs writes, each with a measured result', () => {