Add conformance tests

Author: jakeboone02

AGENTS.md

@@ -218,6 +218,42 @@ it passes whether or not the `snapshot()` is there:
 - A subquery renders **bare `<div>`s** for its group header/body, not a `rule-group` element
   (React's `RuleWithSubQueryGroupComponentsWrapper`), and it is not customizable.
 
+### The conformance harness (step 6)
+
+`vitest.conformance.config.ts` runs **two projects**, because the two fixture layers demand
+opposite render modes. This is structural, not cosmetic — one plugin instance cannot serve both.
+
+| Project           | Compilation                            | Environment | Renders                                                                                 |
+| ----------------- | -------------------------------------- | ----------- | --------------------------------------------------------------------------------------- |
+| `conformance-ssr` | `generate: 'ssr'`, `hydratable: false` | `node`      | `renderToString`, controlled `query` — the **static** layer                             |
+| `conformance-dom` | default `dom`                          | `jsdom`     | testing-library + `flush()`, **uncontrolled** `defaultQuery` — the **post-flush** layer |
+
+- **The static layer is rendered server-side deliberately.** The fixtures come from
+  `renderToStaticMarkup` with no effects run; Solid's client `render()` runs effects, so the
+  "extract before the scheduler flushes" trick both prior ports used is unavailable. SSR runs no
+  effects at all, so this is an _exact_ match rather than an approximation — and it exercises the
+  ssr path in all 50 cases for free.
+- ⚠️ **`solid({ ssr: true, … })` on the ssr project is load-bearing** and is _not_ redundant with
+  `solid: { generate: 'ssr' }`. `vite-plugin-solid@3` injects a `browser` condition in test mode
+  (`isTestMode && !options.ssr`), `@solidjs/web` lists `browser` before `node`, and the browser
+  build's `renderToString` is a stub returning `undefined`. `ssr: true` suppresses the injection
+  (and the plugin's forced `environment: 'jsdom'`); `options.solid` merges last, so the explicit
+  `generate`/`hydratable` still win. Do **not** "fix" this by hand-writing `resolve.conditions`.
+- **`hydratable: false`** — hydration keys land as _attributes_ and would break byte-identical
+  `class` comparison. (Marker comments are ignored by extraction; attributes are not.)
+- **`cases.ts` carries no rendering.** The render helpers are split into `render-ssr.tsx` and
+  `render-dom.tsx` so the ssr project never imports `@solidjs/testing-library` and vice versa.
+- **`scenarios.tsx`, not `.ts`** — `getValueEditorSeparator` returns JSX, which each project must
+  compile for its own target.
+- ⚠️ **`actions.solid.test.ts` must `flush()` once after `createRoot` before replaying.** Effects
+  created inside a root are queued, not run eagerly, so the controlled-`query` sync effect's first
+  run otherwise lands on the flush _after_ the first op and silently reverts it. Found the hard
+  way: 4 of 19 cases failed with the ops apparently never applied.
+- `extract.ts` exposes both `extractFromContainer` and `extractFromMarkup` (upstream's
+  `schemaVersion` 2 split). The markup form builds its own `JSDOM` when there is no global
+  `DOMParser`, which is what lets the ssr project run in the `node` environment and thereby prove
+  a server render needs no document.
+
 ### Types
 
 - **`jsxImportSource` is `"@solidjs/web"`.** `solid-js@2` owns no JSX namespace and no
@@ -241,12 +277,30 @@ it passes whether or not the `snapshot()` is there:
 **Standing rule: every gate must be proven to fail.** When a step adds a gate, deliberately break
 it, record that it went red, then revert. A gate that cannot fail is worse than none.
 
-Current gates (step 8): `check:versions`, `fmt:check`, `build`, `check`, `check:exports`,
+Current gates (steps 6 + 8): `check:versions`, `fmt:check`, `build`, `check`, `check:exports`,
 `lint`, `test:coverage` (global 80% lines, plus a per-directory 90% lines on `packages/*/src/**` —
 widened at step 5 from the step-3 `packages/*/src/reactive/**`, which it subsumes; both
-non-vacuous, both proved red with no injected dead code), `test:ssr` (**both halves**), and
-`check` including the examples. (`conformance` is a stub that exits 0 until step 6; it is not a
-gate yet.)
+non-vacuous, both proved red with no injected dead code), **`conformance`** (237 assertions: 50
+static classnames, 50 accessible descriptions, 50 post-flush classnames, 58 action sequences, 19
+port-side action sequences, plus alignment/drift/format), `test:ssr` (**both halves**), and
+`check` including the examples.
+
+The four step-6 conformance gates were each proved red and reverted:
+
+1. **DOM parity** — ` conformance-gate-probe` appended to `ActionElement.tsx`'s class turned
+   exactly 100 cases red (50 static + 50 post-flush), which is the split the two projects promise.
+2. **`schemaVersion`** — `EXPECTED_SCHEMA_VERSION = 3` made `conformance:fetch` exit 1 with the
+   "update `test/conformance` before bumping the tag" message.
+3. **Scenario drift** — renaming the local `allControls` scenario turned the drift test (and the
+   three case-alignment tests) red while all 50 rendered cases stayed green.
+4. **Value-editor reset** — an early `return` in `createValueEditorReset`'s apply phase left
+   conformance at 237/237 green (as upstream predicts: every case is `differsFromStatic: false`)
+   while turning 5 of the 9 post-mount unit assertions red. That asymmetry is exactly why the
+   plan forbids proving this one through the post-flush fixture.
+
+Separately confirmed: with `test/fixtures/` removed, `bun run test` still passes 284/284 and
+`conformance:test` fails with the actionable "run `bun run conformance:fetch`" message rather than
+an opaque parse error.
 
 The step-8 example gate was proved red twice, independently, and reverted both times:
 `document.title` injected into `QueryBuilder.tsx` turned the served response into a 500 and took

packages/solid-querybuilder/package.json

@@ -59,7 +59,9 @@
     "check:exports": "bun run ./scripts/check-dist-specifiers.ts && attw --pack . --profile esm-only",
     "check:versions": "bun run ./scripts/check-versions.ts",
     "test:ssr": "bun run ./scripts/ssr-smoke.ts",
-    "conformance": "echo 'conformance harness lands at step 6' && exit 0"
+    "conformance": "bun run conformance:fetch && bun run conformance:test",
+    "conformance:fetch": "bun ./scripts/fetch-fixtures.ts",
+    "conformance:test": "vitest run --config vitest.conformance.config.ts"
   },
   "peerDependencies": {
     "solid-js": "^2.0.0-beta.32",

packages/solid-querybuilder/scripts/fetch-fixtures.ts

@@ -0,0 +1,95 @@
+/**
+ * Downloads the conformance fixture set published by `react-querybuilder` as a release asset and
+ * extracts it into `test/fixtures/` (gitignored).
+ *
+ * The fixtures are the contract this port is held to: the `class` attribute of every rendered
+ * element, the accessible description of every rule group, and the result of every curated
+ * mutation sequence. They are *not* generated here — regenerating them locally would make the
+ * tests tautological.
+ *
+ * The pinned tag is a deliberate constant rather than "latest": a fixture set changes when
+ * upstream rendering changes, and that should surface as a reviewed bump here, not as a
+ * mysterious CI failure on an unrelated PR.
+ */
+
+import { mkdir, rm } from 'node:fs/promises';
+import * as path from 'node:path';
+
+/**
+ * The upstream release whose fixtures this port is verified against. Bump deliberately, and
+ * expect a diff in the conformance tests when you do.
+ */
+export const CONFORMANCE_TAG = 'v8.22.4';
+
+/**
+ * The fixture *shape* version. Not the tag: upstream may cut a dozen releases without changing
+ * the schema, and a schema change must fail loudly rather than be mis-read.
+ */
+export const EXPECTED_SCHEMA_VERSION = 2;
+
+const ASSET = 'rqb-conformance-fixtures.tar.gz';
+const RELEASE_URL = `https://github.com/react-querybuilder/react-querybuilder/releases/download/${CONFORMANCE_TAG}`;
+
+export const fixturesDir: string = path.resolve(import.meta.dirname, '../test/fixtures');
+
+const download = async (url: string): Promise<ArrayBuffer> => {
+  const response = await fetch(url);
+  if (!response.ok) {
+    throw new Error(`Failed to download ${url}: ${response.status} ${response.statusText}`);
+  }
+  return response.arrayBuffer();
+};
+
+/**
+ * The tarball is not byte-reproducible across generation runs (BSD `tar` has no `--sort`/
+ * `--mtime`), so the checksum is only meaningful against the sidecar published *with that
+ * archive*. Both are therefore downloaded from the same release, and the check guards transport,
+ * not provenance.
+ */
+const verifyChecksum = (archive: ArrayBuffer, sidecar: string) => {
+  const expected = sidecar.trim().split(/\s+/)[0];
+  const actual = new Bun.CryptoHasher('sha256').update(archive).digest('hex');
+  if (actual !== expected) {
+    throw new Error(`Checksum mismatch for ${ASSET}: expected ${expected}, got ${actual}`);
+  }
+};
+
+export const fetchFixtures = async (): Promise<void> => {
+  const [archive, sidecar] = await Promise.all([
+    download(`${RELEASE_URL}/${ASSET}`),
+    download(`${RELEASE_URL}/${ASSET}.sha256`).then(b => new TextDecoder().decode(b)),
+  ]);
+
+  verifyChecksum(archive, sidecar);
+
+  // Wipe first, so a fixture file dropped upstream cannot linger locally and keep a stale test passing.
+  await rm(fixturesDir, { recursive: true, force: true });
+  await mkdir(fixturesDir, { recursive: true });
+
+  const archivePath = path.join(fixturesDir, ASSET);
+  await Bun.write(archivePath, archive);
+
+  const untar = Bun.spawnSync(['tar', '-xzf', archivePath, '-C', fixturesDir]);
+  if (untar.exitCode !== 0) {
+    throw new Error(`tar failed: ${new TextDecoder().decode(untar.stderr)}`);
+  }
+  await rm(archivePath);
+
+  const index = await Bun.file(path.join(fixturesDir, 'index.json')).json();
+  if (index.schemaVersion !== EXPECTED_SCHEMA_VERSION) {
+    throw new Error(
+      `Fixture schemaVersion is ${index.schemaVersion}, expected ${EXPECTED_SCHEMA_VERSION}. ` +
+        `The fixture format changed upstream; update test/conformance before bumping the tag.`
+    );
+  }
+
+  console.log(
+    `Fetched conformance fixtures from ${CONFORMANCE_TAG} ` +
+      `(core ${index.generator.version}, schema ${index.schemaVersion}): ` +
+      `${index.counts.renderedCases} rendered cases, ${index.counts.actionSequences} sequences.`
+  );
+};
+
+if (import.meta.main) {
+  await fetchFixtures();
+}

packages/solid-querybuilder/test/conformance/accessible-descriptions.test.ts

@@ -0,0 +1,43 @@
+/**
+ * The `title` attribute of every rule group, for all 50 scenario × query pairs.
+ *
+ * This is where `accessibleDescriptionGenerator` surfaces. The `customized` scenario supplies a
+ * non-default generator, so these assertions cover more than the identity function.
+ *
+ * Static layer only, rendered server-side like `classnames.test.ts`. Upstream deliberately
+ * publishes no post-flush descriptions layer: `title` derives from `path` alone and never reads
+ * `value`, so effects cannot change it.
+ */
+
+import { describe, expect, it } from 'vitest';
+import { caseAlignment, loadFixture, renderPairs } from './cases.js';
+import type { AccessibleDescriptionEntry } from './extract.js';
+import { renderAndExtract } from './render-ssr.jsx';
+
+interface AccessibleDescriptionsFixture {
+  cases: {
+    scenario: string;
+    query: string;
+    accessibleDescriptions: AccessibleDescriptionEntry[];
+  }[];
+}
+
+const fixture = await loadFixture<AccessibleDescriptionsFixture>('accessible-descriptions.json');
+
+describe('conformance: accessible descriptions', () => {
+  it('renders the same number of cases the fixture recorded, in the same order', () => {
+    const { local, recorded } = caseAlignment(fixture.cases);
+    expect(renderPairs).toHaveLength(fixture.cases.length);
+    expect(local).toEqual(recorded);
+  });
+
+  for (const [i, pair] of renderPairs.entries()) {
+    const expected = fixture.cases[i];
+
+    it(`${expected.scenario} × ${expected.query}`, () => {
+      const { accessibleDescriptions } = renderAndExtract(pair);
+
+      expect(accessibleDescriptions).toEqual(expected.accessibleDescriptions);
+    });
+  }
+});

packages/solid-querybuilder/test/conformance/actions.solid.test.ts

@@ -0,0 +1,93 @@
+/**
+ * The port-side half of the action conformance suite: the guard-sensitive sequences replayed
+ * through the manager `createQueryBuilderState` builds from `QueryBuilderProps`, rather than
+ * through a manager configured directly.
+ *
+ * This is what catches an option-mapping bug — a `disabled` array that never reaches
+ * `disabledPaths`, or a `maxLevels` that defaults wrong. Two narrowings relative to
+ * `actions.test.ts`, both deliberate:
+ *
+ * - Only the resulting query is asserted. `QueryBuilderProps` has no `onInvalidTarget`, so abort
+ *   reasons are not observable here; a guard that fails to apply shows up anyway as a query that
+ *   changed when it should not have.
+ * - `id`s are stripped before comparing. `createQueryBuilderState` seeds its manager through
+ *   `resolveCandidateQuery`, which draws from the injected generator, so generated `id`s are
+ *   offset by the seeding draws by a constant. That offset is an artifact of construction, not of
+ *   mutation.
+ *
+ * The state is built inside `createRoot(dispose => …)` — Solid's equivalent of Svelte's
+ * `$effect.root` and Vue's `effectScope`. `createQueryBuilderState` registers an `onCleanup` for
+ * its manager subscription and reads a context, both of which need an owner on the stack; a root
+ * supplies one outside a component.
+ *
+ * ⚠️ The root body is an *owned* scope, so the replay itself runs outside it: a reactive write
+ * made under an owner throws `REACTIVE_WRITE_IN_OWNED_SCOPE`. And `flush()` runs between ops,
+ * because a plain read after a write still returns the old value until the next microtask.
+ */
+
+import type { RuleGroupType, RuleGroupTypeAny } from '@react-querybuilder/core';
+import { formatQuery } from '@react-querybuilder/core';
+import { createRoot, flush } from 'solid-js';
+import { describe, expect, it } from 'vitest';
+import { createQueryBuilderState } from '../../src/reactive/index.js';
+import type { QueryBuilderProps } from '../../src/types/index.js';
+import { loadFixture } from './cases.js';
+import { createIdGenerator, queries, type QueryFixtureName } from './queries.js';
+import { applyOp, type ActionCase, type RunOptions } from './replay.js';
+
+const fixture = await loadFixture<{ cases: ActionCase[] }>('actions.json');
+
+const stripIDs = (query: RuleGroupTypeAny): unknown =>
+  JSON.parse(formatQuery(query as RuleGroupType, 'json_without_ids'));
+
+/** `respectDisabled: false` has no prop equivalent, so those cases are skipped. */
+const eligible = fixture.cases.filter(
+  c =>
+    c.options.respectDisabled !== false &&
+    (c.options.disabledPaths !== undefined ||
+      c.options.maxLevels !== undefined ||
+      c.fixture === 'rootDisabled' ||
+      c.fixture === 'withDisabled')
+);
+
+const propsFor = (options: RunOptions): Partial<QueryBuilderProps> => ({
+  ...(options.queryDisabled ? { disabled: true } : {}),
+  ...(options.disabledPaths ? { disabled: options.disabledPaths } : {}),
+  ...(options.maxLevels === undefined ? {} : { maxLevels: options.maxLevels }),
+});
+
+describe('conformance: actions through createQueryBuilderState', () => {
+  it('has guard-sensitive cases to replay', () => {
+    expect(eligible.length).toBeGreaterThan(5);
+  });
+
+  for (const { name, fixture: fixtureName, ops, options, expected } of eligible) {
+    it(name, () => {
+      const root = createRoot(dispose => {
+        const state = createQueryBuilderState({
+          ...propsFor(options),
+          query: structuredClone(queries[fixtureName as QueryFixtureName]),
+          idGenerator: createIdGenerator(),
+          // The fixture corpus mixes `RuleGroupType` and `RuleGroupTypeIC`, which
+          // `QueryBuilderProps` discriminates between. The cast collapses that here; the
+          // discrimination itself is covered by `types.test-d.ts`.
+        } as QueryBuilderProps);
+        return { manager: state.manager, dispose };
+      });
+
+      // Settle the mount before replaying. Effects created inside a root are *queued*, not run
+      // eagerly, so the controlled-`query` sync effect's first run happens at the next `flush()`
+      // — which, without this, would be the one after the first op, silently reverting it. A
+      // mounted component has already had that flush before a user can touch anything.
+      flush();
+
+      for (const op of ops) {
+        applyOp(root.manager, op);
+        flush();
+      }
+
+      expect(stripIDs(root.manager.getQuery())).toEqual(stripIDs(expected.query));
+      root.dispose();
+    });
+  }
+});

packages/solid-querybuilder/test/conformance/actions.test.ts

@@ -0,0 +1,43 @@
+/**
+ * Replays all 58 curated mutation sequences through `QueryManager` and asserts the resulting
+ * query, per-op abort reasons, and refusal flags against the recorded fixtures.
+ *
+ * This exercises core rather than the port. It is here anyway because it pins the core version
+ * the port is built against to the behavior the fixtures describe: a silent behavior change in a
+ * patch release fails here rather than in a consumer's app. The port-side half — that
+ * `createQueryBuilderState` maps props onto the manager's guard options correctly — is in
+ * `actions.solid.test.ts`.
+ */
+
+import { QueryManager } from '@react-querybuilder/core';
+import { describe, expect, it } from 'vitest';
+import { loadFixture } from './cases.js';
+import { createIdGenerator, queries, type QueryFixtureName } from './queries.js';
+import { guardsOf, replay, type ActionCase } from './replay.js';
+
+const fixture = await loadFixture<{ cases: ActionCase[] }>('actions.json');
+
+describe('conformance: actions', () => {
+  it('replays every recorded sequence', () => {
+    expect(fixture.cases).toHaveLength(58);
+  });
+
+  for (const { name, fixture: fixtureName, ops, options, expected } of fixture.cases) {
+    it(name, () => {
+      const result = replay(
+        ops,
+        onInvalidTarget =>
+          new QueryManager(structuredClone(queries[fixtureName as QueryFixtureName]), {
+            idGenerator: createIdGenerator(),
+            ...guardsOf(options),
+            onInvalidTarget,
+          })
+      );
+
+      expect(result.query).toEqual(expected.query);
+      expect(result.aborts).toEqual(expected.aborts);
+      expect(result.allAborts).toEqual(expected.allAborts);
+      expect(result.refused).toEqual(expected.refused);
+    });
+  }
+});