Skip to content

testing.md

Latest commit

 

History

History
219 lines (142 loc) · 14 KB

testing.md

File metadata and controls

219 lines (142 loc) · 14 KB
Raw
summary Acceptance gates, generated-reference drift, menu and render coverage, visual gallery review, golden and byte-fixture rerecording, E2E, package audit, and PowerPoint repair checks
read_when
adding or debugging tests
changing IR, specs, themes, menus, faces, components, motifs, rendering, or export
a snapshot, gallery hash, golden, or byte fixture changed
preparing a release or validating the installed DSH plugin

Testing

Default gates

pnpm check

This is the default acceptance gate. It runs:

  1. pnpm typecheck
  2. pnpm lint
  3. pnpm test:no-tmp-leak, which is vitest run under scripts/tmp-leak-check.mts

Vitest tests are colocated as *.test.ts, *.test.tsx, and script tests under scripts/*.test.mts. The suite covers IR v5, spec v1, theme v2, menu validation, all registered components and faces, SVG rendering, audit, theme lookup and creation, CLI shells, PPTX conversion, ZIP patches, package audit, and the visual gallery matrix.

Run a focused test while developing, then finish with the full gate:

pnpm exec vitest run src/spec/theme-menu.test.ts
pnpm check

Do not replace the configured commands with a guessed build tool.

Temp directories

Vitest runs inside one temp root. src/test-run-root.ts, the globalSetup, creates pptwise-test-run-* in the system temp directory, or in /tmp when the system one is so long that the IPC socket tsx opens under the root would come near macOS's 104-byte cap on a socket path (macOS's own temp directory puts it at about 91), points TMPDIR, TEMP, and TMP at it before any worker starts, and removes it when the run ends. A test can make temp directories with os.tmpdir() and leave them, and so can the CLI children it spawns. A root left by a killed run is removed by a later run once the process that owned it is gone.

pnpm check runs the suite through scripts/tmp-leak-check.mts, which gives the run a private temp directory and fails when anything named pptwise-* is left in it. The same script measures any other command:

pnpm exec tsx scripts/tmp-leak-check.mts --report-only -- pnpm e2e

End-to-end export

pnpm e2e

The E2E script builds the package, drives dist/cli.js, renders real examples and a deck project, exercises validation, audit, preview, placeholders, brand extraction, workspace theme lookup, image formats, package invariants, and native PPTX output. It converts representative output with LibreOffice when soffice is installed. Color changes go through pptwise theme fork, not a render-time overlay.

On macOS the E2E also runs the skill's deck sequence (schema, narratives, themes, theme try, spec validate, inspect, validate, preview, images list, build) on a Chinese showcase deck inside sandbox-exec, with only the project folder and a temp folder writable: once with a writable .git, which may gain only the exclude line, and once with a read-only .git, which must not change. The fake home folder must come out unchanged and git status clean. Elsewhere it prints that it skipped. To keep its logs, run it on its own:

pnpm build && pnpm exec tsx scripts/e2e-sandbox.mts --root <empty dir>

Run E2E when changing themes, menus, faces, components, motifs, render code, audit, CLI target loading, assets, or PPTX export. It is intentionally separate from pnpm check because it builds the package and may use external local applications.

For the installed DSH plugin path:

pnpm e2e:dsh --workspace /tmp/pptwise-dsh-e2e --profile web

Use an isolated workspace and profile. This host-level smoke proves plugin discovery and installed CLI execution. It does not replace the deterministic source E2E.

Generated SKILL references

scripts/gen-skill-refs.mts owns four generated sections:

  • English and Chinese kind tables in skills/pptwise/references/layouts*.md
  • English and Chinese preset tables in skills/pptwise/references/spec*.md

The kind table derives from IR v5 KIND_VALUES plus every built-in theme menu. It contains all eleven kinds, boundary guidance, and preset availability counts.

The preset table derives from THEME_PRESETS plus BUILTIN_THEME_FILES. It contains all 24 ids, labels, occasions, identity strength, content-menu word count, and offered kinds.

After changing any source field that affects those tables, run:

pnpm gen:skill-refs

scripts/gen-skill-refs.test.mts renders the same sections in memory and compares every committed target byte for byte. Drift therefore fails pnpm check. Edit prose outside the generated markers and let the generator own the marked sections.

Theme switch matrix

pnpm theme-switch
pnpm theme-switch --decks=memo/zh --themes=crayon,clinic
pnpm theme-switch --bare-boundaries

Copies each showcase sample into a temporary folder, changes only spec.theme to each built-in theme, builds it, and prints pass or fail with every page that landed on a stand-in kind. --bare-boundaries empties cover, chapter, and ending page files so the run measures content pages alone. It is a maintainer's report, not part of pnpm check: a sample written for one theme's covers and fills is expected to need revision on some others. src/render/kind-landing.test.ts holds the kind guarantee in check: every built-in theme draws a typical page of every kind it lacks.

Visual gallery

pnpm gallery
pnpm gallery --only=theme
pnpm gallery --languages=zh,en
pnpm gallery --bbox

The gallery renders the review matrix through the production validation and SVG path. It has no alternate renderer. Output goes to .gallery/ by default and includes SVG pages, manifest.json, and a self-contained index.html.

The matrix covers themes, internal faces, the shared compositions (at least one page each in the 构图 band, COMPOSITION_PAGES), components, full-load cases, and heading constructions across language tracks. scripts/gallery.test.mts exercises the matrix during pnpm check, including inventory coverage and every page's pinned hash (see Gallery hash goldens). A new component or theme must not disappear from review merely because its builder was forgotten.

--bbox mounts SVG in a real browser and compares getBBox() against declared boxes. It is optional because it needs Playwright and depends on installed fonts. Findings go to .gallery/bbox.json and make the command fail.

For automated gallery audit:

pnpm evals:gallery
pnpm evals:gallery --full
pnpm evals:gallery --l1-only
pnpm evals:gallery --only=cross-language

Default mode compares current page fingerprints with evals/gallery/hashes.json and audits changed or added pages. L1 is deterministic. It grades every text run against the shape actually painted under its glyphs (a badge disc, a caption band, a card), with the same walk pptwise audit runs, and reports a run under its size's floor as low-contrast. evals/gallery/corpus-scan.test.mts holds the whole gallery to zero of them during pnpm check, and src/audit/stress-contrast.test.ts holds the new_components_stress and diagram stress decks to zero on every theme, since the gallery's ordinary content never reached the two defects those decks found. Authored leftover-count phrasing and ellipsis substitutes are rejected by validate before a page is painted, so L1 overflow-marker is the renderer-side backstop. L2 runs only when its local vision CLI is available and the environment permits it. Live findings are written to a gitignored verdict report.

The same command also runs the full 24-theme Latin and mixed-script capacity sweep (every component, chart, and device variant, plus adjacency pairings). A ratchet mismatch fails the process. pnpm evals:gallery --only=cross-language runs only that section. Vitest evals/gallery/cross-language-capacity.test.mts samples four themes (swiss, thesis, crayon, ink). Same assertions, smaller theme axis.

Golden rerecording discipline

A golden diff is behavior evidence, not routine noise. Never accept it blindly and never run a broad snapshot update merely to make the suite green.

Before rerecording any snapshot, hash set, SVG serialization, DrawingML sample, or package fixture:

  1. Reproduce the failing test on the old expectation.
  2. Trace the first source boundary that changed behavior.
  3. State why the new output is intended and which outputs should remain unchanged.
  4. Inspect a focused before and after diff. For visual changes, render the affected gallery pages and review them.
  5. Use the repository's dedicated writer when one exists.
  6. Review the committed diff for unrelated churn, unstable ordering, local paths, dates, and environment-dependent bytes.
  7. Run the focused test, then pnpm check. Run E2E when the render or export chain changed.

Vitest's -u is allowed only after this analysis and only for the smallest intentional target. Do not pass it automatically after a failure.

Gallery hash goldens

evals/gallery/hashes.json pins whole-page, geometry, and color fingerprints. Rerecord it only after the affected pages have passed human review:

pnpm gallery
pnpm evals:gallery --from=.gallery --full
pnpm exec tsx evals/gallery/write-hashes.mts

Inspect the hash diff and page inventory. Do not edit individual hashes by hand.

pnpm check holds every gallery page to its pin. scripts/gallery.test.mts already paints the whole matrix one section at a time, and each section's test compares every page's hash with hashes.json and names the pages that moved. A second test fails when the matrix gains a page the file does not pin or the file pins a page the matrix no longer draws. A drawing change therefore lands together with its rerecorded hashes, in the same commit, before it reaches main. The comparison reads the paint the test makes anyway, so it adds no measurable time: rendering the 2640 pages a second time for it would cost about 18 seconds.

Byte fixtures

One fixture pins the exact output for unassigned theme behavior:

  • src/render/__fixtures__/emphasis-unassigned-bytes.json

Rerecord it through the writer:

pnpm fixtures:unassigned-bytes

Do this only when an intended renderer change reaches those cases. Review the resulting byte-level diff. Never repair these files manually.

Hand-authored goldens

Some tests use hand-derived SVG or structural expectations rather than generated snapshots. Update those expectations by reasoning from the contract and inspecting the emitted structure. Do not convert a precise test into an opaque snapshot to reduce maintenance.

Package-audit hard gate

Every generatePptxBlob export runs src/pptx/package-audit.ts after ZIP patches and before returning bytes. There is no skip switch.

The gate checks required OOXML parts, content types, relationships, slide-list consistency, relationship targets, unique shape ids, finite positive transforms, animation references, and two-sided image-alt preservation for rendered image operations.

src/pptx/package-audit.test.ts creates real packages and surgically breaks one invariant at a time. The E2E script repeats key consistency checks against the built CLI output. When adding a ZIP patch, add a broken-package test that proves the audit rejects the failure mode.

Platform smoke

src/platform/node.smoke.test.ts exercises the installed Node platform seam with real DOM parsing and image re-encoding. Raster tests exercise sharp and SVG fidelity. Browser-safe modules must not import Node-only dependencies through src/index.ts.

Documentation checks

pnpm docs:list

Every operational document keeps summary and read_when front matter. Before adding a document, inspect the list for overlapping scope. When renaming a document, update all incoming links and any count-guard maintenance messages.

PowerPoint repair-dialog gate

LibreOffice and package audit cannot reproduce every native PowerPoint parser decision. Any change to exported XML structure, especially under src/pptx/svg2pptx or animation, gradient, font, relationship, or media ZIP patches, requires the macOS PowerPoint probe before release:

pnpm e2e
osascript scripts/ppt-repair-check.applescript "$PWD/.e2e-out/basic.pptx"
osascript scripts/ppt-repair-check.applescript "$PWD/.e2e-out/webp.pptx"
osascript scripts/ppt-repair-check.applescript "$PWD/.e2e-out/page-numbers.pptx"

page-numbers.pptx is the E2E footer leg: every footer mark on, so its content pages carry PowerPoint's slide-number field (src/pptx/pptx-slide-number.ts). Probe it whenever the footer row or the field patch changes.

The required result is OK. A repair dialog, repaired-title window, or timeout is a release blocker. Record which representative files were probed when handing off an export change.

The probe leaves the maintainer's PowerPoint alone. It copies the file under a unique name into PowerPoint's sandbox temp folder (~/Library/Containers/com.microsoft.Powerpoint/Data/tmp/), because PowerPoint for Mac is sandboxed and a file outside its container opens a "Grant File Access" prompt the probe cannot answer. It opens that copy, reads the verdict off the copy's own window, and closes only that presentation. A PowerPoint that was already running stays open with its other files untouched. One the probe had to launch is quit again. When the copy cannot be closed (a repair dialog is still up, or the open timed out), the probe leaves the copy and PowerPoint as they are for a person to look at.

Acceptance by change type

change minimum evidence
Docs only pnpm docs:list, relevant link and terminology checks, then pnpm check when generated or tested docs are involved.
IR, spec, or menu schema Focused schema tests, red and green behavior, generated refs, pnpm check.
Theme palette or menu Theme validation, theme try, affected gallery review, pnpm check, pnpm e2e.
Face, component, motif, render, or audit Focused tests, affected gallery pages, pnpm check, pnpm e2e.
PPTX or ZIP structure Focused package tests, pnpm check, pnpm e2e, LibreOffice result, PowerPoint repair probe.