Verify generated pages with real Chromium rendering and pixel comparisons

.changepacks/changepack_log_wVMbmtKOkV4yK0ZUQvwQ2.json

@@ -0,0 +1,7 @@
+{
+  "changes": {
+    "Cargo.toml": "Patch"
+  },
+  "note": "Require real Chromium rendering and pinned Figma pixel comparisons before page completion; cover generated DevupUI pages, missing backgrounds, wrong colors and failed assets in the end-to-end harness.",
+  "date": "2026-10-19T14:22:49.794557600Z"
+}

crates/devup-mcp-devup-ui/examples/render_fixture.rs

@@ -0,0 +1,23 @@
+//! Replay a snapshot through production codegen for the browser regression test.
+use std::{error::Error, path::Path};
+
+use devup_mcp_devup_ui::codegen::{
+    CodegenOptions, generate_component, node_ids::annotate_node_ids,
+};
+use devup_mcp_figma::Snapshot;
+
+fn main() -> Result<(), Box<dyn Error>> {
+    let args: Vec<String> = std::env::args().skip(1).collect();
+    if args.len() != 2 {
+        return Err("usage: render_fixture <snapshot.json> <output.tsx>".into());
+    }
+    let snapshot: Snapshot = serde_json::from_slice(&std::fs::read(&args[0])?)?;
+    let options = CodegenOptions {
+        component_name: Some("RenderFixture".into()),
+        ..Default::default()
+    };
+    let mut output = generate_component(&snapshot, &snapshot.roots[0], &options)?;
+    annotate_node_ids(&mut output, &snapshot.nodes.keys().cloned().collect());
+    std::fs::write(Path::new(&args[1]), output.tsx)?;
+    Ok(())
+}

crates/devup-mcp/src/server/skills/devup-implementation-plan/SKILL.md

@@ -378,6 +378,13 @@ including a "view transitions: enabled (mechanism) / skipped (reason)" line.
    with the app running. If the runner cannot be installed, call status with
    `auditPlanPath` and `toolingUnavailable: "<why>"`: that URL's audit is
    inconclusive and the rest of the work goes on.
+   This real-browser step is required before claiming any page is done: build
+   and serve the actual generated app, not a hand-written stand-in. Static
+   checks and TSX snapshots cannot detect missing background images, wrong
+   colors or assets that never load. Run `image-loaded` and `page-health`, read
+   the browser report, and retain screenshots with their viewport and reference
+   identities. A report's `completed` means execution finished; a capture check's
+   `pass` means only that pixels were captured. Neither is a visual verdict.
 9. **Read the status, fix, re-run.** `devup_screen_plan` `action: "status"`
    with `auditPlanPath` and `auditReportPath`. Run every `visualComparisons`
    call (`devup_visual_compare`, each with its `threshold` and `diffPath`)
@@ -421,10 +428,15 @@ including a "view transitions: enabled (mechanism) / skipped (reason)" line.
    with `assume` (`about`: the decision id, `decision`: the recommended
    option). A destructive one (deleting, overwriting, reaching outside the
    project) stays unbuilt and is listed in the report.
-11. **Done means zero fails.** A URL is done when every check and comparison
-   passes at its drawn widths and health and overflow pass at every width -
-   or when each remaining finding is a converter bug with its repro filed, or
-   listed as unresolved with its evidence. Before reporting done, also:
+11. **Done means zero fails.** A URL is done only after its actual Chromium
+   screenshots have been compared with independently exported Figma originals,
+   every check and comparison passes at its drawn widths, and health and
+   overflow pass at every width. Inspect the page/section/asset diffs for
+   background images, colors and loaded assets; feed every result back to
+   status. Missing references, unavailable browser tooling, not-run comparisons,
+   converter bugs and unresolved findings remain incomplete with evidence.
+   Preserve screenshots, reports, Figma reference identities and diffs.
+   Before reporting done, also:
 
    - every confirmed URL is implemented;
    - the build passes: `bun run build` (`vinext build` for every app);
@@ -453,7 +465,8 @@ Hard rules, after confirm:
 
 - Visually verify every screen, section and asset at every drawn width.
 - Never stop to wait; reorder the work.
-- Done = zero fails, or listed as unresolved with evidence.
+- Done = a real browser render and zero failed or missing comparisons;
+  unresolved findings are reported as incomplete with evidence.
 - Layout is exact; a mismatch is drift or a converter bug, never tuned.
 
 ## What the questions mean

crates/devup-mcp/src/server/skills/devup/SKILL.md

@@ -179,7 +179,16 @@ the full rules; these are the steps.
 11. **Never stop to wait.** A decision mid-run: `action: "defer"` with the
     recommended option, keep working, take it with `assume` at the end; a
     destructive one stays unbuilt.
-12. **Done = zero fails.** Every check and comparison passes at the drawn
+12. **Done = zero fails.** A real Chromium render is mandatory before claiming
+    any page is done. Build and serve the generated app, execute the Playwright
+    runner, read its report and compare the actual screenshots with independent
+    Figma references. Static checks, TSX snapshots, a generated audit plan and a
+    capture check's `pass` do not prove visual correctness. Verify background
+    images, colors and loaded assets; inspect page/section/asset diffs and feed
+    every visual result back to status. Missing references, unavailable browser
+    tooling, failed assets and not-run comparisons mean incomplete. Preserve the
+    screenshots, reference identity, report and diffs in the final evidence.
+    Every check and comparison passes at the drawn
     widths and health and overflow pass at every required width. Report filed
     converter bugs and unresolved findings with evidence as incomplete. Also: every
     URL implemented, the production build free of `data-figma-node-id` (the

crates/devup-mcp/src/server/skills/manifest.json

@@ -71,8 +71,8 @@
       "documents": [
         {
           "path": "SKILL.md",
-          "bytes": 35206,
-          "sha256": "a60e55d51cc97c563419dcc50b449c8d83c2a8324470cfc6ade304a84b85f541"
+          "bytes": 36174,
+          "sha256": "3278b142e6780fb99413185b9c01ebabd1293fea1a924fc09aeae5b818c07f0f"
         }
       ],
       "sourceUrl": "https://github.com/dev-five-git/devup-mcp/blob/HEAD/crates/devup-mcp/src/server/skills/devup-implementation-plan",
@@ -89,8 +89,8 @@
       "documents": [
         {
           "path": "SKILL.md",
-          "bytes": 13682,
-          "sha256": "8e9f69df9be904af473844f4db606f2b2f35fe0318da07f142d08a9163a9e783"
+          "bytes": 14375,
+          "sha256": "c866ca964eb42cc01a4477a6922b6187abe3992cee6228476501373c1486a092"
         }
       ],
       "sourceUrl": "https://github.com/dev-five-git/devup-mcp/blob/HEAD/crates/devup-mcp/src/server/skills/devup",

harness/runtime-audit/README.md

@@ -0,0 +1,79 @@
+# Real-render verification
+
+Static validation and a successful screenshot capture do not establish visual
+correctness. With the generated app running, execute its runtime audit, compare
+the actual Chromium screenshots against the independently acquired Figma PNGs,
+and inspect the diffs before calling the page done.
+
+The existing `run.mjs` + `devup_runtime_audit ingest` + `devup_visual_compare`
+workflow supports page, section and asset comparisons. A capture check's `pass`
+means **captured**, not **visually matched**. Feed every comparison result back
+to screen-plan status; missing, failed or inconclusive cells block completion.
+
+For a command-line page gate (including CI):
+
+```sh
+npm ci
+npx playwright install chromium
+cargo build -p devup-mcp-visual
+node verify-render.mjs --plan plan.json --references references.json --out out/page --visual-bin ../../target/debug/devup-mcp-visual
+```
+
+On Windows append `.exe` to the comparator path. With a custom Cargo target
+directory use the binary under `cargo metadata`'s `target_directory`.
+
+Use an audit plan containing the drawn viewports at deviceScaleFactor 1, with
+`page-health`, `image-loaded` and the relevant geometry checks. Use the normal
+runtime audit separately for widths without Figma originals. Each planned
+screen/viewport must have exactly one entry in `references.json`:
+
+```json
+{
+  "references": [{
+    "screen": "home",
+    "width": 1440,
+    "height": 900,
+    "path": "figma/home-1440.png",
+    "sha256": "<SHA-256 of the independently exported Figma PNG>",
+    "threshold": 0.03
+  }]
+}
+```
+
+Paths are relative to the manifest. Export the original frame at scale 1 and
+the drawn dimensions, without resizing it. Never create a reference from the
+generated page, replace an original to pass a test, or raise thresholds to hide
+wrong backgrounds, colors or assets. Pin the original hash and retain its Figma
+file/node/version provenance alongside the manifest.
+
+`verify-render.mjs` launches the runtime-audit Chromium runner and compares every
+page screenshot with `devup-mcp-visual`. It writes the runtime report, screenshots,
+`real-render.json` and `visual-diff-*.png`. Exit codes: 0 means runtime and pixels
+passed; 1 means a runtime or visual failure; 2 means tooling, input or reference
+failure. Missing/duplicate references, changed reference bytes, missing captures,
+browser errors and invalid image dimensions cannot pass. Do not infer success
+from `report.status: completed`, which describes execution, not correctness.
+
+## Offline regression
+
+`npm test` runs all tests, including `test/real-render.test.mjs`. This needs Rust,
+Node and Chromium; it does not silently skip when they are missing. The test
+replays `fixtures/render-snapshot.json` through production Rust codegen, builds
+the result with the real DevupUI plugin, serves the production app, and compares
+the Chromium capture through the production Rust pixel comparator. The correct
+page must match exactly. Removing its background, changing a card color or
+requesting an unavailable asset must fail while the geometry still passes.
+
+The small checked-in PNG oracle is **synthetic**, not a live Figma acquisition:
+its independent pixel construction represents the snapshot's 320x240 frame,
+with `#123456` above y=120, `#abcdef` below, and a red 80x60 card at (32,32).
+`render-hero.png` is the same background without the card. The oracle is never
+rendered from the generated code or updated by the test. This deterministic
+offline contract proves the gate detects rendering failures; verification of an
+actual Figma page requires that page's exported original as described above.
+
+Set `DEVUP_KEEP_RENDER_EVIDENCE=1` to retain regression artifacts under `out/`
+for diagnosis or upload by a CI caller. Run `npm ci`,
+`npx playwright install --with-deps chromium`, and `npm test` to use this suite
+in Linux CI. The repository workflow is not wired to this browser suite yet:
+the GitHub proxy rejected pushes containing that workflow-file modification.

harness/runtime-audit/fixtures/render-snapshot.json

@@ -0,0 +1,28 @@
+{
+  "fileKey": "synthetic-render-contract",
+  "version": "1",
+  "roots": ["1:1"],
+  "diagnostics": [],
+  "nodes": {
+    "1:1": {
+      "id": "1:1", "type": "FRAME",
+      "fields": {
+        "name": "Hero", "width": 320, "height": 240,
+        "layoutMode": "NONE", "layoutSizingHorizontal": "FIXED", "layoutSizingVertical": "FIXED",
+        "childrenIds": ["1:2"],
+        "fills": [{"type": "IMAGE", "imageHash": "hero", "scaleMode": "FILL"}]
+      },
+      "extra": {}, "fieldErrors": {}
+    },
+    "1:2": {
+      "id": "1:2", "type": "FRAME",
+      "fields": {
+        "name": "Card", "parentId": "1:1", "x": 32, "y": 32, "width": 80, "height": 60,
+        "layoutMode": "NONE", "layoutSizingHorizontal": "FIXED", "layoutSizingVertical": "FIXED",
+        "childrenIds": [],
+        "fills": [{"type": "SOLID", "color": {"r": 1, "g": 0, "b": 0, "a": 1}}]
+      },
+      "extra": {}, "fieldErrors": {}
+    }
+  }
+}