From 521d03dae22d59401190acd7cad7fb296f9a1038 Mon Sep 17 00:00:00 2001 From: Paul Clark Date: Mon, 21 Sep 2026 18:41:27 -0400 Subject: [PATCH 1/2] Judge every rendered claim, over the whole set of states that render one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five of the six lane fixes apply here. Two gaps were confirmed by running the mutation against the suite as it stood at ab679b9 and watching it pass. Fix 1 — one helper asserts words, state and plate together. e2e/verdict-assertions.ts adds expectVerdict(page, id, {text, status, tone}) and expectClaim(page, id, {value, text}), and the coverage test now requires every recorded mutation's assertedBy spec to call the helper on that marker, instead of merely mentioning the id. This lab set className and dataset.status in separate expressions, so pinning the class green while the words and the status still followed the run was invisible: at ab679b9 that mutation passed 21/21. tone is checked as the painted background, because .verdict switches class and .mini-verdict switches on [data-status]. Fix 2 — measurements join the coverage loop on the same terms. The page rendered eighteen-odd numbers and not one carried a marker, so the coverage rule enforced itself over one family of a page that had two. Eight data-claim markers now carry the measurement in data-value, are registered in e2e/verdict-mutations.json under "claims", and fail the build both ways: a rendered measurement with no record, and a record naming a measurement the page no longer renders. Fix 3 — the digit-plus-unit scan. findStrayMeasurements walks the result regions for digit-plus-unit text and for bare integers in stats cells that sit outside a marker, with its own injection self-test so it cannot pass by being blind. Hex dumps are exempt as raw material (claims.spec checks them byte for byte against the size markers), and labels and explanatory prose are exempt the same way the verdict-word scan exempts them. The one static number on the page that was really a claim — "AES-128-CTR → 2(λ + 1) bits" — is now measured from a real expandSeed() and marked. Fix 4 — the oracle multiplied where the artifact iterates. The old key-size test asserted root + 17 * domainBits + final === measured at the one default domain: one per-level size multiplied by a level count, both literals, where serializeKey WRITES one correction word per level. It could not tell "sixteen words of seventeen bytes" from "one 272-byte block", which is exactly what the parts list claims. The oracle now measures the per-level cost as the slope between domains, derives λ and the root from it, counts the correction words actually present in the dump by walking it in stride, and SUMS them. Fix 5 — branch protection untouched. Confirmed read-only at the API (build + verdict-coverage required, enforce_admins false, force-push disabled), which is why this is a pull request. Fix 6 — driveEveryState visits every option of every control. Per control, not the cross-product. #alpha-range all 16 indices; the expansion stepper all 5 levels down and back; #new-keys once; #size-domain all 4 domains; #collusion-toggle both states, and again through a fetch for the tile it paints; #tamper-toggle both states with its own fetch; #fetch-record throughout. Two controls are not enumerable option sets and are recorded as such: #shelf-alpha is a 65,536-value number input, so its two rendering classes stand in — the in-range fetch and the rejection that renders no result — and the .key-inspector disclosure is opened because both scans skip elements with no client rects, so a closed disclosure hid its contents from the denominator entirely. No control on this page was judged not to change what renders. That widening is load-bearing, not tidiness: the formula's total was a hard literal correct only at 65,536 records, and at ab679b9 that mutation also passed 21/21 because nothing ever moved #size-domain. Baseline before any edit: 21 passed (15.0s). After: 23 passed (31.7s), plus 24 unit tests. Every mutation below was run with CI=1 so Playwright started its own server on the pinned port 4698. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UH9YUUhdeWXJz141Zy8FxW --- README.md | 8 +- e2e/claims.spec.ts | 218 +++++++++++++++++++++++------ e2e/flows.spec.ts | 18 ++- e2e/verdict-assertions.ts | 90 ++++++++++++ e2e/verdict-coverage.spec.ts | 260 +++++++++++++++++++++++++++++------ e2e/verdict-mutations.json | 66 ++++++++- e2e/verdict-scan.ts | 75 ++++++++++ index.html | 10 +- src/ui/app.ts | 36 +++-- 9 files changed, 671 insertions(+), 110 deletions(-) create mode 100644 e2e/verdict-assertions.ts diff --git a/README.md b/README.md index ad4aeda..ab65c62 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ The teaching implementation supports domains from $2^4$ through $2^{16}$. Its ps 4. **Collusion** — deliberately give both keys to one server and watch α become visible inside that server’s view. With collusion off, the page states the one-key case as a count: how many leaves the single expanded share lights, which is what “no distinguished target” means. 5. **Tampering** — flip one bit in a server answer. Every privacy condition still passes, but the client retrieves the wrong record because this PIR does not authenticate answers. The page names the damage from that run: how many bytes of the record differ, and that reconstruction raised nothing. -Every verdict the page renders carries a `data-verdict` marker and turns on a value the page computed. None of them is a fixed string. +Every verdict the page renders carries a `data-verdict` marker and turns on a value the page computed, and every number it renders carries a `data-claim` marker with the measurement behind the words in `data-value`. None of them is a fixed string, and the key-size meter states the parts it measured for the domain actually selected, not the default one. ## When to Use It @@ -79,7 +79,11 @@ There are **0 published known-answer tests** for this exact randomized BGI tree- The Playwright suite builds before serving and checks the production bundle on the fleet-unique port `4698`. It includes rendered mathematical claims, honest/tampered/colluding flows, retirement and no-op behavior, `[hidden]` containment, desktop/mobile reflow, axe WCAG 2.1 A/AA, arithmetic text contrast, and per-side non-text control contrast. -A separate **verdict coverage** gate walks the page through every state that can render an outcome and derives coverage from the DOM rather than from a list. It fails when a `data-verdict` marker has no mutation recorded in `e2e/verdict-mutations.json`, when a registered mutation names a verdict the page never renders, when a marker renders without a status, and when verdict wording or verdict styling appears outside a marker — the raw banner a careless builder adds later. It also injects such a banner and asserts the scan catches it, so the scan cannot pass by being blind. Each entry in that file names a mutation that has been applied and watched to turn its own assertion red, with the unmutated baseline passing in the same run. +A separate **verdict coverage** gate walks the page through every state that can render an outcome and derives coverage from the DOM rather than from a list. The walk visits every option of every control that changes what renders — each control on its own, never the cross-product — because that walk is the denominator both coverage rules and both stray scans enumerate over: anything reachable only at a setting it never visits is outside the set they judge. + +It fails when a `data-verdict` verdict or a `data-claim` measurement has no mutation recorded in `e2e/verdict-mutations.json`, when a registered mutation names a marker the page never renders, when a marker renders without a status or without a numeric value, and when verdict wording, verdict styling or a rendered measurement appears outside a marker — the raw banner, or the stray number, a careless builder adds later. It injects both and asserts the scans catch them, so neither scan can pass by being blind. Every recorded mutation must also be killed through `expectVerdict`/`expectClaim`, which assert a marker's words, its machine-readable state or value, and its painted plate in one call: text alone is not an assertion of state, and a mutation that pinned the record verdict's plate green while its words read RETRIEVED — AND WRONG used to pass the whole suite. + +Each entry in that file names a mutation that has been applied and watched to turn its own assertion red, with the unmutated baseline passing in the same run. That gate is its own job in `deploy.yml` and a required check on `main`, and both `deploy` and `dependabot-auto-merge` declare `needs: [build, verdict-coverage]`, so nothing ships past it. GitHub Pages deploys only after the unit, build, browser, and verdict-coverage gates pass. diff --git a/e2e/claims.spec.ts b/e2e/claims.spec.ts index 5f96c20..f9dd419 100644 --- a/e2e/claims.spec.ts +++ b/e2e/claims.spec.ts @@ -1,4 +1,5 @@ -import { expect, test } from '@playwright/test'; +import { expect, test, type Page } from '@playwright/test'; +import { claimValue, expectClaim, expectVerdict } from './verdict-assertions'; const SHELF_RECORDS = 65_536; @@ -8,6 +9,35 @@ function foldedCount(text: string): number { return Number((match as RegExpMatchArray)[1].replaceAll(',', '')); } +/** The domain sizes the page itself offers, read off the control. */ +async function domainOptions(page: Page): Promise { + return page + .locator('#size-domain option') + .evaluateAll((nodes) => nodes.map((node) => Number((node as HTMLOptionElement).value))); +} + +async function keyDump(page: Page, domainBits: number): Promise { + await page.locator('#size-domain').selectOption(String(domainBits)); + await expect(page.locator('[data-claim="key-bytes-total"]')).toHaveAttribute('data-value', /^\d+$/); + const hex = (await page.locator('#meter-key-hex').textContent()) ?? ''; + expect(hex, `the key dump for 2^${domainBits} is hex`).toMatch(/^[0-9a-f]+$/); + return hex; +} + +/** + * Walk the serialized key the way the serializer writes it: one correction word + * per level, in stride, each ending in a two-bit control byte. Returns the words + * actually present rather than a count derived by multiplying one size out. + */ +function readCorrectionWords(hex: string, rootBytes: number, wordBytes: number, finalBytes: number): number[] { + const bytes = (hex.match(/../g) ?? []).map((pair) => Number.parseInt(pair, 16)); + const controls: number[] = []; + for (let offset = rootBytes; offset + wordBytes <= bytes.length - finalBytes; offset += wordBytes) { + controls.push(bytes[offset + wordBytes - 1]); + } + return controls; +} + test('honest PIR output equals the indexed shelf record', async ({ page }) => { test.setTimeout(120_000); await page.goto('.'); @@ -15,32 +45,45 @@ test('honest PIR output equals the indexed shelf record', async ({ page }) => { await expect(page.locator('#pir-awaiting')).toBeVisible(); await page.getByRole('button', { name: 'Fetch privately' }).click(); await expect(page.locator('#pir-awaiting')).toBeHidden(); - await expect(page.locator('[data-verdict="record-match"]')).toHaveAttribute('data-status', 'pass'); await expect(page.locator('#server-zero-progress')).toHaveJSProperty('value', SHELF_RECORDS); await expect(page.locator('#server-one-progress')).toHaveJSProperty('value', SHELF_RECORDS); const retrieved = await page.locator('#retrieved-record').textContent(); const expected = await page.locator('#expected-record').textContent(); expect(alpha).toBe(41337); expect(retrieved).toBe(expected); - // The record verdict states a measurement, so check the measurement too. - await expect(page.locator('[data-verdict="record-match"]')).toHaveText( - new RegExp(`match with shelf\\[α\\] across all ${(expected ?? '').length / 2} bytes`) - ); + // Words, state and plate are one claim, and the verdict states a measurement, + // so the measurement is checked with them. + await expectVerdict(page, 'record-match', { + status: 'pass', + tone: 'pass', + text: [ + 'RETRIEVED · byte-for-byte match with shelf[α]', + new RegExp(`match with shelf\\[α\\] across all ${(expected ?? '').length / 2} bytes`) + ] + }); }); test('each server view verdict reports its own measured fold, not a fixed line', async ({ page }) => { test.setTimeout(120_000); await page.goto('.'); + // The key size each tile states is the one the meter measured, not a literal. + const keyBytes = await claimValue(page, 'key-bytes-total'); await page.getByRole('button', { name: 'Fetch privately' }).click(); await expect(page.locator('#pir-result')).toBeVisible(); + await expectVerdict(page, 'server-view-0', { + status: 'pass', + tone: 'pass', + text: [`one party-0 key of ${keyBytes} B`, /folded [\d,]+ of [\d,]+ records/] + }); + await expectVerdict(page, 'server-view-1', { + status: 'pass', + tone: 'pass', + text: [`one party-1 key of ${keyBytes} B`, /folded [\d,]+ of [\d,]+ records/] + }); + const zero = page.locator('[data-verdict="server-view-0"]'); const one = page.locator('[data-verdict="server-view-1"]'); - await expect(zero).toHaveAttribute('data-status', 'pass'); - await expect(one).toHaveAttribute('data-status', 'pass'); - await expect(zero).toContainText('one party-0 key of 290 B'); - await expect(one).toContainText('one party-1 key of 290 B'); - const foldedZero = foldedCount((await zero.textContent()) ?? ''); const foldedOne = foldedCount((await one.textContent()) ?? ''); // Neither view is alpha-blind if it folded one record or the whole shelf, and @@ -51,8 +94,11 @@ test('each server view verdict reports its own measured fold, not a fixed line', expect(foldedOne).toBeLessThan(SHELF_RECORDS); expect(Math.abs(foldedZero - foldedOne)).toBe(1); - await expect(page.locator('[data-verdict="collusion-state"]')).toHaveAttribute('data-status', 'pass'); - await expect(page.locator('[data-verdict="collusion-state"]')).toContainText('Collusion is off'); + await expectVerdict(page, 'collusion-state', { + status: 'pass', + tone: 'pass', + text: 'Collusion is off · neither key left its own server view' + }); }); test('the collusion state verdict follows the live trust boundary', async ({ page }) => { @@ -61,8 +107,11 @@ test('the collusion state verdict follows the live trust boundary', async ({ pag await page.locator('#collusion-toggle').check(); await page.getByRole('button', { name: 'Fetch privately' }).click(); await expect(page.locator('#pir-result')).toBeVisible(); - await expect(page.locator('[data-verdict="collusion-state"]')).toHaveAttribute('data-status', 'alarm'); - await expect(page.locator('[data-verdict="collusion-state"]')).toContainText('Collusion is ON'); + await expectVerdict(page, 'collusion-state', { + status: 'alarm', + tone: 'alarm', + text: 'Collusion is ON · one view holds both keys' + }); await expect(page.locator('#privacy-verdicts [data-status="pass"]')).toHaveCount(2); }); @@ -76,26 +125,100 @@ test('rendered leaf shares XOR to one point at the displayed alpha', async ({ pa expect(reconstruction[alpha]).toBe(1); expect(await page.locator('#tree-xor .node-bit').allTextContents()).toEqual(reconstruction.map(String)); // The headline verdict must report the index the shares actually reconstruct. - await expect(page.locator('[data-verdict="tree-point"]')).toHaveAttribute('data-status', 'pass'); - await expect(page.locator('[data-verdict="tree-point"]')).toContainText('ONE LIT POINT AT'); + await expectVerdict(page, 'tree-point', { + status: 'pass', + tone: 'pass', + text: ['ONE LIT POINT AT', String(reconstruction.indexOf(1))] + }); expect(Number(await page.locator('#tree-alpha-verdict').textContent())).toBe(reconstruction.indexOf(1)); }); -test('displayed serialized size, formula, and parts agree with the hex dump', async ({ page }) => { +/** + * Fix 4, in this lab's shape. The old oracle asserted + * `root + 17 * domainBits + final === measured` at the single default domain: it + * multiplied one per-level size out, with both the size and the level count + * literals, where the serializer WRITES one correction word per level. It could + * not tell "sixteen words of seventeen bytes" from "one 272-byte block", which + * is exactly what the parts list claims, and it agreed only because the suite + * never moved `#size-domain`. This walks every domain the control offers, + * measures the per-level cost as the slope between those measurements, counts + * the words actually present in the dump, and sums them. + */ +test('key-size claims are measured per level across every domain the page offers', async ({ page }) => { + test.setTimeout(120_000); await page.goto('.'); - const measured = Number(await page.locator('#key-bytes').getAttribute('data-count')); - const hex = (await page.locator('#meter-key-hex').textContent()) ?? ''; - expect(hex.length / 2).toBe(measured); - const root = Number.parseInt((await page.locator('#part-root').textContent()) ?? ''); - const levels = Number.parseInt((await page.locator('#part-levels').textContent()) ?? ''); - const final = Number.parseInt((await page.locator('#part-final').textContent()) ?? ''); - expect(root + levels + final).toBe(measured); - const formula = (await page.locator('#size-formula').textContent()) ?? ''; - const values = formula.match(/[\d,]+/g)?.map((value) => Number(value.replaceAll(',', ''))) ?? []; - expect(values).toEqual([17, 17, 65_536, 1, 290]); - const domainBits = Number(await page.locator('#size-domain').inputValue()); - expect(formula).toContain('log₂ 65,536'); - expect(root + 17 * domainBits + final).toBe(measured); + const options = await domainOptions(page); + expect(options.length, 'the meter offers more than one domain to derive a slope from').toBeGreaterThan(1); + + const totals = new Map(); + for (const domainBits of options) { + const hex = await keyDump(page, domainBits); + totals.set(domainBits, hex.length / 2); + } + + // Per-level cost, measured between domains rather than written down. + const sorted = [...options].sort((left, right) => left - right); + const slopes = sorted.slice(1).map((domainBits, index) => { + const previous = sorted[index]; + return ((totals.get(domainBits) as number) - (totals.get(previous) as number)) / (domainBits - previous); + }); + expect(new Set(slopes).size, `the key grows by one constant per level: ${JSON.stringify(slopes)}`).toBe(1); + const perLevel = slopes[0]; + expect(Number.isInteger(perLevel) && perLevel > 1, `per-level cost is a whole number of bytes: ${perLevel}`).toBe( + true + ); + const lambdaBytes = perLevel - 1; // one correction word is a seed plus a control byte + const expectedRoot = 1 + lambdaBytes; // one header byte plus the root seed + + for (const domainBits of sorted) { + const domainSize = 2 ** domainBits; + const hex = await keyDump(page, domainBits); + const total = hex.length / 2; + expect(total, `the dump for 2^${domainBits} is the size it was measured at`).toBe(totals.get(domainBits)); + + const header = Number.parseInt(hex.slice(0, 2), 16); + expect(header & 0x1f, "the dump's header names the domain under test").toBe(domainBits); + + const finalBytes = await expectClaim(page, 'key-part-final', { value: 1, text: 'B' }); + const controls = readCorrectionWords(hex, expectedRoot, perLevel, finalBytes); + expect( + controls.filter((control) => control > 3), + `every stride-aligned correction word ends in two control bits at 2^${domainBits}` + ).toEqual([]); + expect(controls.length, `the key carries one correction word per level at 2^${domainBits}`).toBe(domainBits); + const summedLevels = controls.reduce((sum) => sum + perLevel, 0); + const lastByte = Number.parseInt(hex.slice(-2 * finalBytes), 16); + expect(lastByte, 'the final correction is a bit').toBeLessThanOrEqual(1); + + await expectClaim(page, 'key-part-root', { value: expectedRoot, text: 'B' }); + await expectClaim(page, 'key-part-levels', { value: summedLevels, text: 'B' }); + await expectClaim(page, 'key-bytes-total', { value: total, text: 'B' }); + await expectClaim(page, 'dpf-key-bytes', { value: total, text: 'bytes' }); + expect(expectedRoot + summedLevels + finalBytes, `the parts account for every byte at 2^${domainBits}`).toBe( + total + ); + + const chor = await expectClaim(page, 'chor-query-bytes', { + value: domainSize / 8, + text: `${domainSize.toLocaleString()} bits` + }); + expect(chor * 8, 'the Chor baseline is one bit per record').toBe(domainSize); + + const formulaText = (await page.locator('[data-claim="key-size-formula"]').textContent()) ?? ''; + const formulaNumbers = (formulaText.match(/[\d,]+/g) ?? []).map((value) => Number(value.replaceAll(',', ''))); + expect(formulaNumbers, `the formula states this domain's own measured parts: ${formulaText}`).toEqual([ + expectedRoot, + perLevel, + domainSize, + finalBytes, + total + ]); + await expectClaim(page, 'key-size-formula', { value: total }); + } + + // The PRG width is measured from a real expansion, and λ is the one the key + // sizes just revealed. + await expectClaim(page, 'prg-node-width', { value: 2 * (lambdaBytes * 8 + 1), text: 'bits per node' }); }); test('changing alpha retires stale output while a no-op does not', async ({ page }) => { @@ -106,7 +229,7 @@ test('changing alpha retires stale output while a no-op does not', async ({ page await slider.fill('4'); await expect(page.locator('#retirement')).toContainText('Previous reconstruction for α = 11 retired'); await expect(page.locator('#tree-alpha-verdict')).toHaveText('4'); - await expect(page.locator('[data-verdict="tree-point"]')).toHaveAttribute('data-status', 'pass'); + await expectVerdict(page, 'tree-point', { status: 'pass', tone: 'pass', text: ['ONE LIT POINT AT', '4'] }); const paintedHidden = await page.locator('[hidden]').evaluateAll((nodes) => nodes.filter((node) => getComputedStyle(node).display !== 'none').length); expect(paintedHidden).toBe(0); }); @@ -122,14 +245,15 @@ test('tampering violates integrity while every privacy verdict remains green', a const retrieved = (await page.locator('#retrieved-record').textContent()) ?? ''; const expected = (await page.locator('#expected-record').textContent()) ?? ''; expect(retrieved).not.toBe(expected); - await expect(page.locator('[data-verdict="record-match"]')).toHaveAttribute('data-status', 'alarm'); - await expect(page.locator('[data-verdict="record-match"]')).toHaveText(/RETRIEVED — AND WRONG/); const expectedBytes = expected.match(/../g) ?? []; const actual = (retrieved.match(/../g) ?? []).filter((byte, index) => byte !== expectedBytes[index]).length; - await expect(page.locator('[data-verdict="record-match"]')).toContainText( - `${actual} of ${expectedBytes.length} bytes differ` - ); + // A corrupted record must not be able to keep the green plate. + await expectVerdict(page, 'record-match', { + status: 'alarm', + tone: 'alarm', + text: ['RETRIEVED — AND WRONG', `${actual} of ${expectedBytes.length} bytes differ`] + }); }); test('the negative claim reports the run that exercised it', async ({ page }) => { @@ -141,9 +265,11 @@ test('the negative claim reports the run that exercised it', async ({ page }) => // Evidence from this run, not standing prose: the status has to be the one a // silently-accepted corruption produces, and the byte count has to be real. - const claim = page.locator('[data-verdict="no-authentication"]'); - await expect(claim).toHaveAttribute('data-status', 'alarm'); - await expect(claim).toContainText('does not authenticate what is returned'); + await expectVerdict(page, 'no-authentication', { + status: 'alarm', + tone: 'alarm', + text: 'does not authenticate what is returned' + }); const evidence = (await page.locator('#integrity-evidence').textContent()) ?? ''; const claimed = Number(evidence.match(/and (\d+) of (\d+) bytes now differ/)?.[1] ?? NaN); const retrieved = (await page.locator('#retrieved-record').textContent()) ?? ''; @@ -158,9 +284,17 @@ test('the negative claim reports the run that exercised it', async ({ page }) => test('the negative claim says it was not exercised when nothing was tampered with', async ({ page }) => { test.setTimeout(120_000); await page.goto('.'); - await expect(page.locator('[data-verdict="no-authentication"]')).toHaveAttribute('data-status', 'unexercised'); + await expectVerdict(page, 'no-authentication', { + status: 'unexercised', + tone: 'alarm', + text: 'does not authenticate what is returned' + }); await page.getByRole('button', { name: 'Fetch privately' }).click(); await expect(page.locator('#pir-result')).toBeVisible(); - await expect(page.locator('[data-verdict="no-authentication"]')).toHaveAttribute('data-status', 'unexercised'); + await expectVerdict(page, 'no-authentication', { + status: 'unexercised', + tone: 'alarm', + text: 'does not authenticate what is returned' + }); await expect(page.locator('#integrity-evidence')).toContainText('no answer was altered'); }); diff --git a/e2e/flows.spec.ts b/e2e/flows.spec.ts index 8bdbc0b..4482ec0 100644 --- a/e2e/flows.spec.ts +++ b/e2e/flows.spec.ts @@ -1,4 +1,5 @@ import { expect, test } from '@playwright/test'; +import { expectVerdict } from './verdict-assertions'; test('tree stepper exposes each real expansion level', async ({ page }) => { await page.goto('.'); @@ -22,9 +23,11 @@ test('moving alpha regenerates both keys and moves only the combined point', asy test('collusion deliberately exposes alpha and switching it off removes the alarm', async ({ page }) => { await page.goto('.'); await page.locator('#collusion-toggle').check(); - const joined = page.locator('[data-verdict="collusion-recovery"]'); - await expect(joined).toHaveAttribute('data-status', 'alarm'); - await expect(joined).toContainText('SERVER SEES α = 11'); + await expectVerdict(page, 'collusion-recovery', { + status: 'alarm', + tone: 'alarm', + text: 'SERVER SEES α = 11' + }); // The exposed index is the one the joined view reconstructs, not the slider's. const lit = await page.locator('.collusion-bits .tree-node').evaluateAll((nodes) => nodes.findIndex((node) => node.classList.contains('node-lit')) @@ -34,8 +37,13 @@ test('collusion deliberately exposes alpha and switching it off removes the alar await page.locator('#collusion-toggle').uncheck(); await expect(page.locator('.collusion-alarm')).toHaveCount(0); const alone = page.locator('[data-verdict="single-share"]'); - await expect(alone).toHaveAttribute('data-status', 'pass'); - // "One key hides α" is rendered as a measured lit count over the real share. + // "One key hides α" is rendered as a measured lit count over the real share, + // and the words, the status and the green plate are asserted as one claim. + await expectVerdict(page, 'single-share', { + status: 'pass', + tone: 'pass', + text: [/ONE KEY ONLY · k0 alone lights \d+ of \d+ leaves, naming no point/] + }); const text = (await alone.textContent()) ?? ''; const counted = text.match(/lights (\d+) of (\d+) leaves/); expect(counted, `the single-share verdict reports a measured count: ${text}`).not.toBeNull(); diff --git a/e2e/verdict-assertions.ts b/e2e/verdict-assertions.ts new file mode 100644 index 0000000..35a823a --- /dev/null +++ b/e2e/verdict-assertions.ts @@ -0,0 +1,90 @@ +import { expect, type Page } from '@playwright/test'; + +/** + * A marker's words and its state are ONE claim, so one call asserts all of it. + * + * Asserting text alone records a kill for a mutation that flips the sentence + * while the plate stays green — the marker goes on claiming pass in every way a + * reader can see except the words. This page sets `className` and + * `dataset.status` in separate expressions, so those two really can disagree: + * pinning `className` to `verdict-pass` leaves a green plate reading + * "RETRIEVED — AND WRONG", and it survived the whole suite before this helper + * existed. + * + * `tone` is checked as the painted background rather than as a class name, + * because that is what a reader sees and because the two verdict families style + * themselves differently: `.verdict` switches class, `.mini-verdict` switches on + * `[data-status]`. The plates are the same two colours the stray scan looks for. + */ + +export type VerdictTone = 'pass' | 'alarm'; + +const PLATE: Record = { + pass: 'rgb(15, 53, 43)', + alarm: 'rgb(66, 29, 36)' +}; + +export interface VerdictExpectation { + /** Text the marker must contain — every fragment, all of them required. */ + text: string | RegExp | ReadonlyArray; + /** The machine-readable outcome in `data-status`. */ + status: string; + /** The plate a reader sees. */ + tone: VerdictTone; +} + +export async function expectVerdict(page: Page, id: string, expected: VerdictExpectation): Promise { + const marker = page.locator(`[data-verdict="${id}"]`); + await expect(marker, `${id} renders exactly once`).toHaveCount(1); + const fragments = Array.isArray(expected.text) ? expected.text : [expected.text]; + for (const fragment of fragments as ReadonlyArray) { + await expect(marker, `${id} text`).toContainText(fragment as string & RegExp); + } + await expect(marker, `${id} data-status`).toHaveAttribute('data-status', expected.status); + const painted = await marker.evaluate((node) => getComputedStyle(node).backgroundColor); + expect(painted, `${id} is painted as ${expected.tone} while its text says "${fragments.join(' / ')}"`).toBe( + PLATE[expected.tone] + ); +} + +export interface ClaimExpectation { + /** The measured value behind the words, in `data-value`. */ + value: number; + /** Text fragments the marker must also contain. */ + text?: string | RegExp | ReadonlyArray; +} + +/** + * The measurement equivalent: the rendered sentence and the machine-readable + * value are one claim too, so the value is asserted and the text is required to + * actually state it. A marker whose `data-value` follows the measurement while + * its words say something else is the same defect one family over. + */ +export async function expectClaim(page: Page, id: string, expected: ClaimExpectation): Promise { + const marker = page.locator(`[data-claim="${id}"]`); + await expect(marker, `${id} renders exactly once`).toHaveCount(1); + const raw = await marker.getAttribute('data-value'); + expect(raw, `${id} carries a data-value`).not.toBeNull(); + const value = Number(raw); + expect(Number.isFinite(value), `${id} data-value is numeric, got ${raw}`).toBe(true); + expect(value, `${id} measured value`).toBe(expected.value); + const text = ((await marker.textContent()) ?? '').replace(/\s+/g, ' '); + const plain = String(value); + const grouped = value.toLocaleString('en-US'); + expect( + text.includes(plain) || text.includes(grouped), + `${id} states its own value: data-value ${plain} is not in "${text}"` + ).toBe(true); + const fragments = expected.text === undefined ? [] : Array.isArray(expected.text) ? expected.text : [expected.text]; + for (const fragment of fragments as ReadonlyArray) { + await expect(marker, `${id} text`).toContainText(fragment as string & RegExp); + } + return value; +} + +/** The measured value behind a claim, for oracles that derive from it. */ +export async function claimValue(page: Page, id: string): Promise { + const raw = await page.locator(`[data-claim="${id}"]`).getAttribute('data-value'); + expect(raw, `${id} carries a data-value`).not.toBeNull(); + return Number(raw); +} diff --git a/e2e/verdict-coverage.spec.ts b/e2e/verdict-coverage.spec.ts index 277efb9..f51501d 100644 --- a/e2e/verdict-coverage.spec.ts +++ b/e2e/verdict-coverage.spec.ts @@ -1,12 +1,28 @@ import { readFileSync } from 'node:fs'; import { fileURLToPath } from 'node:url'; import { expect, test, type Page } from '@playwright/test'; -import { findStrayVerdicts, markerStatuses, renderedMarkers } from './verdict-scan'; +import { + claimValues, + findStrayMeasurements, + findStrayVerdicts, + markerStatuses, + renderedClaims, + renderedMarkers +} from './verdict-scan'; + +interface MutationRecord { + renders: string; + computedFrom: string; + mutation: string; + expectedFlip: string; + assertedBy: string; +} const REGISTRY = JSON.parse( readFileSync(fileURLToPath(new URL('./verdict-mutations.json', import.meta.url)), 'utf8') ) as { - markers: Record; + markers: Record; + claims: Record; }; const SPEC_SOURCES = new Map(); @@ -18,88 +34,210 @@ function specSource(relative: string): string { return text; } +interface Walk { + markers: Set; + claims: Set; + statuses: Record; + values: Record; +} + /** - * Drive the page through every state that can render a verdict, accumulating - * what the DOM shows at each one. Coverage is taken off the rendered page, not - * off anyone's enumeration of exhibits, and a verdict that only appears in one - * broken-mode state still counts. + * The DENOMINATOR, not a test. Both coverage rules below and the two stray scans + * enumerate over whatever this walk reaches, so anything renderable only at a + * control setting it never visits is outside the set they judge. + * + * The rule it follows: visit every option of every control that changes what + * renders, each control on its own, never the cross-product. For this lab that + * is 16 secret indices, 5 expansion levels, a key regeneration, 4 domain sizes, + * both collusion states, both tamper states, and both classes of shelf index — + * the in-range fetch and the rejection. `#shelf-alpha` is a 65,536-value number + * input rather than a fixed option set, so its two rendering classes stand in + * for its options; every other control here is enumerable and is enumerated. + * The key inspector's disclosure is opened because the scans skip what has no + * client rects, so leaving it closed would hide its contents from them. */ -async function driveEveryVerdict( - page: Page, - observe: (page: Page) => Promise = async () => {} -): Promise<{ markers: Set; statuses: Record }> { - const markers = new Set(); - const statuses: Record = {}; +async function driveEveryState(page: Page, observe: (page: Page) => Promise = async () => {}): Promise { + const walk: Walk = { markers: new Set(), claims: new Set(), statuses: {}, values: {} }; const sample = async (): Promise => { - for (const marker of await renderedMarkers(page)) markers.add(marker); - Object.assign(statuses, await markerStatuses(page)); + for (const marker of await renderedMarkers(page)) walk.markers.add(marker); + for (const claim of await renderedClaims(page)) walk.claims.add(claim); + Object.assign(walk.statuses, await markerStatuses(page)); + Object.assign(walk.values, await claimValues(page)); await observe(page); }; await page.goto('.'); await sample(); + // The serialized key dump only has client rects once the disclosure is open. + await page.locator('.key-inspector summary').click(); + await sample(); + + // #alpha-range — every secret index it offers. + const range = page.locator('#alpha-range'); + const lowest = Number(await range.getAttribute('min')); + const highest = Number(await range.getAttribute('max')); + for (let alpha = lowest; alpha <= highest; alpha += 1) { + await range.fill(String(alpha)); + await expect(page.locator('#tree-alpha-verdict')).toHaveText(String(alpha)); + await sample(); + } + + // The expansion stepper — every level, down and back up. + const levels = Number((await page.locator('#step-value').textContent())?.split('/')[1]?.trim() ?? '4'); + for (let level = levels - 1; level >= 0; level -= 1) { + await page.getByRole('button', { name: 'Previous' }).click(); + await expect(page.locator('#step-value')).toHaveText(`${level} / ${levels}`); + await sample(); + } + for (let level = 1; level <= levels; level += 1) { + await page.getByRole('button', { name: 'Next level' }).click(); + await expect(page.locator('#step-value')).toHaveText(`${level} / ${levels}`); + await sample(); + } + + // #new-keys — one option, exercised once. + await page.locator('#new-keys').click(); + await sample(); + + // #size-domain — every domain the meter offers. + const domains = await page + .locator('#size-domain option') + .evaluateAll((nodes) => nodes.map((node) => (node as HTMLOptionElement).value)); + for (const domain of domains) { + await page.locator('#size-domain').selectOption(domain); + await expect(page.locator('[data-claim="key-size-formula"]')).toContainText( + (2 ** Number(domain)).toLocaleString() + ); + await sample(); + } + + // #collusion-toggle — both states, which repaint the trust-boundary view. await page.locator('#collusion-toggle').check(); await expect(page.locator('[data-verdict="collusion-recovery"]')).toBeVisible(); await sample(); - await page.locator('#collusion-toggle').uncheck(); await expect(page.locator('[data-verdict="single-share"]')).toBeVisible(); await sample(); - await page.locator('#tamper-toggle').check(); + // #shelf-alpha — the rejected class first, which renders no result at all. + await page.locator('#shelf-alpha').fill('65536'); + await page.getByRole('button', { name: 'Fetch privately' }).click(); + await expect(page.locator('#shelf-alpha')).toHaveAttribute('aria-invalid', 'true'); + await expect(page.locator('#pir-result')).toBeHidden(); + await sample(); + + // ...and an in-range index, which is also #fetch-record's honest state. + await page.locator('#shelf-alpha').fill('7'); await page.getByRole('button', { name: 'Fetch privately' }).click(); await expect(page.locator('#pir-result')).toBeVisible(); - await expect(page.locator('[data-verdict="record-match"]')).toHaveAttribute('data-status', /pass|alarm/); + await expect(page.locator('[data-verdict="record-match"]')).toHaveAttribute('data-status', 'pass'); + await sample(); + + // #tamper-toggle — on, with its own fetch. + await page.locator('#tamper-toggle').check(); + await page.getByRole('button', { name: 'Fetch privately' }).click(); + await expect(page.locator('[data-verdict="record-match"]')).toHaveAttribute('data-status', 'alarm'); + await expect(page.locator('[data-verdict="no-authentication"]')).toHaveAttribute('data-status', 'alarm'); + await sample(); + await page.locator('#tamper-toggle').uncheck(); + + // #collusion-toggle again, this time through a fetch, for the tile it paints. + await page.locator('#collusion-toggle').check(); + await page.getByRole('button', { name: 'Fetch privately' }).click(); + await expect(page.locator('[data-verdict="collusion-state"]')).toHaveAttribute('data-status', 'alarm'); + await sample(); + await page.locator('#collusion-toggle').uncheck(); await sample(); - return { markers, statuses }; + return walk; } -test('every verdict the page renders has a mutation covering it', async ({ page }) => { - test.setTimeout(180_000); - const seen = await driveEveryVerdict(page); - const rendered = [...seen.markers].sort(); - const registered = Object.keys(REGISTRY.markers).sort(); +test('every verdict and every measurement the page renders has a mutation covering it', async ({ page }) => { + test.setTimeout(300_000); + const seen = await driveEveryState(page); - const uncovered = rendered.filter((marker) => !registered.includes(marker)); + const renderedVerdicts = [...seen.markers].sort(); + const registeredVerdicts = Object.keys(REGISTRY.markers).sort(); + expect( + renderedVerdicts.filter((marker) => !registeredVerdicts.includes(marker)), + 'rendered verdict markers with no §4.1c mutation in e2e/verdict-mutations.json' + ).toEqual([]); expect( - uncovered, - `rendered verdict markers with no §4.1c mutation in e2e/verdict-mutations.json: ${JSON.stringify(uncovered)}` + registeredVerdicts.filter((marker) => !renderedVerdicts.includes(marker)), + 'mutations registered for verdicts this page never renders' ).toEqual([]); - const unrendered = registered.filter((marker) => !rendered.includes(marker)); + // Measurement markers are in the same loop on the same terms: a rendered + // number with no mutation record fails, and a record naming a claim the page + // no longer renders fails too. + const renderedClaimIds = [...seen.claims].sort(); + const registeredClaims = Object.keys(REGISTRY.claims).sort(); expect( - unrendered, - `mutations registered for verdicts this page never renders: ${JSON.stringify(unrendered)}` + renderedClaimIds.filter((claim) => !registeredClaims.includes(claim)), + 'rendered data-claim measurements with no §4.1c mutation in e2e/verdict-mutations.json' + ).toEqual([]); + expect( + registeredClaims.filter((claim) => !renderedClaimIds.includes(claim)), + 'mutations registered for measurements this page never renders' ).toEqual([]); - // A marker on an empty element would satisfy the list above while showing - // nothing, so require each one to carry a rendered outcome. - expect(Object.keys(seen.statuses).sort(), 'every marker renders an outcome').toEqual(rendered); + // A marker on an empty element would satisfy the lists above while showing + // nothing, so require each one to carry a rendered outcome or value. + expect(Object.keys(seen.statuses).sort(), 'every verdict marker renders an outcome').toEqual(renderedVerdicts); for (const [marker, status] of Object.entries(seen.statuses)) { expect(status, `${marker} carries a data-status`).not.toBe(''); } + expect(Object.keys(seen.values).sort(), 'every claim marker renders words').toEqual(renderedClaimIds); + for (const [claim, value] of Object.entries(seen.values)) { + expect( + value !== '' && Number.isFinite(Number(value)), + `${claim} carries a numeric data-value, got "${value}"` + ).toBe(true); + } }); -test('each registered mutation names a real assertion on its own marker', async () => { - for (const [marker, entry] of Object.entries(REGISTRY.markers)) { - expect(entry.mutation.length, `${marker} records a mutation`).toBeGreaterThan(20); - expect(entry.expectedFlip.length, `${marker} records the flip it expects`).toBeGreaterThan(20); - const source = specSource(entry.assertedBy); - expect(source, `${entry.assertedBy} asserts on ${marker}`).toContain(marker); +test('each registered mutation is killed through the shared marker helper', async () => { + const families: ReadonlyArray<[string, Record, string]> = [ + ['verdict', REGISTRY.markers, 'expectVerdict'], + ['claim', REGISTRY.claims, 'expectClaim'] + ]; + for (const [family, records, helper] of families) { + for (const [marker, entry] of Object.entries(records)) { + expect(entry.mutation.length, `${marker} records a mutation`).toBeGreaterThan(20); + expect(entry.expectedFlip.length, `${marker} records the flip it expects`).toBeGreaterThan(20); + const source = specSource(entry.assertedBy); + // Mentioning the id is not asserting on it, and asserting its text is not + // asserting its state. The helper asserts words, machine-readable value + // and painted state together, so a kill that only flipped the words is a + // build failure rather than a recorded kill. + const call = new RegExp(`${helper}\\(\\s*page,\\s*['"\`]${marker}['"\`]`); + expect( + call.test(source), + `${entry.assertedBy} must assert the ${family} ${marker} through ${helper}(page, '${marker}', …)` + ).toBe(true); + } } }); test('no verdict word or verdict styling is rendered outside a marker', async ({ page }) => { - test.setTimeout(180_000); + test.setTimeout(300_000); const strays: unknown[] = []; - await driveEveryVerdict(page, async (live) => { + await driveEveryState(page, async (live) => { strays.push(...(await findStrayVerdicts(live))); }); expect(strays, `unmarked verdicts on the page: ${JSON.stringify(strays, null, 2)}`).toEqual([]); }); +test('no measurement is rendered outside a marker', async ({ page }) => { + test.setTimeout(300_000); + const strays: unknown[] = []; + await driveEveryState(page, async (live) => { + strays.push(...(await findStrayMeasurements(live))); + }); + expect(strays, `unmarked measurements on the page: ${JSON.stringify(strays, null, 2)}`).toEqual([]); +}); + test('the stray scan catches a raw unmarked banner, styled or worded', async ({ page }) => { await page.goto('.'); expect(await findStrayVerdicts(page), 'the page starts clean').toEqual([]); @@ -125,7 +263,35 @@ test('the stray scan catches a raw unmarked banner, styled or worded', async ({ expect(worded.map((stray) => stray.id), 'a bare worded banner is caught').toContain('injected-worded-banner'); }); -test('the stray scan clears the same banner once it is marked and covered', async ({ page }) => { +test('the measurement scan catches an unmarked number, with a unit or bare in a stat cell', async ({ page }) => { + await page.goto('.'); + expect(await findStrayMeasurements(page), 'the page starts clean').toEqual([]); + + await page.evaluate(() => { + const line = document.createElement('div'); + line.id = 'injected-measurement'; + line.textContent = '1,632 B'; + document.querySelector('.meter-layout')?.append(line); + }); + expect( + (await findStrayMeasurements(page)).map((stray) => stray.id), + 'a number with a unit is caught' + ).toContain('injected-measurement'); + + await page.evaluate(() => { + document.getElementById('injected-measurement')?.remove(); + const cell = document.createElement('dd'); + cell.id = 'injected-stat'; + cell.textContent = '4,096'; + document.querySelector('.parts-list div')?.append(cell); + }); + expect( + (await findStrayMeasurements(page)).map((stray) => stray.id), + 'a bare integer in a stats cell is caught' + ).toContain('injected-stat'); +}); + +test('both stray scans clear the same injections once they are marked and covered', async ({ page }) => { await page.goto('.'); await page.evaluate(() => { const banner = document.createElement('div'); @@ -135,8 +301,16 @@ test('the stray scan clears the same banner once it is marked and covered', asyn banner.setAttribute('data-status', 'pass'); banner.textContent = 'PROOF ACCEPTED'; document.querySelector('main')?.append(banner); + const line = document.createElement('div'); + line.id = 'injected-marked-measurement'; + line.setAttribute('data-claim', 'injected-size'); + line.setAttribute('data-value', '1632'); + line.textContent = '1,632 B'; + document.querySelector('.meter-layout')?.append(line); }); - expect(await findStrayVerdicts(page), 'the scan is not unconditionally red').toEqual([]); - // ...and the coverage walk still sees it, so marking alone does not hide it. + expect(await findStrayVerdicts(page), 'the verdict scan is not unconditionally red').toEqual([]); + expect(await findStrayMeasurements(page), 'the measurement scan is not unconditionally red').toEqual([]); + // ...and the coverage walk still sees both, so marking alone does not hide them. expect(await renderedMarkers(page)).toContain('injected'); + expect(await renderedClaims(page)).toContain('injected-size'); }); diff --git a/e2e/verdict-mutations.json b/e2e/verdict-mutations.json index 8b662ae..87069e2 100644 --- a/e2e/verdict-mutations.json +++ b/e2e/verdict-mutations.json @@ -1,5 +1,5 @@ { - "note": "One §4.1c mutation per rendered verdict marker. A kill counts only when the unmutated baseline passed in the same run and the failure is that verdict's own assertion. Recorded failures are in the commit body.", + "note": "One §4.1c mutation per rendered marker, in both families: `markers` are the data-verdict outcomes, `claims` are the data-claim measurements. A kill counts only when the unmutated baseline passed in the same run and the failure is that marker's own assertion, and every record's assertion must go through expectVerdict/expectClaim so words, value and painted state are one claim. Recorded failures are in the commit body.", "markers": { "tree-point": { "renders": "#tree-verdict — ONE LIT POINT AT α = n", @@ -12,7 +12,7 @@ "renders": "#collusion-view verdict with the collusion toggle off — ONE KEY ONLY · k0 alone lights n of 16 leaves", "computedFrom": "singleShareView(k0).lit / .namesAPoint", "mutation": "src/attack/single-view.ts — namesAPoint: lit === 1 becomes namesAPoint: lit >= 1", - "expectedFlip": "a share lighting ~8 leaves is reported as naming a point, so data-status becomes alarm", + "expectedFlip": "a share lighting ~8 leaves is reported as naming a point, so data-status becomes alarm and the plate turns red", "assertedBy": "e2e/flows.spec.ts" }, "collusion-recovery": { @@ -46,8 +46,8 @@ "record-match": { "renders": "#record-verdict — RETRIEVED · byte-for-byte match with shelf[α] / RETRIEVED — AND WRONG", "computedFrom": "sameBytes(reconstruct(answer0, answer1), shelf[α]) and the differing byte count", - "mutation": "src/pir/client.ts — reconstruct returns answer0.map((byte, index) => byte ^ (answer1[index] & 0)), dropping server 1 from the XOR", - "expectedFlip": "the reconstruction no longer equals shelf[alpha], so data-status becomes alarm on the honest run", + "mutation": "src/ui/app.ts — verdict.className = `verdict ${matches ? 'verdict-pass' : 'verdict-alarm'}` becomes verdict.className = 'verdict verdict-pass', pinning the plate green while data-status and the words still follow the run", + "expectedFlip": "a corrupted record is announced as RETRIEVED — AND WRONG on a green pass plate; expectVerdict's tone assertion fails where the old text-and-status assertions both passed", "assertedBy": "e2e/claims.spec.ts" }, "no-authentication": { @@ -57,5 +57,63 @@ "expectedFlip": "the tampered run matches shelf[α], so data-status becomes contradicted rather than alarm", "assertedBy": "e2e/claims.spec.ts" } + }, + "claims": { + "dpf-key-bytes": { + "renders": "#key-byte-label in the meter — 290 bytes beside the DPF bar", + "computedFrom": "serializeKey(generateDpf(0, domainBits)[0]).length for the domain under test", + "mutation": "src/ui/app.ts — renderClaim('key-byte-label', serialized.length, `${serialized.length.toLocaleString()} bytes`) becomes renderClaim('key-byte-label', parts.correctionWords, `${parts.correctionWords.toLocaleString()} bytes`)", + "expectedFlip": "the meter states the correction-word region as the whole key, so the claim no longer equals the measured hex dump", + "assertedBy": "e2e/claims.spec.ts" + }, + "chor-query-bytes": { + "renders": "#chor-byte-label in the meter — 8,192 bytes · 65,536 bits", + "computedFrom": "domainSize / 8, the one-bit-per-record Chor baseline", + "mutation": "src/ui/app.ts — const chorBytes = domainSize / 8 becomes const chorBytes = domainSize / 4", + "expectedFlip": "the baseline claims two bits per record, so chor * 8 no longer equals the domain size it names", + "assertedBy": "e2e/claims.spec.ts" + }, + "key-part-root": { + "renders": "#part-root in the parts list — 17 B of root material", + "computedFrom": "serializedKeyParts(domainBits).rootMaterial, checked against the header byte plus the root seed in the dump", + "mutation": "src/ui/app.ts — renderClaim('part-root', parts.rootMaterial, `${parts.rootMaterial} B`) becomes renderClaim('part-root', parts.rootMaterial - 1, `${parts.rootMaterial - 1} B`). The deeper serializer mutation (ROOT_MATERIAL_BYTES = SECURITY_BYTES) was run first and rejected as evidence: it stops the key round-tripping and takes the whole suite red, which is not a readable kill on this marker.", + "expectedFlip": "root material is claimed as 16 B against the 17 B that λ, measured from the per-level slope, requires, and the parts stop accounting for the measured total", + "assertedBy": "e2e/claims.spec.ts" + }, + "key-part-levels": { + "renders": "#part-levels in the parts list — 272 B of per-level corrections", + "computedFrom": "serializedKeyParts(domainBits).correctionWords, checked against the words actually present in the dump, summed", + "mutation": "src/ui/app.ts — renderClaim('part-levels', parts.correctionWords, `${parts.correctionWords} B`) becomes renderClaim('part-levels', parts.correctionWords + perLevel, `${parts.correctionWords + perLevel} B`)", + "expectedFlip": "the parts list claims one correction word more than the key carries, so the summed per-level cost no longer matches and the parts stop accounting for the total", + "assertedBy": "e2e/claims.spec.ts" + }, + "key-part-final": { + "renders": "#part-final in the parts list — 1 B of final correction", + "computedFrom": "serializedKeyParts(domainBits).finalCorrection, the single byte holding the output correction bit", + "mutation": "src/ui/app.ts — renderClaim('part-final', parts.finalCorrection, `${parts.finalCorrection} B`) becomes renderClaim('part-final', 2, '2 B')", + "expectedFlip": "the tail is claimed as two bytes, so the stride walk loses a correction word and the parts overshoot the measured total", + "assertedBy": "e2e/claims.spec.ts" + }, + "key-bytes-total": { + "renders": "#key-bytes in the parts list — the measured total, 290 B", + "computedFrom": "serializeKey(...).length, checked against the length of the rendered hex dump", + "mutation": "src/ui/app.ts — renderClaim('key-bytes', serialized.length, `${serialized.length} B`) becomes renderClaim('key-bytes', serialized.length - 1, `${serialized.length - 1} B`)", + "expectedFlip": "the measured total disagrees by one byte with the hex dump it is measured from", + "assertedBy": "e2e/claims.spec.ts" + }, + "key-size-formula": { + "renders": "#size-formula — 17 + (17 × log₂ 65,536) + 1 = 290 bytes", + "computedFrom": "the same measured parts, restated as the size law for the domain under test", + "mutation": "src/ui/app.ts — the formula's `= ${parts.total} bytes` becomes the literal `= 290 bytes`", + "expectedFlip": "the formula is correct only at the default 65,536-record domain and states 290 bytes at every other one; invisible until the walk visits all four domain options", + "assertedBy": "e2e/claims.spec.ts" + }, + "prg-node-width": { + "renders": "#prg-width in the trace strip — AES-128-CTR → 2(λ + 1) = 258 bits per node", + "computedFrom": "a real expandSeed() of the all-zero seed: both child seeds plus the two control bits", + "mutation": "src/ui/app.ts — const nodeWidthBits = (zeroExpansion.leftSeed.length + zeroExpansion.rightSeed.length) * 8 + 2 becomes the same expression without + 2", + "expectedFlip": "the node width drops the two control bits and claims 256, against 2(λ + 1) derived from the per-level slope the key sizes measured", + "assertedBy": "e2e/claims.spec.ts" + } } } diff --git a/e2e/verdict-scan.ts b/e2e/verdict-scan.ts index 14c15a4..393c6c0 100644 --- a/e2e/verdict-scan.ts +++ b/e2e/verdict-scan.ts @@ -34,6 +34,81 @@ export async function markerStatuses(page: Page): Promise }); } +/** Every `data-claim` measurement marker the page can render. */ +export async function renderedClaims(page: Page): Promise { + return page.evaluate(() => + Array.from(document.querySelectorAll('[data-claim]'), (node) => node.getAttribute('data-claim') ?? '') + ); +} + +/** Claim markers that carry both a rendered value and words, so neither is blank. */ +export async function claimValues(page: Page): Promise> { + return page.evaluate(() => { + const seen: Record = {}; + for (const node of Array.from(document.querySelectorAll('[data-claim]'))) { + const id = node.getAttribute('data-claim') ?? ''; + const value = node.getAttribute('data-value') ?? ''; + if ((node.textContent ?? '').trim().length > 0) seen[id] = value; + } + return seen; + }); +} + +/** + * A rendered number outside a marker is exactly as unchecked as a rendered + * verdict outside one, and it is the easier mistake to make because a number + * does not look like a claim. The verdict-word scan cannot see `1,632 B` or a + * bare `290` in a stats cell, so this walks the result regions for them. + */ +export async function findStrayMeasurements(page: Page): Promise { + return page.evaluate(() => { + // Where this page paints the output of a run. Controls, status prose and + // the awaiting placeholder are not result regions and are not scanned. + const RESULT_REGIONS = '#tree-verdict, .trace-strip, .key-inspector, #pir-result, #collusion-view, .meter-layout, #size-formula'; + const MEASUREMENT = /\d[\d,.]*\s*(?:B|KB|MB|bits?|bytes?|ops?|operations?|ms|s|×|x)\b/i; + const BARE_STAT = /^\d[\d,]*$/; + const STAT_CELL = 'dd,.stat-value,.parts-total'; + // Labels and explanatory prose may carry numbers; a result value may not. + const PROSE = 'p,li,summary,small,h1,h2,h3,h4,label,option,figcaption,blockquote,dt'; + const HEX_DUMP = /^[0-9a-f\s…]{8,}$/i; + + const strays: StrayVerdict[] = []; + const seen = new Set(); + for (const region of Array.from(document.querySelectorAll(RESULT_REGIONS))) { + const nodes = [region, ...Array.from(region.querySelectorAll('*'))]; + for (const node of nodes) { + if (seen.has(node)) continue; + seen.add(node); + if (node.closest('[data-verdict]') !== null || node.closest('[data-claim]') !== null) continue; + if ((node as HTMLElement).getClientRects().length === 0) continue; + + const own = Array.from(node.childNodes) + .filter((child) => child.nodeType === Node.TEXT_NODE) + .map((child) => child.textContent ?? '') + .join(' ') + .replace(/\s+/g, ' ') + .trim(); + if (own.length === 0 || own.length > 160) continue; + // Raw key and record bytes are material, not a stated measurement; the + // claims suite checks them against the size markers byte for byte. + if (HEX_DUMP.test(own)) continue; + const isProse = node.closest(PROSE) !== null; + const bareStat = BARE_STAT.test(own) && node.matches(STAT_CELL); + if (!bareStat && (isProse || !MEASUREMENT.test(own))) continue; + strays.push({ + tag: node.tagName.toLowerCase(), + id: (node as HTMLElement).id, + reason: bareStat + ? `bare measurement in a stats cell outside a data-claim marker: "${own}"` + : `measurement rendered outside a data-claim marker: "${own}"`, + text: own.slice(0, 120) + }); + } + } + return strays; + }); +} + export async function findStrayVerdicts(page: Page): Promise { return page.evaluate(() => { const VERDICT_CLASS = /(^|\s)(verdict|verdict-pass|verdict-alarm|mini-verdict|negative-claim)(\s|$)/; diff --git a/index.html b/index.html index c1b9482..e16877e 100644 --- a/index.html +++ b/index.html @@ -114,7 +114,7 @@

Share the instruction, not the answer

Correction wordfinal output correction
Control bitsβ = 1
-
PrimitiveAES-128-CTR → 2(λ + 1) bits
+
PrimitiveAES-128-CTR → 2(λ + 1) = 258 bits per node
@@ -168,14 +168,14 @@

Share the instruction, not the answer

-
One DPF key290 bytes
-
Chor XOR query8,192 bytes · 65,536 bits
+
One DPF key290 bytes
+
Chor XOR query8,192 bytes · 65,536 bits
-
Root material
17 B
Per-level corrections
272 B
Final correction
1 B
Measured total
290 B
+
Root material
17 B
Per-level corrections
272 B
Final correction
1 B
Measured total
290 B
-

17 + (17 × log₂ 65,536) + 1 = 290 bytes

+

17 + (17 × log₂ 65,536) + 1 = 290 bytes

diff --git a/src/ui/app.ts b/src/ui/app.ts index 88aec1b..75f7b2a 100644 --- a/src/ui/app.ts +++ b/src/ui/app.ts @@ -119,21 +119,35 @@ function regenerateTree(alpha: number, retire: boolean): void { element('collusion-toggle').checked = false; } +/** + * Every number this page paints is a marked claim: `data-claim` names it and + * `data-value` carries the machine-readable measurement behind the words, so a + * rendered sentence and the value it states cannot drift apart unnoticed. + */ +function renderClaim(id: string, value: number, text: string): void { + const node = element(id); + node.dataset.value = String(value); + node.textContent = text; +} + function renderMeter(domainBits: number): void { const domainSize = 2 ** domainBits; const key = generateDpf(0, domainBits)[0]; const serialized = serializeKey(key); const parts = serializedKeyParts(domainBits); const chorBytes = domainSize / 8; - element('key-byte-label').textContent = `${serialized.length.toLocaleString()} bytes`; - element('chor-byte-label').textContent = `${chorBytes.toLocaleString()} bytes · ${domainSize.toLocaleString()} bits`; - element('part-root').textContent = `${parts.rootMaterial} B`; - element('part-levels').textContent = `${parts.correctionWords} B`; - element('part-final').textContent = `${parts.finalCorrection} B`; - const total = element('key-bytes'); - total.textContent = `${serialized.length} B`; - total.dataset.count = String(serialized.length); - element('size-formula').textContent = `17 + (17 × log₂ ${domainSize.toLocaleString()}) + 1 = ${parts.total} bytes`; + const perLevel = parts.correctionWords / domainBits; + renderClaim('key-byte-label', serialized.length, `${serialized.length.toLocaleString()} bytes`); + renderClaim('chor-byte-label', chorBytes, `${chorBytes.toLocaleString()} bytes · ${domainSize.toLocaleString()} bits`); + renderClaim('part-root', parts.rootMaterial, `${parts.rootMaterial} B`); + renderClaim('part-levels', parts.correctionWords, `${parts.correctionWords} B`); + renderClaim('part-final', parts.finalCorrection, `${parts.finalCorrection} B`); + renderClaim('key-bytes', serialized.length, `${serialized.length} B`); + renderClaim( + 'size-formula', + parts.total, + `${parts.rootMaterial} + (${perLevel} × log₂ ${domainSize.toLocaleString()}) + ${parts.finalCorrection} = ${parts.total} bytes` + ); element('meter-key-hex').textContent = toHex(serialized); const baseline = Math.max(chorBytes, serialized.length); element('dpf-bar').style.width = `${Math.max(3, (serialized.length / baseline) * 100)}%`; @@ -293,6 +307,10 @@ export function boot(): void { renderMeter(16); const zeroExpansion = expandSeed(new Uint8Array(16)); element('zero-seed-proof').textContent = `L ${toHex(zeroExpansion.leftSeed).slice(0, 8)} · R ${toHex(zeroExpansion.rightSeed).slice(0, 8)}`; + // The PRG's node width used to be painted as the constant "2(λ + 1) bits". + // It is a measurement of the real expansion, so it is rendered as one. + const nodeWidthBits = (zeroExpansion.leftSeed.length + zeroExpansion.rightSeed.length) * 8 + 2; + renderClaim('prg-width', nodeWidthBits, `AES-128-CTR → 2(λ + 1) = ${nodeWidthBits} bits per node`); element('alpha-range').addEventListener('input', (event) => { regenerateTree(Number((event.currentTarget as HTMLInputElement).value), true); From 8f39f7cde807b325a9442b77394b25ae7528c64b Mon Sep 17 00:00:00 2001 From: Paul Clark Date: Tue, 22 Sep 2026 08:01:32 -0400 Subject: [PATCH 2/2] Record the assertions that ran, and judge coverage on those MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit D6, in this lab. Fix 1 as it shipped at 521d03d did not satisfy Fix 1. The rule was FILE-granular and read with a regex: a record naming e2e/claims.spec.ts was satisfied by the string expectVerdict(page, 'record-match' appearing ANYWHERE in that file — inside a comment, inside a call handed the page's own values, inside a call asserting a different state in a different test. And the shipped tree already contained an instance: tree-point's recorded mutation was killed by two raw assertions that never reached the helper, so the expectedFlip its record describes was a flip no run had ever demonstrated. expectVerdict/expectClaim now append the (spec, test, marker) triple of every call they EXECUTE to a run-scoped ledger under test-results/verdict-ledger/, cleared once per run in e2e/global-setup.ts. e2e/verdict-ledger.spec.ts reads it back and fails when a record's triple is not among them. Playwright runs each spec in its own worker process, so a module-level Set cannot aggregate; the sink is one append-only file per process, and the reader is its own project with dependencies: ['claims'] rather than an ordinary test. That project is what npm run test:verdicts runs now, so the required check reads the ledger with the specs that write it — .github/ is untouched. assertedBy stops being a filename. It is {spec, test, status, says}: the triple, plus the part of the expectation the RECORD decides — the outcome the killing assertion must require, and one fragment it must hand the helper verbatim (a "re:" prefix is a RegExp source). That second half is the only thing that can catch a call kept but made tautological, and it has to work that way round: on an unmutated tree an expectation read off the page and one decided in advance are the same values, so no amount of observing the run separates them. Only something decided before the run can. The plate is no longer an argument. tone: 'pass' | 'alarm' is gone and the plate is derived from the asserted status through one table, because a reader is never shown an alarm outcome on the success plate — so a test cannot expect one either. That closes the residue: a tautologist who has read the record and hands the declared fragment still cannot mask record-match's mutation, because the plate follows the status it read rather than the paint it read. Two recorded mutations were killed by a precursor rather than by the helper, and both are reordered so the helper is reached first: - tree-point — reconstruction[alpha], the lit-bit count and the #tree-xor readback all failed on the direction-bit mutation before line 232's expectVerdict ran. - key-size-formula — found by running the set: the formulaNumbers comparison caught the pinned-290 mutation at claims.spec.ts:209, before expectClaim. Through the helper it now fails on its own terms, "data-value 86 is not in 17 + (17 × log₂ 16) + 1 = 290 bytes". Escapes. Each run in an isolated tree archived from this commit, node_modules symlinked, port moved to 4931, spec-only rewrites with the built bundle IDENTICAL before and after (index-RVWjpVch.js 3a8b3a777965e297ba523fe90f68143b), baseline 21 passed / exit 0 in the same tree: - Commented out — verdict-ledger fails: "e2e/verdict-mutations.json says its mutation is killed by expectVerdict in e2e/claims.spec.ts › 'tampering violates integrity…', and that call did not execute this run. expectVerdict ran for record-match in: e2e/claims.spec.ts › honest PIR output equals the indexed shelf record (status 'pass')." - Kept but tautological — expectVerdict fails in the test itself: "declares the words this assertion must require — 'RETRIEVED — AND WRONG' — and it was handed ['RETRIEVED — AND WRONG · 1 of 16 bytes differ from shelf[α]; no PIR authentication failure was raised']." With the declared fragment handed over and the mutation applied, it fails one layer down instead: "record-match is painted as pass while its outcome is 'alarm'", expected rgb(66, 29, 36), received rgb(15, 53, 43). - Satisfied from an unrelated line in the same file — this lab's own failure mode, the killing assertion rewritten as text + data-status with the honest-fetch call left in place. Same ledger failure as the first, naming the test the surviving call actually ran in. At 521d03d this shipped 23/23. All 16 recorded mutations re-run against this tree with CI=1: none survived, and every failure is now inside e2e/verdict-assertions.ts — line 110 (text), 112 (data-status), 118 (plate), 154 (measured value), 161 (the sentence states its own value). README: the count is 47 (24 Vitest + 23 Playwright), the gate paragraph describes the ledger, and README:17's "every number it renders carries a data-claim marker" is corrected. #fetch-status renders "Both full-domain folds completed in 1,019 ms" outside every marker; verdict-scan.ts scans result regions and not status prose, so the rule is met as scoped and it was the sentence that was false. The figure is left unmarked on purpose — it measures the device, not the protocol. Branch protection untouched, in either direction. Co-Authored-By: Claude Opus 5 --- README.md | 12 +++- e2e/claims.spec.ts | 37 +++++----- e2e/flows.spec.ts | 2 - e2e/global-setup.ts | 11 +++ e2e/verdict-assertions.ts | 103 +++++++++++++++++++++++++--- e2e/verdict-coverage.spec.ts | 54 ++------------- e2e/verdict-ledger.spec.ts | 73 ++++++++++++++++++++ e2e/verdict-ledger.ts | 128 +++++++++++++++++++++++++++++++++++ e2e/verdict-mutations.json | 106 ++++++++++++++++++++++++----- package.json | 2 +- playwright.config.ts | 13 +++- 11 files changed, 444 insertions(+), 97 deletions(-) create mode 100644 e2e/global-setup.ts create mode 100644 e2e/verdict-ledger.spec.ts create mode 100644 e2e/verdict-ledger.ts diff --git a/README.md b/README.md index ab65c62..a8b7fc4 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ The teaching implementation supports domains from $2^4$ through $2^{16}$. Its ps 4. **Collusion** — deliberately give both keys to one server and watch α become visible inside that server’s view. With collusion off, the page states the one-key case as a count: how many leaves the single expanded share lights, which is what “no distinguished target” means. 5. **Tampering** — flip one bit in a server answer. Every privacy condition still passes, but the client retrieves the wrong record because this PIR does not authenticate answers. The page names the damage from that run: how many bytes of the record differ, and that reconstruction raised nothing. -Every verdict the page renders carries a `data-verdict` marker and turns on a value the page computed, and every number it renders carries a `data-claim` marker with the measurement behind the words in `data-value`. None of them is a fixed string, and the key-size meter states the parts it measured for the domain actually selected, not the default one. +Every verdict the page renders carries a `data-verdict` marker and turns on a value the page computed, and every number it renders **in a result region** carries a `data-claim` marker with the measurement behind the words in `data-value`. One figure sits outside that rule on purpose: the elapsed time in `#fetch-status`, which reports how long this device took rather than anything the protocol determines, and which `verdict-scan.ts` excludes by scanning result regions rather than status prose. The scope is written into the checker, not assumed. None of the marked claims is a fixed string, and the key-size meter states the parts it measured for the domain actually selected, not the default one. ## When to Use It @@ -73,7 +73,7 @@ npm run test:a11y ## Build & Verify -The repository has **45 automated checks**: 24 Vitest unit/property tests and 21 Playwright browser tests. Unit tests cover every supported domain size, independent point-vector reconstruction, single-point versus full-domain evaluation, strict serialization, server API isolation, progressive and synchronous PIR reconstruction, the measured fold count each server's own share produces, the single-key view, record widths beyond one AES block, malformed inputs, collusion, and answer tampering. Current measured core coverage is 95% statements, 96.51% lines, 97.77% functions, and 88.88% branches. +The repository has **47 automated checks**: 24 Vitest unit/property tests and 23 Playwright browser tests. Unit tests cover every supported domain size, independent point-vector reconstruction, single-point versus full-domain evaluation, strict serialization, server API isolation, progressive and synchronous PIR reconstruction, the measured fold count each server's own share produces, the single-key view, record widths beyond one AES block, malformed inputs, collusion, and answer tampering. Current measured core coverage is 95% statements, 96.51% lines, 97.77% functions, and 88.88% branches. There are **0 published known-answer tests** for this exact randomized BGI tree-DPF variant: a search of the primary paper and appendices found proofs and algorithms but no fixed seed/key/output vectors. No vectors were invented. Correctness instead uses randomized properties at every supported domain size plus an independent naive point vector compared leaf by leaf. @@ -81,7 +81,13 @@ The Playwright suite builds before serving and checks the production bundle on t A separate **verdict coverage** gate walks the page through every state that can render an outcome and derives coverage from the DOM rather than from a list. The walk visits every option of every control that changes what renders — each control on its own, never the cross-product — because that walk is the denominator both coverage rules and both stray scans enumerate over: anything reachable only at a setting it never visits is outside the set they judge. -It fails when a `data-verdict` verdict or a `data-claim` measurement has no mutation recorded in `e2e/verdict-mutations.json`, when a registered mutation names a marker the page never renders, when a marker renders without a status or without a numeric value, and when verdict wording, verdict styling or a rendered measurement appears outside a marker — the raw banner, or the stray number, a careless builder adds later. It injects both and asserts the scans catch them, so neither scan can pass by being blind. Every recorded mutation must also be killed through `expectVerdict`/`expectClaim`, which assert a marker's words, its machine-readable state or value, and its painted plate in one call: text alone is not an assertion of state, and a mutation that pinned the record verdict's plate green while its words read RETRIEVED — AND WRONG used to pass the whole suite. +It fails when a `data-verdict` verdict or a `data-claim` measurement has no mutation recorded in `e2e/verdict-mutations.json`, when a registered mutation names a marker the page never renders, when a marker renders without a status or without a numeric value, and when verdict wording, verdict styling or a rendered measurement appears outside a marker — the raw banner, or the stray number, a careless builder adds later. It injects both and asserts the scans catch them, so neither scan can pass by being blind. + +Every recorded mutation must also be killed through `expectVerdict`/`expectClaim`, and that is asserted over the run rather than over the spec's source. Those helpers assert a marker's words, its machine-readable state or value, and its painted plate in one call — the plate **derived** from the asserted outcome, never supplied beside it, because a reader is never shown an alarm outcome on the success plate. Text alone is not an assertion of state: a mutation that pinned the record verdict's plate green while its words read RETRIEVED — AND WRONG used to pass the whole suite. + +Each helper appends the `(spec, test, marker)` triple of every call it executes to a run-scoped ledger, cleared once per run, and `e2e/verdict-ledger.spec.ts` reads those back and fails when a record's triple is not among them. It is its own Playwright project with `dependencies:` on the specs that write the ledger, because Playwright runs tests in separate worker processes and a module-level set cannot aggregate across them. `npm run test:verdicts` runs that project, and its dependency with it. + +The rule this replaces was a regex over spec source naming a FILE, and a regex over source cannot tell a live assertion from one commented out, from one in an unrelated test elsewhere in the same file, or from one handed the page's own values. All three shipped a live mutation green here. Each record therefore also declares the outcome and one text fragment its killing assertion must require, which is what an expectation lifted out of the marker it judges fails against: on an unmutated tree a tautological expectation and an honest one are the same values, so only something decided before the run can separate them. Each entry in that file names a mutation that has been applied and watched to turn its own assertion red, with the unmutated baseline passing in the same run. diff --git a/e2e/claims.spec.ts b/e2e/claims.spec.ts index f9dd419..58a4975 100644 --- a/e2e/claims.spec.ts +++ b/e2e/claims.spec.ts @@ -55,7 +55,6 @@ test('honest PIR output equals the indexed shelf record', async ({ page }) => { // so the measurement is checked with them. await expectVerdict(page, 'record-match', { status: 'pass', - tone: 'pass', text: [ 'RETRIEVED · byte-for-byte match with shelf[α]', new RegExp(`match with shelf\\[α\\] across all ${(expected ?? '').length / 2} bytes`) @@ -73,12 +72,10 @@ test('each server view verdict reports its own measured fold, not a fixed line', await expectVerdict(page, 'server-view-0', { status: 'pass', - tone: 'pass', text: [`one party-0 key of ${keyBytes} B`, /folded [\d,]+ of [\d,]+ records/] }); await expectVerdict(page, 'server-view-1', { status: 'pass', - tone: 'pass', text: [`one party-1 key of ${keyBytes} B`, /folded [\d,]+ of [\d,]+ records/] }); @@ -96,7 +93,6 @@ test('each server view verdict reports its own measured fold, not a fixed line', await expectVerdict(page, 'collusion-state', { status: 'pass', - tone: 'pass', text: 'Collusion is off · neither key left its own server view' }); }); @@ -109,7 +105,6 @@ test('the collusion state verdict follows the live trust boundary', async ({ pag await expect(page.locator('#pir-result')).toBeVisible(); await expectVerdict(page, 'collusion-state', { status: 'alarm', - tone: 'alarm', text: 'Collusion is ON · one view holds both keys' }); await expect(page.locator('#privacy-verdicts [data-status="pass"]')).toHaveCount(2); @@ -121,15 +116,20 @@ test('rendered leaf shares XOR to one point at the displayed alpha', async ({ pa const share0 = await page.locator('#tree-zero .node-bit').allTextContents(); const share1 = await page.locator('#tree-one .node-bit').allTextContents(); const reconstruction = share0.map((bit, index) => Number(bit) ^ Number(share1[index])); - expect(reconstruction.filter((bit) => bit === 1)).toHaveLength(1); - expect(reconstruction[alpha]).toBe(1); - expect(await page.locator('#tree-xor .node-bit').allTextContents()).toEqual(reconstruction.map(String)); - // The headline verdict must report the index the shares actually reconstruct. + // The headline verdict must report the index the shares actually reconstruct, + // and it is asserted FIRST. verdict-mutations.json records this test as the + // one whose expectVerdict kills the direction-bit mutation, and every raw + // assertion that used to sit above this line — the lit-bit count, + // reconstruction[alpha], the #tree-xor readback — fails on that mutation + // before the helper is ever reached, so the flip the record describes was a + // flip no run had demonstrated. await expectVerdict(page, 'tree-point', { status: 'pass', - tone: 'pass', text: ['ONE LIT POINT AT', String(reconstruction.indexOf(1))] }); + expect(reconstruction.filter((bit) => bit === 1)).toHaveLength(1); + expect(reconstruction[alpha]).toBe(1); + expect(await page.locator('#tree-xor .node-bit').allTextContents()).toEqual(reconstruction.map(String)); expect(Number(await page.locator('#tree-alpha-verdict').textContent())).toBe(reconstruction.indexOf(1)); }); @@ -200,10 +200,17 @@ test('key-size claims are measured per level across every domain the page offers const chor = await expectClaim(page, 'chor-query-bytes', { value: domainSize / 8, - text: `${domainSize.toLocaleString()} bits` + text: ['bits', `${domainSize.toLocaleString()} bits`] }); expect(chor * 8, 'the Chor baseline is one bit per record').toBe(domainSize); + // Helper first, for the reason the tree-point call is first: the recorded + // mutation pins the formula's total to a literal 290, and the raw + // formulaNumbers comparison below catches that at every domain but the + // default — before expectClaim is reached, leaving the record's flip a flip + // no run demonstrated. expectClaim catches it on its own terms, because the + // sentence stops stating the data-value beside it. + await expectClaim(page, 'key-size-formula', { value: total, text: 'bytes' }); const formulaText = (await page.locator('[data-claim="key-size-formula"]').textContent()) ?? ''; const formulaNumbers = (formulaText.match(/[\d,]+/g) ?? []).map((value) => Number(value.replaceAll(',', ''))); expect(formulaNumbers, `the formula states this domain's own measured parts: ${formulaText}`).toEqual([ @@ -213,7 +220,6 @@ test('key-size claims are measured per level across every domain the page offers finalBytes, total ]); - await expectClaim(page, 'key-size-formula', { value: total }); } // The PRG width is measured from a real expansion, and λ is the one the key @@ -228,8 +234,9 @@ test('changing alpha retires stale output while a no-op does not', async ({ page await expect(page.locator('#retirement')).toBeHidden(); await slider.fill('4'); await expect(page.locator('#retirement')).toContainText('Previous reconstruction for α = 11 retired'); + // Helper before readback here too: the readback fails on the same mutation. + await expectVerdict(page, 'tree-point', { status: 'pass', text: ['ONE LIT POINT AT', '4'] }); await expect(page.locator('#tree-alpha-verdict')).toHaveText('4'); - await expectVerdict(page, 'tree-point', { status: 'pass', tone: 'pass', text: ['ONE LIT POINT AT', '4'] }); const paintedHidden = await page.locator('[hidden]').evaluateAll((nodes) => nodes.filter((node) => getComputedStyle(node).display !== 'none').length); expect(paintedHidden).toBe(0); }); @@ -251,7 +258,6 @@ test('tampering violates integrity while every privacy verdict remains green', a // A corrupted record must not be able to keep the green plate. await expectVerdict(page, 'record-match', { status: 'alarm', - tone: 'alarm', text: ['RETRIEVED — AND WRONG', `${actual} of ${expectedBytes.length} bytes differ`] }); }); @@ -267,7 +273,6 @@ test('the negative claim reports the run that exercised it', async ({ page }) => // silently-accepted corruption produces, and the byte count has to be real. await expectVerdict(page, 'no-authentication', { status: 'alarm', - tone: 'alarm', text: 'does not authenticate what is returned' }); const evidence = (await page.locator('#integrity-evidence').textContent()) ?? ''; @@ -286,14 +291,12 @@ test('the negative claim says it was not exercised when nothing was tampered wit await page.goto('.'); await expectVerdict(page, 'no-authentication', { status: 'unexercised', - tone: 'alarm', text: 'does not authenticate what is returned' }); await page.getByRole('button', { name: 'Fetch privately' }).click(); await expect(page.locator('#pir-result')).toBeVisible(); await expectVerdict(page, 'no-authentication', { status: 'unexercised', - tone: 'alarm', text: 'does not authenticate what is returned' }); await expect(page.locator('#integrity-evidence')).toContainText('no answer was altered'); diff --git a/e2e/flows.spec.ts b/e2e/flows.spec.ts index 4482ec0..528f4b2 100644 --- a/e2e/flows.spec.ts +++ b/e2e/flows.spec.ts @@ -25,7 +25,6 @@ test('collusion deliberately exposes alpha and switching it off removes the alar await page.locator('#collusion-toggle').check(); await expectVerdict(page, 'collusion-recovery', { status: 'alarm', - tone: 'alarm', text: 'SERVER SEES α = 11' }); // The exposed index is the one the joined view reconstructs, not the slider's. @@ -41,7 +40,6 @@ test('collusion deliberately exposes alpha and switching it off removes the alar // and the words, the status and the green plate are asserted as one claim. await expectVerdict(page, 'single-share', { status: 'pass', - tone: 'pass', text: [/ONE KEY ONLY · k0 alone lights \d+ of \d+ leaves, naming no point/] }); const text = (await alone.textContent()) ?? ''; diff --git a/e2e/global-setup.ts b/e2e/global-setup.ts new file mode 100644 index 0000000..b37800f --- /dev/null +++ b/e2e/global-setup.ts @@ -0,0 +1,11 @@ +import { resetLedger } from './verdict-ledger'; + +/** + * Clear the executed-assertion ledger once per run. A sink left over from an + * earlier run would make the coverage proof answer a question about a tree that + * is no longer on disk — the same shape as a mutation reported as a survivor + * because the patch never applied. + */ +export default function globalSetup(): void { + resetLedger(); +} diff --git a/e2e/verdict-assertions.ts b/e2e/verdict-assertions.ts index 35a823a..088e008 100644 --- a/e2e/verdict-assertions.ts +++ b/e2e/verdict-assertions.ts @@ -1,4 +1,5 @@ -import { expect, type Page } from '@playwright/test'; +import { expect, test, type Page } from '@playwright/test'; +import { declarationFor, fragmentSources, recordAssertion, specPathOf, type Family } from './verdict-ledger'; /** * A marker's words and its state are ONE claim, so one call asserts all of it. @@ -11,10 +12,24 @@ import { expect, type Page } from '@playwright/test'; * "RETRIEVED — AND WRONG", and it survived the whole suite before this helper * existed. * - * `tone` is checked as the painted background rather than as a class name, + * The plate is checked as the painted background rather than as a class name, * because that is what a reader sees and because the two verdict families style * themselves differently: `.verdict` switches class, `.mini-verdict` switches on * `[data-status]`. The plates are the same two colours the stray scan looks for. + * + * **The plate is DERIVED from the asserted outcome, never supplied.** It used to + * be a `tone` argument, and a caller that read both off the page could then + * assert "alarm words on a pass plate" as if that were the expected rendering — + * which is exactly the state the `record-match` mutation produces. A reader is + * never shown an alarm outcome on the success plate, so a test cannot expect one + * either: the only expectation this helper accepts is that the plate agrees with + * the outcome. + * + * Every call that passes appends the triple `(spec, test, marker)` it actually + * executed to the run-scoped ledger. `verdict-ledger.spec.ts` reads those back + * and fails when a registry record's triple is not among them, so "the mutation + * is killed through the helper" is a statement about the run rather than about + * the spec's source text. */ export type VerdictTone = 'pass' | 'alarm'; @@ -24,16 +39,70 @@ const PLATE: Record = { alarm: 'rgb(66, 29, 36)' }; +/** + * Every outcome this page sets in `dataset.status`, and the plate each is + * painted on. `pass` is the only one that earns the success plate: `mismatch`, + * `contradicted` and `unexercised` are all states in which the page has NOT + * demonstrated what the marker exists to demonstrate, and all three are styled + * on the alarm plate. An outcome missing from this table fails rather than + * defaulting, because a status the helper does not know is a status it cannot + * check the paint against. + */ +const TONE_OF_STATUS: Record = { + pass: 'pass', + alarm: 'alarm', + mismatch: 'alarm', + contradicted: 'alarm', + unexercised: 'alarm' +}; + export interface VerdictExpectation { /** Text the marker must contain — every fragment, all of them required. */ text: string | RegExp | ReadonlyArray; - /** The machine-readable outcome in `data-status`. */ + /** The machine-readable outcome in `data-status`. The plate follows from it. */ status: string; - /** The plate a reader sees. */ - tone: VerdictTone; +} + +/** + * When the executing triple is the one a registry record pins as the assertion + * that kills its mutation, the record's own declaration is enforced against the + * arguments before anything is read from the page. + * + * This is the only defence against a call that is kept but made tautological, + * and it has to work this way round: on an unmutated tree an expectation read + * off the page and an expectation decided in advance are THE SAME VALUES, so no + * amount of observing the run can separate them. What separates them is that + * the record decided part of the expectation before the run existed. + */ +function enforceDeclaration(family: Family, marker: string, asserted: { status?: string; says: string[] }): void { + const declared = declarationFor(family, marker); + if (declared === undefined) return; + const info = test.info(); + if (declared.spec !== specPathOf(info.file) || declared.test !== info.title) return; + + if (declared.status !== undefined) { + expect( + asserted.status, + `${marker}: e2e/verdict-mutations.json pins this test as the assertion that kills its mutation, and ` + + `declares the outcome that assertion must require as "${declared.status}". It was handed ` + + `"${asserted.status}". An expectation read off the page instead of decided in advance fails here.` + ).toBe(declared.status); + } + expect( + asserted.says, + `${marker}: e2e/verdict-mutations.json declares the words this assertion must require — ` + + `${JSON.stringify(declared.says)} — and it was handed ${JSON.stringify(asserted.says)}. The fragment ` + + `has to be written in the test, not lifted out of the marker it is judging.` + ).toContain(declared.says); } export async function expectVerdict(page: Page, id: string, expected: VerdictExpectation): Promise { + const says = fragmentSources(expected.text); + enforceDeclaration('verdict', id, { status: expected.status, says }); + + const tone = TONE_OF_STATUS[expected.status]; + expect(tone, `${id}: "${expected.status}" is not an outcome this page paints`).not.toBeUndefined(); + const marker = page.locator(`[data-verdict="${id}"]`); await expect(marker, `${id} renders exactly once`).toHaveCount(1); const fragments = Array.isArray(expected.text) ? expected.text : [expected.text]; @@ -42,9 +111,21 @@ export async function expectVerdict(page: Page, id: string, expected: VerdictExp } await expect(marker, `${id} data-status`).toHaveAttribute('data-status', expected.status); const painted = await marker.evaluate((node) => getComputedStyle(node).backgroundColor); - expect(painted, `${id} is painted as ${expected.tone} while its text says "${fragments.join(' / ')}"`).toBe( - PLATE[expected.tone] - ); + expect( + painted, + `${id} is painted as ${tone === 'pass' ? 'alarm' : 'pass'} while its outcome is "${expected.status}" and ` + + `its text says "${says.join(' / ')}"` + ).toBe(PLATE[tone]); + + const info = test.info(); + recordAssertion({ + family: 'verdict', + marker: id, + spec: specPathOf(info.file), + test: info.title, + status: expected.status, + says + }); } export interface ClaimExpectation { @@ -61,6 +142,9 @@ export interface ClaimExpectation { * its words say something else is the same defect one family over. */ export async function expectClaim(page: Page, id: string, expected: ClaimExpectation): Promise { + const says = fragmentSources(expected.text); + enforceDeclaration('claim', id, { says }); + const marker = page.locator(`[data-claim="${id}"]`); await expect(marker, `${id} renders exactly once`).toHaveCount(1); const raw = await marker.getAttribute('data-value'); @@ -79,6 +163,9 @@ export async function expectClaim(page: Page, id: string, expected: ClaimExpecta for (const fragment of fragments as ReadonlyArray) { await expect(marker, `${id} text`).toContainText(fragment as string & RegExp); } + + const info = test.info(); + recordAssertion({ family: 'claim', marker: id, spec: specPathOf(info.file), test: info.title, value, says }); return value; } diff --git a/e2e/verdict-coverage.spec.ts b/e2e/verdict-coverage.spec.ts index f51501d..83594a9 100644 --- a/e2e/verdict-coverage.spec.ts +++ b/e2e/verdict-coverage.spec.ts @@ -1,5 +1,3 @@ -import { readFileSync } from 'node:fs'; -import { fileURLToPath } from 'node:url'; import { expect, test, type Page } from '@playwright/test'; import { claimValues, @@ -9,30 +7,13 @@ import { renderedClaims, renderedMarkers } from './verdict-scan'; +import { REGISTRY } from './verdict-ledger'; -interface MutationRecord { - renders: string; - computedFrom: string; - mutation: string; - expectedFlip: string; - assertedBy: string; -} - -const REGISTRY = JSON.parse( - readFileSync(fileURLToPath(new URL('./verdict-mutations.json', import.meta.url)), 'utf8') -) as { - markers: Record; - claims: Record; -}; - -const SPEC_SOURCES = new Map(); -function specSource(relative: string): string { - const cached = SPEC_SOURCES.get(relative); - if (cached !== undefined) return cached; - const text = readFileSync(fileURLToPath(new URL(`../${relative}`, import.meta.url)), 'utf8'); - SPEC_SOURCES.set(relative, text); - return text; -} +// Whether each registered mutation is actually killed THROUGH expectVerdict or +// expectClaim is asserted in e2e/verdict-ledger.spec.ts, over the triples those +// helpers recorded as they ran. It used to be asserted here as a regex over +// spec source, and source cannot tell a live assertion from a commented-out +// one, from one in an unrelated test, or from one handed the page's own values. interface Walk { markers: Set; @@ -197,29 +178,6 @@ test('every verdict and every measurement the page renders has a mutation coveri } }); -test('each registered mutation is killed through the shared marker helper', async () => { - const families: ReadonlyArray<[string, Record, string]> = [ - ['verdict', REGISTRY.markers, 'expectVerdict'], - ['claim', REGISTRY.claims, 'expectClaim'] - ]; - for (const [family, records, helper] of families) { - for (const [marker, entry] of Object.entries(records)) { - expect(entry.mutation.length, `${marker} records a mutation`).toBeGreaterThan(20); - expect(entry.expectedFlip.length, `${marker} records the flip it expects`).toBeGreaterThan(20); - const source = specSource(entry.assertedBy); - // Mentioning the id is not asserting on it, and asserting its text is not - // asserting its state. The helper asserts words, machine-readable value - // and painted state together, so a kill that only flipped the words is a - // build failure rather than a recorded kill. - const call = new RegExp(`${helper}\\(\\s*page,\\s*['"\`]${marker}['"\`]`); - expect( - call.test(source), - `${entry.assertedBy} must assert the ${family} ${marker} through ${helper}(page, '${marker}', …)` - ).toBe(true); - } - } -}); - test('no verdict word or verdict styling is rendered outside a marker', async ({ page }) => { test.setTimeout(300_000); const strays: unknown[] = []; diff --git a/e2e/verdict-ledger.spec.ts b/e2e/verdict-ledger.spec.ts new file mode 100644 index 0000000..077161e --- /dev/null +++ b/e2e/verdict-ledger.spec.ts @@ -0,0 +1,73 @@ +import { expect, test } from '@playwright/test'; +import { readLedger, recordsFor, type Family, type LedgerEntry } from './verdict-ledger'; + +/** + * "Every recorded mutation is killed through the shared helper", asserted over + * the run that just happened. + * + * This replaces a regex over spec source, and it replaces it because that regex + * was satisfiable three ways with the mutation still shipping green: the call + * commented out, the call kept but handed the page's own values, and — this + * lab's own instance — the killing assertion rewritten while an unrelated call + * for the same marker, in a different test, stayed in the file. The rule was + * FILE-granular, so the unrelated line answered for the rewritten one. + * + * The ledger is written by expectVerdict/expectClaim as they execute, so the + * denominator here is what RAN. This project `dependencies:` on the specs that + * write it; running it without them is the one way it could report clean while + * seeing nothing, and the emptiness check below is that guard. + */ + +const FAMILIES: ReadonlyArray<[Family, string, string]> = [ + ['verdict', 'markers', 'expectVerdict'], + ['claim', 'claims', 'expectClaim'] +]; + +function describeEntry(entry: LedgerEntry): string { + return `${entry.spec} › ${entry.test}${entry.status === undefined ? '' : ` (status "${entry.status}")`}`; +} + +test('every recorded mutation was killed through the helper, in the test its record names', () => { + const ledger = readLedger(); + expect( + ledger.length, + 'the executed-assertion ledger is empty: expectVerdict/expectClaim recorded nothing this run, so ' + + 'nothing below is evidence about anything. Run this project through its `claims` dependency.' + ).toBeGreaterThan(0); + + for (const [family, key, helper] of FAMILIES) { + for (const [marker, record] of Object.entries(recordsFor(family))) { + expect(record.mutation.length, `${marker} records a mutation`).toBeGreaterThan(20); + expect(record.expectedFlip.length, `${marker} records the flip it expects`).toBeGreaterThan(20); + + const declared = record.assertedBy; + expect(declared, `${key}.${marker} declares the assertion that kills it`).toBeTruthy(); + expect(declared.spec, `${marker}: assertedBy.spec names a spec file`).toMatch(/^e2e\/.+\.spec\.ts$/); + expect(declared.test.length, `${marker}: assertedBy.test names a test`).toBeGreaterThan(10); + expect(declared.says.length, `${marker}: assertedBy.says declares words the assertion must require`).toBeGreaterThan(0); + + const forMarker = ledger.filter((entry) => entry.family === family && entry.marker === marker); + const exact = forMarker.filter((entry) => entry.spec === declared.spec && entry.test === declared.test); + expect( + exact.length, + `${marker}: e2e/verdict-mutations.json says its mutation is killed by ${helper} in ` + + `${declared.spec} › "${declared.test}", and that call did not execute this run. ` + + `${helper} ran for ${marker} in: ${ + forMarker.length === 0 ? 'no test at all' : forMarker.map(describeEntry).join(' | ') + }. A call in another test, in a comment, or deleted outright all look the same from here — ` + + 'which is the point.' + ).toBeGreaterThan(0); + + if (declared.status !== undefined) { + expect( + exact.map((entry) => entry.status), + `${marker}: the killing assertion must require the outcome its record declares, "${declared.status}"` + ).toContain(declared.status); + } + expect( + exact.flatMap((entry) => entry.says), + `${marker}: the killing assertion must require the words its record declares, ${JSON.stringify(declared.says)}` + ).toContain(declared.says); + } + } +}); diff --git a/e2e/verdict-ledger.ts b/e2e/verdict-ledger.ts new file mode 100644 index 0000000..ba11488 --- /dev/null +++ b/e2e/verdict-ledger.ts @@ -0,0 +1,128 @@ +import { appendFileSync, mkdirSync, readFileSync, readdirSync, rmSync } from 'node:fs'; +import { dirname, relative, resolve, sep } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +/** + * The denominator has to come from what RAN, not from what the source says ran. + * + * The rule this replaces was file-granular and read with a regex: a record + * naming `e2e/claims.spec.ts` was satisfied by the string `expectVerdict(page, + * 'record-match'` appearing ANYWHERE in that file — inside a comment, inside a + * call asserting a different state in a different test, inside a call fed the + * page's own values. All three of those shipped a live mutation green. + * + * So the helpers now append the `(spec, test, marker)` triples they actually + * execute to a run-scoped sink, and `verdict-ledger.spec.ts` asserts that every + * registry record's triple is in it. Playwright runs tests in separate worker + * processes, so a module-level Set cannot aggregate; the sink is one append-only + * file per process, cleared once per run in `global-setup.ts` and read back + * after the whole suite by a project that `dependencies:` on it. + */ + +const HERE = dirname(fileURLToPath(import.meta.url)); + +export const REPO_ROOT = resolve(HERE, '..'); +export const LEDGER_DIR = resolve(REPO_ROOT, 'test-results', 'verdict-ledger'); + +export type Family = 'verdict' | 'claim'; + +/** + * What the registry pins about the assertion that kills a mutation. `spec` and + * `test` are the triple's other two thirds. `status` and `says` are the parts of + * the expectation the record DECIDES IN ADVANCE, so that an expectation read off + * the page instead — which is indistinguishable from an honest one by + * observation alone, on an unmutated tree — fails against the record. + */ +export interface Declaration { + spec: string; + test: string; + /** The outcome the killing assertion must require. Verdict records only. */ + status?: string; + /** One fragment the killing assertion must hand the helper verbatim. */ + says: string; +} + +export interface MutationRecord { + renders: string; + computedFrom: string; + mutation: string; + expectedFlip: string; + assertedBy: Declaration; +} + +export interface Registry { + note: string; + markers: Record; + claims: Record; +} + +export const REGISTRY = JSON.parse(readFileSync(resolve(HERE, 'verdict-mutations.json'), 'utf8')) as Registry; + +export function recordsFor(family: Family): Record { + return family === 'verdict' ? REGISTRY.markers : REGISTRY.claims; +} + +export function declarationFor(family: Family, marker: string): Declaration | undefined { + return recordsFor(family)[marker]?.assertedBy; +} + +export interface LedgerEntry { + family: Family; + marker: string; + spec: string; + test: string; + /** The outcome the call required, for verdicts. */ + status?: string; + /** The measured value the call required, for claims. */ + value?: number; + /** Every text fragment the call handed the helper, as written. */ + says: string[]; +} + +/** A spec path as the registry writes it: repo-relative, forward slashes. */ +export function specPathOf(file: string): string { + return relative(REPO_ROOT, file).split(sep).join('/'); +} + +/** Fragments as SOURCES, so a RegExp is comparable and a page-read string is not. */ +export function fragmentSources(text: unknown): string[] { + const fragments = text === undefined ? [] : Array.isArray(text) ? text : [text]; + return (fragments as unknown[]).map((fragment) => + fragment instanceof RegExp ? `re:${fragment.source}` : String(fragment) + ); +} + +let sink: string | undefined; + +/** Append one executed triple. One file per process; one JSON object per line. */ +export function recordAssertion(entry: LedgerEntry): void { + if (sink === undefined) { + mkdirSync(LEDGER_DIR, { recursive: true }); + sink = resolve(LEDGER_DIR, `worker-${process.pid}.jsonl`); + } + appendFileSync(sink, `${JSON.stringify(entry)}\n`, 'utf8'); +} + +/** Clear the sink at the start of a run, so a stale file can never be evidence. */ +export function resetLedger(): void { + rmSync(LEDGER_DIR, { recursive: true, force: true }); + mkdirSync(LEDGER_DIR, { recursive: true }); +} + +export function readLedger(): LedgerEntry[] { + let files: string[]; + try { + files = readdirSync(LEDGER_DIR); + } catch { + return []; + } + const entries: LedgerEntry[] = []; + for (const name of files.filter((file) => file.endsWith('.jsonl'))) { + const text = readFileSync(resolve(LEDGER_DIR, name), 'utf8'); + for (const line of text.split('\n')) { + if (line.trim().length === 0) continue; + entries.push(JSON.parse(line) as LedgerEntry); + } + } + return entries; +} diff --git a/e2e/verdict-mutations.json b/e2e/verdict-mutations.json index 87069e2..cc354df 100644 --- a/e2e/verdict-mutations.json +++ b/e2e/verdict-mutations.json @@ -1,61 +1,101 @@ { - "note": "One §4.1c mutation per rendered marker, in both families: `markers` are the data-verdict outcomes, `claims` are the data-claim measurements. A kill counts only when the unmutated baseline passed in the same run and the failure is that marker's own assertion, and every record's assertion must go through expectVerdict/expectClaim so words, value and painted state are one claim. Recorded failures are in the commit body.", + "note": "One §4.1c mutation per rendered marker, in both families: `markers` are the data-verdict outcomes, `claims` are the data-claim measurements. A kill counts only when the unmutated baseline passed in the same run and the failure is that marker's own assertion. `assertedBy` names the ONE test whose call to expectVerdict/expectClaim must kill it, and declares the part of that call's expectation the record decides in advance: `status` is the outcome the assertion must require, `says` one text fragment it must hand the helper verbatim (a `re:` prefix is a RegExp source). The helpers append every triple they execute to a run-scoped ledger and e2e/verdict-ledger.spec.ts asserts each record's triple is in it, so the guarantee rests on the run rather than on a regex over spec source — which a comment, a tautological argument, or an unrelated line elsewhere in the same file all satisfied. Recorded failures are in the commit body.", "markers": { "tree-point": { "renders": "#tree-verdict — ONE LIT POINT AT α = n", "computedFrom": "evaluateAll(k0) XOR evaluateAll(k1), reading back the lit index", "mutation": "src/dpf/gen.ts — flip every direction bit: ((alpha >>> (domainBits - level - 1)) & 1) becomes (((alpha >>> (domainBits - level - 1)) & 1) ^ 1)", "expectedFlip": "point lands at 15 - α, so data-status becomes alarm and the text becomes POINT AT THE WRONG INDEX", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "rendered leaf shares XOR to one point at the displayed alpha", + "status": "pass", + "says": "ONE LIT POINT AT" + } }, "single-share": { "renders": "#collusion-view verdict with the collusion toggle off — ONE KEY ONLY · k0 alone lights n of 16 leaves", "computedFrom": "singleShareView(k0).lit / .namesAPoint", "mutation": "src/attack/single-view.ts — namesAPoint: lit === 1 becomes namesAPoint: lit >= 1", "expectedFlip": "a share lighting ~8 leaves is reported as naming a point, so data-status becomes alarm and the plate turns red", - "assertedBy": "e2e/flows.spec.ts" + "assertedBy": { + "spec": "e2e/flows.spec.ts", + "test": "collusion deliberately exposes alpha and switching it off removes the alarm", + "status": "pass", + "says": "re:ONE KEY ONLY · k0 alone lights \\d+ of \\d+ leaves, naming no point" + } }, "collusion-recovery": { "renders": "#collusion-view verdict with the collusion toggle on — SERVER SEES α = n", "computedFrom": "collude(k0, k1).alpha compared with the client's secret α", "mutation": "src/attack/collude.ts — alpha: lit[0][0] becomes alpha: (lit[0][0] + 1) % reconstruction.length", "expectedFlip": "recovered index no longer equals α, so data-status becomes mismatch and the text becomes JOIN MISSED", - "assertedBy": "e2e/flows.spec.ts" + "assertedBy": { + "spec": "e2e/flows.spec.ts", + "test": "collusion deliberately exposes alpha and switching it off removes the alarm", + "status": "alarm", + "says": "SERVER SEES α = 11" + } }, "server-view-0": { "renders": "first #privacy-verdicts tile — Server 0 · one party-0 key of 290 B · folded n of 65,536 records", "computedFrom": "deserializeKey(keys[0]).party, keys[0].length, and fold0.foldedRecords from the real fold", "mutation": "src/pir/server.ts — in serverAnswerProgressive, if (share[index] === 0) continue becomes if (share[index] === 0 || index !== 0) continue", "expectedFlip": "the server folds at most the first record, so folded > 1 fails and data-status becomes alarm", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "each server view verdict reports its own measured fold, not a fixed line", + "status": "pass", + "says": "re:folded [\\d,]+ of [\\d,]+ records" + } }, "server-view-1": { "renders": "second #privacy-verdicts tile — Server 1 · one party-1 key of 290 B · folded n of 65,536 records", "computedFrom": "deserializeKey(keys[1]).party, keys[1].length, and fold1.foldedRecords from the real fold", "mutation": "src/dpf/serialize.ts — bytes[0] = (key.party << 7) | key.domainBits becomes bytes[0] = key.domainBits, dropping the party bit from the wire format", "expectedFlip": "server 1's key deserializes as party 0, so only this tile's data-status becomes alarm while server-view-0 stays pass", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "each server view verdict reports its own measured fold, not a fixed line", + "status": "pass", + "says": "re:folded [\\d,]+ of [\\d,]+ records" + } }, "collusion-state": { "renders": "third #privacy-verdicts tile — Collusion is off / Collusion is ON", "computedFrom": "the live state of #collusion-toggle at fetch time", "mutation": "src/ui/app.ts — const colluding = element('collusion-toggle').checked becomes const colluding = false", "expectedFlip": "a fetch run with collusion switched on still reports the tile as pass", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "the collusion state verdict follows the live trust boundary", + "status": "alarm", + "says": "Collusion is ON · one view holds both keys" + } }, "record-match": { "renders": "#record-verdict — RETRIEVED · byte-for-byte match with shelf[α] / RETRIEVED — AND WRONG", "computedFrom": "sameBytes(reconstruct(answer0, answer1), shelf[α]) and the differing byte count", "mutation": "src/ui/app.ts — verdict.className = `verdict ${matches ? 'verdict-pass' : 'verdict-alarm'}` becomes verdict.className = 'verdict verdict-pass', pinning the plate green while data-status and the words still follow the run", "expectedFlip": "a corrupted record is announced as RETRIEVED — AND WRONG on a green pass plate; expectVerdict's tone assertion fails where the old text-and-status assertions both passed", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "tampering violates integrity while every privacy verdict remains green", + "status": "alarm", + "says": "RETRIEVED — AND WRONG" + } }, "no-authentication": { "renders": "#negative-claim — the run-level evidence in #integrity-evidence", "computedFrom": "whether a tampered answer reconstructed to something other than shelf[α] with no error raised", "mutation": "src/attack/tamper.ts — tampered[0] ^= 1 becomes tampered[0] ^= 0", "expectedFlip": "the tampered run matches shelf[α], so data-status becomes contradicted rather than alarm", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "the negative claim reports the run that exercised it", + "status": "alarm", + "says": "does not authenticate what is returned" + } } }, "claims": { @@ -64,56 +104,88 @@ "computedFrom": "serializeKey(generateDpf(0, domainBits)[0]).length for the domain under test", "mutation": "src/ui/app.ts — renderClaim('key-byte-label', serialized.length, `${serialized.length.toLocaleString()} bytes`) becomes renderClaim('key-byte-label', parts.correctionWords, `${parts.correctionWords.toLocaleString()} bytes`)", "expectedFlip": "the meter states the correction-word region as the whole key, so the claim no longer equals the measured hex dump", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "key-size claims are measured per level across every domain the page offers", + "says": "bytes" + } }, "chor-query-bytes": { "renders": "#chor-byte-label in the meter — 8,192 bytes · 65,536 bits", "computedFrom": "domainSize / 8, the one-bit-per-record Chor baseline", "mutation": "src/ui/app.ts — const chorBytes = domainSize / 8 becomes const chorBytes = domainSize / 4", "expectedFlip": "the baseline claims two bits per record, so chor * 8 no longer equals the domain size it names", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "key-size claims are measured per level across every domain the page offers", + "says": "bits" + } }, "key-part-root": { "renders": "#part-root in the parts list — 17 B of root material", "computedFrom": "serializedKeyParts(domainBits).rootMaterial, checked against the header byte plus the root seed in the dump", "mutation": "src/ui/app.ts — renderClaim('part-root', parts.rootMaterial, `${parts.rootMaterial} B`) becomes renderClaim('part-root', parts.rootMaterial - 1, `${parts.rootMaterial - 1} B`). The deeper serializer mutation (ROOT_MATERIAL_BYTES = SECURITY_BYTES) was run first and rejected as evidence: it stops the key round-tripping and takes the whole suite red, which is not a readable kill on this marker.", "expectedFlip": "root material is claimed as 16 B against the 17 B that λ, measured from the per-level slope, requires, and the parts stop accounting for the measured total", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "key-size claims are measured per level across every domain the page offers", + "says": "B" + } }, "key-part-levels": { "renders": "#part-levels in the parts list — 272 B of per-level corrections", "computedFrom": "serializedKeyParts(domainBits).correctionWords, checked against the words actually present in the dump, summed", "mutation": "src/ui/app.ts — renderClaim('part-levels', parts.correctionWords, `${parts.correctionWords} B`) becomes renderClaim('part-levels', parts.correctionWords + perLevel, `${parts.correctionWords + perLevel} B`)", "expectedFlip": "the parts list claims one correction word more than the key carries, so the summed per-level cost no longer matches and the parts stop accounting for the total", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "key-size claims are measured per level across every domain the page offers", + "says": "B" + } }, "key-part-final": { "renders": "#part-final in the parts list — 1 B of final correction", "computedFrom": "serializedKeyParts(domainBits).finalCorrection, the single byte holding the output correction bit", "mutation": "src/ui/app.ts — renderClaim('part-final', parts.finalCorrection, `${parts.finalCorrection} B`) becomes renderClaim('part-final', 2, '2 B')", "expectedFlip": "the tail is claimed as two bytes, so the stride walk loses a correction word and the parts overshoot the measured total", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "key-size claims are measured per level across every domain the page offers", + "says": "B" + } }, "key-bytes-total": { "renders": "#key-bytes in the parts list — the measured total, 290 B", "computedFrom": "serializeKey(...).length, checked against the length of the rendered hex dump", "mutation": "src/ui/app.ts — renderClaim('key-bytes', serialized.length, `${serialized.length} B`) becomes renderClaim('key-bytes', serialized.length - 1, `${serialized.length - 1} B`)", "expectedFlip": "the measured total disagrees by one byte with the hex dump it is measured from", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "key-size claims are measured per level across every domain the page offers", + "says": "B" + } }, "key-size-formula": { "renders": "#size-formula — 17 + (17 × log₂ 65,536) + 1 = 290 bytes", "computedFrom": "the same measured parts, restated as the size law for the domain under test", "mutation": "src/ui/app.ts — the formula's `= ${parts.total} bytes` becomes the literal `= 290 bytes`", "expectedFlip": "the formula is correct only at the default 65,536-record domain and states 290 bytes at every other one; invisible until the walk visits all four domain options", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "key-size claims are measured per level across every domain the page offers", + "says": "bytes" + } }, "prg-node-width": { "renders": "#prg-width in the trace strip — AES-128-CTR → 2(λ + 1) = 258 bits per node", "computedFrom": "a real expandSeed() of the all-zero seed: both child seeds plus the two control bits", "mutation": "src/ui/app.ts — const nodeWidthBits = (zeroExpansion.leftSeed.length + zeroExpansion.rightSeed.length) * 8 + 2 becomes the same expression without + 2", "expectedFlip": "the node width drops the two control bits and claims 256, against 2(λ + 1) derived from the per-level slope the key sizes measured", - "assertedBy": "e2e/claims.spec.ts" + "assertedBy": { + "spec": "e2e/claims.spec.ts", + "test": "key-size claims are measured per level across every domain the page offers", + "says": "bits per node" + } } } } diff --git a/package.json b/package.json index 55e5de4..3b29ca3 100644 --- a/package.json +++ b/package.json @@ -10,7 +10,7 @@ "test": "vitest run --coverage", "test:unit": "vitest run", "test:a11y": "playwright test", - "test:verdicts": "playwright test e2e/verdict-coverage.spec.ts" + "test:verdicts": "playwright test --project=verdict-ledger" }, "dependencies": { "@noble/ciphers": "^2.0.1" diff --git a/playwright.config.ts b/playwright.config.ts index cc6592a..6134c01 100644 --- a/playwright.config.ts +++ b/playwright.config.ts @@ -7,14 +7,25 @@ export default defineConfig({ forbidOnly: !!process.env.CI, retries: process.env.CI ? 1 : 0, reporter: 'list', + globalSetup: './e2e/global-setup.ts', use: { baseURL: `http://localhost:${PORT}/crypto-lab-split-point/`, colorScheme: 'dark', trace: 'retain-on-failure' }, + // `verdict-ledger` reads back the (spec, test, marker) triples the shared + // helpers actually executed, so it has to run after everything that calls + // them. Playwright puts each spec in its own worker process, which is why the + // ledger is a file rather than a Set, and why this is a project with + // `dependencies:` rather than an ordinary test. + projects: [ + { name: 'claims', testMatch: /(?:claims|flows|verdict-coverage)\.spec\.ts$/ }, + { name: 'a11y', testMatch: /a11y\.spec\.ts$/ }, + { name: 'verdict-ledger', testMatch: /verdict-ledger\.spec\.ts$/, dependencies: ['claims'] } + ], webServer: { command: `npm run build && npm run preview -- --port ${PORT} --strictPort`, url: `http://localhost:${PORT}/crypto-lab-split-point/`, reuseExistingServer: !process.env.CI } -}); \ No newline at end of file +});