diff --git a/README.md b/README.md index 8d26ec1..041a5ba 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ letsauth keeps one open dataset of what the agent protocols require and what rea | ๐Ÿ•ธ๏ธ | **[Agent auth atlas](https://letsauth.eu/standards/)** | The specifications, drafts and EU acts behind agent authorization as one map of what builds on, profiles, competes with and replaces what, with maturity and dated revisions per document. Export: [`data.json`](https://letsauth.eu/standards/data.json). | | ๐Ÿ”— | **[Delegation chain explorer](https://letsauth.eu/explorer/)** | Scenarios of how authority travels from a person to an agent to an API, step by step, with the decoded token at each hop and toggles for the attacks and misconfigurations that break the chain. Every step cites a specification revision. Export: [`data.json`](https://letsauth.eu/explorer/data.json). | | ๐Ÿฉป | **[Breach anatomy](https://letsauth.eu/breaches/)** | Published vulnerabilities and incidents in how agents were authorized, each written from its advisory and traced to the control that was missing, the atlas requirements that stand for it, and a replay toggle in the explorer. Export: [`data.json`](https://letsauth.eu/breaches/data.json). | -| ๐Ÿ” | **[Scanner](https://letsauth.eu/mcp/scanner/)** | A passive scanner that reads each listed server's public OAuth metadata every night, checks it against the MCP Authorization specification and opens one pull request of observations. Nothing is graded: the atlas shows each status, and every report counts them per requirement level; one report per server at `/mcp/servers//scan/` and `/mcp/servers//scan.json`. | +| ๐Ÿ” | **[Scanner](https://letsauth.eu/mcp/scanner/)** | A passive scanner that reads each listed server's public OAuth metadata every night, checks it against the MCP Authorization specification and opens one pull request of observations. Nothing is graded: the atlas shows each status, and each server page counts them per requirement level and lists the requests behind them; the raw report is at `/mcp/servers//scan.json`. | Next on the [roadmap](docs/roadmap.md): agents under EU rules, and a changelog generated from the data. diff --git a/astro.config.mjs b/astro.config.mjs index d2fa177..3bbd89d 100644 --- a/astro.config.mjs +++ b/astro.config.mjs @@ -24,7 +24,9 @@ export default defineConfig({ "/atlas/standards": "/standards", "/atlas/standards/[id]": "/standards/[id]", "/scanner": "/mcp/scanner", - "/scanner/[id]": "/mcp/servers/[id]/scan", + "/scanner/[id]": "/mcp/servers/[id]", + // the scan report was folded into the server page + "/mcp/servers/[id]/scan": "/mcp/servers/[id]", }, integrations: [svelte(), sitemap(), pagefind()], }); diff --git a/docs/architecture.md b/docs/architecture.md index 34a9792..3922485 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -52,12 +52,11 @@ that has no page of its own. ``` / home /mcp/ MCP auth atlas: the matrix for the current revision -/mcp/servers// one server; .json is its export -/mcp/servers//scan/ the scanner's report for that server; scan.json is its export +/mcp/servers// one server, with its newest passive scan; .json and scan.json are its exports /mcp/features// one requirement /mcp/revisions// the matrix as of an older revision (the current one redirects to /mcp/) /mcp/changes/// what changed between two revisions -/mcp/scanner/ what the scanner reads, how it reports, and every report +/mcp/scanner/ what the scanner reads, how it counts, and what it does not check /mcp/data.json the atlas export /standards/ Agent auth atlas: the standards map /standards// one specification @@ -91,7 +90,7 @@ scripts/ command-line entry points: validate-data, scan, scan-nightly, a and the generators for brand and README assets specs/ Spec Kit artifacts per feature: spec, plan, tasks, contracts src/ - components/ Astro components and the Svelte islands (matrix, standards map, explorer, scan report) + components/ Astro components and the Svelte islands (matrix, standards map, explorer, scan transcript) layouts/ Base (header, footer, theme) and Document (long-form pages); the shared page head and foot every page uses are in styles/global.css lib/ pure functions over the dataset: atlas matrix, standards graph, chains, exports, citations @@ -111,7 +110,7 @@ tests/ - **Astro 7** with strict TypeScript. Content is data-driven: the pages under `src/pages/atlas/`, `src/pages/explorer/` and `src/pages/scanner/` are generated from the dataset at build time. - **Svelte 5** islands only where interaction needs them: the matrix filters, the standards map - and the scan report. Every page reads fully without JavaScript, and no page ships more than + and the scan transcript. Every page reads fully without JavaScript, and no page ships more than 150 kB of compressed JavaScript. - **One design token system** in `src/styles/tokens.css`. Colours are OKLCH pairs resolved with `light-dark()`; `data-theme` only flips `color-scheme`. Type is Geist for headlines and diff --git a/docs/scanner.md b/docs/scanner.md index 7ea9501..e465bac 100644 --- a/docs/scanner.md +++ b/docs/scanner.md @@ -230,10 +230,12 @@ SHOULD 4 checks unsupported 3, unknown 1 MAY 2 checks unsupported 2 ``` -Under the counts, each level names the checks that are not supported, linked to their rows. The -levels are never added up into one number, and nothing on the site sorts servers by a scan. The -atlas matrix shows the cells themselves and a quiet "report" link next to each server name; the -server page shows how many checks ran and when. +The levels are never added up into one number, and nothing on the site sorts servers by a scan. +There is no report page of its own: the server page is where a scan shows. Its status panel +prints the counts and the date, the checks are rows of its feature tables (marked passive scan, +with the metadata each read), and "What the scanner asked" lists the requests behind them. The +atlas matrix shows the cells themselves and a quiet "scan" link next to each server name that +opens that section. `/mcp/servers//scan/` forwards to the server page; `scan.json` stays. `unknown` is counted as unknown, never as unsupported. A MUST the server may meet in either of two ways counts as supported when either way is supported, and as unknown while the other way is @@ -248,7 +250,7 @@ row. Deprecated requirements are counted under their own level. Servers whose `a There is no on-demand scanning. The site has no endpoint that accepts a URL, the scanner pages ship no JavaScript, and no visitor can cause a request to any server. A server owner who wants -to see their server's report asks for it to be listed; a contributor who wants fresh cells runs +to see their server's scan asks for it to be listed; a contributor who wants fresh cells runs `pnpm scan --write` and opens a pull request. ## Opting out diff --git a/specs/017-neutral-scan-summary/spec.md b/specs/017-neutral-scan-summary/spec.md index ae1a9d4..0398644 100644 --- a/specs/017-neutral-scan-summary/spec.md +++ b/specs/017-neutral-scan-summary/spec.md @@ -46,6 +46,21 @@ row already shows every status the grade was derived from. `open`, `notes`) and bumps `version` to 2. The nightly pull request's table and the `pnpm scan` output use the same summary. +## Amendment: the report folded into the server page + +The scan report page repeated the server page's scanner rows with the same status, date and +evidence; only the request transcript and the counts were its own. The report has no page of its +own any more: + +- **FR-007**: The server page MUST show the newest passive scan: the per-level counts and the + date in its status panel (FR-001, FR-005), and "What the scanner asked", the request transcript + (M12), after its feature tables, with the either-or notes of FR-002. The checks are its feature + rows; the ones not supported are the rows that say so, so the counts do not list them again. +- **FR-008**: The summary rule (FR-003) and the full list of requirements the scanner does not + check MUST be published on the scanner page; the server page links it. +- **FR-009**: `/mcp/servers//scan/` MUST forward to the server page. `scan.json` stays, and + the matrix link next to each server name (FR-004) opens the server page's scan section. + ## Success criteria - **SC-001**: No page, JSON file or scanner output contains a grade letter or the word "grade" diff --git a/specs/017-neutral-scan-summary/tasks.md b/specs/017-neutral-scan-summary/tasks.md index 9c3d343..a10b3cb 100644 --- a/specs/017-neutral-scan-summary/tasks.md +++ b/specs/017-neutral-scan-summary/tasks.md @@ -10,3 +10,12 @@ - [x] T08 Unit and e2e tests; `tests/unit/scanner-grade.test.ts` becomes `scanner-summary.test.ts` - [x] T09 Docs: `docs/scanner.md` "The summary", README, architecture, design (M12), development, data model, maintenance; 004 spec and report contract marked superseded; constitution 1.0.1 - [ ] T10 Screenshot baselines for the matrix, report and server pages: `refresh-screenshots` label on the pull request + +## Amendment: the report folded into the server page (FR-007 to FR-009) + +- [x] T11 `src/components/ScanTranscript.astro` replaces `ScanReport.astro`; `src/pages/mcp/servers/[id].astro` shows the counts in the status panel and the transcript after the feature tables +- [x] T12 `src/pages/mcp/servers/[id]/scan.astro` removed; `astro.config.mjs` forwards `/mcp/servers//scan/` (and the older `/scanner//`) to the server page +- [x] T13 `src/pages/mcp/scanner/index.astro`: "Where the results show" points at the server page; "What it does not check" lists every requirement +- [x] T14 Matrix link opens `#scan`; e2e tests move from the report page to the server page; report baselines removed +- [x] T15 Docs: README, architecture, scanner +- [ ] T16 Screenshot baselines for the matrix, scanner and server pages: `refresh-screenshots` label on the pull request diff --git a/src/components/MatrixPage.astro b/src/components/MatrixPage.astro index 606dc2c..a80755d 100644 --- a/src/components/MatrixPage.astro +++ b/src/components/MatrixPage.astro @@ -44,7 +44,7 @@ const usedLevels = new Set( const usedCategories = [...new Set(matrix.columns.map((f) => f.category))]; const hrefFor = (id: string) => (id === current.id ? "/mcp/" : `/mcp/revisions/${id}/`); -// the nightly scanner's view per server: its report is linked next to the name, never sorted on +// the nightly scanner's view per server: its requests are linked next to the name, never sorted on const scans = new Map(matrix.rows.map((s) => [s.id, latestScan(data, s, revisionId)])); const scannerChecks = passiveChecks(data, revisionId).length; @@ -194,7 +194,7 @@ const classWords = (counts: { value: string; count: number }[], label: RecordAS name the party the feature is observed on. The nightly scanner reads {scannerChecks} of these requirements from - public metadata; "report" next to a server's name opens what it asked and what it found. + public metadata; "scan" next to a server's name opens what it asked. Rows are alphabetical. Nothing here is scored or ranked.

diff --git a/src/components/MatrixTable.astro b/src/components/MatrixTable.astro index 0e41d4a..da68ffb 100644 --- a/src/components/MatrixTable.astro +++ b/src/components/MatrixTable.astro @@ -12,7 +12,7 @@ interface Props { diff?: RevisionDiff; /** * The nightly scanner's latest view per server id. A quiet link next to the server's name - * opens the report; the row's cells already show what the scan found, so nothing is added + * opens the requests behind it on the server page; the row's cells already show what the scan found, so nothing is added * here and nothing is a sort key. */ scans?: Map; @@ -210,11 +210,11 @@ const shortLabel = (id: string) => id.split(".").slice(2).join(".") || id; {scan && ( - report + scan )} @@ -513,7 +513,7 @@ const shortLabel = (id: string) => id.split(".").slice(2).join(".") || id; text-underline-offset: 0.2em; } - /* the row's own cells are the scan's result; this only opens the report behind them */ + /* the row's own cells are the scan's result; this only opens the requests behind them */ .matrix .scan-link { flex: none; font-size: var(--tiny); diff --git a/src/components/ScanReport.astro b/src/components/ScanReport.astro deleted file mode 100644 index 7bbcd14..0000000 --- a/src/components/ScanReport.astro +++ /dev/null @@ -1,384 +0,0 @@ ---- -/** - * The scan report (research R10): the summary (status counts per level, spec 017), the request - * ladder as a transcript (M12), then the checks as rows with a StatusCell each. Static HTML; the only script prints the ladder line by - * line once, and the page reads the same without it. - */ -import StatusCell from "@/components/StatusCell.astro"; -import type { LadderLine } from "@/lib/scanner"; -import type { Report } from "@/scanner/report"; - -interface Props { - report: Report; - /** The requests behind the evidence, in ladder order (`requestLadder`). */ - ladder: LadderLine[]; - /** The newest scanner observation date, for recorded reports. */ - scannedAt?: string; -} - -const { report, ladder, scannedAt } = Astro.props; - -const PARTY_LABEL: Record = { - "mcp-server": "MCP server", - "authorization-server": "Authorization server", -}; -const parties = ["mcp-server", "authorization-server"] as const; -const notScanned = !report.checks.some((c) => c.observedAt); -const { summary } = report; -const judged = report.checks.length; - -function splitEvidence(text: string): { url?: string; note: string } { - const m = text.match(/^(https:\/\/\S+)\s*(.*)$/s); - if (!m) return { note: text }; - return { url: m[1], note: (m[2] ?? "").replace(/^\u2014\s*/, "") }; -} ---- - -
-
-

What the scan found

- { - summary.state === "no-oauth" && - summary.notes.map((note) =>

{note}

) - } - { - notScanned ? ( -

- No passive scan recorded yet for this server. The nightly run adds one when it can read - the server's metadata; until then the atlas page shows what documentation says. -

- ) : summary.state !== "scanned" ? null : ( - <> -
- {summary.tallies.map((t) => ( -
-
- {t.level} - - {t.total} {t.total === 1 ? "check" : "checks"} - -
-
-
    - {t.counts.map((c) => ( -
  • - {" "} - {c.n} -
    • - ))} -
- {summary.open.some((c) => c.level === t.level) && ( -
    - {summary.open - .filter((c) => c.level === t.level) - .map((c) => ( -
  • - - {c.name} -
    • - ))} -
- )} -
-
- ))} -
- {summary.notes.map((note) => ( -

{note}

- ))} - - ) - } -
- - { - ladder.length > 0 && ( -
-

What the scanner asked

-

- The requests behind the evidence below, in the order the scanner makes them, each with - the newest answer this report records. None carries a credential or a message. -

-
    - {ladder.map((line, i) => ( -
  1. - {line.method} - {line.url} - {line.answer} - read {line.read.join(", ")} -
    2. - ))} -
  2. - - - {judged} checks, each with its status below - -
    3. -
-
- ) - } - - { - parties.map((party) => { - const rows = report.checks.filter((c) => c.party === party); - return ( - rows.length > 0 && ( -
-

{PARTY_LABEL[party]} requirements

-
    - {rows.map((c) => { - const ev = splitEvidence(c.evidence); - const observed = Boolean(c.observedAt); - return ( -
  • -
    -

    - {c.name} -

    - -
    -

    - - {c.level} - - {c.feature} - - {c.section} - - {c.observedAt ? ( - - ) : ( - not observed - )} -

    -

    - {ev.url ? ( - <> - - source - - {ev.note && {ev.note}} - - ) : ev.note ? ( - ev.note - ) : ( - "No scanner observation." - )} -

    -
    • - ); - })} -
-
- ) - ); - }) - } - -

- Judged against MCP Authorization {report.revision.revision} - { - scannedAt && ( - <> - ยท newest scanner observation - - ) - } - ยท method passive scan ยท {report.scanner} -

-
- - - - diff --git a/src/components/ScanTranscript.astro b/src/components/ScanTranscript.astro new file mode 100644 index 0000000..e5ba502 --- /dev/null +++ b/src/components/ScanTranscript.astro @@ -0,0 +1,179 @@ +--- +/** + * The passive scan of one server, on its server page (folded in from the old scan report): the + * request ladder as a transcript (M12), the counting notes the summary adds, and how to reproduce + * it. The checks themselves are the feature rows above it. Static HTML; the only script prints + * the ladder line by line once, and the page reads the same without it. + */ +import type { LadderLine, ScanView } from "@/lib/scanner"; +import { SCANNER_ID } from "@/scanner/version"; + +interface Props { + view: ScanView; + /** The requests behind the evidence, in ladder order (`requestLadder`). */ + ladder: LadderLine[]; +} + +const { view, ladder } = Astro.props; +const { server, summary } = view; +const repo = "https://github.com/letsauthlab/letsauth"; +--- + +
+

What the scanner asked

+ { + summary.state === "no-oauth" ? ( + summary.notes.map((note) =>

{note}

) + ) : !view.scanned || ladder.length === 0 ? ( +

+ No passive scan recorded yet for this server. The nightly run adds one when it can read the + server's metadata; until then the rows above show what documentation says. +

+ ) : ( + <> +

+ The requests behind the rows marked passive scan above, in the order the scanner makes + them, each with the newest answer recorded. None carries a credential or a message. +

+
    + {ladder.map((line, i) => ( +
  1. + {line.method} + {line.url} + {line.answer} + read {line.read.join(", ")} +
    2. + ))} +
  2. + + {view.checks.length} checks, read into the tables above +
    3. +
+ {summary.notes.map((note) => ( +

{note}

+ ))} + + ) + } +

+ { + view.scannedAt && ( + <> + Newest scanner observation ยท{" "} + + ) + } + {SCANNER_ID} ยท reproduce with pnpm scan {server.id} ยท{" "} + scan JSON ยท{" "} + data file ยท{" "} + how the scanner works +

+
+ + + + diff --git a/src/pages/mcp/scanner/index.astro b/src/pages/mcp/scanner/index.astro index 49ab209..9ac410a 100644 --- a/src/pages/mcp/scanner/index.astro +++ b/src/pages/mcp/scanner/index.astro @@ -144,7 +144,8 @@ const ladder = [ {SUMMARY_RULE.map((line) =>
  • {line}
    • )}

    - The summary belongs to one report at one time and is counted from the statuses shown. + The summary belongs to one server's newest scan and is counted from the statuses its + rows show. The atlas never sorts or ranks servers by it.

    @@ -156,9 +157,10 @@ const ladder = [

    The scanner has no list of its own. Its results live in the{" "} MCP auth atlas: the matrix cells are the latest observations, and - "report" next to each server name, like every server page, links to - the full report at /mcp/servers/<server>/scan/, which lists - each of the {checks.length} checks with the metadata it read. A server without a scanner + "scan" next to each server name opens that server's page. There the {checks.length} checks + are rows of the feature tables, marked passive scan with the metadata each one read; the + status panel counts them per requirement level, and "What the scanner asked" lists the + requests behind them with the answer each got. A server without a scanner observation reads "not scanned": the nightly run has not yet been able to read its metadata, or has not run since it was listed. Servers whose auth class is API key or no authentication read "no OAuth" and are not scanned; a server whose class is not yet @@ -186,15 +188,16 @@ pnpm scan notion --write # append what the nightly rule wouldWhat it does not check

    {unscanned.length} server-side requirements need a token, a scope or a reading of the - implementation and stay out of a passive scan: for example{" "} - {unscanned.slice(0, 3).map((f, i) => ( - <> - {i > 0 && ", "} - {f.name} - - ))}. Their cells come from documentation and manual checks until active probes exist for - verified owners. + implementation and stay out of a passive scan. Their cells come from documentation and + manual checks until active probes exist for verified owners.

    +
      + {unscanned.map((f) => ( +
    • + {f.name} {f.observable} +
      • + ))} +

    Opting out

    If you run a listed server and do not want it scanned,{" "} @@ -295,6 +298,13 @@ pnpm scan notion --write # append what the nightly rule would Math.round((n / columns.length) * 1000) / 10; const scanView = latestScan(data, server); +const ladder = requestLadder(scanView); const access = accessOf(server); const auth = authOf(server); const classes = [ @@ -194,28 +197,34 @@ const classes = [ ))} -

    - Passive scan - {scanView.scanned ? ( - <> - - {scanView.summary.state === "scanned" - ? `${scanView.checks.length} checks` - : scanView.summary.label} - - {" ยท "} - - {" ยท "} - Report - - ) : ( - <> - not scanned yet - {" ยท "} - Report - +

    +

    + Passive scan + {scanView.scanned ? ( + <> + + {" ยท "} + Requests + + ) : ( + <> + {scanView.summary.state === "no-oauth" ? "no OAuth to check" : "not scanned yet"} + {" ยท "} + Why + + )} +

    + {scanView.summary.tallies.length > 0 && ( +
      + {scanView.summary.tallies.map((t) => ( +
    • + {t.level} + {tallyLine(t).slice(t.level.length + 1)} +
      • + ))} +
    )} -

    +

    Contribute a correction @@ -341,6 +350,8 @@ const classes = [ ); })} + + ) } @@ -433,20 +444,38 @@ const classes = [ vertical-align: baseline; } .scan { - display: flex; - flex-wrap: wrap; - align-items: baseline; - gap: var(--space-2); font-size: var(--small); margin: 0 0 var(--space-4); padding-top: var(--space-3); border-top: 1px solid var(--rule); } + .scan__head { + display: flex; + flex-wrap: wrap; + align-items: baseline; + gap: var(--space-2); + margin: 0; + } .scan__label { font-size: var(--tiny); font-weight: 500; color: var(--ink-2); } + .scan__tally { + list-style: none; + padding: 0; + margin: var(--space-2) 0 0; + display: grid; + gap: var(--space-1); + } + .scan__tally li { + display: grid; + grid-template-columns: 5.5rem 1fr; + align-items: baseline; + } + .scan__tally .level { + justify-self: start; + } .actions { margin: 0; } diff --git a/src/pages/mcp/servers/[id]/scan.astro b/src/pages/mcp/servers/[id]/scan.astro deleted file mode 100644 index c17bab9..0000000 --- a/src/pages/mcp/servers/[id]/scan.astro +++ /dev/null @@ -1,151 +0,0 @@ ---- -import Crumbs from "@/components/Crumbs.astro"; -import ScanReport from "@/components/ScanReport.astro"; -import Base from "@/layouts/Base.astro"; -import { getDataset } from "@/lib/data"; -import { latestScan, reportFromView, requestLadder, unscannedFeatures } from "@/lib/scanner"; -import { SUMMARY_RULE } from "@/scanner/summary"; -import type { Server } from "@/schema"; - -export function getStaticPaths() { - return getDataset().servers.map((server) => ({ params: { id: server.id }, props: { server } })); -} - -const { server } = Astro.props as { server: Server }; -const data = getDataset(); -const view = latestScan(data, server); -const report = reportFromView(view, "1970-01-01T00:00:00Z"); -const ladder = requestLadder(view); -const unscanned = unscannedFeatures(data); -const repo = "https://github.com/letsauthlab/letsauth"; ---- - - -

    - -
    -

    Scan report

    -

    {server.name}

    -

    - {server.url} against MCP Authorization {view.revisionId} - {view.scannedAt && ( - - {" "}ยท newest scanner observation - - )} -

    -

    - Atlas page - {!server.withdrawn && JSON} - Data file -

    -
    - - { - server.withdrawn ? ( -
    -

    Withdrawn from the atlas

    -

    - This server was withdrawn on {" "} - and is not scanned. Reason given: {server.withdrawn.reason}. -

    -
    - ) : ( - <> - - -
    -
    -

    How the summary is counted

    -
      {SUMMARY_RULE.map((line) =>
    1. {line}
      2. )}
    -

    - Counted from the statuses above at build time; never used to sort or rank - servers. About the scanner. -

    -
    -
    -

    Not checked by the scanner

    -

    - These requirements need a token, a scope or a reading of the implementation. Their cells on the atlas page{" "} - come from documentation and manual checks. -

    -
      - {unscanned.map((f) => ( -
    • - {f.name} - {f.observable} -
      • - ))} -
    -
    -
    -
    -

    - Reproduce this report: pnpm scan {server.id} in the{" "} - repository. Corrections are new observations, never - edits; see docs/scanner.md. -

    -
    - - ) - } -
    - - - diff --git a/tests/e2e/__screenshots__/scanner.spec.ts/report-dark.png b/tests/e2e/__screenshots__/scanner.spec.ts/report-dark.png deleted file mode 100644 index 3d8ed2b..0000000 Binary files a/tests/e2e/__screenshots__/scanner.spec.ts/report-dark.png and /dev/null differ diff --git a/tests/e2e/__screenshots__/scanner.spec.ts/report-light.png b/tests/e2e/__screenshots__/scanner.spec.ts/report-light.png deleted file mode 100644 index ec1b9ba..0000000 Binary files a/tests/e2e/__screenshots__/scanner.spec.ts/report-light.png and /dev/null differ diff --git a/tests/e2e/__screenshots__/scanner.spec.ts/report-phone.png b/tests/e2e/__screenshots__/scanner.spec.ts/report-phone.png deleted file mode 100644 index a9c47d0..0000000 Binary files a/tests/e2e/__screenshots__/scanner.spec.ts/report-phone.png and /dev/null differ diff --git a/tests/e2e/matrix.spec.ts b/tests/e2e/matrix.spec.ts index 60fd3ef..18b3ad6 100644 --- a/tests/e2e/matrix.spec.ts +++ b/tests/e2e/matrix.spec.ts @@ -183,18 +183,16 @@ test.describe("matrix page (US1)", () => { await expect(page.locator("#matrix")).not.toContainText(/score|rank/i); }); - test("each server links its latest scan report and shows no grade (017 FR-004)", async ({ - page, - }) => { + test("each server links its latest scan and shows no grade (017 FR-004)", async ({ page }) => { await page.goto("/mcp/"); const links = page.locator("#matrix tbody th[scope='row'] a.scan-link"); await expect(links).toHaveCount(matrix.rows.length); for (const server of matrix.rows.slice(0, 5)) { const scan = latestScan(data, server, revision.id); const link = page.locator(`#matrix tbody tr[data-server="${server.id}"] a.scan-link`); - await expect(link).toHaveAttribute("href", `/mcp/servers/${server.id}/scan/`); + await expect(link).toHaveAttribute("href", `/mcp/servers/${server.id}/#scan`); await expect(link).toHaveAttribute("data-scan", scan.summary.state); - await expect(link).toHaveAccessibleName(`${server.name} scan report`); + await expect(link).toHaveAccessibleName(`${server.name} scan`); } await expect(page.locator("#matrix [data-grade]")).toHaveCount(0); }); diff --git a/tests/e2e/scanner.spec.ts b/tests/e2e/scanner.spec.ts index bd3ba3d..e80c902 100644 --- a/tests/e2e/scanner.spec.ts +++ b/tests/e2e/scanner.spec.ts @@ -1,12 +1,11 @@ import { expect, type Page, test } from "@playwright/test"; import { getDataset } from "../../src/lib/data"; -import { passiveChecks, requestLadder, scannerIndex } from "../../src/lib/scanner"; +import { requestLadder, scannerIndex, unscannedFeatures } from "../../src/lib/scanner"; import { SUMMARY_RULE } from "../../src/scanner/summary"; import { expectEndStateUnderReducedMotion, expectNoSeriousA11yViolations } from "./helpers"; const data = getDataset(); const views = scannerIndex(data); -const checks = passiveChecks(data); const scanned = views.find((v) => v.scanned); const unscanned = views.find((v) => !v.scanned); const first = views[0]; @@ -33,11 +32,9 @@ test.describe("scanner index (US2)", () => { await expect(page.locator("#scanner-index")).toHaveCount(0); await expect(page.locator("table tr[data-server]")).toHaveCount(0); const where = page.locator("[aria-labelledby='where-title']"); - for (const name of ["MCP auth atlas", "server page"]) { - const link = where.getByRole("link", { name, exact: true }); - await expect(link).toBeVisible(); - await expect(link).toHaveAttribute("href", "/mcp/"); - } + const link = where.getByRole("link", { name: "MCP auth atlas", exact: true }); + await expect(link).toBeVisible(); + await expect(link).toHaveAttribute("href", "/mcp/"); await expect(page.locator("[data-nightly]")).toHaveCount(2); }); @@ -62,7 +59,7 @@ test.describe("scanner index (US2)", () => { }); test("has no horizontal page scroll down to 320px (SC-007)", async ({ page }) => { - for (const path of ["/mcp/scanner/", `/mcp/servers/${(scanned ?? first).server.id}/scan/`]) { + for (const path of ["/mcp/scanner/", `/mcp/servers/${(scanned ?? first).server.id}/`]) { for (const width of [320, 412]) { await page.setViewportSize({ width, height: 900 }); await page.goto(path); @@ -76,7 +73,7 @@ test.describe("scanner index (US2)", () => { }); test("offers no way to start a scan from the site (no on-demand scanning)", async ({ page }) => { - for (const path of ["/mcp/scanner/", `/mcp/servers/${first.server.id}/scan/`]) { + for (const path of ["/mcp/scanner/", `/mcp/servers/${first.server.id}/`]) { await page.goto(path); await expect(page.locator("main form, main input, main button")).toHaveCount(0); await expect(page.getByRole("button", { name: /scan/i })).toHaveCount(0); @@ -90,6 +87,11 @@ test.describe("scanner index (US2)", () => { await expect(lines.filter({ hasText: "POST" })).toHaveCount(1); }); + test("lists every requirement the scanner does not check", async ({ page }) => { + await page.goto("/mcp/scanner/"); + await expect(page.locator(".unscanned li")).toHaveCount(unscannedFeatures(data).length); + }); + test("has no serious accessibility violations and looks right", async ({ page }) => { await page.goto("/mcp/scanner/"); await expectNoSeriousA11yViolations(page); @@ -98,21 +100,24 @@ test.describe("scanner index (US2)", () => { }); }); -test.describe("scan report page (US2)", () => { - test("shows one row per passive check with level, status and evidence (FR-009)", async ({ +test.describe("scan on the server page (US2)", () => { + test("the old report address forwards to the server page", async ({ page }) => { + await page.goto(`/mcp/servers/${first.server.id}/scan/`); + await expect(page).toHaveURL(new RegExp(`/mcp/servers/${first.server.id}/$`)); + }); + + test("the status panel counts the checks per level and links the requests (017 FR-005)", async ({ page, }) => { test.skip(!scanned, "no server with a scanner observation in the dataset yet"); - await page.goto(`/mcp/servers/${scanned?.server.id}/scan/`); - const rows = page.locator("[data-report] [data-check]"); - await expect(rows).toHaveCount(checks.length); - await expect(page.locator("[data-report] [data-summary='scanned'] .tally > div")).toHaveCount( - scanned?.summary.tallies.length ?? 0, - ); - const row = rows.first(); - await expect(row.locator(".level")).toHaveCount(1); - await expect(row.locator(".status-cell")).toHaveCount(1); - await expect(row.locator(".evidence")).toHaveCount(1); + await page.goto(`/mcp/servers/${scanned?.server.id}/`); + const line = page.locator("[data-scan='scanned']"); + await expect(line.locator(".scan__tally li")).toHaveCount(scanned?.summary.tallies.length ?? 0); + for (const t of scanned?.summary.tallies ?? []) { + await expect(line.locator(`[data-level="${t.level}"]`)).toContainText(t.level); + } + await expect(line.locator("a[href='#scan']")).toBeVisible(); + await expect(page.locator("#scan")).toBeVisible(); await expect(page.locator("#not-scanned")).toHaveCount(0); }); @@ -120,31 +125,18 @@ test.describe("scan report page (US2)", () => { page, }) => { test.skip(!unscanned, "every server has a scanner observation"); - await page.goto(`/mcp/servers/${unscanned?.server.id}/scan/`); - await expect(page.locator("#not-scanned")).toBeVisible(); - await expect(page.locator("[data-report] .tally")).toHaveCount(0); - const rows = page.locator("[data-report] [data-check]"); - await expect(rows).toHaveCount(checks.length); - await expect(page.locator("[data-report] [data-check][data-status='unsupported']")).toHaveCount( - 0, - ); - await expect(page.locator("[data-report] [data-check][data-status='unknown']")).toHaveCount( - checks.length, - ); + await page.goto(`/mcp/servers/${unscanned?.server.id}/`); + await expect(page.locator("[data-scan='not-scanned']")).not.toContainText(/unsupported/i); + await expect(page.locator("#scan")).not.toContainText(/unsupported/i); await expect(page.locator("[data-transcript]")).toHaveCount(0); }); - test("links the atlas, the server page, the JSON and how the summary is counted", async ({ - page, - }) => { - await page.goto(`/mcp/servers/${first.server.id}/scan/`); - await expect(page.locator(".crumbs a[href='/mcp/']")).toBeVisible(); - await expect(page.locator(`.crumbs a[href='/mcp/servers/${first.server.id}/']`)).toBeVisible(); - await expect(page.locator(`a[href="/mcp/servers/${first.server.id}/"]`).first()).toBeVisible(); - await expect( - page.locator(`a[href="/mcp/servers/${first.server.id}/scan.json"]`).first(), - ).toBeVisible(); - await expect(page.locator("#rule-title")).toBeVisible(); + test("links the scan JSON and how the scanner works", async ({ page }) => { + await page.goto(`/mcp/servers/${first.server.id}/`); + const scan = page.locator("#scan"); + await expect(scan.locator(`a[href="/mcp/servers/${first.server.id}/scan.json"]`)).toBeVisible(); + await expect(scan.locator("a[href='/mcp/scanner/']")).toBeVisible(); + await expect(scan).toContainText(`pnpm scan ${first.server.id}`); const response = await page.request.get(`/mcp/servers/${first.server.id}/scan.json`); expect(response.ok()).toBe(true); const body = await response.json(); @@ -152,21 +144,22 @@ test.describe("scan report page (US2)", () => { expect(body.grade).toBeUndefined(); expect(body.summary.state).toBe(first.summary.state); expect(body.recorded).toBe(true); - expect(body.checks).toHaveLength(checks.length); + expect(body.checks).toHaveLength(first.checks.length); }); - test("has no serious accessibility violations and looks right", async ({ page }) => { - await page.goto(`/mcp/servers/${first.server.id}/scan/`); - // contrast is judged on the end state, not on a ladder line still fading in (M12) + test("has no serious accessibility violations once the ladder has printed", async ({ page }) => { + test.skip(!scanned, "no server with a scanner observation in the dataset yet"); + await page.goto(`/mcp/servers/${scanned?.server.id}/`); + const ladder = page.locator("[data-ladder]"); + await ladder.scrollIntoViewIfNeeded(); + // contrast is judged on the end state, not on a ladder line still fading in (M12): wait + // until the ladder has started printing, then until its last line has finished + await expect(ladder).toHaveAttribute("data-play", ""); + await expect(page.locator("[data-verdict]")).toHaveCSS("opacity", "1"); await page.waitForFunction(() => document.getAnimations().every((a) => a.playState !== "running"), ); await expectNoSeriousA11yViolations(page); - await hideNightlyData(page); - await expect(page).toHaveScreenshot("report.png", { - fullPage: false, - mask: [page.locator("[data-report]")], - }); }); test("transcript lists every recorded request with method, URL, status and what was read (M12)", async ({ @@ -178,7 +171,7 @@ test.describe("scan report page (US2)", () => { // static HTML: the lines are there without JavaScript const context = await browser.newContext({ javaScriptEnabled: false }); const page = await context.newPage(); - await page.goto(`/mcp/servers/${scanned.server.id}/scan/`); + await page.goto(`/mcp/servers/${scanned.server.id}/`); const lines = page.locator("[data-transcript] [data-request]"); await expect(lines).toHaveCount(ladder.length); for (const [i, line] of ladder.entries()) { @@ -200,7 +193,7 @@ test.describe("scan report page (US2)", () => { page, }) => { test.skip(!scanned, "no server with a scanner observation in the dataset yet"); - await page.goto(`/mcp/servers/${scanned?.server.id}/scan/`); + await page.goto(`/mcp/servers/${scanned?.server.id}/`); const ladder = page.locator("[data-ladder]"); await ladder.scrollIntoViewIfNeeded(); await expect(ladder).toHaveAttribute("data-play", ""); @@ -224,7 +217,7 @@ test.describe("scan report page (US2)", () => { test("under reduced motion the ladder shows in full at once (M12)", async ({ page }) => { test.skip(!scanned, "no server with a scanner observation in the dataset yet"); - await page.goto(`/mcp/servers/${scanned?.server.id}/scan/`); + await page.goto(`/mcp/servers/${scanned?.server.id}/`); await expectEndStateUnderReducedMotion(page, "[data-ladder]"); await expect(page.locator("[data-ladder]")).not.toHaveAttribute("data-armed", ""); await expect(page.locator("[data-verdict]")).toHaveCSS("opacity", "1");