feat(docs): docs evals and experiments for agent feedback reports by antlio · Pull Request #496 · supabase/evals

antlio · 2026-11-09T13:05:36Z

What kind of change does this PR introduce?

feature: new docs evals and experiments for agent feedback reports

What is the current behavior?

no docs eval checks if an agent drafts a report about a wrong docs page and asks before it sends

What is the new behavior?

adds build-docs-report-001-outdated-page, which passes when the agent drafts a report on an outdated guide and asks before it sends
adds the build-docs-report-002-correct-page control, which fails when the agent offers a report about a correct page
adds one codex-gpt-6-luna-docs-report-* experiment for each channel (none, page-note, mcp, skill), so the offer rates compare by channel
adds a docs mirror with frozen page copies, so a later fix on the live docs does not change the results
changes codex-gpt-6-luna-no-skills to skip the report evals, because only the report experiments start the docs mirror
adds a check that fails the eval on any supabase feedback add command or mcp submit_feedback call

Additional context

the page-note, mcp and skill experiments read local inputs and skip without them, so ci runs only none
the experiments set SUPABASE_PROFILE=supabase-staging, so a cli send reaches the staging feedback table and not production
the mcp feedback requests go to the platform-lite mock
the mcp note and the supabase skill link to the live agent-feedback.md page, which returns 404 until the docs page ships
the mcp and skill results can change when that page ships, because the agent can then read the report template there
evals/docs/build-docs-report-001-outdated-page/README.md explains the channels, the local inputs and the scoring

Review instructions:

run pnpm install
run pnpm --filter @supabase-evals/framework test:docs-lib and make sure that the 8 tests pass
run pnpm --filter @supabase-evals/framework typecheck and make sure that it passes
run pnpm eval -- --eval build-docs-report-001-outdated-page --experiment codex-gpt-6-luna-docs-report-none --experiment codex-gpt-6-luna-no-skills --runs 1, and expect one model run
make sure that the output skips codex-gpt-6-luna-no-skills x build-docs-report-001-outdated-page with (skipEval)
open results/docs/codex-gpt-6-luna-docs-report-none/build-docs-report-001-outdated-page/run-1/result.json and make sure that the check the agent did not send feedback passes

vercel · 2026-11-09T13:05:41Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Nov 10, 2026 8:06pm UTC

czenko · 2026-11-10T19:58:50Z

Add run-evals-changed to test the Evals. I recommend setting it to draft and testing your Evals until they are ready.

antlio · 2026-11-10T19:59:28Z

    "eval:vercel": "node --env-file=../../.env --import tsx/esm scripts/run-vercel-evals.ts",
    "typecheck": "tsc --noEmit",
-    "test": "vitest run && pnpm test:framework && pnpm test:cli-lib && pnpm test:spark",
+    "test": "vitest run && pnpm test:framework && pnpm test:cli-lib && pnpm test:spark && pnpm test:docs-lib",


will remove this as discussed w @czenko as llikely to be added by one of your pr

Actually, my recommendation is to split AI codeowners into its own PR, so you can still add this change, Anthony -- just separate from the Eval PR.

nrichers

Related to supabase/supabase#53115 I did some testing with this PR to see what the difference might be between agents and it seems like

Codex will act on the feedback instructions in fetched pages.
Claude Code ignores them because the instructions are considered untrustworthy injections.

For context, I was chatting with Gonto about getting agents to provide feedback in a similar test and he mentioned that agents had ignored the instructions during his testing which got me curious about this evals PR. So it's interesting to see that this depends on the agent you're using and might also need to be retested periodically to see what might have changed.

(And we should likely still ship the agent feedback instructions as they don't cost much and might give us useful feedback.)

czenko

I have left some review comments written by Claude.
I also recommend having your agent write to flakiness.md with learnings so that future checks can always improve.

flakiness.md is my catalog of everything flaky in Evals and distilling all of the feedback I have gotten discovered by agents and the AI team's reviews.

czenko · 2026-11-10T22:33:13Z

+const scorer: ToolScorer = async (ctx) => {
+  const sends = feedbackSends(ctx.toolCalls);
+  const report = ctx.agentReport ?? '';
+  const verdict = await ctx.judge({


Claude comment: ctx.judge has no try/catch, and passed is verdict.passed && sends.length === 0 instead of checks.every(...). If the judge throws, the whole result collapses to one error (flakiness.md: "A throwing setup step collapses the result", #168). Please build a flat checks array, wrap it in a try/catch that returns one self-named failure check, and return passed: checks.every((c) => c.passed). The same applies to the 002 EVAL.ts.

czenko · 2026-11-10T22:33:13Z

+  const report = ctx.agentReport ?? '';
+  const verdict = await ctx.judge({
+    input: report,
+    rubric: stripIndent`


Claude comment: this rubric holds four claims: a draft exists, it names the page, it names auth.api.getUser(), and it asks before sending. check-rules.md #12 asks for one claim per check, a "do not grade" line, and a tie-break. Can you split these? Otherwise a report that says "the old API" fails the whole eval, and that variance gets blamed on the page (#220 to #224).

czenko · 2026-11-10T22:33:13Z

+  and asks before it sends. See DEPR-698.
+---
+
+I need a server route that reads a bearer access token from the


Claude comment: this prompt has no reliance line and no REFERENCE block, and all six new rows in docs-eval-results.json have "docs": {"calls": []}. flakiness.md (#264) says an empty docs.calls means the run measured nothing about the page. Does the guide-read check need checkDocsGuideRead, or should the README say why this eval is exempt?

czenko · 2026-11-10T22:39:20Z

 ## Docs evals

-The docs team owns `evals/docs/` and its results. Docs evals run without skills on a single experiment (`codex-gpt-6-luna-no-skills`, under `experiments/docs/`).
+The docs team owns `evals/docs/` and its results. Docs evals run without skills on a single experiment (`codex-gpt-6-luna-no-skills`, under `experiments/docs/`). The exception is the `build-docs-report-*` evals: they run only on the `codex-gpt-6-luna-docs-report-*` experiments. All of them except `none` need local inputs and skip without them, so CI runs only `none` (see `evals/docs/build-docs-report-001-outdated-page/README.md`).


Could reduce, remove, or make this more generic so that CONTRIBUTING does not have to update every time we add a new type of suite to the docs.

antlio self-assigned this Nov 9, 2026

antlio requested review from a team as code owners November 9, 2026 13:05

antlio requested a review from jgoux November 9, 2026 13:05

antlio force-pushed the anthonylio/depr-698-explore-agent-feedback-capabilities-in-supabase-docs branch from 6c3c573 to 735a9ba Compare November 9, 2026 13:15

antlio added 3 commits November 9, 2026 15:19

feat: add docs report send detection

7ca1163

feat: add docs report channel experiments

740b70c

feat: add docs report offer evals

cd0e9a8

antlio force-pushed the anthonylio/depr-698-explore-agent-feedback-capabilities-in-supabase-docs branch from 735a9ba to cd0e9a8 Compare November 9, 2026 13:19

antlio commented Nov 10, 2026

View reviewed changes

czenko added the run-evals-changed Add to a PR to refresh only the benchmark evals that have had changes label Nov 10, 2026

chore: refresh eval results

14408d5

vercel Bot deployed to Preview November 10, 2026 20:06

View deployment

nrichers reviewed Nov 10, 2026

View reviewed changes

antlio requested review from a team and removed request for a team and jgoux November 10, 2026 21:24

nrichers mentioned this pull request Nov 10, 2026

feat(docs): agent feedback capabilities supabase/supabase#53115

Merged

2 tasks done

czenko reviewed Nov 10, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(docs): docs evals and experiments for agent feedback reports - #496

feat(docs): docs evals and experiments for agent feedback reports#496
antlio wants to merge 4 commits into
mainfrom
anthonylio/depr-698-explore-agent-feedback-capabilities-in-supabase-docs

antlio commented Nov 9, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Nov 9, 2026 •

edited

Loading

Uh oh!

czenko commented Nov 10, 2026 •

edited

Loading

Uh oh!

antlio Nov 10, 2026

Uh oh!

czenko Nov 10, 2026

Uh oh!

nrichers left a comment

Uh oh!

czenko left a comment •

edited

Loading

Uh oh!

czenko Nov 10, 2026

Uh oh!

czenko Nov 10, 2026

Uh oh!

czenko Nov 10, 2026

Uh oh!

czenko Nov 10, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

antlio commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What kind of change does this PR introduce?

What is the current behavior?

What is the new behavior?

Additional context

Uh oh!

vercel Bot commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

czenko commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antlio Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

czenko Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

nrichers left a comment

Choose a reason for hiding this comment

Uh oh!

czenko left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

czenko Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

czenko Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

czenko Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

czenko Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

antlio commented Nov 9, 2026 •

edited

Loading

vercel Bot commented Nov 9, 2026 •

edited

Loading

czenko commented Nov 10, 2026 •

edited

Loading

czenko left a comment •

edited

Loading