[Coly010]

Current Behavior

19 of the 25 cli evals already set area and journeyPhase in their PROMPT.md frontmatter, but the six original local-stack evals don't. evalFrontmatterSchema also strips both keys, so none of them reach the exported results or Braintrust, and the coverage grid and dashboard have nothing to slice by.

Expected Behavior

• The six local-stack evals are tagged area: local-stacks. The four build-* evals are journeyPhase: design and the two resolve-* evals are journeyPhase: resolve. Every cli eval now carries both keys.
• area and journeyPhase are optional, closed-set fields on the eval metadata schema, normalised the same way as stage/topic:
  • journeyPhase: the KPI 2 phases design, deploy, observe, detect, resolve. These don't match stage, which is why it's a separate field.
  • area: the 10 CLI areas from the coverage grid.
• Both are passed through to eval-results.json (via toEvalResult), to Braintrust row metadata as area / journey_phase, and back out of fetch-braintrust.ts.
• Evals without the keys are unchanged.
• The "the schema strips them" notes are removed from three eval READMEs, since they're no longer true.

A couple of things I'd like a second opinion on from the AI team, since this touches packages/core and apps/framework:

1. I made area a closed enum like topic, because it catches typos. It's CLI-specific at the minute though, so if another suite wants areas later it'd need extending. Happy to loosen it to a string, or rename it to something like cliArea, if you'd prefer.
2. This is the "official optional journeyPhase field" we asked about. If you'd rather own the shape of it, I'm more than happy to adjust 🙂

The new fields will only show up in Braintrust and the published results once this merges and the next CLI run uploads.

Related Issue(s)

Fixes CLI-2700

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
evals	Ignored	Preview	Nov 12, 2026 9:48am UTC

[jgoux]

Thanks for this. The six local-stack evals are tagged as described, and all 26 evals/cli prompts now carry enum-valid area/journeyPhase values. No other suite sets these keys, so nothing else starts failing validation. The fields reach eval-results.json and Braintrust metadata, and the core and framework tests for this change pass, as does pnpm -r typecheck.

There's one small inline consistency note; it doesn't block merging. One FYI: fetch-braintrust.ts only assembles benchmark-suite rows today, so its new area/journey_phase mapping won't show up for CLI evals until that fetcher covers the cli suite.

On your two questions: a closed area enum fits how topic works, and widening it when a second suite needs areas seems fine. A separate journeyPhase field next to stage makes sense because the two sets don't line up.

[jgoux]

P3 (non-blocking): these two fields only come from promptData, but stage/product/topic/suite/interface above all fall back to the recorded result. The result file does record these tags now: run-eval.ts writes ...ev.metadata into result.json, and rawEvalResultSchema keeps area/journeyPhase.

So if a saved result is exported after its PROMPT.md is deleted or renamed (readPrompt() returns undefined), its tags are dropped silently while its stage and topic survive. The same gap exists in upload-braintrust.ts (area: promptData?.area, journey_phase: promptData?.journeyPhase).

The normal CI path isn't affected, because export and upload run in the same checkout as the eval run. Suggest matching the neighbouring fields for consistency:

area: promptData?.area ?? parsedResult.area,
journeyPhase: promptData?.journeyPhase ?? parsedResult.journeyPhase,

(and ?? result.area / ?? result.journeyPhase in the upload row), plus an export test where only the recorded result carries the tags.

[Coly010]

good shout, thanks! both now fall back to the recorded result in export-results.ts and upload-braintrust.ts, with an export test where only the result carries the tags