docs: document Factory Benchmarks (GROW-6129)

[rachaelrenk]

What this feature does

Benchmarks let Warp Factories teams compare model and runner configurations on representative tasks before changing production work. This update documents creating suites, inspecting results, and storing reusable suites in code. Shipped in v0.2026.09.02.08.27.stable_01 (2026-09-02).

Summary

Adds the public documentation for the Warp Factories Benchmarks launch in GROW-6129. The page follows the factory Early Access audience, covers the dashboard workflow, and links it to the Factory definition reference.

Changes

• Adds src/content/docs/factories/benchmarks.mdx with the feature model, suite and task workflow, launch guidance, result interpretation, the current third-party harness limitation, and related pages.
• Documents benchmarks/<suite-slug>/suite.yaml and task files in src/content/docs/factories/factory-as-code.mdx.
• Updates the factory overview, dashboard, measurement page, and sidebar so Benchmarks is discoverable without duplicating the workflow.

Content design plan

Audience and JTBD: A member of a team with Warp Factories Early Access who administers a factory and needs evidence from representative work before changing an agent's model or runner.

Problem: The factory dashboard exposes Benchmarks, but the docs did not explain how to build a suite, compare configurations, inspect results, or store a reusable suite in a factory definition.

Goals:

• Create and run a benchmark suite with representative tasks and explicit correctness criteria.
• Compare baseline and candidate model or runner configurations using quality, cost, and time results.
• Apply a reviewed configuration change and preserve a production baseline.

Purpose and value: A dedicated page documents a multi-step, cost-bearing workflow that the existing measurement overview could not cover without obscuring its Scorer and Self-improvement guidance.

Content type: Feature documentation (combined) — readers need both a concise model and the procedure for creating, launching, and reviewing a benchmark.

Skill and template: draft_feature_doc / .agents/templates/feature-doc.md

High-impact scenarios:

• Covers: dashboard-managed suites from completed runs or authored tasks; model and runner comparisons; Scorers, repetitions, results, and code-managed suite files.
• Excludes: universal model claims, WarpBench internal results, cross-harness comparisons, scheduled benchmark runs, and warehouse or API analytics. Third-party harness comparison is marked as coming soon in the released UI.

Unverified claims

None — all UI labels, workflow behavior, result states, cost semantics, and definition-file fields were verified against the released Factory source and public launch material.

Documentation risk

Risk: engineering-review-required
Rationale: This PR adds a new feature page and changes published factory configuration reference content.
Source files consulted: warp-server/client/packages/factory/src/pages/FactoryBenchmarks/SuiteEditor.tsx, TaskSheet.tsx, LaunchRunDialog.tsx, and RunDetail.tsx at 17ccf4e151c87d9bca07cd46e9dccab8510f1838; warp-server/logic/factoryfile/schema/v1alpha1/benchmark_suite.schema.json and benchmark_suite_task.schema.json at the same revision.
Requested engineering reviewers: none — source-owner resolution returned no match, and no reviewer is requested during the author’s interactive review.
Engineering review status: pending
Docs override: none

Validation

• python3 .agents/skills/check_for_broken_links/check_links.py --internal-only — passed; 4,064 internal links and anchors checked.
• npm run build — passed.
• python3 .agents/skills/style_lint/style_lint.py --changed — no blocking findings; remaining output is unrecognized-term suggestions on pre-existing UI labels and the existing Early Access note.
• npm run lint — not run successfully because Trunk is not installed in this environment.

Related: https://linear.app/warpdotdev/issue/GROW-6129/benchmark-launch-docs

Co-Authored-By: Warp agent@warp.dev

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 4, 2026 9:26pm UTC

[warp-for-oss]

@rachaelrenk

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[warp-for-oss]

Overview

This PR adds a Factory Benchmarks page, links it from the factory overview/dashboard/measurement pages, and documents benchmark suite files in the factory definition reference. I found one documentation-convention issue in the new page: the web-app product surface and URL are hardcoded instead of using the repository's centralized variable system.

Concerns

• The new benchmarks page hardcodes Warp Factories web app and https://platform.warp.dev instead of the existing VARS.FACTORY_WEB_APP and VARS.FACTORY_WEB_APP_URL variables used by neighboring factory docs.

Verdict

Found: 0 critical, 1 important, 0 suggestions

Request changes

Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• hardcoded-var (1)
• missing-source-citations (1)
• ui-copy-clarity (1)

Verdict

Request changes

Review signal

[SIGNAL:pr-review] {"branch": "rachaelrenk/grow-6129-benchmark-launch-docs", "critical": 0, "date": "2026-09-04", "head_sha": "1cfe2f8cb5a59043f13b4452bac540ffca033fb4", "important": 1, "nits": 0, "pr": "685", "reviewer_login": "github-actions[bot]", "skill_used": "draft_feature_doc", "suggestions": 1, "top_categories": ["hardcoded-var (1)", "missing-source-citations (1)", "ui-copy-clarity (1)"], "verdict": "Request changes"}

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• missing-source-citations (1)
• description-length (1)
• ui-copy-clarity (1)

Verdict

Request changes

Review signal

[SIGNAL:pr-review] {"branch": "rachaelrenk/grow-6129-benchmark-launch-docs", "critical": 0, "date": "2026-09-04", "head_sha": "6bfea0667fcc91607320ed5129d94adcd2483785", "important": 1, "nits": 0, "pr": "685", "reviewer_login": "github-actions[bot]", "skill_used": "draft_feature_doc", "suggestions": 2, "top_categories": ["missing-source-citations (1)", "description-length (1)", "ui-copy-clarity (1)"], "verdict": "Request changes"}

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• source-verification-gap (1)
• frontmatter-description-length (1)
• screenshot-width-precedent (1)

Verdict

Request changes

Review signal

[SIGNAL:pr-review] {"branch": "rachaelrenk/grow-6129-benchmark-launch-docs", "critical": 0, "date": "2026-09-04", "head_sha": "faacecc7136bfd5b5b08bd8a0e59cbdae7bcf71e", "important": 1, "nits": 0, "pr": "685", "reviewer_login": "github-actions[bot]", "skill_used": "draft_feature_doc", "suggestions": 2, "top_categories": ["source-verification-gap (1)", "frontmatter-description-length (1)", "screenshot-width-precedent (1)"], "verdict": "Request changes"}

[rachaelrenk]

[style-rule-gap]
Static UI labels such as “Tasks” should use quotation marks, while interactive controls such as Add task should use bold. The style guidance covers field labels, but an explicit section-label example would make this distinction easier to apply.

[rachaelrenk]

[skill-feedback]
For procedural UI docs, verify state transitions in the live product as well as control labels. The first Benchmarks draft missed that Add task saves a new benchmark before task setup opens, and it briefly documented the alternate empty-state button instead of the persistent New button.

[rachaelrenk]

General feedback (already implemented, saved for log and future improvements)
Tighten the Benchmarks page’s opening and procedure:
• Rewrite the frontmatter description in clearer, plain language; two sentences are fine.
• Move the Early Access note before the opening paragraph.
• Define both a benchmark suite and a trial. Replace “holds” with more direct language.
• Start the procedure with “To use Benchmarks, you need a factory with an agent to evaluate,” and use active voice for the completed-run guidance.
• Document the persistent New button as the primary creation path. The empty state also has New benchmark, but that is not the primary instruction.
• Simplify task creation. Don’t format the static “Tasks” section label as an interactive UI element, and don’t enumerate self-explanatory task fields.
• Explain that clicking Add task saves a new benchmark before task setup opens.
• Shorten the baseline, configuration, repetition, cost, and run-status guidance without dropping the important behavior.
• Add the cropped benchmark editor screenshot after the name and agent-selection step, with descriptive alt text and a short caption.

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• missing-source-citations (1)
• screenshot-width-precedent (1)
• frontmatter-description-length (1)

Verdict

Request changes

Review signal

[SIGNAL:pr-review] {"branch": "rachaelrenk/grow-6129-benchmark-launch-docs", "critical": 0, "date": "2026-09-04", "head_sha": "725298d028abbd40f634cfa2b930367467b3fe83", "important": 2, "nits": 0, "pr": "685", "reviewer_login": "github-actions[bot]", "skill_used": "draft_feature_doc", "suggestions": 2, "top_categories": ["missing-source-citations (1)", "screenshot-width-precedent (1)", "frontmatter-description-length (1)"], "verdict": "Request changes"}

[rachaelrenk]

/warp-agent-review

[warp-for-oss]

@rachaelrenk

I'm re-reviewing this pull request in response to a review request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[warp-for-oss]

Overview

This PR adds Factory Benchmarks documentation, wires the page into the Factory docs navigation, and extends the factory definition-as-code reference with benchmark suite files.

Concerns

• No blocking concerns found in the attached diff.
• The supplemental security pass found no secrets, unsafe configuration defaults, dependency changes, or security-sensitive code paths in the changed documentation.
• spec_context.md reported no approved or repository spec context, so there was no spec drift to evaluate.

Verdict

Found: 0 critical, 0 important, 0 suggestions

Approve

Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• contract-completeness (1)
• technical-accuracy (1)
• screenshot-sizing (1)

Verdict

Request changes

[github-actions]

Review summary

The independent agent completed its review for this commit.

Findings

• wording-clarity (1)
• compression-contract-note (1)

Verdict

Approve