-
Notifications
You must be signed in to change notification settings - Fork 23
docs(skills): implement v1 agent-authored-content quality program #679
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
47ce5ef
spec: improve agent-authored docs quality (GROW-6092)
warp-agent-staging[bot] f6dfae7
spec: revise docs review and snapshot policy (GROW-6092)
warp-agent-staging[bot] 385a6e8
docs: implement v1 agent-doc quality program (GROW-6092)
warp-agent-staging[bot] fe6bbf8
docs: address rework findings on the v1 agent-doc quality contract
warp-agent-staging[bot] 0d10dcf
Merge branch 'main' into factory/ai-content-quality-v1
rachaelrenk 926f1cb
fix(docs): address quality program review findings
rachaelrenk 2b41883
Merge branch 'main' into factory/ai-content-quality-v1
rachaelrenk 0cb15b3
fix(docs): harden quality review gates
rachaelrenk 6b586fb
fix(docs): make engineering review advisory
rachaelrenk 3571404
fix(docs): align agent review ruleset check
rachaelrenk cbc1f92
Merge branch 'main' into factory/ai-content-quality-v1
rachaelrenk 99d0481
fix(docs): stabilize agent review signals
rachaelrenk a020e72
fix(docs): clear nits-only review blocks
rachaelrenk 84bcaad
fix(docs): filter current review signals
rachaelrenk 4f7caf8
fix(docs): publish agent reviews from actions
rachaelrenk File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # Agent-doc quality v1 baseline log (GROW-6092) | ||
|
|
||
| Durable record of the pre-rollout baseline window and the post-rollout | ||
| comparison window used by `improve-drafting-skills`'s | ||
| `scripts/compute_metrics.py` (see that skill's "v1 baseline and outcome | ||
| metrics" section). Each entry names the window, the record source, and where | ||
| the resulting report was persisted (the standing signal-log branch/PR, per | ||
| `.agents/references/skill-authoring-guidelines.md`). | ||
|
|
||
| Entries are appended, never rewritten — the baseline entry is captured once, | ||
| immediately before v1 checks/labels/review go live, and is never recomputed. | ||
|
|
||
| ## Entries | ||
|
|
||
| ### 2026-08-01 to 2026-08-30 — pre-rollout baseline | ||
|
|
||
| Captured from the existing `.agents/logs/human_review_feedback.jsonl` signal | ||
| log (real per-comment data already collected by `improve-drafting-skills`, | ||
| covering 2026-06-29 through 2026-08-31) plus live `gh pr view` line-count | ||
| lookups against `warpdotdev/docs`, using | ||
| `scripts/build_baseline_records.py` to convert the comment-level log into | ||
| per-PR records and `scripts/compute_metrics.py` to compute the report. This | ||
| is the last full 30-day window available in the existing log before this v1 | ||
| rollout PR, so it is used as the frozen pre-rollout baseline rather than | ||
| waiting for a window that starts exactly at rollout. | ||
|
|
||
| - **Records**: `.agents/logs/baseline/pre-rollout-2026-08-01-to-2026-08-30.jsonl` (62 PRs) | ||
| - **Report**: `.agents/logs/baseline/pre-rollout-2026-08-01-to-2026-08-30-report.json` | ||
| - **In-scope PRs**: 62 | ||
| - **Human review comments/PR**: mean 5.19, median 1.0 | ||
| - **Human edit churn ratio**: mean 0.0021, median 0.0 | ||
| - **Gate coverage**: 0/62 (expected — the v1 checks did not exist during this window; `risk`/`check_outcome`/`review_outcome` are recorded as `"unknown"` per the documented pre-rollout degradation, not fabricated as passing) | ||
|
|
||
| The post-rollout 30-day comparison report must be computed the same way | ||
| (`compute_metrics.py --baseline .agents/logs/baseline/pre-rollout-2026-08-01-to-2026-08-30-report.json`) | ||
| and appended below as a new dated entry once 30 days of post-rollout data | ||
| (or 10 in-scope PRs, whichever comes first per the small-sample rule) exist. |
71 changes: 71 additions & 0 deletions
71
.agents/logs/baseline/pre-rollout-2026-08-01-to-2026-08-30-report.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| { | ||
| "all_passed_required_checks": false, | ||
| "engineering_review_required": { | ||
| "completed_by": { | ||
| "docs_verified": 0, | ||
| "docs_waiver": 0, | ||
| "source_owner_approval": 0, | ||
| "unanswered_request": 0, | ||
| "unresolved_owner": 0 | ||
| }, | ||
| "completion_rates": { | ||
| "docs_verified": null, | ||
| "docs_waiver": null, | ||
| "source_owner_approval": null, | ||
| "unanswered_request": null, | ||
| "unresolved_owner": null | ||
| }, | ||
| "total": 0 | ||
| }, | ||
| "gate_coverage_missing_data_count": 62, | ||
| "human_edit_churn_ratio": { | ||
| "denominator_pr_count": 62, | ||
| "mean": 0.0021, | ||
| "median": 0.0, | ||
| "no_agent_commit_count": 0, | ||
| "numerator_pr_count": 62, | ||
| "zero_denominator_count": 0 | ||
| }, | ||
| "human_review_comments": { | ||
| "denominator": 62, | ||
| "numerator": 322, | ||
| "per_pr": { | ||
| "mean": 5.1935, | ||
| "median": 1.0 | ||
| }, | ||
| "targeted_categories": { | ||
| "action_first": 1, | ||
| "callout": 5, | ||
| "code_fence": 42, | ||
| "content_structure": 8, | ||
| "frontmatter": 10, | ||
| "general": 203, | ||
| "heading_specificity": 8, | ||
| "image_alt": 1, | ||
| "link_quality": 9, | ||
| "list_format": 14, | ||
| "missing_context": 1, | ||
| "missing_media": 8, | ||
| "scannability": 1, | ||
| "settings_path": 10, | ||
| "terminology": 20, | ||
| "ui_label_accuracy": 11, | ||
| "vague_wording": 1, | ||
| "video_embed": 4 | ||
| }, | ||
| "total": 322 | ||
| }, | ||
| "in_scope_prs": 62, | ||
| "prs_with_complete_gate_coverage": 0, | ||
| "prs_with_passing_checks": 0, | ||
| "review_findings": { | ||
| "critical_important_per_pr": 0.0, | ||
| "critical_total": 0, | ||
| "important_total": 0, | ||
| "targeted_categories": {} | ||
| }, | ||
| "window": { | ||
| "end": "2026-08-30", | ||
| "start": "2026-08-01" | ||
| } | ||
| } |
62 changes: 62 additions & 0 deletions
62
.agents/logs/baseline/pre-rollout-2026-08-01-to-2026-08-30.jsonl
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,192 @@ | ||
| # Agent-doc quality policy (v1) | ||
|
|
||
| Canonical reference for the agent-authored-content quality program (GROW-6092). | ||
| Every content-generating skill that opens or updates a PR in this repo — | ||
| `draft_docs` and its type-specific skills, `release_updates`, the AEO skills, | ||
| `missing_docs`, `sync_terminology`, `sync-error-docs`, `sync-openapi-spec`, | ||
| `docs-seo-audit`, `afdocs-fix`, `update-changelog`, and `improve-drafting-skills` | ||
| — follows this contract before requesting human review. `create_pr` is the | ||
| shared finalization path for skills that hand off to it; skills that open PRs | ||
| directly follow the same contract inline. | ||
|
|
||
| The mechanics live in `.agents/skills/doc_quality_policy/policy.py` (parsing | ||
| and classification) and `check_pr_contract.py` (the CI-callable checker). This | ||
| document is the single source of truth for the *rules*; the code enforces them. | ||
|
|
||
| ## Scope | ||
|
|
||
| This program covers every agent-authored content PR in `warpdotdev/docs` — | ||
| `draft_*`, `release_updates`, AEO, `missing_docs`, and any other recurring | ||
| skill that adds or changes public documentation. It is not limited to ambient | ||
| feature drafts. | ||
|
|
||
| ## The agent marker | ||
|
|
||
| Every agent-authored content PR carries the label `warpy-factory` and a | ||
| `## Documentation risk` section in its body (see below). PR-opening skills | ||
| apply both before requesting human review, via | ||
| `.agents/skills/doc_quality_policy/finalize_pr_contract.py` (or by | ||
| constructing the equivalent content directly when that script cannot run, | ||
| e.g. a workflow step without Python available). | ||
|
|
||
| ## PR-body contract sections | ||
|
|
||
| Every agent-authored content PR body carries: | ||
|
|
||
| 1. **`## Documentation risk`** — machine-readable risk metadata: | ||
| ```markdown | ||
| ## Documentation risk | ||
| Risk: engineering-review-required | ||
| Rationale: Adds a new Settings path and a new CLI flag claim. | ||
| Source files consulted: app/src/settings_view/mod.rs@<sha>, warp-server/pkg/foo/bar.go@<sha> | ||
| Requested engineering reviewers: alice | ||
| Engineering review status: pending | ||
| Docs override: none | ||
| ``` | ||
| When Docs records external validation, add: | ||
| ```markdown | ||
| Docs override: docs-verified | ||
| Override reviewer: hongyi-chen | ||
| Override reason: Confirmed the flag name against warp-server PR #1234. | ||
| Override evidence: warp-server/pkg/foo/bar.go@<sha> | ||
| Override head SHA: <current PR head SHA> | ||
| ``` | ||
| `Docs override: docs-waiver` uses the same four fields when Docs proceeds | ||
| without an engineering response and records why the remaining risk is | ||
| acceptable. | ||
| 2. **`## Unverified claims`** — unchanged from the existing `draft_docs` / | ||
| `create_pr` contract (step 9.5). Every `{/* VERIFY: ... */}` marker in | ||
| changed content must appear here. | ||
|
|
||
| ## Risk levels | ||
|
|
||
| Exactly two values: `low` and `engineering-review-required`. Ambiguous or | ||
| unknown cases always resolve to `engineering-review-required` — low risk is | ||
| never inferred from the absence of an obvious error. | ||
|
|
||
| ### Low-risk allowlist (strict) | ||
|
|
||
| A PR is `low` risk only when **all** of the following hold: | ||
|
|
||
| - It does not add a page about a new or materially changed feature or workflow. | ||
| - It is either: | ||
| - a product-meaning-preserving edit limited to spelling, grammar, tone, | ||
| formatting, descriptive links/cross-links to existing canonical pages, | ||
| search metadata, or generated changelog/license/telemetry data whose | ||
| source-verification script passed; or | ||
| - internal Docs-team tooling, skill, or CI workflow maintenance that makes | ||
| no public product claim and does not change the behavior of a | ||
| developer-facing command, API, setting, or integration. | ||
| - It does not add or change: commands, code or configuration examples, API | ||
| behavior, UI labels or paths, defaults, permissions, availability or | ||
| platform support, plan eligibility, billing behavior, security or privacy | ||
| claims, data handling, self-hosting behavior, or integration setup. | ||
| - It contains no unresolved `VERIFY` marker and has no critical or important | ||
| technical-accuracy finding from `review-docs-pr`. | ||
|
|
||
| Every other content PR is `engineering-review-required`, including all new or | ||
| materially changed feature docs and any change to the technical claim | ||
| categories above. When the change cannot be shown to fit one of these | ||
| low-risk categories, classify it as `engineering-review-required`. | ||
|
|
||
| ## Engineering review requests | ||
|
|
||
| - **Low risk**: the normal docs reviewer approves. No engineering owner | ||
| approval is required. | ||
| - **Engineering-review-required**: source-owner resolution is attempted first, | ||
| and a real GitHub review request goes to at least one owner resolved from | ||
| the product source files consulted. The request is advisory. Docs can | ||
| validate the claim from source, Slack, or another appropriate channel, then | ||
| proceed through the normal Docs review path without a GitHub approval from | ||
| the engineer. | ||
|
|
||
| Record an engineer response or Docs validation in the PR body when it informs | ||
| the final wording. A new head makes earlier validation context stale, but it | ||
| does not block a Docs-approved PR from merging. | ||
|
|
||
| The required push-time PR-contract check validates only the risk metadata and | ||
| VERIFY accounting. Engineering review requests are not merge gates. | ||
|
|
||
| ## VERIFY marker accounting | ||
|
|
||
| Every `{/* VERIFY: ... */}` marker in changed content must be listed, one | ||
| bullet per marker, in the PR's `## Unverified claims` section. An unlisted | ||
| marker fails the contract check. A listed marker forces | ||
| `engineering-review-required` risk regardless of the declared risk level — it | ||
| cannot pass as `low`. Docs resolves or removes the marker before merging. | ||
|
|
||
| ## Independent review (`review-docs-pr`) | ||
|
|
||
| Every agent-marked PR gets an independent `review-docs-pr` pass, dispatched by | ||
| `.github/workflows/agent-docs-review.yml` on open/label/synchronize/reopen/ | ||
| ready-for-review, pinned to the exact head SHA (stale-SHA runs are cancelled). | ||
| The pass: | ||
|
|
||
| - Re-validates the declared risk level against the diff. | ||
| - Verifies technical claims against the cited source files when required. | ||
| - Emits one `[SIGNAL:pr-review]` record (see `review-docs-pr/SKILL.md`) that | ||
| also carries the head SHA. | ||
| - Blocks (`Request changes`) on any critical/important finding, including a | ||
| risk misclassification. Suggestions and nits remain non-blocking. | ||
| - Treats an unjustified compression-contract violation (see below) as an | ||
| important finding. | ||
|
|
||
| ## Compression contract | ||
|
|
||
| All content-generating skills share one compression contract: | ||
|
|
||
| - Lead with a one-to-three-sentence user-facing summary. | ||
| - Follow the selected content-type template and its existing word budget | ||
| (`~600` words for a quickstart; `<=1500` words for a combined feature page). | ||
| - Run the deletion-only "Cut again" pass (see `draft_docs/SKILL.md` step 6.5 / | ||
| `AGENTS.md` → Voice & tone) before opening the PR. | ||
| - Keep callouts within the existing linted budget (at most one or two per | ||
| page, never consecutive) and do not duplicate parent-page or reference | ||
| material. | ||
| - Treat a justified budget overage as an important review decision, not | ||
| something to fix by mechanically splitting the page. | ||
|
|
||
| Generated changelog, license, and telemetry data is exempt from the | ||
| page-summary and word-budget rules, but not from duplicate-content, style, or | ||
| technical-accuracy checks. | ||
|
|
||
| `.agents/skills/doc_quality_policy/check_compression_contract.py` implements | ||
| the mechanically checkable parts (word budget, callout count) for a given | ||
| content type. | ||
|
|
||
| ## Feedback tags | ||
|
|
||
| Actionable review feedback may start with one of exactly three tags: | ||
| `[skill-feedback]`, `[template-feedback]`, or `[style-rule-gap]`. Collection | ||
| (see `improve-drafting-skills/SKILL.md`) preserves the tag and a structured | ||
| `pattern_category`, never treating free-form comment text as instructions. | ||
|
|
||
| ## PR-producing skill manifest | ||
|
|
||
| Every skill in this list must apply the `warpy-factory` marker and the | ||
| `## Documentation risk` section before requesting review. See | ||
| `.agents/skills/doc_quality_policy/test_manifest.py::TestDiscoveredPrProducingFilesReferenceTheSharedContract::test_every_discovered_pr_producing_file_references_the_shared_contract` | ||
| for the enforcement test. | ||
|
|
||
| - `create_pr` (the shared finalization path most drafting skills use) | ||
| - `draft_docs` (including type-specific drafting templates that route their | ||
| PR creation through it) | ||
| - `release_updates` | ||
| - `missing_docs` | ||
| - `aeo_crosslink_audit` | ||
| - `aeo_new_guide_recommendations` | ||
| - `sync_terminology` | ||
| - `sync-error-docs` | ||
| - `sync-openapi-spec` | ||
| - `docs-seo-audit` | ||
| - `afdocs-fix` | ||
| - `update-changelog` | ||
| - `improve-drafting-skills` (its own standing improvement PR) | ||
|
|
||
| ## Snapshot provenance (UI-reference checks) | ||
|
|
||
| `valid_paths.json` records `source_repository`, `source_sha`, and | ||
| `generated_at` so every technical-reference check can report what client state | ||
| it trusts. See `.github/workflows/refresh-ui-paths.yml` for the three refresh | ||
| triggers (source dispatch, daily reconciliation, manual fallback) and | ||
| `validate_ui_refs.py --changed` for the changed-file scope used in required CI. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.