[aw] Claude Code User Documentation Review reported incomplete result

[github-actions]

Workflow Failure

Workflow: Claude Code User Documentation Review
Branch: main
Run: https://github.com/github/gh-aw/actions/runs/33252752408

Warning

Credential Authentication Failed: The firewall audit log detected authentication rejections (HTTP 401/403) from AI provider APIs. The following provider credentials appear to be missing, expired, or invalid:

• Anthropic Claude (ANTHROPIC_API_KEY)

To resolve this:

1. Verify that the required API keys are configured as repository secrets
2. Confirm the secrets have not expired and are still valid
3. Check that the secret names in your workflow frontmatter match the configured repository secrets

For details on configuring engine credentials, see: Engine Reference

Warning

Task Could Not Be Completed: The agent reported that the task could not be performed due to an infrastructure or tool failure.

Reasons:

• create_discussion quota (1/run) was accidentally consumed by a test/placeholder call before the real report was ready; the real discussion could not be created this run

  Executive Summary

A Claude Code-only developer (no Copilot) can complete the gh-aw quickstart end-to-end using engine: claude, ANTHROPIC_API_KEY, and manual auth — the basics are documented. But every layer past that first step is visibly thinner for non-Copilot engines: no auto-generated agent/MCP scaffolding, sparse tool-compatibility notes, and a 7th consecutive day of an unresolved auth footgun. Key finding: CLAUDE_CODE_OAUTH_TOKEN (the token from claude login) is silently ignored — first flagged in this recurring review on 2026-08-22, still present at cli.md:284.

Severity Findings (click to expand)

Critical Blockers

1. CLAUDE_CODE_OAUTH_TOKEN silently ignored, unresolved for 7 days. A Claude Code user who already ran claude login will try to reuse that token with gh aw secrets set; it's silently dropped and the run instead fails with a generic Claude CLI auth error that "never mentions the token" (cli.md:284). No troubleshooting step tells the user this is the actual cause.

Major Obstacles

1. No auto-generated agent/MCP scaffolding for Claude. gh aw init --engine copilot auto-creates .github/agents/agentic-workflows.md plus MCP wiring (.github/mcp.json, copilot-setup-steps.yml); for --engine claude this step is skipped entirely and the user is told only to "author an agent file in your own agent's format," with no template or command given (cli.md:163, cli.md:179-184).
2. Tool compatibility undocumented per-engine except for Copilot/Codex. tools.md states an explicit Codex carve-out (web-search disabled by default, tools.md:65-67) and a Claude/Codex-only timeout table (60s/120s, tools.md:178), but never says whether Claude supports/limits playwright, cache-memory, drive-memory, or qmd (tools.md:92-99, tools.md:112-123). A Claude user reading this file alone cannot determine tool availability.
3. No Claude-equivalent config example for network/firewall. architecture.mdx's only concrete network:/tools example uses engine: copilot (architecture.mdx:252-262); nothing parallel exists for engine: claude in that file.
4. Thin Claude auth walkthrough vs. Copilot's dual-path detail. Copilot's quickstart tab documents two distinct auth paths (org billing vs. PAT) in depth (quick-start.mdx:121-130); Claude's tab is 2 steps (quick-start.mdx:132-136) and defers deeper detail to an external, unreviewed auth reference page.

Minor Confusion

1. Secrets-bootstrap example commands only show --engine copilot; no parallel --engine claude example, even though it's a documented valid value (cli.md:291-296).
2. Architecture Mermaid diagram node is literally id: COPILOT despite a generic "Agent CLI" label, hinting the diagram was written Copilot-first (architecture.mdx:206).
3. tools.timeout defaults are only given for Claude (60s) and Codex (120s); Copilot/Gemini/Pi are unspecified (tools.md:178).

Engine & Tool Matrix

	Copilot	Claude	Codex	Custom
Setup	Auto-scaffolds agent file + MCP wiring via gh aw init (cli.md:159, cli.md:179-184)	Manual — init skips Copilot artifacts, no template provided (cli.md:163,183-184)	Same manual path as Claude (cli.md:177)	Undocumented beyond 1 shared snippet
Examples	114/224 (~51%)	63/224 (~28%)	46/224 (~21%)	1/224 (<1%, shared/genaiscript.md)
Auth	copilot-requests:write or COPILOT_GITHUB_TOKEN PAT, two documented paths (quick-start.mdx:121-130)	ANTHROPIC_API_KEY; OAuth token path silently broken (cli.md:284)	OPENAI_API_KEY/CODEX_API_KEY (latter wins), 2-step (quick-start.mdx:138-142)	N/A
Tool docs	Only engine with concrete network/firewall example (architecture.mdx:252-262)	No stated support for playwright/cache-memory/drive-memory/qmd (tools.md:92-123)	Only engine with explicit web-search carve-out (tools.md:67)	None
Score	5/5 — full parity, default path	3/5 — works, but real friction at every layer past basic auth	3/5 — same structural gaps as Claude, no known bug	1/5 — essentially undocumented

Parity note from example-count scan: Copilot's share is roughly double Claude's and more than double Codex's; Copilot's frontmatter also more often carries extra sub-keys (copilot-sdk, max-continuations), suggesting richer copy-paste starting points exist only for that engine.

Auth Gaps

• Claude: ANTHROPIC_API_KEY documented (quick-start.mdx:133-135, cli.md:284); CLAUDE_CODE_OAUTH_TOKEN explicitly unsupported but the failure mode is buried in one CLI-reference sentence, not surfaced as an error-message troubleshooting entry (cli.md:284).
• Codex: OPENAI_API_KEY/CODEX_API_KEY documented clearly (quick-start.mdx:138-142); no equivalent gaps found.
• Copilot: fully documented, two auth paths (quick-start.mdx:121-130, cli.md:205).
• Cross-engine: no doc states whether non-Copilot engines get any MCP auto-wiring equivalent — a Claude/Codex user must "register gh aw mcp-server" themselves with no example shown (cli.md:183-184).

Recommended Actions

Priority 1

• Fix or explicitly document the CLAUDE_CODE_OAUTH_TOKEN silent-ignore behavior — either support it or surface a clear error naming the token (cli.md:284). This is the 7th straight day this has been flagged.

Priority 2

• Add a concrete gh aw init --engine claude template/example for the agent file + MCP wiring step (cli.md:183-184).
• Extend tools.md's per-engine notes beyond the Codex-only carve-out to cover Claude explicitly for playwright/cache-memory/drive-memory/qmd (tools.md:92-123).

Priority 3

• Add a Claude-engine network/firewall example alongside the existing Copilot one in architecture.mdx:252-262.
• Add --engine claude variants next to the --engine copilot examples in cli.md:291-296.

_{Automated daily documentation review — see prior runs for trend history.}

This is a structured incompletion signal (report_incomplete), not a real task outcome. Any other safe outputs emitted alongside this signal (e.g., comments) describe the failure state, not a completed review or action.

Action Required

Assign this issue to an agent to debug and fix the issue.

Debug with any coding agent

Use this prompt with any coding agent (GitHub Copilot, Claude, Gemini, etc.):

Debug the agentic workflow failure using https://raw.githubusercontent.com/github/gh-aw/main/debug.md

The failed workflow run is at https://github.com/github/gh-aw/actions/runs/33252752408

Manually invoke the agent

Debug this workflow failure using your favorite Agent CLI and the agentic-workflows prompt.

• Start your agent
• Load the agentic-workflows skill from .github/skills/agentic-workflows/SKILL.md or https://github.com/github/gh-aw/blob/main/.github/skills/agentic-workflows/SKILL.md
• Type debug the agentic workflow claude-code-user-docs-review failure in https://github.com/github/gh-aw/actions/runs/33252752408

Tip

Stop reporting this workflow as a failure

To stop a workflow from creating failure issues, set report-failure-as-issue: false in its frontmatter:

safe-outputs:
  report-failure-as-issue: false

Generated from Claude Code User Documentation Review · claude · 190.2 AIC · ◷

• expires on Aug 30, 2026, 12:43 AM UTC