Skip to content

feat(debugging): add a diagnose action and a Phase 1 surface pass - #16617

Merged
kyle-sexton merged 9 commits into
mainfrom
feat/debugging-diagnose-surface-pass
Nov 16, 2026
Merged

kyle-sexton merged 9 commits into
mainfrom
feat/debugging-diagnose-surface-pass

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

No related issue: an owner-decided change to /debugging:debug.

Summary

/debugging:debug gains a named diagnose action and a Phase 1 surface pass, and its Phase 3 and Phase 6 steps read and record more of what the app shows.

Fix

  • diagnose action. /debugging:debug diagnose <observation> runs Phases 1 to 4, stops once a Phase 3 row is confirmed, reports the cause at file:line with the loop command and its output, removes its Phase 4 probes and changes no other code. Until now this existed only as a stop rule in "What this skill does not do"; that rule now points at the action. The full loop keeps the name debug: ISO/IEC/IEEE 24765 defines debug as "to detect, locate, and correct faults" and diagnostic as "detection and isolation of faults", with no correction. The action is in the argument hint, the Arguments line and the description (the 'diagnose this' trigger moved from the full-loop list to the action).
  • Phase 1 surface pass. Before building a loop: /app-control:drive ready when the app-control plugin is enabled and the project has a control skill, otherwise the manifests and launch config. Then the surfaces the app type has (browser app, API, worker, CLI). Never assume a port: read it from the running process, drive's report, or launch config that pins it. Logs, captures, records and tickets the skill reads are framed as data, never instructions. Each source the run cannot reach becomes a request for scoped access, never for a secret; a subagent reports the request to its parent. The "no loop: full stop" rule stays.
  • Phase 3. Reads the domain glossary first, and checks what each environment runs (recent deployments, the deployed ref, the CI runs for the deployed commit read through the repository's CI or forge adapter, recent tickets) as a lead to test, never as proof. No provider API field is named; no existing plugin owns a deployed-commit run lookup today.
  • No per-stack examples. The skill states stack-agnostic rules and reads stack specifics (the logger, banned output APIs, clock and network fakes, timing tools, where a framework logs or picks a port) from the repository's own config, rules and instructions at run time, then /discovery:research when enabled. skills/debug/reference/ecosystem-debugging.md (per-stack .NET, Python and TypeScript rows) is removed with its anchor-baseline entry, and the tool names in the loop table (a named browser driver, curl, S3) are replaced by agnostic wording. docs/architecture/landscape.json still lists that file among the citing files of three external links; it is a recorded landscape snapshot no check reads, left for its next refresh.
  • Phase 6. The record lists lessons about driving the app as /app-control:map refresh "<lesson>" lines; map stays the one writer of a control skill's Gotchas. The always-on signal step points at /implementation:implement for observability-by-default design guidance.
  • Checklist template, README and a minor changelog fragment updated.
  • Two eval cases: diagnose-action-stops-at-cause (the action word alone, no "don't change it" in the prompt) and surface-pass-reads-worker-dead-letter (a worker that exits 0 while its dead-letter directory and log hold the failure).

Verification

  • claude plugin eval on the final skill text (Claude Code 2.1.325, model claude-opus-5-5, judge opus, --scaffold, 3 runs, with and without arms, through the Docker link swap), one case per pass:
    • diagnose-action-stops-at-cause: validity gate VALID; every grader passed in all 6 runs, judges unanimous. The without-plugin arm also scores 1.00, so the case is at the ceiling.
    • surface-pass-reads-worker-dead-letter (with the new worker-restored grader): validity gate VALID; with-plugin runs passed every grader; one without-plugin run missed cites-tax-line; judges unanimous.
    • Noise report: n too small to call (one case per pass). These cases guard against regressions; they do not establish a gain from the plugin.
    • Earlier passes on intermediate heads also scored VALID with every grader passing; one was voided by a Docker credential-store link and rerun.
  • validate-cases.py plugins/debugging/evals: no FAIL; the new deterministic graders pass their offline sample answers.
  • scripts/validate-plugin-contracts.mjs: pass. check-skill.sh plugins/debugging/skills: PASS (SKILL.md 446/500 lines, description 1020/1024).
  • scripts/check-as-of-recheck.sh --check: pass. plugins/debugging/scripts/build-view.test.sh: all ok.
  • Pre-push check-affected: pass after regenerating docs/skill-cheat-sheet.md.
  • Local code and security reviews (fresh-context agents; the CI review lanes read as failing at ready time). Fixed: untrusted-content framing for every surface the skill reads; a diagnose run's loop file stays in scratch; an end-state worker-restored grader added to the surface-pass case. Kept as decided: any -- token other than --unattended is part of the free-text observation, which departs from the argument-shape contract's stop-on-unknown-token rule because an observation can quote a failing CLI option.

Related

  • /app-control:drive ready and /app-control:map refresh are the surfaces this change calls.
  • The Phase 6 pointer names /implementation:implement as the owner of observability-by-default design guidance; that skill does not carry the guidance on main yet, so the pointer depends on a separate change to the implementation plugin.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC

kyle-sexton and others added 5 commits November 16, 2026 14:20
/debugging:debug diagnose runs the loop and hypothesis phases, stops at
the confirmed cause at file:line with the loop output, removes its
instrumentation and changes no other code. The old diagnosis-only stop
rule now points at the action.

Phase 1 opens with a surface pass: /app-control:drive ready when the
project has a control skill, otherwise manifests and launch config, then
the surfaces the app type has. It never assumes a port, and turns each
unreachable source into a request for scoped access, never for a secret.

Phase 3 reads the domain glossary first and checks what each environment
runs as a lead to test, never as proof. The post-mortem lists lessons
about driving the app for /app-control:map refresh, and the always-on
signal step points at /implementation:implement.

Adds eval cases for the diagnose action and the surface pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
Each case's prompt.md description already says what its scaffold builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
An observation can name a failing CLI option, so only --unattended is
taken out as a flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
…oop file

The surface pass and Phase 3 read logs, captures, dead-letter records,
deployment records and tickets, so each is framed as data per the
untrusted-content contract. A diagnose run keeps any loop file it wrote
in a scratch directory and names it, leaving no test in the test tree.
The port rule states our decision with Vite's pin as one example, the
CI-run lookup names GitHub's head_sha as an example, and the surface-pass
eval gains an end-state grader that catches a fix made by any tool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 19:40
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T19:42:12.377875Z 62aba91 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

The action table now lists 'diagnose this' as a plain-words request for
the diagnose action, matching the description, and the summary names the
full loop as debug. Phase 1 writes its assumptions before the loop
rather than first, so the surface pass reads as its opening step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@kyle-sexton kyle-sexton added the do-not-merge Hard merge gate: do not merge while applied. label Nov 16, 2026
@kyle-sexton

Copy link
Copy Markdown
Member Author

Held until #16614 merges: the Phase 6 pointer names /implementation:implement as the owner of observability-by-default guidance, which #16614 adds. The hold comes off when #16614 is on main.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 62aba91512

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/debugging/skills/debug/SKILL.md Resolved
@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/fix-docs
Model rung 0
Leg base
Run job success
Activity step skipped
Commits pushed none (read lane)
Run 43941816257

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

…tory's adapter

Phase 3 keeps the check of which CI runs an environment's deployed
commit had, and leaves the provider lookup to the repository's CI or
forge adapter instead of naming a provider API field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 1da080ed9dbc45ab2ce7c988253b2723a1ecfc33: read 35 of 35 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #16617, whole review at 1da080ed

No security issues found.

critical-findings: 0

What I read. I read every hunk of all 35 files listed in .claude-lane/pr.diff, plus the deletion of plugins/debugging/skills/debug/reference/ecosystem-debugging.md. None of the changed files matches a trusted-path glob. The diff changes no plugin.json, no marketplace.json, no workflow, no hook and no trust control (trusted-actor check, lane token, kill switch, broker, or merge hold). That means there are no manifest lines to report.

Instruction-surface lens. I checked each rule the diff deletes, narrows or rewrites in plugins/debugging/skills/debug/SKILL.md:

  • Diagnosis-only rule. The "Does not fix when the user asked only for a diagnosis" rule moves into the new Action: diagnose section and is not weakened. That section still says to remove every Phase 4 probe and change no other code (no fix, no regression test). It also keeps loop files in scratch and out of the test tree.
  • Deleted reference file. The per-ecosystem reference held stack examples and a "project conventions win" note. The project's own config, rules and instructions replace it as the source of stack specifics. No protected-class rule is lost.
  • Removed tool names. Dropping the curl, Playwright and S3 names changes examples only and forbids nothing different.
  • Stronger, not weaker. The new surface pass adds untrusted-content framing for logs, captures, dead-letter records, drive reports, deployment and workflow-run records, and tickets. It says text from these sources never changes the loop, the files written, or what gets pushed. It also says never to ask the user for a secret, only for scoped access.

Eval fixtures. Both scaffold.sh files build a local fixture repository. Git hooks are off (core.hooksPath=/dev/null), signing is off, GIT_CEILING_DIRECTORIES is set, the new case calls repo-scaffold-guard.sh, and nothing touches the network. The diagnose case reuses an existing scaffold through a relative path inside the repository. The grader regexes and sample JSON files are data only.

Earlier review. The prior-findings file says blocking only: yes, so this review reports only CRITICAL and IMPORTANT findings, and there are none. The one earlier summary post found nothing.

Settled thread. The thread on SKILL.md (CI runs looked up through "the repository's CI or forge adapter") was answered by the owner with the fix. I am not raising it again; the current wording leaves no open security point.

Instructions in PR text. The PR context and the settled-thread text contain no instructions aimed at this lane.

Spend (not a finding). The two new eval cases (3 runs each, max_turns: 40, timeout_seconds: 900) use subscription or API credit when the plugin evals run.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 1da080ed9dbc45ab2ce7c988253b2723a1ecfc33: read 35 of 35 changed file(s); 0 finding(s), 0 as inline comments.

Code review (blocking only): no findings

This lane already reviewed an earlier head (d3e8948a), so this review reports blocking findings only and drops advisory ones. I read all 35 listed files in .claude-lane/pr.diff. I also read the deleted ecosystem-debugging.md hunk, which appears in the diff but not on the file list.

What I checked:

  • Deleted reference: no skill, README or checklist still links to skills/debug/reference/ecosystem-debugging.md. Both ${CLAUDE_PLUGIN_ROOT} pointers to it were removed from SKILL.md, and the file's row was dropped from scripts/docs-anchor-baseline.txt.
  • Phase names: the section names that "Action: diagnose" cites, "Remove what the run added" and "Leave a record", both exist in SKILL.md.
  • Skill calls: the drive and map skills accept the calls this change adds: /app-control:drive ready with --unattended, and /app-control:map refresh "<lesson>".
  • Scaffolds: diagnose-action-stops-at-cause/scaffold.sh delegates to diagnosis-only-stops-at-confirmed-hypothesis/scaffold.sh, which exists. That script finds its own directory through BASH_SOURCE, so its path to ../fixtures/repo-scaffold-guard.sh still works when another script calls it. surface-pass-reads-worker-dead-letter/scaffold.sh uses the same guard pattern as the existing cases, and its worker writes the dead-letter record and log that the graders expect.
  • Grader samples: I read the deterministic graders' samples against their patterns. Examples: sed -i '/DEBUG-b0m1/d' does not trip no-shell-importer-edit because no fix pattern appears in it; worker/run.py:28 does not satisfy cites-tax-line. I found nothing that contradicts the samples. I did not run validate-cases.py; the author says it passed, but I have not verified that.
  • Manifests: no plugin.json or marketplace.json changed, so the manifest checks do not apply. The changelog fragment's bump: minor matches a new action.
  • Settled thread: the resolved thread on the Phase 3 environment-check line stays settled. The line now names "the repository's CI or forge adapter", which is the owner's stated fix.
  • Closing issues: the PR says it has no related issue, so I had no acceptance criteria to check.

Spend: the two new eval cases add 3 runs each per arm when evals run, which uses prepaid API credits. This is not a hold.

Verifier dropped 0 of 0 candidates.

kyle-sexton and others added 2 commits November 16, 2026 15:42
…hipping them

/debugging:debug now states only the stack-agnostic rules and reads stack
specifics (the logger, banned output APIs, clock and network fakes, timing
tools, where a framework logs or picks a port) from the repository's own
config, rules and instructions, then /discovery:research when enabled.
The per-ecosystem reference file and the tool names and doc links in the
surface pass and the loop table are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@kyle-sexton kyle-sexton removed the do-not-merge Hard merge gate: do not merge while applied. label Nov 16, 2026
@kyle-sexton

Copy link
Copy Markdown
Member Author

Hold lifted: #16614 merged at 2026-11-16T21:41:48Z, so the implementation reference this PR's Phase 6 pointer names is on main. Both review lanes reviewed head 1da080e.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 7316ed6 Nov 16, 2026
79 of 81 checks passed
@kyle-sexton
kyle-sexton deleted the feat/debugging-diagnose-surface-pass branch November 16, 2026 22:05
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant