Skip to content

chore(agents): add review checks and a review skill - #108

Merged
scottlovegrove merged 2 commits into
scottlovegrove/agents-3-skillsfrom
scottlovegrove/agents-4-review
Nov 5, 2026
Merged

scottlovegrove merged 2 commits into
scottlovegrove/agents-3-skillsfrom
scottlovegrove/agents-4-review

Conversation

@scottlovegrove

@scottlovegrove scottlovegrove commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Context

Part 4 of 5. Adds review checks and a review-changes skill for when you say a change is ready.

What was changed

  • Six checks in .agents/checks/: tests, architecture, security, privacy, command surface, evidence. Each says what to report and what is fine. AGENTS.md links them, so Doistbot applies them too.
  • The security check covers private channels: nothing from one is shown without --include-private-channels.
  • review-changes runs, in parallel:
    • a task review: a fresh agent that gets your request, never the author's reasoning;
    • a blast-radius review, when shared code changed (src/lib/, src/index.ts, the cli-core or SDK version);
    • Doistbot, or one agent per check without it.
  • Every finding is fixed, dismissed with evidence, or left open for you.
  • npm run review:worktree -- <dir> gives a reviewer its own copy of the change, so the shared checkout stays untouched. It works with a target inside the checkout and ignores textconv drivers.
  • Before any review, every untracked file is checked for credentials and personal data; gitignored files are never sent.

Please check

  • Whether the checks match what you want reviewed.

Proof

  • Tests for the worktree script (diffs, binary files, untracked files, cleanup on failure, paths with spaces) and for the shape of every check.
  • One real npm run review:worktree run, removed cleanly afterwards.

Not verified

  • Doistbot inside review-changes: the skill needs Doistbot 1.0.12 or newer, and the version I have is older, so it falls back to per-check subagents.

@scottlovegrove scottlovegrove added the 🚢 Ship PR requires passing checks before merge, but no independent human review. label Nov 5, 2026
@scottlovegrove scottlovegrove self-assigned this Nov 5, 2026

@doistbot doistbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds six review checks, a review-changes skill, and a review:worktree script for giving reviewers isolated copies of a change. Few things worth tightening:

  • The list-output check says ids go after the name, but existing list commands print the id first (workspace.ts, channel/list.ts, channel/members.ts) — either update those commands or reword the check as a new-rule-for-new-output convention.
  • Exclude the destination worktree from the untracked-file copy (capture untracked paths before creating the worktree, or filter it out), since a destination inside the checkout currently aborts setup.
  • Disable textconv when generating the patch (--no-textconv) so configured drivers (e.g. for PNGs) don't produce patches git apply can't reproduce; a test would pin this.
  • Fix paths that will fail the instructions.test.ts broken-reference check: skills/comms-cli/SKILL.md and oauth/client-metadata.json in security.md resolve relative to .agents/checks/ — add them to ROOTED or write them repo-root-relative.
  • Also worth addressing: untracked files go to reviewers and LLM providers unmasked before the privacy check runs, so secrets/PII (.env, logs, real user fixtures) can leak — filter or exclude them before any review step.

I also included a few optional follow-up notes in the details below.

Optional follow-up notes (5)
  • P3 .agents/checks/privacy.md:16: Alan Grant and Ellie Sattler aren't cli-core fixtures; they're defined in this repo's src/lib/__fixtures__/accounts.ts. Point reviewers at the local fixtures so the "Fine" list doesn't send them looking in the wrong package.
  • P3 .agents/scripts/lib/review-worktree.ts:47: The cleanup path ignores the result of git worktree remove --force and then unconditionally rmSyncs the target. If removal fails (for example, a locked worktree), the files are still deleted while the repository keeps its worktree registration, leaving stale metadata and breaking the "retry is not refused" guarantee in the comment. Check the result and skip rmSync (or surface the removal failure) when removal did not succeed.
  • P3 .agents/scripts/lib/review-worktree.test.ts:98: The test name promises a checkout whose path contains a space, but the test never establishes that repo.dir has a space; it depends on createTempRepo's hardcoded 'agent tools-' prefix. If that prefix changes, the test silently stops covering the scenario it describes. Add an explicit expect(repo.dir).toContain(' ') (or build the temp repo under a path with a space).
  • P3 docs/agents.md:87: The blast-radius trigger is now restated in review-changes/SKILL.md, here in docs/agents.md, and in .agents/checks/evidence.md, with different details: only the skill includes argv handling in src/index.ts and build/CI config. Reviewers reading different documents will apply different criteria, and the lists will drift. Point to the skill as the single definition, or make the three lists match.
  • P3 .agents/scripts/lib/review-worktree.test.ts:86: The skip guard only covers root. On Windows, chmodSync(path, 0o000) drops the write bit but the file stays readable, so createReviewWorktree succeeds and this test fails with no error thrown. Extend the condition to process.getuid?.() === 0 || process.platform === 'win32' (and mention Windows in the comment) so the test skips where it can't run rather than failing.

Share Feedback • Review Logs

Comment thread .agents/checks/command-surface.md Resolved
Comment thread .agents/scripts/lib/review-worktree.ts Resolved
Comment thread .agents/scripts/lib/review-worktree.ts Resolved
Comment thread .agents/checks/command-surface.md Resolved
Comment thread .agents/skills/review-changes/SKILL.md Resolved
@scottlovegrove
scottlovegrove force-pushed the scottlovegrove/agents-4-review branch from c456861 to 74fe46a Compare November 5, 2026 11:22
@scottlovegrove
scottlovegrove force-pushed the scottlovegrove/agents-4-review branch from 74fe46a to c2e0de1 Compare November 5, 2026 11:27
Adds six review checks in .agents/checks/: tests, architecture,
security, privacy, command surface and evidence. Each says what to
report and what is fine. AGENTS.md links them, so Doistbot applies them
to pull request reviews too. The security check covers private
channels, and the architecture check the batch helpers in
src/lib/api.ts.

The review-changes skill runs, in parallel, a task review (a fresh agent
that gets the owner's request, never the author's reasoning), a
blast-radius review when shared code changed, and Doistbot or one agent
per check without it. Every finding is fixed, dismissed with evidence,
or left open for the owner.

npm run review:worktree -- <dir> gives a reviewer its own copy of the
change, so the shared checkout stays untouched.
…h textconv

createReviewWorktree reads the diff and the untracked files before the
worktree exists, so a target inside the checkout is no longer copied
into itself, and diffs with --no-textconv, so a configured textconv
driver can't produce hunks git apply rejects.

The reviewer brief names the untracked files explicitly, since git diff
leaves them out, and review-changes asks for every untracked file to be
checked for credentials and personal data before any review runs.

The architecture and review rules no longer mention batch helpers that
no code has, and the command surface check no longer asks for names
before ids in lists, which the existing list commands don't do.
@scottlovegrove
scottlovegrove force-pushed the scottlovegrove/agents-4-review branch from c2e0de1 to b9b4b22 Compare November 5, 2026 11:53
@scottlovegrove
scottlovegrove added this pull request to stack #110 November 5, 2026 11:53
@scottlovegrove
scottlovegrove merged commit 0fc5a8d into main Nov 5, 2026
6 checks passed
@scottlovegrove
scottlovegrove deleted the scottlovegrove/agents-4-review branch November 5, 2026 11:55
@doist-release-bot

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 3.10.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

released 🚢 Ship PR requires passing checks before merge, but no independent human review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants