Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Do not substitute one review for the other. If the user asks for both, run and r

### 2. Read the agent checks

Read {{AGENT_CHECKS_PATH}} completely, including its top-level `instructions` and every check. Each check has an `id`, a `version`, a `severity`, and a `prompt`.
Read {{AGENT_CHECKS_PATH}} completely, including its top-level `instructions` and every check. Each check has an `id`, a `version`, a `severity`, a `docs_url`, and a `prompt`. The `docs_url` is the check's page on shopify.dev, which explains the issue and shows how to fix it.

Use separate sub-agents or isolated evaluation passes when available so each check is assessed independently and receives enough context. Determine applicability only from the check's prompt and the repository evidence it directs you to inspect. Do not force a check onto an app capability that is absent.

Expand Down Expand Up @@ -115,7 +115,7 @@ It combines {{DETERMINISTIC_FINDINGS_PATH}} with {{AGENT_FINDINGS_PATH}} into on

- CLI and ruleset versions;
- finding counts per check, grouped by severity and source;
- each verified finding's impact and concise file/line evidence;
- each verified finding's impact and concise file/line evidence, with a link to its check's `docs_url`;
- skipped or incomplete coverage and unresolved checks;
- prioritized remediation steps.

Expand Down

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import {EMBEDDED_CHECK_SOURCES} from './embedded.js'
import {redactScope, redactText} from '../rules/secret-rules.js'
import {RULE_CATALOG} from '../rules/catalog.js'
import {RULE_CATALOG, type RuleCatalogEntry} from '../rules/catalog.js'
import {
AGENT_CHECKS_SCHEMA_VERSION,
ENGINE_NAME,
Expand Down Expand Up @@ -112,13 +112,14 @@ export interface AgentChecks {
version: string
}
generated_at: string
checks: Pick<Check, 'id' | 'version' | 'severity' | 'prompt'>[]
checks: (Pick<Check, 'id' | 'version' | 'severity' | 'prompt'> & {docs_url: string})[]
instructions: string
}

const INSTRUCTIONS = `Each check below is a prompt for you to run against this
codebase. For each one, explore the repository, find real instances of what
it describes, and report them with evidence.
it describes, and report them with evidence. Each check's docs_url is its
page on shopify.dev: link it when you explain a finding to the user.

Write ONE findings document that covers every check you ran. Copy each check's
\`version\` from this file into \`check_version\`:
Expand Down Expand Up @@ -152,6 +153,13 @@ Rules:
- If you can't prove exploitability or affected authority, record the check as unresolved instead of reporting a finding.
- Don't report things you couldn't confirm — uncertainty is not a finding.`

/** registry/index.ts guarantees every agent check has a catalog entry ("Orphan agent implementation"). */
const catalogEntry = (checkId: string): RuleCatalogEntry => {
const entry = RULE_CATALOG.find((candidate) => candidate.id === checkId)
if (!entry) throw new Error(`Agent check has no catalog entry: ${checkId}`)
return entry
}

/**
* Build agent-checks.json — the prompts for the developer's agent.
* No candidates, no scan output. The agent explores independently.
Expand All @@ -164,6 +172,7 @@ export const buildAgentChecks = (engineVersion: string): AgentChecks => ({
id: check.id,
version: check.version,
severity: check.severity,
docs_url: catalogEntry(check.id).docsUrl,
prompt: check.prompt,
})),
instructions: INSTRUCTIONS,
Expand Down Expand Up @@ -431,14 +440,13 @@ function groupFindingsByCheck(

/** Check metadata as it stands in the catalog and frontmatter right now. */
function snapshotCheck(check: Check): CheckSnapshot {
// registry/index.ts guarantees every agent check has a catalog entry ("Orphan agent implementation").
const entry = RULE_CATALOG.find((catalogEntry) => catalogEntry.id === check.id)
if (!entry) throw new Error(`Agent check has no catalog entry: ${check.id}`)
const entry = catalogEntry(check.id)
return {
title: entry.title,
severity: check.severity,
description: entry.description,
...(entry.guide ? {guide: entry.guide} : {}),
docs_url: entry.docsUrl,
current_version: check.version,
// Always written, even for the default, so the stored file describes itself without the current catalog.
precedence: check.precedence,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ export {
SEVERITY_RANK,
} from './types.js'
export {groupIssues, skippedFileCounts} from './output/group-issues.js'
export {checkDocsUrl} from './rules/catalog.js'
export type {IssueGroup} from './output/group-issues.js'
export type {
AgentFindingsDocument,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ export interface CombinedCheck {
severity: Severity
description: string
guide?: string
/** The check's page on shopify.dev. Absent when the describing results file predates check docs pages. */
docs_url?: string
/** The agent snapshot's precedence, or 'union' when the agent didn't record the check. */
precedence: CheckPrecedence
/**
Expand Down Expand Up @@ -221,6 +223,7 @@ function combineCheck(id: string, {deterministic, agent, describingCheck}: Sourc
severity: snapshot.severity,
description: snapshot.description,
...(snapshot.guide ? {guide: snapshot.guide} : {}),
...(snapshot.docs_url ? {docs_url: snapshot.docs_url} : {}),
precedence,
applied_precedence: appliedPrecedence,
status: agentReplacesDeterministic
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ const checkSnapshotSchema = zod.object({
severity: severitySchema,
description: zod.string(),
guide: zod.string().optional(),
docs_url: zod.string().optional(),
Comment thread
jplhomer marked this conversation as resolved.
current_version: zod.number(),
precedence: checkPrecedenceSchema.optional(),
})
Expand Down
Loading
Loading