diff --git a/.github/workflows/issue-triage.yml b/.github/workflows/issue-triage.yml index f0e988b..ccbfe81 100644 --- a/.github/workflows/issue-triage.yml +++ b/.github/workflows/issue-triage.yml @@ -253,6 +253,9 @@ jobs: # Live progress comment, so a run in flight is visible. Silence reads # the same whether the job is working or broken. track-progress: 'true' + # An issue triaged before keeps its old state, so the check below + # cannot tell a skipped run from a finished one. This can. + fail-on-self-skip: 'true' prompt: | Triage issue #${{ inputs.issue-number || github.event.issue.number }} in ${{ github.repository }}. Mode: ${{ needs.gate.outputs.mode }} @@ -486,10 +489,9 @@ jobs: ## Finally - Remove the `agent:triage` label if it is present. It means "triage - requested / in progress"; leaving it on makes a finished issue look - like it is still being worked. Do NOT add it yourself. It is an - input, not a record of what you did. + Leave the `agent:triage` label alone. A step after you removes it + on every path. Do NOT add it yourself. It is an input, not a + record of what you did. Keep the comment tight. No preamble, no restating the issue back. @@ -501,7 +503,7 @@ jobs: # The read is repo-scoped on purpose. It must use the same token the agent # had, or it would pass where the agent failed. No `if:`, so it runs only # when the agent step succeeded, which is the one case where a green job - # would otherwise mislead. + # would otherwise mislead. A self-skip already failed in run-agent. - name: Verify triage recorded its state env: GH_TOKEN: ${{ github.token }} @@ -577,3 +579,25 @@ jobs: token: ${{ steps.org.outputs.token }} authors: claude[bot] ${{ steps.org.outputs.login }} blocker-repos: ${{ inputs.blocker-repos }} + + # `agent:triage` is a REQUEST, never a record, so it goes on every path: + # an agent that never ran or failed cannot remove it, and a label left on + # cannot be applied again, which is the only way to ask twice. Removing + # it with github.token cannot loop, because GitHub starts no workflow run + # from GITHUB_TOKEN activity. A cancelled run clears it only if the label + # started it: `cancel-in-progress` may have handed the issue to a newer + # run that the label requested, while a timeout or a manual cancel of a + # labeled run must still free the request. + - name: Clear the request label + if: always() && (!cancelled() || github.event.action == 'labeled') + env: + GH_TOKEN: ${{ github.token }} + REPO: ${{ github.repository }} + NUM: ${{ inputs.issue-number || github.event.issue.number }} + run: | + # Percent-encoded: the colon is part of the label name. A 404 is a + # label that was not there; any other failure leaves the request on. + if ! err=$(gh api -X DELETE "/repos/$REPO/issues/$NUM/labels/agent%3Atriage" 2>&1 >/dev/null) \ + && [[ $err != *"HTTP 404"* ]]; then + echo "::warning title=agent:triage not removed::#$NUM keeps agent:triage, so applying it again will not re-run triage: ${err//$'\n'/ }" + fi diff --git a/AGENTS.md b/AGENTS.md index 2b821a6..a2222ca 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,7 +47,7 @@ why, not history. | `.github/workflows/deps-verify.yml`, `actions/push-round/` | Verification never merges or approves. Its fix rounds are pushed only by `push-round`, never by the agent: at most two, one commit each, by the fix author, code plus companion versions in setup-stack's `version-files` (each recorded, carried as `Deps-Verify-Bump:`), never a line Renovate's commits added, never a hand-edited lockfile. Lockfiles come from setup-stack's `lock` command, run before the App token is minted. A repo whose Renovate config does not ignore the fix author gets no push | Evidence is only useful if it is allowed to be wrong; merging is deps-automerge's call, from the labels set here. A blanket manifest ban left updates stuck on the companion bump their own notes asked for. The agent is told the rules, not bound by them, so the action checks them. Renovate stops rebasing a branch an unignored author touched. `push-round/test.sh` covers every refusal; `run-agent/test.sh` the publish, actor and renovate steps. | | `.github/workflows/deps-verify.yml` | The label and the comment are written by the job from the agent's files, never by the agent. A red build that was not fixed is `deps:needs-manual`, and a branch with fix commits is `deps:fixed`, never `deps:verified` | An agent that labels its own verdict left two labels on one PR and none on another. A later automerge rule must be able to tell a clean bump from one a machine edited. A `deps:needs-manual` that the PR's own author applied (a Renovate package rule asking for manual review) is never removed, because the last `labeled` event for it names that author and the job's labels come from the App or `GITHUB_TOKEN`. | | `actions/run-agent/` | With `report-file`, the report exists and passes `report-valid` after every run: run-agent writes `report-fallback` plus a `why` (the workflow-file mismatch it checks before starting the action, the turn cap, or the error the action hides) whenever the agent left none. Callers word `why`; they never parse the execution file | "The agent wrote no report" told nobody what to do, and claude-code-action hides both its errors and its own skip. `test.sh` runs `validate`, `outcome` and `report`. | -| `actions/run-agent/`, `actions/pick-model/prices.tsv` | Every run that reached the agent writes ONE `Agent run` notice, a failed run with no result included. Its keys only grow (`stage model turns cost priced wall`, then `pr` or `issue` when the event names one, then the rule in parentheses). `cost` is the tokens in `modelUsage` times `prices.tsv` (`priced=table`); a model with no row falls back to the reported cost (`priced=reported`) with a warning; anything else is `cost=?` and `priced=none`, never 0. A run the action skipped itself on is not a run and writes no notice. Every model `pick.sh` can choose has a row in `prices.tsv`, which lives beside the ids | The factory metrics read these notices, so a run with no notice is missing spend, not zero spend, and a renamed key breaks every reader. The reported cost depends on the CLI version and the login type; one list prices every run the same way. `prices.tsv` sits in pick-model because a model bump and its price are one change. `test.sh` runs `outcome` with priced, fallback, unpriced and resultless runs, and checks the rows. | +| `actions/run-agent/`, `actions/pick-model/prices.tsv` | Every run that reached the agent writes ONE `Agent run` notice, a failed run with no result included. Its keys only grow (`stage model turns cost priced wall`, then `pr` or `issue` when the event names one, then the rule in parentheses). `cost` is the tokens in `modelUsage` times `prices.tsv` (`priced=table`); a model with no row falls back to the reported cost (`priced=reported`) with a warning; anything else is `cost=?` and `priced=none`, never 0. A run the action skipped itself on is not a run and writes no notice, only an `Agent never ran` warning, `self-skipped=true`, and a failure with `fail-on-self-skip`. Every model `pick.sh` can choose has a row in `prices.tsv`, which lives beside the ids | The factory metrics read these notices, so a run with no notice is missing spend, not zero spend, and a renamed key breaks every reader. The reported cost depends on the CLI version and the login type; one list prices every run the same way. `prices.tsv` sits in pick-model because a model bump and its price are one change. `test.sh` runs `outcome` with priced, fallback, unpriced and resultless runs, and checks the rows. | | `actions/pr-facts/`, `actions/app-token/` | Every workflow that resolves a PR from an event, or checks it is still the same open PR at the head it acted on, uses `pr-facts`. Every App token comes from `app-token`, which returns empty when no App is configured | One copy of "which PR, and is it still that PR" and of the mint. deps-verify checks out the head SHA and pushes or publishes only while `pr-facts` still finds the PR open at it. `pr-facts/test.sh` covers each refusal. | | `.github/workflows/deps-verify.yml` | The build verdict is the caller's `pr-checks` run on the head commit. The job builds only when none of the repo's workflows calls `ci-workflow`, and re-runs a `pr-checks` run that failed on the runner once, failed jobs only | A second build of the same commit beside `pr-checks`, on the same runner hosts, ran both out of memory, so a fine update came out red twice over. An exit 137 says nothing about the update. More than one retry would hide a real capacity problem. | | `.github/workflows/deps-verify.yml` | Nothing waits for pr-checks. The caller also triggers on `workflow_run` (PR Checks, completed), and the `gate` (light pool of a private repo, hosted for a public one or a fork PR) reads the finished run once: no run yet, still running, cancelled, or just re-run after an infra failure all stop the run, and that run's completion starts the next one. There are exactly two jobs, `gate` and `verify`. `verify` runs on light for a green verdict and takes the heavy pool only when the verdict is not green or the repo builds here | A job that polled pr-checks held a runner for up to 45 minutes. On heavy it sat idle for 5 to 15 minutes on every green Renovate PR and took a slot from the builds it waited on; on light it deadlocked, because with as many open dependency PRs as light runners every slot held a waiting job and the pr-checks jobs could never start. Hosted polling would cost hosted minutes for nothing, and self-hosted capacity is not for sleeping. The gate resolves the PR from either event through the API and drops forks, closed PRs and a run for a commit the PR has moved past. A caller without the `workflow_run` trigger never gets verified; the onboarding skill's static checks catch that, so the gate does not re-read the caller's file at run time. `Self test` asserts the two jobs, no `sleep`, and the gate's proceed rule. | @@ -207,6 +207,17 @@ why, not history. agent mode instead, so a reopened issue or a manual dispatch is triaged rather than reported red. Narrow either list if upstream narrows; never widen it. +- **Tag mode re-checks the trigger even with a prompt.** On `issues/labeled` it + compares the label to `label_trigger`, default `claude`, and on a mismatch it + logs "No trigger found", skips itself and reports success. `issue-triage` + runs in tag mode for its progress comment, so `agent:triage` did nothing, + green. `run-agent` passes the routed event's own label when a prompt is set, + because the gate already chose it. Agent mode does not check. +- **A self-skip is a green step.** `run-agent` warns `Agent never ran`, sets + `self-skipped=true`, and fails with `fail-on-self-skip: 'true'`. `issue-triage` + sets it, because an issue triaged before keeps its old state and its own check + would pass. An edited or deleted comment is not a self-skip: the action + refuses it by design. - **Agent mode starts the inline-comment MCP server only when its tool is named in the allowlist.** `mcp__github_inline_comment__create_inline_comment` must be in `allowed-tools`, or the server never starts and the tool does not exist. @@ -215,9 +226,10 @@ why, not history. was NOT provided, stop here". The review depth this repo computes travels in the system prompt, because an extra word on the slash command is discarded. - **A label already present emits no `labeled` event.** So a request label must - be cleared when the job finishes, or asking again does nothing. `pr-review` - clears `agent:review`, `issue-triage` clears `agent:triage`, and - `issue-decompose` clears `agent:decompose`. + be cleared when the job finishes, or asking again does nothing. A step clears + it on every path, never the agent, which cannot when it never ran. `pr-review` + clears `agent:review`, `issue-triage` clears `agent:triage`, `issue-implement` + clears `agent:implement`, and `issue-decompose` clears `agent:decompose`. - **The org runners API says busy and offline separately.** `status` is `online` or `offline`, and `busy` is a per-runner boolean that is `false` on an offline runner. A pool is available when a runner is `online` and not `busy`, diff --git a/README.md b/README.md index bbcfd7f..e083e28 100644 --- a/README.md +++ b/README.md @@ -399,7 +399,7 @@ Renovate marker, and a bump releases as `chore(deps)`. |---|---| | `actions/agent-gate` | Decides whether an agent may run. Evaluates `agent:no-touch` first, always. | | `actions/decompose` | Files the confirmed children of an Epic or an Initiative. `decompose.sh parse` is the pure check of the proposal and `run` the writes; `DRY_RUN=1` writes nothing | -| `actions/run-agent` | Invokes the agent with the org's tool allowlist and reporting defaults, on the model `actions/pick-model` chooses for its `stage`. Does not start it when the running workflow file differs from the default branch, says why a run stopped (`why`), and with `report-file` guarantees a valid report file on every path | +| `actions/run-agent` | Invokes the agent with the org's tool allowlist and reporting defaults, on the model `actions/pick-model` chooses for its `stage`. Does not start it when the running workflow file differs from the default branch, says why a run stopped (`why`), warns when claude-code-action skipped itself (and fails with `fail-on-self-skip`), and with `report-file` guarantees a valid report file on every path | | `actions/pr-facts` | Resolves the PR from a number or a pull_request, workflow_run or status event, and checks it is open, not a fork, and at the expected head | | `actions/judge-verdict` | Marks an `agent:judge` request pending, turns a review by the exact judge login on the current head into `judge / verdict` with the org App token, or ends unanswered requests in `error`. Fails closed on anything it cannot trust; `verdict.sh` holds the rules | | `actions/request-review` | Applies `agent:review` with the App token when the head commit has no `review / verdict` from the org App yet | diff --git a/actions/run-agent/action.yml b/actions/run-agent/action.yml index 00b9591..17078a5 100644 --- a/actions/run-agent/action.yml +++ b/actions/run-agent/action.yml @@ -28,6 +28,10 @@ # prompt before the agent runs. The guard below drops the progress comment # for those cases rather than the run. # +# Tag mode re-checks the trigger even with a prompt: on issues/labeled the +# label must equal `label_trigger` (default `claude`), or the action skips +# itself, green. The gate already chose the label, so it is passed through. +# # claude-code-action sets its own git identity (`claude[bot]`) on the # checkout, over whatever the caller configured earlier. A caller that checks # who authored the agent's commits has to pass the identity it expects as @@ -207,6 +211,13 @@ inputs: github-token: description: 'Reads the running workflow file at two refs, for the mismatch check below. Needs contents: read.' default: ${{ github.token }} + fail-on-self-skip: + description: >- + Fail the step when claude-code-action skipped itself and reported + success, for a caller whose job means nothing without an agent run. + An edited or deleted comment is not a self-skip: the action refuses it + by design. + default: 'false' # `execution-file` and `conclusion` are EMPTY when the agent never ran, which is # not the same as a run that found nothing. claude-code-action skips itself, and @@ -226,6 +237,9 @@ outputs: skipped: description: '`workflow-mismatch` when the agent was not started because the running workflow file differs from the default branch; empty otherwise.' value: ${{ steps.validate.outputs.skipped }} + self-skipped: + description: '`true` when claude-code-action was started, left no result and reported success; empty otherwise.' + value: ${{ steps.outcome.outputs.self-skipped }} why: description: 'One line saying why the agent did not start, stopped without a result, or left no valid report. Empty for a run that finished normally.' value: ${{ steps.report.outputs.why || steps.outcome.outputs.why || steps.validate.outputs.why }} @@ -409,6 +423,9 @@ runs: bot_name: ${{ inputs.bot-name }} bot_id: ${{ inputs.bot-id }} branch_prefix: ${{ inputs.branch-prefix }} + # See the header. Only with a prompt: in pure tag mode the label is + # the request itself, and any label would then start a run. + label_trigger: ${{ inputs.prompt != '' && github.event_name == 'issues' && github.event.action == 'labeled' && github.event.label.name || 'claude' }} # Never optional. See the header. display_report: true claude_args: >- @@ -455,6 +472,8 @@ runs: STAGE: ${{ inputs.stage }} MODEL: ${{ steps.model.outputs.model }} ROUTE: ${{ steps.model.outputs.route }} + EVENT: ${{ github.event_name }} + ACTION: ${{ github.event.action }} run: | set -euo pipefail exe="${EXECUTION_FILE:-$RUNNER_TEMP/claude-execution-output.json}" @@ -462,8 +481,15 @@ runs: clock=''; [ -z "${STARTED:-}" ] || clock=$(( $(date +%s) - STARTED )) turns='' cost='?' priced=none wall='' denied='' why='' if [ -z "$r" ]; then - if [ "$OUTCOME" = success ]; then - echo "why=claude-code-action skipped itself and reported success." >> "$GITHUB_OUTPUT" + # Not a run, so no notice either way. An edited or deleted comment is + # refused ahead of any trigger, by design: its text may not be the + # actor's. Anything else is a skip a green job would hide. + if [ "$OUTCOME" = success ] && [[ ${EVENT:-} == *comment && ${ACTION:-created} != created ]]; then + echo "why=claude-code-action does not act on an ${ACTION} comment." >> "$GITHUB_OUTPUT" + exit 0 + elif [ "$OUTCOME" = success ]; then + printf 'why=claude-code-action skipped itself and reported success.\nself-skipped=true\n' >> "$GITHUB_OUTPUT" + echo "::warning title=Agent never ran::claude-code-action skipped itself and reported success. Its log names the check that stopped it, usually 'No trigger found'." exit 0 fi why="claude-code-action failed before the agent produced a result." @@ -525,3 +551,11 @@ runs: echo "why=$why" } >> "$GITHUB_OUTPUT" echo "::warning title=No agent report::$why" + + # Last, so the report and the outputs above exist on this path too. + - name: Fail a run the agent never reached + if: always() && inputs.fail-on-self-skip == 'true' && steps.outcome.outputs.self-skipped == 'true' + shell: bash + run: | + echo "::error title=Agent never ran::claude-code-action skipped itself and reported success, so this job did nothing. Its log names the check that stopped it." + exit 1 diff --git a/actions/run-agent/test.sh b/actions/run-agent/test.sh index 46510f8..7f47794 100755 --- a/actions/run-agent/test.sh +++ b/actions/run-agent/test.sh @@ -105,6 +105,48 @@ EOF } expect "run-agent turns the repo's hooks off with --settings" hooks_off +# Tag mode compares an issues/labeled label to `label_trigger` (default `claude`), +# so `agent:triage` skipped itself green. One parse of both files for the wiring. +self_skip_wired() { +python3 - "$ROOT" "$WORK" <<'EOF' +import sys, yaml +root = sys.argv[1] +doc = yaml.safe_load(open(root + '/actions/run-agent/action.yml')) +steps = doc['runs']['steps'] +lt = [s for s in steps if 'claude-code-action@' in str(s.get('uses', ''))][0]['with'].get('label_trigger', '') +for part in ("inputs.prompt != ''", "github.event.action == 'labeled'", 'github.event.label.name', "|| 'claude'"): + assert part in lt, (part, lt) +assert doc['inputs']['fail-on-self-skip']['default'] == 'false', doc['inputs'].get('fail-on-self-skip') +fail = steps[-1] +assert "inputs.fail-on-self-skip == 'true'" in fail['if'] and 'steps.outcome.outputs.self-skipped' in fail['if'], fail +assert 'exit 1' in fail['run'], fail +triage = yaml.safe_load(open(root + '/.github/workflows/issue-triage.yml'))['jobs']['triage']['steps'] +agent = [s for s in triage if 'run-agent@' in str(s.get('uses', ''))][0]['with'] +assert agent.get('fail-on-self-skip') == 'true', agent.get('fail-on-self-skip') +clear = triage[-1] +assert clear.get('name') == 'Clear the request label', 'the label is cleared last' +# Not on a cancel: the run that cancelled this one may be the one the label asked for. +assert clear.get('if') == "always() && (!cancelled() || github.event.action == 'labeled')", clear.get('if') +assert 'agent%3Atriage' in clear['run'], clear['run'] +open(sys.argv[2] + '/clear.sh', 'w').write(clear['run']) +EOF +} +expect "a routed label reaches label_trigger, triage fails on a self-skip, and clears agent:triage last" self_skip_wired + +# The clear step, run for real: a 404 is a label that was not there, anything +# else leaves the request on and says so. +clear() { + mkdir -p "$WORK/cbin" + printf '#!/bin/sh\n%s\n' "$1" > "$WORK/cbin/gh"; chmod +x "$WORK/cbin/gh" + ( env -i PATH="$WORK/cbin:$PATH" REPO=o/r NUM=7 bash -e "$WORK/clear.sh" ) > "$WORK/log" 2>&1 +} +clear 'exit 0' +expect "a removed label is quiet" bash -c '! grep -q warning "$1"' _ "$WORK/log" +clear 'echo "gh: Label does not exist (HTTP 404)" >&2; exit 1' +expect "an absent label is not a warning" bash -c '! grep -q warning "$1"' _ "$WORK/log" +clear 'echo "gh: Resource not accessible by integration (HTTP 403)" >&2; exit 1' +expect "any other failure warns that the request stays on" grep -q '::warning title=agent:triage not removed::#7 keeps agent:triage.*HTTP 403' "$WORK/log" + # The files the agent writes outside the checkout reach the tool list. That # pr-review names both of its own is a Self test check. writable_wired() { @@ -263,6 +305,7 @@ outcome() { [ -n "${2:-}" ] && printf '%s' "$2" > "$WORK/exe.json" ( cd "$WORK" && env -i PATH="$PATH" HOME="$WORK" RUNNER_TEMP="$WORK/none" OUTCOME="$1" \ GITHUB_ACTION_PATH="$ROOT/actions/run-agent" STARTED="${STARTED:-}" STAGE=review MODEL=m ROUTE=r PR="${PR:-}" ISSUE="${ISSUE:-}" \ + EVENT="${EVENT:-issues}" ACTION="${ACTION:-labeled}" \ EXECUTION_FILE="${2:+$WORK/exe.json}" GITHUB_OUTPUT="$WORK/out" bash "$WORK/outcome.sh" ) > "$WORK/log" 2>&1 } out_why() { sed -n 's/^why=//p' "$WORK/out"; } @@ -296,8 +339,15 @@ expect "no refusals, no warning" bash -c 'grep -qx "denied=" "$1" && ! grep -q " outcome success expect "no result after a success is the action skipping itself" grep -q 'skipped itself' <(out_why) expect "which is not a run, so it writes no notice" bash -c '! grep -q "title=Agent run" "$1"' _ "$WORK/log" +expect "but says so in a warning, so the green job does not read as a run" grep -q '::warning title=Agent never ran::' "$WORK/log" +expect "and marks it for the caller" grep -qx 'self-skipped=true' "$WORK/out" +EVENT=issue_comment ACTION=edited outcome success +expect "an edited comment is refused by design, not a self-skip" bash -c '! grep -q "^self-skipped=" "$1" && ! grep -q "never ran" "$2"' _ "$WORK/out" "$WORK/log" +EVENT=issue_comment ACTION=created outcome success +expect "a new comment that skipped is one" grep -qx 'self-skipped=true' "$WORK/out" STARTED=$(( $(date +%s) - 7 )) outcome failure expect "no result after a failure says the action failed first" grep -q 'failed before the agent produced a result' <(out_why) +expect "a failure is not a self-skip" bash -c '! grep -q "^self-skipped=" "$1"' _ "$WORK/out" expect "and is still a run: counted unpriced, timed by the step's clock" \ grep -Eq '::notice title=Agent run::stage=review model=m turns=\? cost=\? priced=none wall=(7|8) \(r\)$' "$WORK/log" expect "with cost ? and priced none in the outputs" bash -c 'grep -qx "cost=?" "$1" && grep -qx "priced=none" "$1"' _ "$WORK/out"