diff --git a/AGENTS.md b/AGENTS.md index 66c2d3d1ef8..e12d208ae69 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,6 +74,7 @@ config/crew-harness crewmate harness override; LOCAL, gitignored; absent or "de config/claude-permission-mode optional one-token permission posture for every Claude worker launch: absent or "bypass" keeps --dangerously-skip-permissions, "auto" launches with --permission-mode auto; LOCAL, gitignored; inherited by secondmate homes; see docs/configuration.md "Claude permission mode" config/crew-dispatch.json optional crewmate dispatch profiles; LOCAL, gitignored; firstmate-maintained but human-editable natural-language rules that choose a per-task harness/model/effort profile (section 4). Inherited by secondmate homes config/house-rules.json optional opt-in list of project names, and optional rules, for the advisory house-rules check: a ship brief asks its worker to run the check only for a project named in the file's "projects" list, and only when TYPESAFE_API_KEY is present; absent means off for every project; LOCAL, gitignored, and not inherited; see docs/configuration.md "House-rules check" +config/jev-code-projects optional list, one entry per line, of the projects whose code or failure logs a Jev check may send to typesafe.ai: `/` for a check keyed by repository, matched exactly, or a firstmate project name for a check keyed by project; absent means none; LOCAL, gitignored, and not inherited; see docs/configuration.md "Failed check sort" config/secondmate-harness harness the PRIMARY uses to launch SECONDMATE agents, optionally followed by a model and effort token on the same line (" [] []"; section 4); LOCAL, gitignored; absent or "default" harness falls back to config/crew-harness then firstmate's own. The primary's own setting; NOT inherited into secondmate homes (secondmates do not spawn secondmates) config/backlog-backend backlog backend override; LOCAL, gitignored; absent or "tasks-axi" = the configured tasks-axi backend, "manual" = force routine backlog updates to hand-editing; inherited by secondmate homes (section 10) config/backend runtime session-provider backend override for new tasks; LOCAL, gitignored; absent = falls through to runtime auto-detection (the runtime firstmate itself is executing inside), then tmux; tmux is the verified reference backend (docs/tmux-backend.md), herdr has its own required CI lane (docs/herdr-backend.md), while zellij, orca, and cmux remain experimental with no dedicated real-backend CI lane (docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md) - herdr and cmux can also be selected by runtime auto-detection, zellij and orca never are (always explicit), and codex-app is not accepted; see docs/codex-app-backend.md; inherited by secondmate homes under the primary-authoritative contract in secondmate-provisioning diff --git a/bin/fm-check-sort-lib.sh b/bin/fm-check-sort-lib.sh new file mode 100644 index 00000000000..a2e6bc3caa0 --- /dev/null +++ b/bin/fm-check-sort-lib.sh @@ -0,0 +1,150 @@ +# shellcheck shell=bash +# Advisory sort of a failed required check: code bug, flaky, or environment. +# Usage: . bin/fm-check-sort-lib.sh (source it before the caller runs any child) +# +# bin/fm-pr-state.sh is the one caller, and only under its +# --sort-failed-checks option. It hands over the required checks it already +# read as failed, and this file prints one extra line per check: +# FAILED CHECK SORT: : () +# where is `flaky`, `environment`, `code bug`, or `unknown`. The line is +# a label for the reader. It never re-runs a check, never changes what a check +# reported, and nothing merges, blocks, or discards on it. +# +# The sort runs only when TYPESAFE_API_KEY is available (bin/fm-jev-lib.sh owns +# the key); without it nothing is read and nothing is printed. Fixed rules +# decide first, from GitHub reads alone: +# 1. Another attempt of the same workflow run passed this check on the same +# head commit, so the same code and the same trigger both passed and +# failed -> flaky. A pass from a different run does not count, because a +# check triggered again by a pull request edit saw a different input, and +# neither does a pass from the same attempt, which is another job that +# shares the check's name. +# 2. The failed steps' log shows a connection error (FM_CHECK_SORT_MARKERS) and +# the same check is failing on the base branch, which does not carry this +# pull request's change -> environment. +# A failure no rule decides goes to Jev as one fixed-choice question over the +# check's name and the log of the job's failed steps. Only the last +# FM_CHECK_SORT_LOG_LINES lines are read; of those, the lines that name a +# failure (FM_CHECK_SORT_EVIDENCE) are sent, or every line when none does, cut +# to the last FM_CHECK_SORT_TAIL_CHARS characters, because a job states its +# failure last and ends in summary lines that say nothing about the cause. A failure log quotes the project's +# code, so it is sent only for a repository fm_jev_code_allowed finds in +# config/jev-code-projects as its exact `/`; a bare name there +# never matches a repository. An unlisted repository's unclear failure is +# `unknown` with no call. `unknown` is also the answer for a check with no +# GitHub Actions job log, an unreadable read, a Jev timeout or error, a +# malformed answer, an `unclear` choice, and a confidence under the floor; the +# worker then investigates as it does without the label. `code bug` needs the +# shared FM_JEV_CONFIDENCE_FLOOR. `flaky` and `environment` need the stricter +# FM_CHECK_SORT_AWAY_FLOOR, because those two tell a worker the failure is not +# in its code, and a live call labelled a pull request process failure +# `environment` above the shared floor (docs/verification/failed-check-sort.md). +# +# ponytail: only the first FM_CHECK_SORT_MAX failed checks are sorted, so a +# pull request with many red checks costs a bounded number of reads; each of +# the rest is printed as `unknown (not sorted)` with nothing read. +# +# fm_check_sort +# Reads failed check names, one per line, on stdin. Always returns 0. + +_FM_CHECK_SORT_DIR=${BASH_SOURCE[0]%/*} +[ "$_FM_CHECK_SORT_DIR" != "${BASH_SOURCE[0]}" ] || _FM_CHECK_SORT_DIR=. +# shellcheck source=bin/fm-jev-lib.sh +. "$_FM_CHECK_SORT_DIR/fm-jev-lib.sh" + +FM_CHECK_SORT_MAX=3 +FM_CHECK_SORT_LOG_LINES=2000 +FM_CHECK_SORT_TAIL_CHARS=4000 +# A live call labelled a process failure `environment` at 0.65 to 0.73 (docs/verification/failed-check-sort.md). +FM_CHECK_SORT_AWAY_FLOOR=0.8 +FM_CHECK_SORT_EVIDENCE='error|fail|not ok|panic|exception|assert|timed out|timeout|refused|denied|killed|cannot|unable' +FM_CHECK_SORT_MARKERS='ECONNREFUSED|ECONNRESET|ETIMEDOUT|EAI_AGAIN|ENOTFOUND|could not resolve host|temporary failure in name resolution|connection (refused|reset|timed out)|TLS handshake timeout|50[234] (bad gateway|service unavailable|gateway time-?out)' +# shellcheck disable=SC2016 # The backticks are literal markup in the question text. +FM_CHECK_SORT_INSTRUCTIONS='An automated check named in `check` failed on a pull request, and `log_tail` holds the last lines of the failed job'"'"'s log that name a failure. Decide from the words of the log why it failed. Choose `unclear` whenever the log does not plainly show one of the other three.' +FM_CHECK_SORT_CRITERIA='{"code_bug":"The log shows the project'"'"'s own code or tests failing on their merits: a compile, type, or lint error, a failed assertion, or a test that fails for a reason in the code.","flaky":"The log shows a failure that depends on timing, ordering, or chance rather than on the code: a race, a timing-dependent assertion, or a test that timed out waiting with no error in the code.","environment":"The log shows the runner or something outside the project failing: a network or connection error, a failed download or dependency install, an unavailable service, a rate limit, a missing credential, or a runner out of disk, memory, or time.","unclear":"The log does not show enough to tell which of the others applies."}' + +# Prints " ()" or "unknown" for one failed check. +_fm_check_sort_one() { # + local listed=$1 repo=$2 head=$3 url=$4 name=$5 runs run passed job log evidence base base_state + # shellcheck disable=SC2016 # jq variables are literal filter syntax. + runs=$(gh api -X GET "/repos/$repo/commits/$head/check-runs" \ + -f check_name="$name" -f filter=all -f per_page=100 --jq ' + def run: (.details_url // "") | split("/job/") | .[0] // ""; + ([.check_runs[] | select(.app.slug == "github-actions" and .status == "completed" and .conclusion != "success")] + | max_by(.id)) as $failed + | (($failed // {}) | run) as $run + | ($run | split("/") | last // ""), + (($failed.id // "") | tostring), + ([.check_runs[] | select($run != "" and .conclusion == "success" and run == $run) | .id | tostring] | join(" "))' \ + 2>/dev/null) || { echo unknown; return 0; } + { + IFS= read -r run + IFS= read -r job + IFS= read -r passed + } </dev/null \ + | awk -v job="$job" -v passed=" $passed " ' + { attempt[$1] = $2 } + END { + if (attempt[job] == "") exit 1 + for (id in attempt) if (index(passed, " " id " ") && attempt[id] != "" && attempt[id] != attempt[job]) exit 0 + exit 1 + }'; then + echo 'flaky (rule: another attempt of the same run passed)' + return 0 + fi ;; + esac + # Each line is "\t\t "; only the text is kept. + log=$(gh run view --job "$job" --log-failed -R "$repo" 2>/dev/null \ + | tail -n "$FM_CHECK_SORT_LOG_LINES" | cut -f 3- | sed -E 's/^[0-9]{4}-[0-9T:.-]+Z //') || true + [ -n "$log" ] || { echo unknown; return 0; } + if printf '%s\n' "$log" | grep -Eiq -- "$FM_CHECK_SORT_MARKERS"; then + base=$(gh pr view "$url" --json baseRefName --jq .baseRefName 2>/dev/null) || base= + base_state= + [ -z "$base" ] || base_state=$(gh api -X GET "/repos/$repo/commits/$base/check-runs" \ + -f check_name="$name" --jq ' + [.check_runs[] | select(.status == "completed")] | max_by(.id) | .conclusion // ""' 2>/dev/null) || base_state= + case "$base_state" in + failure|timed_out) + echo 'environment (rule: connection error in its log, and it fails on the base branch too)' + return 0 ;; + esac + fi + [ "$listed" -eq 1 ] || { echo unknown; return 0; } + evidence=$(printf '%s\n' "$log" | grep -Ei -- "$FM_CHECK_SORT_EVIDENCE") || evidence=$log + fm_jev_choice failed_check "$FM_CHECK_SORT_INSTRUCTIONS" \ + <(printf '%s' "$evidence" | tail -c "$FM_CHECK_SORT_TAIL_CHARS" | jq -Rsc --arg check "$name" '{check: $check, log_tail: .}') \ + <(printf '%s' "$FM_CHECK_SORT_CRITERIA") || { echo unknown; return 0; } + jq -r --argjson floor "$FM_JEV_CONFIDENCE_FLOOR" --argjson away "$FM_CHECK_SORT_AWAY_FLOOR" ' + if (.choice | IN("code_bug", "flaky", "environment")) and .confidence >= (if .choice == "code_bug" then $floor else $away end) + then "\(.choice | sub("_"; " ")) (jev, confidence \(.confidence))" else "unknown" end' \ + <<<"$FM_JEV_ANSWER" 2>/dev/null || echo unknown +} + +fm_check_sort() { # ; failed check names on stdin + local home=$1 repo=$2 head=$3 url=$4 listed=0 name names=() n=0 + while IFS= read -r name; do + [ -z "$name" ] || names+=("$name") + done + [ "${#names[@]}" -gt 0 ] || return 0 + fm_jev_key_load "$home" || return 0 + command -v jq >/dev/null 2>&1 || return 0 + ! FM_HOME=$home fm_jev_code_allowed "$repo" || listed=1 + for name in "${names[@]}"; do + n=$((n + 1)) + if [ "$n" -gt "$FM_CHECK_SORT_MAX" ]; then + printf 'FAILED CHECK SORT: %s: unknown (not sorted)\n' "$name" + continue + fi + printf 'FAILED CHECK SORT: %s: %s\n' "$name" \ + "$(_fm_check_sort_one "$listed" "$repo" "$head" "$url" "$name" # 0 when a key is available, 1 when it is absent from both sources. # +# fm_jev_code_allowed +# 0 only when is a whole line of $FM_HOME/config/jev-code-projects +# (one entry per line; blank lines and lines starting with # are ignored). +# A caller keyed by repository passes `/`, and a caller keyed by +# a firstmate project passes that project's name; neither form matches the +# other, so a bare name never allows a repository by its basename. +# The key alone lets a caller send routing text and status lines; a caller +# that would send a project's code, file list, or pull request text asks this +# first. An absent or empty file allows no project. +# # fm_jev_choice # One POST to $FM_JEV_BASE/v1/systemone. holds the JSON # value sent as `state`; holds one JSON object mapping @@ -82,6 +93,11 @@ fm_jev_key_load() { # [ -n "$TYPESAFE_API_KEY_PRIVATE" ] } +fm_jev_code_allowed() { # + [ -n "$1" ] && [ "${1#\#}" = "$1" ] \ + && grep -qxF -- "$1" "${FM_HOME:-.}/config/jev-code-projects" 2>/dev/null +} + _fm_jev_fail() { # FM_JEV_STATUS=error FM_JEV_ERROR=$1 diff --git a/bin/fm-pr-state.sh b/bin/fm-pr-state.sh index b7b1c2b5367..d46e450e082 100755 --- a/bin/fm-pr-state.sh +++ b/bin/fm-pr-state.sh @@ -17,8 +17,15 @@ # STALE when it was left at a superseded head. # A closed or merged pull request reports that terminal state and nothing else. # Unresolved review-thread state is out of this command's scope. +# With --sort-failed-checks and TYPESAFE_API_KEY available, each failed +# required check also gets one advisory FAILED CHECK SORT line labelling it a +# code bug, flaky, environment, or unknown; bin/fm-check-sort-lib.sh owns that +# sort. The label reads more of GitHub and changes nothing a check reported. +# Firstmate passes the option when it reads a pull request's failed checks by +# hand. The watcher's routine-wake triage does not, so its read costs and +# prints what it does without the sort. # -# Usage: fm-pr-state.sh +# Usage: fm-pr-state.sh [--sort-failed-checks] # Prints one line per blocker it can see and nothing when it sees none. # Blockers do not change the successful exit status; lookup or usage refusal # exits non-zero. @@ -26,6 +33,9 @@ set -eu SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# Sourced before any child runs, so no gh call inherits the Jev key. +# shellcheck source=bin/fm-check-sort-lib.sh +. "$SCRIPT_DIR/fm-check-sort-lib.sh" # shellcheck source=bin/fm-pr-lib.sh . "$SCRIPT_DIR/fm-pr-lib.sh" @@ -42,7 +52,12 @@ if [ "${1:-}" = --help ] || [ "${1:-}" = -h ]; then usage exit 0 fi -[ "$#" -eq 1 ] || die "usage: fm-pr-state.sh " +SORT=0 +if [ "${1:-}" = --sort-failed-checks ]; then + SORT=1 + shift +fi +[ "$#" -eq 1 ] || die "usage: fm-pr-state.sh [--sort-failed-checks] " command -v gh >/dev/null 2>&1 || die "gh is required" URL=$1 @@ -106,10 +121,15 @@ esac GH_STDERR=$(mktemp "${TMPDIR:-/tmp}/fm-pr-state.XXXXXX") \ || die "could not create temporary file" trap 'rm -f "$GH_STDERR"' EXIT INT TERM -if ! REQUIRED=$(gh pr checks "$URL" --required --json name,state,bucket --jq ' +FAILED= +if REQUIRED=$(gh pr checks "$URL" --required --json name,state,bucket --jq ' .[] | select(.bucket != "pass" and .bucket != "skipping") - | "REQUIRED CHECK: \(.name) (\(.state))"' 2>"$GH_STDERR"); then + | "REQUIRED CHECK: \(.name) (\(.state))", + (select(.bucket == "fail") | "FAILED CHECK NAME: \(.name)")' 2>"$GH_STDERR"); then + FAILED=$(printf '%s\n' "$REQUIRED" | sed -n 's/^FAILED CHECK NAME: //p') + REQUIRED=$(printf '%s\n' "$REQUIRED" | sed '/^FAILED CHECK NAME: /d') +else # These two sentences are gh's own human-readable error text, verified against # gh 2.100.0 on 2026-09-12. gh reports "nothing reported" as an error rather # than as structured data, so matching its text is the only way to tell that @@ -125,6 +145,11 @@ if ! REQUIRED=$(gh pr checks "$URL" --required --json name,state,bucket --jq ' fi fi [ -z "$REQUIRED" ] || printf '%s\n' "$REQUIRED" +if [ "$SORT" -eq 1 ]; then + fm_check_sort "${FM_HOME:-$SCRIPT_DIR/..}" "$PATH_PART" "$HEAD" "$URL" <` adds one advisory line under each failed required check it reports, labelling the failure a code bug, flaky, environment, or unknown, so a worker is not sent to fix a failure that is not in its code. +The sort runs only when the caller passes that option: firstmate passes it when it reads a pull request's failed checks by hand, and the watcher's routine-wake triage never does, so its recheck reads and prints what it does without the feature and sends nothing. +With the option it is still off unless `TYPESAFE_API_KEY` is available under the same environment-then-`.env` contract as [typed dispatch resolution](#typed-dispatch-resolution-env-typesafe_api_key); with the key absent the command reads and prints exactly what it does without the feature. +The label never re-runs a check, never changes what a check reported, and never changes the command's blocker lines or exit status, so nothing merges, blocks, or discards on it. + +Fixed rules decide first from GitHub reads alone: a check that another attempt of the same workflow run passed is flaky, which a same-named job passing in the same attempt is not, and a check whose failed steps log a connection error while the same check also fails on the base branch is environment. +Only a failure neither rule decides is put to typesafe.ai's System One model (Jev) as one fixed-choice question, `code_bug`, `flaky`, `environment`, or `unclear`. +A `code_bug` answer needs the shared 0.6 confidence floor, and a `flaky` or `environment` answer needs 0.8, because those two point a worker away from the failure. +A timeout, a transport or API error, a malformed answer, an `unclear` choice, an answer under its floor, a check with no GitHub Actions job log, and an unreadable GitHub read all print `unknown`, and the worker investigates as it does without the label. + +The question carries the check's name and the failure-naming lines of the job's failed steps, cut to their last 4000 characters. +A failure log quotes the project's code, so it is sent only for a repository listed in the optional local, gitignored `config/jev-code-projects`, one entry per line, with blank lines and lines starting with `#` ignored. +An entry takes one of two forms: `/` matches a caller keyed by repository, such as this sort, only when it equals the pull request's owner and repository exactly, so a same-named repository under another owner is not allowed; a bare name matches only a caller that passes a firstmate project name, and never a repository's basename. +Only the first three failed required checks are sorted, and each one after them is printed as `unknown (not sorted)` with nothing read. +An unlisted repository still gets the two fixed rules, and its undecided failures are `unknown` with no call. + +The sort covers GitHub pull requests only, which is all `bin/fm-pr-state.sh` reads, and it is the same on every harness and runtime backend because the command reads the forge and no worker surface. +`bin/fm-check-sort-lib.sh`'s header owns the exact rules, bounds, and output, `bin/fm-jev-lib.sh` owns the request, the answer validation, the key handling, and the `fm_jev_code_allowed` project list read, and [`verification/failed-check-sort.md`](verification/failed-check-sort.md) records the live evidence. + ## Gate defaults (.no-mistakes.yaml) The tracked `.no-mistakes.yaml` sets `test.evidence.store_in_repo: true` and pins `commands.lint` to `bin/fm-lint.sh`, the same owner CI invokes. @@ -631,7 +652,7 @@ By accepted design, a `clear` result does not enforce catalog/authentication, re Firstmate passes its profile line unless it states a reason to override, such as the brief's reasoning class or an eligible-unranked-candidate note; every non-clear result returns to the full existing intake. The resolver, the watcher, the house-rules check, the finished check, and bootstrap copy an environment-provided key into a non-exported private variable and unset `TYPESAFE_API_KEY` before launching child processes, so the secret is absent from child environments. -`bin/fm-jev-lib.sh` is the single owner of the Jev request, the answer validation, and that key handling for the resolver, for [routine-wake triage](#routine-wake-triage-env-typesafe_api_key), for the [house-rules check](#house-rules-check-env-typesafe_api_key-confighouse-rulesjson), and for the [finished check](#finished-check-env-typesafe_api_key); it sends the key to `curl` only as a header read from a file descriptor, never on argv, and nothing prints, logs, or writes it. +`bin/fm-jev-lib.sh` is the single owner of the Jev request, the answer validation, and that key handling for the resolver, for [routine-wake triage](#routine-wake-triage-env-typesafe_api_key), for the [house-rules check](#house-rules-check-env-typesafe_api_key-confighouse-rulesjson), for the [finished check](#finished-check-env-typesafe_api_key), and for the [failed check sort](#failed-check-sort-env-typesafe_api_key-configjev-code-projects); it sends the key to `curl` only as a header read from a file descriptor, never on argv, and nothing prints, logs, or writes it. It fixes the endpoint at `https://api.typesafe.ai`, model at `jev-latest`, confidence floor at 0.6, and request timeout at 5 seconds; `TYPESAFE_API_KEY` is the resolver's only resolver-specific environment setting. The live rule-match evidence is recorded in [`verification/dispatch-resolve.md`](verification/dispatch-resolve.md). @@ -1222,7 +1243,7 @@ FMX_RELAY_URL=https://myfirstmate.io # optional Relay endpoint override, mainl FMX_ENV_FILE= # optional alternate .env file for direct Relay client invocations; bootstrap still checks $FM_HOME/.env FMX_DRY_RUN= # truthy previews Relay replies and dismissals to state/x-outbox/ without posting or requiring a token FMX_X_REPLY_MAX_CHARS=280 # X reply per-message split budget; values below 50 clamp to 50 -TYPESAFE_API_KEY= # typed dispatch resolution opt-in, from the environment or .env; absent means bin/fm-dispatch-resolve.sh is off (docs/configuration.md "Typed dispatch resolution"), and so are routine-wake triage, the finished check, and the house-rules check, which also needs a project opted in in config/house-rules.json +TYPESAFE_API_KEY= # typed dispatch resolution opt-in, from the environment or .env; absent means bin/fm-dispatch-resolve.sh is off (docs/configuration.md "Typed dispatch resolution"), and so are routine-wake triage, the finished check, the failed check sort, and the house-rules check, which also needs a project opted in in config/house-rules.json FMX_DISCORD_REPLY_MAX_CHARS=1900 # Discord reply per-message split budget; values below 50 clamp to 50, values above 2000 reset to 1900 FMX_X_THREAD_MAX=25 # maximum messages in one auto-split reply thread FMX_FOLLOWUP_MAX_AGE_SECS=604800 # local window for posting Relay completion follow-ups (7 days) diff --git a/docs/documentation-audiences.json b/docs/documentation-audiences.json index 2df3a97adc6..5c50eebf906 100644 --- a/docs/documentation-audiences.json +++ b/docs/documentation-audiences.json @@ -464,6 +464,10 @@ "path": "docs/verification/dispatch-resolve.md", "audience": "maintainer-verification" }, + { + "path": "docs/verification/failed-check-sort.md", + "audience": "maintainer-verification" + }, { "path": "docs/verification/finished-check.md", "audience": "maintainer-verification" diff --git a/docs/scripts.md b/docs/scripts.md index 6136a5fdbc7..8e3042b5ee6 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -142,7 +142,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-contributions.sh` | Observe owned publications, retain exact-head judgments, measure required actors, and wake on maintainer signals | | `fm-pr-check.sh` | Record validated `pr=` and `pr_head=` values, then atomically arm a static merge poll | | `fm-pr-merge.sh` | Record PR metadata, merge a task's canonical full GitHub or GitLab URL, then refuse an outcome it cannot prove landed or queued | -| `fm-pr-state.sh` | Read-only: print one line per GitHub pull-request blocker it can see, reporting on checks that have reported rather than verdicting merge-readiness | +| `fm-pr-state.sh` | Read-only: print one line per GitHub pull-request blocker it can see, reporting on checks that have reported rather than verdicting merge-readiness; `--sort-failed-checks` adds the advisory failed check sort line under each failed required check | | `fm-pr-reviewers.sh` | Read-only: suggest reviewers from GitHub's own author mapping of recent commits on a pull request's changed files, never requesting one | | `fm-merge-outcome-lib.sh` | Publish a confirmed merge's durable, role-routed supervision outcome | | `fm-merge-authority-lib.sh` | Resolve merge authority at the gate, persist it against the accepted canonical PR, and identity-check its later poll consumption | diff --git a/docs/verification/failed-check-sort.md b/docs/verification/failed-check-sort.md new file mode 100644 index 00000000000..73d20f8f85a --- /dev/null +++ b/docs/verification/failed-check-sort.md @@ -0,0 +1,56 @@ +# Failed check sort verification + +Audience: maintainer verification. + +This record supports the key-gated failed check sort owned by [`../configuration.md`](../configuration.md) ("Failed check sort"). +It records only facts that must be re-established when the typesafe.ai model, GitHub's check-run or job-log reads, or the rules and question in `bin/fm-check-sort-lib.sh` change. +The API shape itself is recorded in [`dispatch-resolve.md`](dispatch-resolve.md). + +## Live answers on real failed checks + +Run 2026-10-06 against `https://api.typesafe.ai`, model `jev-latest` answering as `jev-1.13.0`, timeout 5 s, with gh 2.100.0. +Each row is `fm_check_sort` from `bin/fm-check-sort-lib.sh` run three times on one real failed check of `talktejas/firstmate`, with its rules, question, and criteria unmodified, the key read from the environment by `bin/fm-jev-lib.sh`, and a fixture home whose `config/jev-code-projects` lists `firstmate`. +The list has since been matched as the exact `/`, so the same run now needs the entry `talktejas/firstmate`. +Every read of GitHub was real: the check runs of the commit, and the failed steps' log through `gh run view --job --log-failed`. +No rule fired on any row, so each run is one real question; the 18 calls took 298 to 808 ms. + +| Commit | Check | What its log shows | Choice and confidence, three runs | Printed label | +| --- | --- | --- | --- | --- | +| 30fc910f | Behavior portable serial 4 | one `not ok` test assertion | code_bug 0.88, 0.89, 0.89 | code bug | +| dfa2369b | Behavior portable serial 4 | one `not ok` test assertion | code_bug 0.99, 0.98, 0.99 | code bug | +| 30fc910f | Behavior portable serial 2 | one `not ok` test assertion among summary lines | code_bug 0.41, 0.48, 0.48 | unknown | +| f84f2f35 | Behavior portable serial 1 | one `not ok` timing assertion | flaky 0.33, 0.28, 0.37 | unknown | +| f84f2f35 | Behavior portable serial 3 | summary lines only | unclear 0.59, 0.55, 0.58 | unknown | +| 30fc910f | PR must be raised via no-mistakes | a pull request process check refusing a head it did not validate | environment 0.73, 0.72, 0.65 | unknown | + +The last row is the reason `flaky` and `environment` need 0.8 rather than the shared 0.6: that failure is the author's to fix, and the model called it `environment` above 0.6 on every run. +The choice was the same on all three runs of every row, and the confidence moved by at most 0.09. + +The same check on the same commit also has a passing run that a later pull request edit triggered. +Rule 1 does not call it flaky, because that pass belongs to a different workflow run: + +```text +$ gh api "repos/talktejas/firstmate/commits/30fc910f8dac43b550cf46a9099d94ea90e2131b/check-runs?filter=all&per_page=100" --jq '.check_runs[] | select(.name|startswith("PR must")) | [.id,.conclusion,.details_url]|@tsv' +110078953481 failure https://github.com/talktejas/firstmate/actions/runs/36771558348/job/110078953481 +110071234065 success https://github.com/talktejas/firstmate/actions/runs/36769277507/job/110071234065 +110064348154 failure https://github.com/talktejas/firstmate/actions/runs/36767230944/job/110064348154 +``` + +With the same commit and check but a fixture home that lists no project, the label was `unknown` and no question was sent. + +gh 2.100.0 refuses to print a job's raw log from `gh api /repos///actions/jobs//logs` ("the response contains terminal escape sequences"), which is why the log is read through `gh run view --log-failed`. + +## Stub-only coverage + +No open pull request had a failing required check on the run date, so these were not observed live. +`tests/fm-pr-state.test.sh` covers them with a fake `gh` that answers in GitHub's JSON shapes and `fm_jev_choice` stubbed at the library boundary, against fixture homes with no key in the environment: + +- the `flaky` rule firing on a pass from another attempt of the same run, and not on a pass from a different run or from a same-named job of the same attempt; +- the `environment` rule firing on a connection error whose check also fails on the base branch; +- `bin/fm-pr-state.sh` itself printing the label beside its unchanged blocker lines, with a fake `curl` that fails, and the key reaching neither `curl`'s argv or environment nor any `gh` call; +- a call without `--sort-failed-checks`, key and listed repository present, making no check-run read, no log download, and no model call; +- no key meaning no extra GitHub read and no extra line; +- an unlisted or commented-out repository, a bare repository name, a same-named repository under another owner, a check with no job log, a failed call, an `unclear` choice, and an answer under either floor all printing `unknown`; +- only the failure-naming lines being sent, cut to the last 4000 characters, and only the first three failed checks being sorted, with a fourth printed as `unknown (not sorted)`. + +`FM_PR_STATE_LIVE_E2E=1 bash tests/fm-pr-state-live-e2e.test.sh` is the command that confirms gh's own jq engine still accepts the script's pull-request read programs. diff --git a/tests/fm-pr-state.test.sh b/tests/fm-pr-state.test.sh index d911f674147..2560c57fb78 100755 --- a/tests/fm-pr-state.test.sh +++ b/tests/fm-pr-state.test.sh @@ -50,25 +50,81 @@ serve() { checks='[{"name":"lint","state":"SUCCESS","bucket":"pass","workflow":"ci"},{"name":"optional","state":"SKIPPED","bucket":"skipping","workflow":"ci"}]' printf '%s\n' "${FM_TEST_REQUIRED_CHECKS:-$checks}" ;; + "api -X GET /repos/o/r/commits/$head/check-runs -f check_name="*" -f filter=all -f per_page=100 --jq "*) + printf '%s\n' "${FM_TEST_HEAD_RUNS:-{\"check_runs\":[]\}}" + ;; + "api -X GET /repos/o/r/actions/runs/5/jobs -f filter=all -f per_page=100 --jq "*) + printf '%s\n' "${FM_TEST_RUN_JOBS:-{\"jobs\":[]\}}" + ;; + "run view --job 41 --log-failed -R o/r") + printf '%s\n' "${FM_TEST_LOG-}" + ;; + "pr view "*" --json baseRefName --jq .baseRefName") + printf '{"baseRefName":"main"}\n' + ;; + "api -X GET /repos/o/r/commits/main/check-runs -f check_name="*" --jq "*) + printf '%s\n' "${FM_TEST_BASE_RUNS:-{\"check_runs\":[]\}}" + ;; *) printf 'unexpected gh call: %s\n' "$*" >&2 exit 91 ;; esac } +[ -z "${FM_TEST_GH_CALLS-}" ] || printf '%s key=%s\n' "$*" "${TYPESAFE_API_KEY-}" >> "$FM_TEST_GH_CALLS" prog= prev= for arg in "$@"; do [ "$prev" != --jq ] || prog=$arg prev=$arg done -serve "$@" | jq -r "$prog" +if [ -n "$prog" ]; then serve "$@" | jq -r "$prog"; else serve "$@"; fi SH chmod +x "$FAKEBIN/gh" +# Every run is pinned to a fixture home with no key in the environment, so no +# real key can load and the sort stays off unless a test turns it on. +HOME_OFF="$TMP_ROOT/home-off" +HOME_ON="$TMP_ROOT/home-on" +mkdir -p "$HOME_OFF" "$HOME_ON/config" +printf 'TYPESAFE_API_KEY=fixture-key\n' > "$HOME_ON/.env" +GH_CALLS="$TMP_ROOT/gh-calls" +JEV_CALLS="$TMP_ROOT/jev-calls" +JEV_STATE="$TMP_ROOT/jev-state" +FAILING='[{"name":"CI Status","state":"FAILURE","bucket":"fail","workflow":"ci"},{"name":"slow","state":"IN_PROGRESS","bucket":"pending","workflow":"ci"}]' +RUN_URL=https://github.com/o/r/actions/runs +FAILED_RUN='{"check_runs":[{"id":41,"status":"completed","conclusion":"failure","details_url":"'$RUN_URL'/5/job/41","app":{"slug":"github-actions"}}]}' +passed_too() { # + jq -c --arg url "$RUN_URL/$1/job/40" '.check_runs += [{id: 40, status: "completed", conclusion: "success", details_url: $url, app: {slug: "github-actions"}}]' <<<"$FAILED_RUN" +} + run_state() { - PATH="$FAKEBIN:$PATH" "$SCRIPT" https://github.com/o/r/pull/7 + env -u TYPESAFE_API_KEY FM_HOME="${FM_TEST_HOME:-$HOME_OFF}" PATH="$FAKEBIN:$PATH" \ + "$SCRIPT" "$@" https://github.com/o/r/pull/7 +} + +# Runs fm_check_sort over one failed check with fm_jev_choice replaced at the +# library boundary. STUB_CHOICE and STUB_CONF are the answer; STUB_FAIL makes +# the call an error. Each call's state lands in $JEV_STATE. +run_sort() { + rm -f "$GH_CALLS" "$JEV_CALLS" "$JEV_STATE" + # shellcheck disable=SC2016 # The inner shell expands its own variables. + env -u TYPESAFE_API_KEY FM_TEST_GH_CALLS="$GH_CALLS" JEV_CALLS="$JEV_CALLS" JEV_STATE="$JEV_STATE" \ + PATH="$FAKEBIN:$PATH" bash -c ' + set -eu + # shellcheck source=/dev/null + . "$1" + fm_jev_choice() { + echo "$1" >> "$JEV_CALLS" + cat "$3" > "$JEV_STATE" + jq -e "has(\"code_bug\") and has(\"flaky\") and has(\"environment\") and has(\"unclear\") and length == 4" "$4" >/dev/null + [ -z "${STUB_FAIL:-}" ] || { FM_JEV_STATUS=error; return 1; } + FM_JEV_ANSWER=$(jq -cn --arg c "${STUB_CHOICE:-code_bug}" --argjson k "${STUB_CONF:-0.9}" "{choice: \$c, confidence: \$k}") + } + printf "%s\n" "${FAILED_NAMES:-CI Status}" | fm_check_sort "$2" o/r "$3" https://github.com/o/r/pull/7 + ' _ "$ROOT/bin/fm-check-sort-lib.sh" "${FM_TEST_HOME:-$HOME_ON}" "$HEAD" } +jev_calls() { [ -f "$JEV_CALLS" ] && wc -l < "$JEV_CALLS" | tr -d ' ' || echo 0; } # reviews " "... prints the JSON array # GitHub's reviews endpoint returns for those submissions. @@ -196,6 +252,160 @@ test_required_failure_is_a_blocker() { pass "required failure blocks readiness" } +test_failed_check_sort_is_off_unless_asked_for() { + local out + rm -f "$GH_CALLS" "$TMP_ROOT/curl-calls" + cat > "$FAKEBIN/curl" <<'SH' +#!/bin/sh +echo called >> "$FM_TEST_CURL_CALLS" +exit 7 +SH + chmod +x "$FAKEBIN/curl" + printf 'o/r\n' > "$HOME_ON/config/jev-code-projects" + out=$(FM_TEST_HOME="$HOME_ON" FM_TEST_REQUIRED_CHECKS=$FAILING FM_TEST_HEAD_RUNS=$FAILED_RUN \ + FM_TEST_LOG='assert failed' FM_TEST_GH_CALLS="$GH_CALLS" FM_TEST_CURL_CALLS="$TMP_ROOT/curl-calls" run_state) \ + || fail "blocked fixture was refused" + rm -f "$FAKEBIN/curl" "$HOME_ON/config/jev-code-projects" + [ "$out" = "$(printf 'REQUIRED CHECK: CI Status (FAILURE)\nREQUIRED CHECK: slow (IN_PROGRESS)')" ] \ + || fail "without the option the report is exactly the blocker lines, got: $out" + assert_no_grep 'check-runs' "$GH_CALLS" "without the option no check run is read" + assert_no_grep 'log-failed' "$GH_CALLS" "without the option no job log is downloaded" + [ ! -e "$TMP_ROOT/curl-calls" ] || fail "without the option the model is never asked" + pass "the default call reads and prints what it does without the sort, key and listed repository present" +} + +test_failed_check_sort_is_off_without_a_key() { + local out + rm -f "$GH_CALLS" + out=$(FM_TEST_REQUIRED_CHECKS=$FAILING FM_TEST_GH_CALLS="$GH_CALLS" run_state --sort-failed-checks) \ + || fail "blocked fixture was refused" + [ "$out" = "$(printf 'REQUIRED CHECK: CI Status (FAILURE)\nREQUIRED CHECK: slow (IN_PROGRESS)')" ] \ + || fail "without a key the report is exactly the blocker lines, got: $out" + ! grep -q 'check-runs\|log-failed' "$GH_CALLS" || fail "without a key nothing more is read from GitHub" + out=$(FM_TEST_HOME="$HOME_OFF" FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG='boom' run_sort) + [ -z "$out" ] || fail "without a key the sort prints nothing, got: $out" + [ "$(jev_calls)" = 0 ] || fail "without a key the model is never asked" + pass "without a key the report and the GitHub reads are unchanged" +} + +test_failed_check_sort_rules_decide_first() { + local out + out=$(FM_TEST_HEAD_RUNS=$(passed_too 5) \ + FM_TEST_RUN_JOBS='{"jobs":[{"id":40,"run_attempt":1},{"id":41,"run_attempt":2}]}' run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: flaky (rule: another attempt of the same run passed)' ] \ + || fail "a check that passed on another attempt of its run is flaky, got: $out" + assert_no_grep '--log-failed' "$GH_CALLS" "a flaky verdict needs no log" + out=$(FM_TEST_HEAD_RUNS=$(passed_too 5) FM_TEST_LOG='boom' \ + FM_TEST_RUN_JOBS='{"jobs":[{"id":40,"run_attempt":1},{"id":41,"run_attempt":1}]}' run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ + || fail "a same-named job that passed in the same attempt is not a retry, got: $out" + out=$(FM_TEST_HEAD_RUNS=$(passed_too 5) FM_TEST_LOG='boom' run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ + || fail "a pass whose attempt cannot be read is not a retry, got: $out" + out=$(FM_TEST_HEAD_RUNS=$(passed_too 6) FM_TEST_LOG='boom' run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ + || fail "a pass from a different run, such as one a pull request edit triggered, is not a retry, got: $out" + + printf 'o/r\n' > "$HOME_ON/config/jev-code-projects" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN \ + FM_TEST_LOG=$(printf 'build\tinstall\t2026-10-06T10:00:00.1234567Z curl: (6) Could not resolve host: registry.example') \ + FM_TEST_BASE_RUNS='{"check_runs":[{"id":9,"status":"completed","conclusion":"failure"}]}' run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: environment (rule: connection error in its log, and it fails on the base branch too)' ] \ + || fail "a connection error on a check the base branch fails too is environment, got: $out" + [ "$(jev_calls)" = 0 ] || fail "a failure a rule decides is never sent to the model" + rm -f "$HOME_ON/config/jev-code-projects" + pass "fixed rules label flaky and environment failures without asking the model" +} + +test_failed_check_sort_asks_only_for_a_listed_project() { + local out log i=0 + log=$(printf 'build\ttest\t2026-10-06T10:00:00.0000000Z not ok - early failure\n' + while [ "$i" -lt 400 ]; do i=$((i + 1)); printf 'build\ttest\t2026-10-06T10:00:00.0000000Z ok - passing line %s of a long job log\n' "$i"; done + printf '2026-10-06T10:00:01.0000000Z ECONNRESET\n2026-10-06T10:00:02.0000000Z assertion: expected 2 but got 3 THE-END\n') + + rm -f "$HOME_ON/config/jev-code-projects" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ + || fail "an unclear failure in an unlisted repository is unknown, got: $out" + [ "$(jev_calls)" = 0 ] || fail "an unlisted repository's log is never sent" + + printf '# o/r\n' > "$HOME_ON/config/jev-code-projects" + FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log run_sort >/dev/null + [ "$(jev_calls)" = 0 ] || fail "a commented-out entry does not list the repository" + + printf 'r\no\nother/r\no/r2\n' > "$HOME_ON/config/jev-code-projects" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ + || fail "a bare name or a same-named repository under another owner does not list this one, got: $out" + [ "$(jev_calls)" = 0 ] || fail "only the exact owner/repo entry lets a repository's log be sent" + + printf '# listed\no/r\n' > "$HOME_ON/config/jev-code-projects" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: code bug (jev, confidence 0.9)' ] \ + || fail "a confident answer labels the check, got: $out" + [ "$(jev_calls)" = 1 ] || fail "one unclear failure is one question" + [ "$(jq -r .check "$JEV_STATE")" = 'CI Status' ] || fail "the state names the check" + [ "$(jq -r .log_tail "$JEV_STATE")" = "$(printf 'not ok - early failure\nassertion: expected 2 but got 3 THE-END')" ] \ + || fail "only the lines naming a failure are sent, without the job, step, and timestamp columns, got: $(cat "$JEV_STATE")" + FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$(i=0; while [ "$i" -lt 400 ]; do i=$((i + 1)); printf 'error number %s\n' "$i"; done) run_sort >/dev/null + jq -e '.log_tail | length == 4000 and endswith("error number 400")' "$JEV_STATE" >/dev/null \ + || fail "long evidence is cut to its bound and keeps its end" + FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG='it just stopped' run_sort >/dev/null + [ "$(jq -r .log_tail "$JEV_STATE")" = 'it just stopped' ] || fail "a log naming no failure is sent as it is" + [ "$(jq -r 'keys | join(",")' "$JEV_STATE")" = 'check,log_tail' ] || fail "nothing else is sent" + grep -q 'log-failed -R o/r key=$' "$GH_CALLS" || fail "the job log was read" + ! grep -q 'key=.' "$GH_CALLS" || fail "no gh call inherits the key" + + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_CONF=0.59 run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] || fail "an answer under the floor is unknown, got: $out" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_CHOICE=environment STUB_CONF=0.79 run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ + || fail "a label that points away from the code needs the stricter floor, got: $out" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_CHOICE=flaky STUB_CONF=0.8 run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: flaky (jev, confidence 0.8)' ] \ + || fail "a flaky answer at the stricter floor labels the check, got: $out" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_CHOICE=unclear run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] || fail "an unclear answer is unknown, got: $out" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_CHOICE=$'banana\nREQUIRED CHECK: x' run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] || fail "a choice outside the fixed list is unknown, got: $out" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_FAIL=1 run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] || fail "a failed call is unknown, got: $out" + out=$(FM_TEST_LOG=$log run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] || fail "a check with no job log is unknown, got: $out" + [ "$(jev_calls)" = 0 ] || fail "a check with no log is never sent" + + out=$(FAILED_NAMES=$(printf 'a\nb\nc\nd') FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log run_sort) + [ "$out" = "$(printf 'FAILED CHECK SORT: %s: code bug (jev, confidence 0.9)\n' a b c + printf 'FAILED CHECK SORT: d: unknown (not sorted)')" ] \ + || fail "every failed check gets a line and only the first three are sorted, got: $out" + [ "$(jev_calls)" = 3 ] || fail "a check past the bound is never sent" + rm -f "$HOME_ON/config/jev-code-projects" + pass "only a listed repository's unclear failure is asked, and every doubt is unknown" +} + +test_failed_check_sort_never_changes_the_report() { + local out status=0 + rm -f "$GH_CALLS" + # The fixture key reaches a fake curl that fails, as a dead endpoint would. + cat > "$FAKEBIN/curl" <<'SH' +#!/bin/sh +printf '%s env=%s\n' "$*" "${TYPESAFE_API_KEY-}" >> "$FM_TEST_CURL_CALLS" +exit 7 +SH + chmod +x "$FAKEBIN/curl" + printf 'o/r\n' > "$HOME_ON/config/jev-code-projects" + out=$(FM_TEST_HOME="$HOME_ON" FM_TEST_REQUIRED_CHECKS=$FAILING FM_TEST_HEAD_RUNS=$FAILED_RUN \ + FM_TEST_LOG='assert failed' FM_TEST_GH_CALLS="$GH_CALLS" FM_TEST_CURL_CALLS="$TMP_ROOT/curl-calls" run_state --sort-failed-checks) || status=$? + rm -f "$FAKEBIN/curl" "$HOME_ON/config/jev-code-projects" + [ "$status" -eq 0 ] || fail "a failed model call must not change the exit status" + [ "$out" = "$(printf 'REQUIRED CHECK: CI Status (FAILURE)\nREQUIRED CHECK: slow (IN_PROGRESS)\nFAILED CHECK SORT: CI Status: unknown')" ] \ + || fail "the blocker lines stay and only the failed check gains a label, got: $out" + assert_present "$TMP_ROOT/curl-calls" "the model was asked through the shared caller" + assert_no_grep 'fixture-key' "$TMP_ROOT/curl-calls" "the key reaches neither curl's argv nor its environment" + assert_no_grep 'fixture-key' "$GH_CALLS" "the key reaches no gh call" + pass "with the model down the blocker lines are intact and the check is unknown" +} + # The next two cases supply gh's own "nothing reported" sentences through # FM_TEST_CHECKS_ERROR, so they prove the behaviour GIVEN those strings and # nothing about the strings themselves. A gh reword is invisible to this @@ -279,6 +489,11 @@ test_changes_requested_decision_is_never_silent test_authors_own_changes_requested_review_is_not_a_blocker test_pending_approval_is_not_a_blocker test_required_failure_is_a_blocker +test_failed_check_sort_is_off_unless_asked_for +test_failed_check_sort_is_off_without_a_key +test_failed_check_sort_rules_decide_first +test_failed_check_sort_asks_only_for_a_listed_project +test_failed_check_sort_never_changes_the_report test_unreported_required_checks_are_unconfirmed test_no_reported_checks_is_unverified test_help_states_what_silence_means_and_what_is_out_of_scope