From 153eb02fd9a41b7cebd4688c5465bc67c22f7e08 Mon Sep 17 00:00:00 2001 From: Tejas Shah Date: Tue, 6 Oct 2026 02:28:18 +0700 Subject: [PATCH 1/6] feat(bin): label a failed required check as code bug, flaky, or environment bin/fm-pr-state.sh adds one advisory FAILED CHECK SORT line under each failed required check when TYPESAFE_API_KEY is available. Fixed rules decide first from GitHub reads (another attempt of the same run passed means flaky; a connection error on a check the base branch also fails means environment); only an undecided failure is put to Jev as one fixed-choice question through bin/fm-jev-lib.sh, and only for a repository listed in config/jev-code-projects. Every doubt prints unknown. Without the key the command reads and prints exactly what it did before. The label never re-runs a check and never changes a blocker line or the exit status. --- AGENTS.md | 1 + bin/fm-check-sort-lib.sh | 126 ++++++++++++++++++ bin/fm-jev-lib.sh | 17 ++- bin/fm-pr-state.sh | 20 ++- bin/fm-test-run.sh | 9 +- docs/configuration.md | 18 +++ docs/documentation-audiences.json | 4 + docs/verification/failed-check-sort.md | 54 ++++++++ tests/fm-pr-state.test.sh | 174 ++++++++++++++++++++++++- 9 files changed, 415 insertions(+), 8 deletions(-) create mode 100644 bin/fm-check-sort-lib.sh create mode 100644 docs/verification/failed-check-sort.md diff --git a/AGENTS.md b/AGENTS.md index 66c2d3d1ef8..678ff26ec90 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,6 +74,7 @@ config/crew-harness crewmate harness override; LOCAL, gitignored; absent or "de config/claude-permission-mode optional one-token permission posture for every Claude worker launch: absent or "bypass" keeps --dangerously-skip-permissions, "auto" launches with --permission-mode auto; LOCAL, gitignored; inherited by secondmate homes; see docs/configuration.md "Claude permission mode" config/crew-dispatch.json optional crewmate dispatch profiles; LOCAL, gitignored; firstmate-maintained but human-editable natural-language rules that choose a per-task harness/model/effort profile (section 4). Inherited by secondmate homes config/house-rules.json optional opt-in list of project names, and optional rules, for the advisory house-rules check: a ship brief asks its worker to run the check only for a project named in the file's "projects" list, and only when TYPESAFE_API_KEY is present; absent means off for every project; LOCAL, gitignored, and not inherited; see docs/configuration.md "House-rules check" +config/jev-code-projects optional list of project names, one per line, whose code or failure logs a Jev check may send to typesafe.ai; absent means none; LOCAL, gitignored, and not inherited; see docs/configuration.md "Failed check sort" config/secondmate-harness harness the PRIMARY uses to launch SECONDMATE agents, optionally followed by a model and effort token on the same line (" [] []"; section 4); LOCAL, gitignored; absent or "default" harness falls back to config/crew-harness then firstmate's own. The primary's own setting; NOT inherited into secondmate homes (secondmates do not spawn secondmates) config/backlog-backend backlog backend override; LOCAL, gitignored; absent or "tasks-axi" = the configured tasks-axi backend, "manual" = force routine backlog updates to hand-editing; inherited by secondmate homes (section 10) config/backend runtime session-provider backend override for new tasks; LOCAL, gitignored; absent = falls through to runtime auto-detection (the runtime firstmate itself is executing inside), then tmux; tmux is the verified reference backend (docs/tmux-backend.md), herdr has its own required CI lane (docs/herdr-backend.md), while zellij, orca, and cmux remain experimental with no dedicated real-backend CI lane (docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md) - herdr and cmux can also be selected by runtime auto-detection, zellij and orca never are (always explicit), and codex-app is not accepted; see docs/codex-app-backend.md; inherited by secondmate homes under the primary-authoritative contract in secondmate-provisioning diff --git a/bin/fm-check-sort-lib.sh b/bin/fm-check-sort-lib.sh new file mode 100644 index 00000000000..f82bd041221 --- /dev/null +++ b/bin/fm-check-sort-lib.sh @@ -0,0 +1,126 @@ +# shellcheck shell=bash +# Advisory sort of a failed required check: code bug, flaky, or environment. +# Usage: . bin/fm-check-sort-lib.sh (source it before the caller runs any child) +# +# bin/fm-pr-state.sh is the one caller. It hands over the required checks it +# already read as failed, and this file prints one extra line per check: +# FAILED CHECK SORT: : () +# where is `flaky`, `environment`, `code bug`, or `unknown`. The line is +# a label for the reader. It never re-runs a check, never changes what a check +# reported, and nothing merges, blocks, or discards on it. +# +# The sort runs only when TYPESAFE_API_KEY is available (bin/fm-jev-lib.sh owns +# the key); without it nothing is read and nothing is printed. Fixed rules +# decide first, from GitHub reads alone: +# 1. Another attempt of the same workflow run passed this check on the same +# head commit, so the same code and the same trigger both passed and +# failed -> flaky. A pass from a different run does not count, because a +# check triggered again by a pull request edit saw a different input. +# 2. The failed steps' log shows a connection error (FM_CHECK_SORT_MARKERS) and +# the same check is failing on the base branch, which does not carry this +# pull request's change -> environment. +# A failure no rule decides goes to Jev as one fixed-choice question over the +# check's name and the log of the job's failed steps. Only the last +# FM_CHECK_SORT_LOG_LINES lines are read; of those, the lines that name a +# failure (FM_CHECK_SORT_EVIDENCE) are sent, or every line when none does, cut +# to the last FM_CHECK_SORT_TAIL_CHARS characters, because a job states its +# failure last and ends in summary lines that say nothing about the cause. A failure log quotes the project's +# code, so it is sent only for a repository whose name fm_jev_code_allowed +# lists in config/jev-code-projects; an unlisted repository's unclear failure +# is `unknown` with no call. `unknown` is also the answer for a check with no +# GitHub Actions job log, an unreadable read, a Jev timeout or error, a +# malformed answer, an `unclear` choice, and a confidence under the floor; the +# worker then investigates as it does without the label. `code bug` needs the +# shared FM_JEV_CONFIDENCE_FLOOR. `flaky` and `environment` need the stricter +# FM_CHECK_SORT_AWAY_FLOOR, because those two tell a worker the failure is not +# in its code, and a live call labelled a pull request process failure +# `environment` at 0.66 (docs/verification/failed-check-sort.md). +# +# ponytail: only the first FM_CHECK_SORT_MAX failed checks are sorted, so a +# pull request with many red checks costs a bounded number of reads; the rest +# keep their REQUIRED CHECK line and get no label. +# +# fm_check_sort +# Reads failed check names, one per line, on stdin. Always returns 0. + +_FM_CHECK_SORT_DIR=${BASH_SOURCE[0]%/*} +[ "$_FM_CHECK_SORT_DIR" != "${BASH_SOURCE[0]}" ] || _FM_CHECK_SORT_DIR=. +# shellcheck source=bin/fm-jev-lib.sh +. "$_FM_CHECK_SORT_DIR/fm-jev-lib.sh" + +FM_CHECK_SORT_MAX=3 +FM_CHECK_SORT_LOG_LINES=2000 +FM_CHECK_SORT_TAIL_CHARS=4000 +FM_CHECK_SORT_AWAY_FLOOR=0.8 +FM_CHECK_SORT_EVIDENCE='error|fail|not ok|panic|exception|assert|timed out|timeout|refused|denied|killed|cannot|unable' +FM_CHECK_SORT_MARKERS='ECONNREFUSED|ECONNRESET|ETIMEDOUT|EAI_AGAIN|ENOTFOUND|could not resolve host|temporary failure in name resolution|connection (refused|reset|timed out)|TLS handshake timeout|50[234] (bad gateway|service unavailable|gateway time-?out)' +# shellcheck disable=SC2016 # The backticks are literal markup in the question text. +FM_CHECK_SORT_INSTRUCTIONS='An automated check named in `check` failed on a pull request, and `log_tail` holds the last lines of the failed job'"'"'s log that name a failure. Decide from the words of the log why it failed. Choose `unclear` whenever the log does not plainly show one of the other three.' +FM_CHECK_SORT_CRITERIA='{"code_bug":"The log shows the project'"'"'s own code or tests failing on their merits: a compile, type, or lint error, a failed assertion, or a test that fails for a reason in the code.","flaky":"The log shows a failure that depends on timing, ordering, or chance rather than on the code: a race, a timing-dependent assertion, or a test that timed out waiting with no error in the code.","environment":"The log shows the runner or something outside the project failing: a network or connection error, a failed download or dependency install, an unavailable service, a rate limit, a missing credential, or a runner out of disk, memory, or time.","unclear":"The log does not show enough to tell which of the others applies."}' + +# Prints " ()" or "unknown" for one failed check. +_fm_check_sort_one() { # + local listed=$1 repo=$2 head=$3 url=$4 name=$5 runs passed job log evidence base base_state + runs=$(gh api -X GET "/repos/$repo/commits/$head/check-runs" \ + -f check_name="$name" -f filter=all -f per_page=100 --jq ' + def run: (.details_url // "") | split("/job/") | .[0] // ""; + ([.check_runs[] | select(.app.slug == "github-actions" and .status == "completed" and .conclusion != "success")] + | max_by(.id)) as $failed + | (($failed // {}) | run) as $run + | ($run != "" and any(.check_runs[]; .conclusion == "success" and run == $run) | tostring), + (($failed.id // "") | tostring)' 2>/dev/null) || { echo unknown; return 0; } + { + IFS= read -r passed + IFS= read -r job + } <\t\t "; only the text is kept. + log=$(gh run view --job "$job" --log-failed -R "$repo" 2>/dev/null \ + | tail -n "$FM_CHECK_SORT_LOG_LINES" | cut -f 3- | sed -E 's/^[0-9]{4}-[0-9T:.-]+Z //') || true + [ -n "$log" ] || { echo unknown; return 0; } + if printf '%s\n' "$log" | grep -Eiq -- "$FM_CHECK_SORT_MARKERS"; then + base=$(gh pr view "$url" --json baseRefName --jq .baseRefName 2>/dev/null) || base= + base_state= + [ -z "$base" ] || base_state=$(gh api -X GET "/repos/$repo/commits/$base/check-runs" \ + -f check_name="$name" --jq ' + [.check_runs[] | select(.status == "completed")] | max_by(.id) | .conclusion // ""' 2>/dev/null) || base_state= + case "$base_state" in + failure|timed_out) + echo 'environment (rule: connection error in its log, and it fails on the base branch too)' + return 0 ;; + esac + fi + [ "$listed" -eq 1 ] || { echo unknown; return 0; } + evidence=$(printf '%s\n' "$log" | grep -Ei -- "$FM_CHECK_SORT_EVIDENCE") || evidence=$log + fm_jev_choice failed_check "$FM_CHECK_SORT_INSTRUCTIONS" \ + <(printf '%s' "$evidence" | tail -c "$FM_CHECK_SORT_TAIL_CHARS" | jq -Rsc --arg check "$name" '{check: $check, log_tail: .}') \ + <(printf '%s' "$FM_CHECK_SORT_CRITERIA") || { echo unknown; return 0; } + jq -r --argjson floor "$FM_JEV_CONFIDENCE_FLOOR" --argjson away "$FM_CHECK_SORT_AWAY_FLOOR" ' + if .choice != "unclear" and .confidence >= (if .choice == "code_bug" then $floor else $away end) + then "\(.choice | sub("_"; " ")) (jev, confidence \(.confidence))" else "unknown" end' \ + <<<"$FM_JEV_ANSWER" 2>/dev/null || echo unknown +} + +fm_check_sort() { # ; failed check names on stdin + local home=$1 repo=$2 head=$3 url=$4 listed=0 name names=() n=0 + while IFS= read -r name; do + [ -z "$name" ] || names+=("$name") + done + [ "${#names[@]}" -gt 0 ] || return 0 + fm_jev_key_load "$home" || return 0 + command -v jq >/dev/null 2>&1 || return 0 + ! FM_HOME=$home fm_jev_code_allowed "${repo##*/}" || listed=1 + for name in "${names[@]}"; do + n=$((n + 1)) + [ "$n" -le "$FM_CHECK_SORT_MAX" ] || break + printf 'FAILED CHECK SORT: %s: %s\n' "$name" \ + "$(_fm_check_sort_one "$listed" "$repo" "$head" "$url" "$name" # 0 when a key is available, 1 when it is absent from both sources. # +# fm_jev_code_allowed +# 0 only when is a line of $FM_HOME/config/jev-code-projects (one +# project name per line; blank lines and lines starting with # are ignored). +# The key alone lets a caller send routing text and status lines; a caller +# that would send a project's code, file list, or pull request text asks this +# first. An absent or empty file allows no project. +# # fm_jev_choice # One POST to $FM_JEV_BASE/v1/systemone. holds the JSON # value sent as `state`; holds one JSON object mapping @@ -82,6 +90,11 @@ fm_jev_key_load() { # [ -n "$TYPESAFE_API_KEY_PRIVATE" ] } +fm_jev_code_allowed() { # + [ -n "$1" ] && [ "${1#\#}" = "$1" ] \ + && grep -qxF -- "$1" "${FM_HOME:-.}/config/jev-code-projects" 2>/dev/null +} + _fm_jev_fail() { # FM_JEV_STATUS=error FM_JEV_ERROR=$1 diff --git a/bin/fm-pr-state.sh b/bin/fm-pr-state.sh index b7b1c2b5367..2b4fd6c275f 100755 --- a/bin/fm-pr-state.sh +++ b/bin/fm-pr-state.sh @@ -17,6 +17,10 @@ # STALE when it was left at a superseded head. # A closed or merged pull request reports that terminal state and nothing else. # Unresolved review-thread state is out of this command's scope. +# When TYPESAFE_API_KEY is available, each failed required check also gets one +# advisory FAILED CHECK SORT line labelling it a code bug, flaky, environment, +# or unknown; bin/fm-check-sort-lib.sh owns that sort. The label reads more of +# GitHub and changes nothing a check reported. # # Usage: fm-pr-state.sh # Prints one line per blocker it can see and nothing when it sees none. @@ -26,6 +30,9 @@ set -eu SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# Sourced before any child runs, so no gh call inherits the Jev key. +# shellcheck source=bin/fm-check-sort-lib.sh +. "$SCRIPT_DIR/fm-check-sort-lib.sh" # shellcheck source=bin/fm-pr-lib.sh . "$SCRIPT_DIR/fm-pr-lib.sh" @@ -106,10 +113,16 @@ esac GH_STDERR=$(mktemp "${TMPDIR:-/tmp}/fm-pr-state.XXXXXX") \ || die "could not create temporary file" trap 'rm -f "$GH_STDERR"' EXIT INT TERM -if ! REQUIRED=$(gh pr checks "$URL" --required --json name,state,bucket --jq ' +FAILED= +if REQUIRED=$(gh pr checks "$URL" --required --json name,state,bucket --jq ' .[] | select(.bucket != "pass" and .bucket != "skipping") - | "REQUIRED CHECK: \(.name) (\(.state))"' 2>"$GH_STDERR"); then + | [.bucket, .state, .name] + | @tsv' 2>"$GH_STDERR"); then + FAILED=$(printf '%s\n' "$REQUIRED" | awk -F '\t' '$1 == "fail" { print $3 }') + REQUIRED=$(printf '%s\n' "$REQUIRED" \ + | awk -F '\t' 'NF >= 3 { printf "REQUIRED CHECK: %s (%s)\n", $3, $2 }') +else # These two sentences are gh's own human-readable error text, verified against # gh 2.100.0 on 2026-09-12. gh reports "nothing reported" as an error rather # than as structured data, so matching its text is the only way to tell that @@ -125,6 +138,9 @@ if ! REQUIRED=$(gh pr checks "$URL" --required --json name,state,bucket --jq ' fi fi [ -z "$REQUIRED" ] || printf '%s\n' "$REQUIRED" +fm_check_sort "${FM_HOME:-$SCRIPT_DIR/..}" "$PATH_PART" "$HEAD" "$URL" < --log-failed`. +No rule fired on any row, so each run is one real question; the 18 calls took 298 to 808 ms. + +| Commit | Check | What its log shows | Choice and confidence, three runs | Printed label | +| --- | --- | --- | --- | --- | +| 30fc910f | Behavior portable serial 4 | one `not ok` test assertion | code_bug 0.88, 0.89, 0.89 | code bug | +| dfa2369b | Behavior portable serial 4 | one `not ok` test assertion | code_bug 0.99, 0.98, 0.99 | code bug | +| 30fc910f | Behavior portable serial 2 | one `not ok` test assertion among summary lines | code_bug 0.41, 0.48, 0.48 | unknown | +| f84f2f35 | Behavior portable serial 1 | one `not ok` timing assertion | flaky 0.33, 0.28, 0.37 | unknown | +| f84f2f35 | Behavior portable serial 3 | summary lines only | unclear 0.59, 0.55, 0.58 | unknown | +| 30fc910f | PR must be raised via no-mistakes | a pull request process check refusing a head it did not validate | environment 0.73, 0.72, 0.65 | unknown | + +The last row is the reason `flaky` and `environment` need 0.8 rather than the shared 0.6: that failure is the author's to fix, and the model called it `environment` above 0.6 on every run. +The choice was the same on all three runs of every row, and the confidence moved by at most 0.09. + +The same check on the same commit also has a passing run that a later pull request edit triggered. +Rule 1 does not call it flaky, because that pass belongs to a different workflow run: + +```text +$ gh api "repos/talktejas/firstmate/commits/30fc910f8dac43b550cf46a9099d94ea90e2131b/check-runs?filter=all&per_page=100" --jq '.check_runs[] | select(.name|startswith("PR must")) | [.id,.conclusion,.details_url]|@tsv' +110078953481 failure https://github.com/talktejas/firstmate/actions/runs/36771558348/job/110078953481 +110071234065 success https://github.com/talktejas/firstmate/actions/runs/36769277507/job/110071234065 +110064348154 failure https://github.com/talktejas/firstmate/actions/runs/36767230944/job/110064348154 +``` + +With the same commit and check but a fixture home that lists no project, the label was `unknown` and no question was sent. + +gh 2.100.0 refuses to print a job's raw log from `gh api /repos///actions/jobs//logs` ("the response contains terminal escape sequences"), which is why the log is read through `gh run view --log-failed`. + +## Stub-only coverage + +No open pull request had a failing required check on the run date, so these were not observed live. +`tests/fm-pr-state.test.sh` covers them with a fake `gh` that answers in GitHub's JSON shapes and `fm_jev_choice` stubbed at the library boundary, against fixture homes with no key in the environment: + +- the `flaky` rule firing on a pass from another attempt of the same run, and not on a pass from a different run; +- the `environment` rule firing on a connection error whose check also fails on the base branch; +- `bin/fm-pr-state.sh` itself printing the label beside its unchanged blocker lines, with a fake `curl` that fails, and the key reaching neither `curl`'s argv or environment nor any `gh` call; +- no key meaning no extra GitHub read and no extra line; +- an unlisted or commented-out repository name, a check with no job log, a failed call, an `unclear` choice, and an answer under either floor all printing `unknown`; +- only the failure-naming lines being sent, cut to the last 4000 characters, and only the first three failed checks being sorted. + +`FM_PR_STATE_LIVE_E2E=1 bash tests/fm-pr-state-live-e2e.test.sh` is the command that confirms gh's own jq engine still accepts the script's pull-request read programs. diff --git a/tests/fm-pr-state.test.sh b/tests/fm-pr-state.test.sh index d911f674147..37b8c36513d 100755 --- a/tests/fm-pr-state.test.sh +++ b/tests/fm-pr-state.test.sh @@ -50,26 +50,79 @@ serve() { checks='[{"name":"lint","state":"SUCCESS","bucket":"pass","workflow":"ci"},{"name":"optional","state":"SKIPPED","bucket":"skipping","workflow":"ci"}]' printf '%s\n' "${FM_TEST_REQUIRED_CHECKS:-$checks}" ;; + "api -X GET /repos/o/r/commits/$head/check-runs -f check_name="*" -f filter=all -f per_page=100 --jq "*) + printf '%s\n' "${FM_TEST_HEAD_RUNS:-{\"check_runs\":[]\}}" + ;; + "run view --job 41 --log-failed -R o/r") + printf '%s\n' "${FM_TEST_LOG-}" + ;; + "pr view "*" --json baseRefName --jq .baseRefName") + printf '{"baseRefName":"main"}\n' + ;; + "api -X GET /repos/o/r/commits/main/check-runs -f check_name="*" --jq "*) + printf '%s\n' "${FM_TEST_BASE_RUNS:-{\"check_runs\":[]\}}" + ;; *) printf 'unexpected gh call: %s\n' "$*" >&2 exit 91 ;; esac } +[ -z "${FM_TEST_GH_CALLS-}" ] || printf '%s key=%s\n' "$*" "${TYPESAFE_API_KEY-}" >> "$FM_TEST_GH_CALLS" prog= prev= for arg in "$@"; do [ "$prev" != --jq ] || prog=$arg prev=$arg done -serve "$@" | jq -r "$prog" +if [ -n "$prog" ]; then serve "$@" | jq -r "$prog"; else serve "$@"; fi SH chmod +x "$FAKEBIN/gh" +# Every run is pinned to a fixture home with no key in the environment, so no +# real key can load and the sort stays off unless a test turns it on. +HOME_OFF="$TMP_ROOT/home-off" +HOME_ON="$TMP_ROOT/home-on" +mkdir -p "$HOME_OFF" "$HOME_ON/config" +printf 'TYPESAFE_API_KEY=fixture-key\n' > "$HOME_ON/.env" +GH_CALLS="$TMP_ROOT/gh-calls" +JEV_CALLS="$TMP_ROOT/jev-calls" +JEV_STATE="$TMP_ROOT/jev-state" +FAILING='[{"name":"CI Status","state":"FAILURE","bucket":"fail","workflow":"ci"},{"name":"slow","state":"IN_PROGRESS","bucket":"pending","workflow":"ci"}]' +RUN_URL=https://github.com/o/r/actions/runs +FAILED_RUN='{"check_runs":[{"id":41,"status":"completed","conclusion":"failure","details_url":"'$RUN_URL'/5/job/41","app":{"slug":"github-actions"}}]}' +passed_too() { # + jq -c --arg url "$RUN_URL/$1/job/40" '.check_runs += [{id: 40, status: "completed", conclusion: "success", details_url: $url, app: {slug: "github-actions"}}]' <<<"$FAILED_RUN" +} + run_state() { - PATH="$FAKEBIN:$PATH" "$SCRIPT" https://github.com/o/r/pull/7 + env -u TYPESAFE_API_KEY FM_HOME="${FM_TEST_HOME:-$HOME_OFF}" PATH="$FAKEBIN:$PATH" \ + "$SCRIPT" https://github.com/o/r/pull/7 } +# Runs fm_check_sort over one failed check with fm_jev_choice replaced at the +# library boundary. STUB_CHOICE and STUB_CONF are the answer; STUB_FAIL makes +# the call an error. Each call's state lands in $JEV_STATE. +run_sort() { + rm -f "$GH_CALLS" "$JEV_CALLS" "$JEV_STATE" + # shellcheck disable=SC2016 # The inner shell expands its own variables. + env -u TYPESAFE_API_KEY FM_TEST_GH_CALLS="$GH_CALLS" JEV_CALLS="$JEV_CALLS" JEV_STATE="$JEV_STATE" \ + PATH="$FAKEBIN:$PATH" bash -c ' + set -eu + # shellcheck source=/dev/null + . "$1" + fm_jev_choice() { + echo "$1" >> "$JEV_CALLS" + cat "$3" > "$JEV_STATE" + jq -e "has(\"code_bug\") and has(\"flaky\") and has(\"environment\") and has(\"unclear\") and length == 4" "$4" >/dev/null + [ -z "${STUB_FAIL:-}" ] || { FM_JEV_STATUS=error; return 1; } + FM_JEV_ANSWER=$(jq -cn --arg c "${STUB_CHOICE:-code_bug}" --argjson k "${STUB_CONF:-0.9}" "{choice: \$c, confidence: \$k}") + } + printf "%s\n" "${FAILED_NAMES:-CI Status}" | fm_check_sort "$2" o/r "$3" https://github.com/o/r/pull/7 + ' _ "$ROOT/bin/fm-check-sort-lib.sh" "${FM_TEST_HOME:-$HOME_ON}" "$HEAD" +} +jev_calls() { [ -f "$JEV_CALLS" ] && wc -l < "$JEV_CALLS" | tr -d ' ' || echo 0; } + # reviews " "... prints the JSON array # GitHub's reviews endpoint returns for those submissions. reviews() { @@ -196,6 +249,119 @@ test_required_failure_is_a_blocker() { pass "required failure blocks readiness" } +test_failed_check_sort_is_off_without_a_key() { + local out + rm -f "$GH_CALLS" + out=$(FM_TEST_REQUIRED_CHECKS=$FAILING FM_TEST_GH_CALLS="$GH_CALLS" run_state) \ + || fail "blocked fixture was refused" + [ "$out" = "$(printf 'REQUIRED CHECK: CI Status (FAILURE)\nREQUIRED CHECK: slow (IN_PROGRESS)')" ] \ + || fail "without a key the report is exactly the blocker lines, got: $out" + ! grep -q 'check-runs\|log-failed' "$GH_CALLS" || fail "without a key nothing more is read from GitHub" + out=$(FM_TEST_HOME="$HOME_OFF" FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG='boom' run_sort) + [ -z "$out" ] || fail "without a key the sort prints nothing, got: $out" + [ "$(jev_calls)" = 0 ] || fail "without a key the model is never asked" + pass "without a key the report and the GitHub reads are unchanged" +} + +test_failed_check_sort_rules_decide_first() { + local out + out=$(FM_TEST_HEAD_RUNS=$(passed_too 5) run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: flaky (rule: another attempt of the same run passed)' ] \ + || fail "a check that passed on another attempt of its run is flaky, got: $out" + assert_no_grep '--log-failed' "$GH_CALLS" "a flaky verdict needs no log" + out=$(FM_TEST_HEAD_RUNS=$(passed_too 6) FM_TEST_LOG='boom' run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ + || fail "a pass from a different run, such as one a pull request edit triggered, is not a retry, got: $out" + + printf 'o\nr\n' > "$HOME_ON/config/jev-code-projects" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN \ + FM_TEST_LOG=$(printf 'build\tinstall\t2026-10-06T10:00:00.1234567Z curl: (6) Could not resolve host: registry.example') \ + FM_TEST_BASE_RUNS='{"check_runs":[{"id":9,"status":"completed","conclusion":"failure"}]}' run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: environment (rule: connection error in its log, and it fails on the base branch too)' ] \ + || fail "a connection error on a check the base branch fails too is environment, got: $out" + [ "$(jev_calls)" = 0 ] || fail "a failure a rule decides is never sent to the model" + rm -f "$HOME_ON/config/jev-code-projects" + pass "fixed rules label flaky and environment failures without asking the model" +} + +test_failed_check_sort_asks_only_for_a_listed_project() { + local out log i=0 + log=$(printf 'build\ttest\t2026-10-06T10:00:00.0000000Z not ok - early failure\n' + while [ "$i" -lt 400 ]; do i=$((i + 1)); printf 'build\ttest\t2026-10-06T10:00:00.0000000Z ok - passing line %s of a long job log\n' "$i"; done + printf '2026-10-06T10:00:01.0000000Z ECONNRESET\n2026-10-06T10:00:02.0000000Z assertion: expected 2 but got 3 THE-END\n') + + rm -f "$HOME_ON/config/jev-code-projects" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ + || fail "an unclear failure in an unlisted repository is unknown, got: $out" + [ "$(jev_calls)" = 0 ] || fail "an unlisted repository's log is never sent" + + printf '# r\n' > "$HOME_ON/config/jev-code-projects" + FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log run_sort >/dev/null + [ "$(jev_calls)" = 0 ] || fail "a commented-out name does not list the repository" + + printf '# listed\nr\n' > "$HOME_ON/config/jev-code-projects" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: code bug (jev, confidence 0.9)' ] \ + || fail "a confident answer labels the check, got: $out" + [ "$(jev_calls)" = 1 ] || fail "one unclear failure is one question" + [ "$(jq -r .check "$JEV_STATE")" = 'CI Status' ] || fail "the state names the check" + [ "$(jq -r .log_tail "$JEV_STATE")" = "$(printf 'not ok - early failure\nassertion: expected 2 but got 3 THE-END')" ] \ + || fail "only the lines naming a failure are sent, without the job, step, and timestamp columns, got: $(cat "$JEV_STATE")" + FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$(i=0; while [ "$i" -lt 400 ]; do i=$((i + 1)); printf 'error number %s\n' "$i"; done) run_sort >/dev/null + jq -e '.log_tail | length == 4000 and endswith("error number 400")' "$JEV_STATE" >/dev/null \ + || fail "long evidence is cut to its bound and keeps its end" + FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG='it just stopped' run_sort >/dev/null + [ "$(jq -r .log_tail "$JEV_STATE")" = 'it just stopped' ] || fail "a log naming no failure is sent as it is" + [ "$(jq -r 'keys | join(",")' "$JEV_STATE")" = 'check,log_tail' ] || fail "nothing else is sent" + grep -q 'log-failed -R o/r key=$' "$GH_CALLS" || fail "the job log was read" + ! grep -q 'key=.' "$GH_CALLS" || fail "no gh call inherits the key" + + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_CONF=0.59 run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] || fail "an answer under the floor is unknown, got: $out" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_CHOICE=environment STUB_CONF=0.79 run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ + || fail "a label that points away from the code needs the stricter floor, got: $out" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_CHOICE=flaky STUB_CONF=0.8 run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: flaky (jev, confidence 0.8)' ] \ + || fail "a flaky answer at the stricter floor labels the check, got: $out" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_CHOICE=unclear run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] || fail "an unclear answer is unknown, got: $out" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_FAIL=1 run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] || fail "a failed call is unknown, got: $out" + out=$(FM_TEST_LOG=$log run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] || fail "a check with no job log is unknown, got: $out" + [ "$(jev_calls)" = 0 ] || fail "a check with no log is never sent" + + out=$(FAILED_NAMES=$(printf 'a\nb\nc\nd') FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log run_sort) + [ "$(printf '%s\n' "$out" | wc -l | tr -d ' ')" = 3 ] || fail "only the first three failed checks are sorted, got: $out" + rm -f "$HOME_ON/config/jev-code-projects" + pass "only a listed repository's unclear failure is asked, and every doubt is unknown" +} + +test_failed_check_sort_never_changes_the_report() { + local out status=0 + rm -f "$GH_CALLS" + # The fixture key reaches a fake curl that fails, as a dead endpoint would. + cat > "$FAKEBIN/curl" <<'SH' +#!/bin/sh +printf '%s env=%s\n' "$*" "${TYPESAFE_API_KEY-}" >> "$FM_TEST_CURL_CALLS" +exit 7 +SH + chmod +x "$FAKEBIN/curl" + printf 'r\n' > "$HOME_ON/config/jev-code-projects" + out=$(FM_TEST_HOME="$HOME_ON" FM_TEST_REQUIRED_CHECKS=$FAILING FM_TEST_HEAD_RUNS=$FAILED_RUN \ + FM_TEST_LOG='assert failed' FM_TEST_GH_CALLS="$GH_CALLS" FM_TEST_CURL_CALLS="$TMP_ROOT/curl-calls" run_state) || status=$? + rm -f "$FAKEBIN/curl" "$HOME_ON/config/jev-code-projects" + [ "$status" -eq 0 ] || fail "a failed model call must not change the exit status" + [ "$out" = "$(printf 'REQUIRED CHECK: CI Status (FAILURE)\nREQUIRED CHECK: slow (IN_PROGRESS)\nFAILED CHECK SORT: CI Status: unknown')" ] \ + || fail "the blocker lines stay and only the failed check gains a label, got: $out" + assert_present "$TMP_ROOT/curl-calls" "the model was asked through the shared caller" + assert_no_grep 'fixture-key' "$TMP_ROOT/curl-calls" "the key reaches neither curl's argv nor its environment" + assert_no_grep 'fixture-key' "$GH_CALLS" "the key reaches no gh call" + pass "with the model down the blocker lines are intact and the check is unknown" +} + # The next two cases supply gh's own "nothing reported" sentences through # FM_TEST_CHECKS_ERROR, so they prove the behaviour GIVEN those strings and # nothing about the strings themselves. A gh reword is invisible to this @@ -279,6 +445,10 @@ test_changes_requested_decision_is_never_silent test_authors_own_changes_requested_review_is_not_a_blocker test_pending_approval_is_not_a_blocker test_required_failure_is_a_blocker +test_failed_check_sort_is_off_without_a_key +test_failed_check_sort_rules_decide_first +test_failed_check_sort_asks_only_for_a_listed_project +test_failed_check_sort_never_changes_the_report test_unreported_required_checks_are_unconfirmed test_no_reported_checks_is_unverified test_help_states_what_silence_means_and_what_is_out_of_scope From 40b1808e2bf3d33412405635bcc4025bd4a7e644 Mon Sep 17 00:00:00 2001 From: Tejas Shah Date: Tue, 6 Oct 2026 02:35:15 +0700 Subject: [PATCH 2/6] no-mistakes(review): make failed check sort opt-in, match owner/repo, label all --- AGENTS.md | 2 +- bin/fm-check-sort-lib.sh | 24 ++++++++----- bin/fm-jev-lib.sh | 7 ++-- bin/fm-pr-state.sh | 33 ++++++++++------- docs/configuration.md | 9 +++-- docs/verification/failed-check-sort.md | 6 ++-- tests/fm-pr-state.test.sh | 50 +++++++++++++++++++++----- 7 files changed, 93 insertions(+), 38 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 678ff26ec90..e12d208ae69 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,7 +74,7 @@ config/crew-harness crewmate harness override; LOCAL, gitignored; absent or "de config/claude-permission-mode optional one-token permission posture for every Claude worker launch: absent or "bypass" keeps --dangerously-skip-permissions, "auto" launches with --permission-mode auto; LOCAL, gitignored; inherited by secondmate homes; see docs/configuration.md "Claude permission mode" config/crew-dispatch.json optional crewmate dispatch profiles; LOCAL, gitignored; firstmate-maintained but human-editable natural-language rules that choose a per-task harness/model/effort profile (section 4). Inherited by secondmate homes config/house-rules.json optional opt-in list of project names, and optional rules, for the advisory house-rules check: a ship brief asks its worker to run the check only for a project named in the file's "projects" list, and only when TYPESAFE_API_KEY is present; absent means off for every project; LOCAL, gitignored, and not inherited; see docs/configuration.md "House-rules check" -config/jev-code-projects optional list of project names, one per line, whose code or failure logs a Jev check may send to typesafe.ai; absent means none; LOCAL, gitignored, and not inherited; see docs/configuration.md "Failed check sort" +config/jev-code-projects optional list, one entry per line, of the projects whose code or failure logs a Jev check may send to typesafe.ai: `/` for a check keyed by repository, matched exactly, or a firstmate project name for a check keyed by project; absent means none; LOCAL, gitignored, and not inherited; see docs/configuration.md "Failed check sort" config/secondmate-harness harness the PRIMARY uses to launch SECONDMATE agents, optionally followed by a model and effort token on the same line (" [] []"; section 4); LOCAL, gitignored; absent or "default" harness falls back to config/crew-harness then firstmate's own. The primary's own setting; NOT inherited into secondmate homes (secondmates do not spawn secondmates) config/backlog-backend backlog backend override; LOCAL, gitignored; absent or "tasks-axi" = the configured tasks-axi backend, "manual" = force routine backlog updates to hand-editing; inherited by secondmate homes (section 10) config/backend runtime session-provider backend override for new tasks; LOCAL, gitignored; absent = falls through to runtime auto-detection (the runtime firstmate itself is executing inside), then tmux; tmux is the verified reference backend (docs/tmux-backend.md), herdr has its own required CI lane (docs/herdr-backend.md), while zellij, orca, and cmux remain experimental with no dedicated real-backend CI lane (docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md) - herdr and cmux can also be selected by runtime auto-detection, zellij and orca never are (always explicit), and codex-app is not accepted; see docs/codex-app-backend.md; inherited by secondmate homes under the primary-authoritative contract in secondmate-provisioning diff --git a/bin/fm-check-sort-lib.sh b/bin/fm-check-sort-lib.sh index f82bd041221..8a4b27d0efa 100644 --- a/bin/fm-check-sort-lib.sh +++ b/bin/fm-check-sort-lib.sh @@ -2,8 +2,9 @@ # Advisory sort of a failed required check: code bug, flaky, or environment. # Usage: . bin/fm-check-sort-lib.sh (source it before the caller runs any child) # -# bin/fm-pr-state.sh is the one caller. It hands over the required checks it -# already read as failed, and this file prints one extra line per check: +# bin/fm-pr-state.sh is the one caller, and only under its +# --sort-failed-checks option. It hands over the required checks it already +# read as failed, and this file prints one extra line per check: # FAILED CHECK SORT: : () # where is `flaky`, `environment`, `code bug`, or `unknown`. The line is # a label for the reader. It never re-runs a check, never changes what a check @@ -25,9 +26,10 @@ # failure (FM_CHECK_SORT_EVIDENCE) are sent, or every line when none does, cut # to the last FM_CHECK_SORT_TAIL_CHARS characters, because a job states its # failure last and ends in summary lines that say nothing about the cause. A failure log quotes the project's -# code, so it is sent only for a repository whose name fm_jev_code_allowed -# lists in config/jev-code-projects; an unlisted repository's unclear failure -# is `unknown` with no call. `unknown` is also the answer for a check with no +# code, so it is sent only for a repository fm_jev_code_allowed finds in +# config/jev-code-projects as its exact `/`; a bare name there +# never matches a repository. An unlisted repository's unclear failure is +# `unknown` with no call. `unknown` is also the answer for a check with no # GitHub Actions job log, an unreadable read, a Jev timeout or error, a # malformed answer, an `unclear` choice, and a confidence under the floor; the # worker then investigates as it does without the label. `code bug` needs the @@ -37,8 +39,8 @@ # `environment` at 0.66 (docs/verification/failed-check-sort.md). # # ponytail: only the first FM_CHECK_SORT_MAX failed checks are sorted, so a -# pull request with many red checks costs a bounded number of reads; the rest -# keep their REQUIRED CHECK line and get no label. +# pull request with many red checks costs a bounded number of reads; each of +# the rest is printed as `unknown (not sorted)` with nothing read. # # fm_check_sort # Reads failed check names, one per line, on stdin. Always returns 0. @@ -51,6 +53,7 @@ _FM_CHECK_SORT_DIR=${BASH_SOURCE[0]%/*} FM_CHECK_SORT_MAX=3 FM_CHECK_SORT_LOG_LINES=2000 FM_CHECK_SORT_TAIL_CHARS=4000 +# A live call labelled a process failure `environment` at 0.65 to 0.73 (docs/verification/failed-check-sort.md). FM_CHECK_SORT_AWAY_FLOOR=0.8 FM_CHECK_SORT_EVIDENCE='error|fail|not ok|panic|exception|assert|timed out|timeout|refused|denied|killed|cannot|unable' FM_CHECK_SORT_MARKERS='ECONNREFUSED|ECONNRESET|ETIMEDOUT|EAI_AGAIN|ENOTFOUND|could not resolve host|temporary failure in name resolution|connection (refused|reset|timed out)|TLS handshake timeout|50[234] (bad gateway|service unavailable|gateway time-?out)' @@ -115,10 +118,13 @@ fm_check_sort() { # ; failed check names [ "${#names[@]}" -gt 0 ] || return 0 fm_jev_key_load "$home" || return 0 command -v jq >/dev/null 2>&1 || return 0 - ! FM_HOME=$home fm_jev_code_allowed "${repo##*/}" || listed=1 + ! FM_HOME=$home fm_jev_code_allowed "$repo" || listed=1 for name in "${names[@]}"; do n=$((n + 1)) - [ "$n" -le "$FM_CHECK_SORT_MAX" ] || break + if [ "$n" -gt "$FM_CHECK_SORT_MAX" ]; then + printf 'FAILED CHECK SORT: %s: unknown (not sorted)\n' "$name" + continue + fi printf 'FAILED CHECK SORT: %s: %s\n' "$name" \ "$(_fm_check_sort_one "$listed" "$repo" "$head" "$url" "$name" -# 0 only when is a line of $FM_HOME/config/jev-code-projects (one -# project name per line; blank lines and lines starting with # are ignored). +# 0 only when is a whole line of $FM_HOME/config/jev-code-projects +# (one entry per line; blank lines and lines starting with # are ignored). +# A caller keyed by repository passes `/`, and a caller keyed by +# a firstmate project passes that project's name; neither form matches the +# other, so a bare name never allows a repository by its basename. # The key alone lets a caller send routing text and status lines; a caller # that would send a project's code, file list, or pull request text asks this # first. An absent or empty file allows no project. diff --git a/bin/fm-pr-state.sh b/bin/fm-pr-state.sh index 2b4fd6c275f..d46e450e082 100755 --- a/bin/fm-pr-state.sh +++ b/bin/fm-pr-state.sh @@ -17,12 +17,15 @@ # STALE when it was left at a superseded head. # A closed or merged pull request reports that terminal state and nothing else. # Unresolved review-thread state is out of this command's scope. -# When TYPESAFE_API_KEY is available, each failed required check also gets one -# advisory FAILED CHECK SORT line labelling it a code bug, flaky, environment, -# or unknown; bin/fm-check-sort-lib.sh owns that sort. The label reads more of -# GitHub and changes nothing a check reported. +# With --sort-failed-checks and TYPESAFE_API_KEY available, each failed +# required check also gets one advisory FAILED CHECK SORT line labelling it a +# code bug, flaky, environment, or unknown; bin/fm-check-sort-lib.sh owns that +# sort. The label reads more of GitHub and changes nothing a check reported. +# Firstmate passes the option when it reads a pull request's failed checks by +# hand. The watcher's routine-wake triage does not, so its read costs and +# prints what it does without the sort. # -# Usage: fm-pr-state.sh +# Usage: fm-pr-state.sh [--sort-failed-checks] # Prints one line per blocker it can see and nothing when it sees none. # Blockers do not change the successful exit status; lookup or usage refusal # exits non-zero. @@ -49,7 +52,12 @@ if [ "${1:-}" = --help ] || [ "${1:-}" = -h ]; then usage exit 0 fi -[ "$#" -eq 1 ] || die "usage: fm-pr-state.sh " +SORT=0 +if [ "${1:-}" = --sort-failed-checks ]; then + SORT=1 + shift +fi +[ "$#" -eq 1 ] || die "usage: fm-pr-state.sh [--sort-failed-checks] " command -v gh >/dev/null 2>&1 || die "gh is required" URL=$1 @@ -117,11 +125,10 @@ FAILED= if REQUIRED=$(gh pr checks "$URL" --required --json name,state,bucket --jq ' .[] | select(.bucket != "pass" and .bucket != "skipping") - | [.bucket, .state, .name] - | @tsv' 2>"$GH_STDERR"); then - FAILED=$(printf '%s\n' "$REQUIRED" | awk -F '\t' '$1 == "fail" { print $3 }') - REQUIRED=$(printf '%s\n' "$REQUIRED" \ - | awk -F '\t' 'NF >= 3 { printf "REQUIRED CHECK: %s (%s)\n", $3, $2 }') + | "REQUIRED CHECK: \(.name) (\(.state))", + (select(.bucket == "fail") | "FAILED CHECK NAME: \(.name)")' 2>"$GH_STDERR"); then + FAILED=$(printf '%s\n' "$REQUIRED" | sed -n 's/^FAILED CHECK NAME: //p') + REQUIRED=$(printf '%s\n' "$REQUIRED" | sed '/^FAILED CHECK NAME: /d') else # These two sentences are gh's own human-readable error text, verified against # gh 2.100.0 on 2026-09-12. gh reports "nothing reported" as an error rather @@ -138,9 +145,11 @@ else fi fi [ -z "$REQUIRED" ] || printf '%s\n' "$REQUIRED" -fm_check_sort "${FM_HOME:-$SCRIPT_DIR/..}" "$PATH_PART" "$HEAD" "$URL" <` adds one advisory line under each failed required check it reports, labelling the failure a code bug, flaky, environment, or unknown, so a worker is not sent to fix a failure that is not in its code. +The sort runs only when the caller passes that option: firstmate passes it when it reads a pull request's failed checks by hand, and the watcher's routine-wake triage never does, so its recheck reads and prints what it does without the feature and sends nothing. +With the option it is still off unless `TYPESAFE_API_KEY` is available under the same environment-then-`.env` contract as [typed dispatch resolution](#typed-dispatch-resolution-env-typesafe_api_key); with the key absent the command reads and prints exactly what it does without the feature. The label never re-runs a check, never changes what a check reported, and never changes the command's blocker lines or exit status, so nothing merges, blocks, or discards on it. Fixed rules decide first from GitHub reads alone: a check that another attempt of the same workflow run passed is flaky, and a check whose failed steps log a connection error while the same check also fails on the base branch is environment. @@ -338,7 +339,9 @@ A `code_bug` answer needs the shared 0.6 confidence floor, and a `flaky` or `env A timeout, a transport or API error, a malformed answer, an `unclear` choice, an answer under its floor, a check with no GitHub Actions job log, and an unreadable GitHub read all print `unknown`, and the worker investigates as it does without the label. The question carries the check's name and the failure-naming lines of the job's failed steps, cut to their last 4000 characters. -A failure log quotes the project's code, so it is sent only for a repository listed in the optional local, gitignored `config/jev-code-projects`, one project name per line, with blank lines and lines starting with `#` ignored; the name compared is the repository name in the pull request's address. +A failure log quotes the project's code, so it is sent only for a repository listed in the optional local, gitignored `config/jev-code-projects`, one entry per line, with blank lines and lines starting with `#` ignored. +An entry takes one of two forms: `/` matches a caller keyed by repository, such as this sort, only when it equals the pull request's owner and repository exactly, so a same-named repository under another owner is not allowed; a bare name matches only a caller that passes a firstmate project name, and never a repository's basename. +Only the first three failed required checks are sorted, and each one after them is printed as `unknown (not sorted)` with nothing read. An unlisted repository still gets the two fixed rules, and its undecided failures are `unknown` with no call. The sort covers GitHub pull requests only, which is all `bin/fm-pr-state.sh` reads, and it is the same on every harness and runtime backend because the command reads the forge and no worker surface. diff --git a/docs/verification/failed-check-sort.md b/docs/verification/failed-check-sort.md index 797461dd961..865e6478880 100644 --- a/docs/verification/failed-check-sort.md +++ b/docs/verification/failed-check-sort.md @@ -10,6 +10,7 @@ The API shape itself is recorded in [`dispatch-resolve.md`](dispatch-resolve.md) Run 2026-10-06 against `https://api.typesafe.ai`, model `jev-latest` answering as `jev-1.13.0`, timeout 5 s, with gh 2.100.0. Each row is `fm_check_sort` from `bin/fm-check-sort-lib.sh` run three times on one real failed check of `talktejas/firstmate`, with its rules, question, and criteria unmodified, the key read from the environment by `bin/fm-jev-lib.sh`, and a fixture home whose `config/jev-code-projects` lists `firstmate`. +The list has since been matched as the exact `/`, so the same run now needs the entry `talktejas/firstmate`. Every read of GitHub was real: the check runs of the commit, and the failed steps' log through `gh run view --job --log-failed`. No rule fired on any row, so each run is one real question; the 18 calls took 298 to 808 ms. @@ -47,8 +48,9 @@ No open pull request had a failing required check on the run date, so these were - the `flaky` rule firing on a pass from another attempt of the same run, and not on a pass from a different run; - the `environment` rule firing on a connection error whose check also fails on the base branch; - `bin/fm-pr-state.sh` itself printing the label beside its unchanged blocker lines, with a fake `curl` that fails, and the key reaching neither `curl`'s argv or environment nor any `gh` call; +- a call without `--sort-failed-checks`, key and listed repository present, making no check-run read, no log download, and no model call; - no key meaning no extra GitHub read and no extra line; -- an unlisted or commented-out repository name, a check with no job log, a failed call, an `unclear` choice, and an answer under either floor all printing `unknown`; -- only the failure-naming lines being sent, cut to the last 4000 characters, and only the first three failed checks being sorted. +- an unlisted or commented-out repository, a bare repository name, a same-named repository under another owner, a check with no job log, a failed call, an `unclear` choice, and an answer under either floor all printing `unknown`; +- only the failure-naming lines being sent, cut to the last 4000 characters, and only the first three failed checks being sorted, with a fourth printed as `unknown (not sorted)`. `FM_PR_STATE_LIVE_E2E=1 bash tests/fm-pr-state-live-e2e.test.sh` is the command that confirms gh's own jq engine still accepts the script's pull-request read programs. diff --git a/tests/fm-pr-state.test.sh b/tests/fm-pr-state.test.sh index 37b8c36513d..f106aeea2d3 100755 --- a/tests/fm-pr-state.test.sh +++ b/tests/fm-pr-state.test.sh @@ -97,7 +97,7 @@ passed_too() { # run_state() { env -u TYPESAFE_API_KEY FM_HOME="${FM_TEST_HOME:-$HOME_OFF}" PATH="$FAKEBIN:$PATH" \ - "$SCRIPT" https://github.com/o/r/pull/7 + "$SCRIPT" "$@" https://github.com/o/r/pull/7 } # Runs fm_check_sort over one failed check with fm_jev_choice replaced at the @@ -249,10 +249,32 @@ test_required_failure_is_a_blocker() { pass "required failure blocks readiness" } +test_failed_check_sort_is_off_unless_asked_for() { + local out + rm -f "$GH_CALLS" "$TMP_ROOT/curl-calls" + cat > "$FAKEBIN/curl" <<'SH' +#!/bin/sh +echo called >> "$FM_TEST_CURL_CALLS" +exit 7 +SH + chmod +x "$FAKEBIN/curl" + printf 'o/r\n' > "$HOME_ON/config/jev-code-projects" + out=$(FM_TEST_HOME="$HOME_ON" FM_TEST_REQUIRED_CHECKS=$FAILING FM_TEST_HEAD_RUNS=$FAILED_RUN \ + FM_TEST_LOG='assert failed' FM_TEST_GH_CALLS="$GH_CALLS" FM_TEST_CURL_CALLS="$TMP_ROOT/curl-calls" run_state) \ + || fail "blocked fixture was refused" + rm -f "$FAKEBIN/curl" "$HOME_ON/config/jev-code-projects" + [ "$out" = "$(printf 'REQUIRED CHECK: CI Status (FAILURE)\nREQUIRED CHECK: slow (IN_PROGRESS)')" ] \ + || fail "without the option the report is exactly the blocker lines, got: $out" + assert_no_grep 'check-runs' "$GH_CALLS" "without the option no check run is read" + assert_no_grep 'log-failed' "$GH_CALLS" "without the option no job log is downloaded" + [ ! -e "$TMP_ROOT/curl-calls" ] || fail "without the option the model is never asked" + pass "the default call reads and prints what it does without the sort, key and listed repository present" +} + test_failed_check_sort_is_off_without_a_key() { local out rm -f "$GH_CALLS" - out=$(FM_TEST_REQUIRED_CHECKS=$FAILING FM_TEST_GH_CALLS="$GH_CALLS" run_state) \ + out=$(FM_TEST_REQUIRED_CHECKS=$FAILING FM_TEST_GH_CALLS="$GH_CALLS" run_state --sort-failed-checks) \ || fail "blocked fixture was refused" [ "$out" = "$(printf 'REQUIRED CHECK: CI Status (FAILURE)\nREQUIRED CHECK: slow (IN_PROGRESS)')" ] \ || fail "without a key the report is exactly the blocker lines, got: $out" @@ -273,7 +295,7 @@ test_failed_check_sort_rules_decide_first() { [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ || fail "a pass from a different run, such as one a pull request edit triggered, is not a retry, got: $out" - printf 'o\nr\n' > "$HOME_ON/config/jev-code-projects" + printf 'o/r\n' > "$HOME_ON/config/jev-code-projects" out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN \ FM_TEST_LOG=$(printf 'build\tinstall\t2026-10-06T10:00:00.1234567Z curl: (6) Could not resolve host: registry.example') \ FM_TEST_BASE_RUNS='{"check_runs":[{"id":9,"status":"completed","conclusion":"failure"}]}' run_sort) @@ -296,11 +318,17 @@ test_failed_check_sort_asks_only_for_a_listed_project() { || fail "an unclear failure in an unlisted repository is unknown, got: $out" [ "$(jev_calls)" = 0 ] || fail "an unlisted repository's log is never sent" - printf '# r\n' > "$HOME_ON/config/jev-code-projects" + printf '# o/r\n' > "$HOME_ON/config/jev-code-projects" FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log run_sort >/dev/null - [ "$(jev_calls)" = 0 ] || fail "a commented-out name does not list the repository" + [ "$(jev_calls)" = 0 ] || fail "a commented-out entry does not list the repository" + + printf 'r\no\nother/r\no/r2\n' > "$HOME_ON/config/jev-code-projects" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ + || fail "a bare name or a same-named repository under another owner does not list this one, got: $out" + [ "$(jev_calls)" = 0 ] || fail "only the exact owner/repo entry lets a repository's log be sent" - printf '# listed\nr\n' > "$HOME_ON/config/jev-code-projects" + printf '# listed\no/r\n' > "$HOME_ON/config/jev-code-projects" out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log run_sort) [ "$out" = 'FAILED CHECK SORT: CI Status: code bug (jev, confidence 0.9)' ] \ || fail "a confident answer labels the check, got: $out" @@ -334,7 +362,10 @@ test_failed_check_sort_asks_only_for_a_listed_project() { [ "$(jev_calls)" = 0 ] || fail "a check with no log is never sent" out=$(FAILED_NAMES=$(printf 'a\nb\nc\nd') FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log run_sort) - [ "$(printf '%s\n' "$out" | wc -l | tr -d ' ')" = 3 ] || fail "only the first three failed checks are sorted, got: $out" + [ "$out" = "$(printf 'FAILED CHECK SORT: %s: code bug (jev, confidence 0.9)\n' a b c + printf 'FAILED CHECK SORT: d: unknown (not sorted)')" ] \ + || fail "every failed check gets a line and only the first three are sorted, got: $out" + [ "$(jev_calls)" = 3 ] || fail "a check past the bound is never sent" rm -f "$HOME_ON/config/jev-code-projects" pass "only a listed repository's unclear failure is asked, and every doubt is unknown" } @@ -349,9 +380,9 @@ printf '%s env=%s\n' "$*" "${TYPESAFE_API_KEY-}" >> "$FM_TEST_CURL_CALLS" exit 7 SH chmod +x "$FAKEBIN/curl" - printf 'r\n' > "$HOME_ON/config/jev-code-projects" + printf 'o/r\n' > "$HOME_ON/config/jev-code-projects" out=$(FM_TEST_HOME="$HOME_ON" FM_TEST_REQUIRED_CHECKS=$FAILING FM_TEST_HEAD_RUNS=$FAILED_RUN \ - FM_TEST_LOG='assert failed' FM_TEST_GH_CALLS="$GH_CALLS" FM_TEST_CURL_CALLS="$TMP_ROOT/curl-calls" run_state) || status=$? + FM_TEST_LOG='assert failed' FM_TEST_GH_CALLS="$GH_CALLS" FM_TEST_CURL_CALLS="$TMP_ROOT/curl-calls" run_state --sort-failed-checks) || status=$? rm -f "$FAKEBIN/curl" "$HOME_ON/config/jev-code-projects" [ "$status" -eq 0 ] || fail "a failed model call must not change the exit status" [ "$out" = "$(printf 'REQUIRED CHECK: CI Status (FAILURE)\nREQUIRED CHECK: slow (IN_PROGRESS)\nFAILED CHECK SORT: CI Status: unknown')" ] \ @@ -445,6 +476,7 @@ test_changes_requested_decision_is_never_silent test_authors_own_changes_requested_review_is_not_a_blocker test_pending_approval_is_not_a_blocker test_required_failure_is_a_blocker +test_failed_check_sort_is_off_unless_asked_for test_failed_check_sort_is_off_without_a_key test_failed_check_sort_rules_decide_first test_failed_check_sort_asks_only_for_a_listed_project From 457b52e13ee32511b1d3869108aacdce38455807 Mon Sep 17 00:00:00 2001 From: Tejas Shah Date: Tue, 6 Oct 2026 02:37:42 +0700 Subject: [PATCH 3/6] no-mistakes(review): label only fixed-list choices; index the sort option --- bin/fm-check-sort-lib.sh | 2 +- docs/scripts.md | 2 +- tests/fm-pr-state.test.sh | 2 ++ 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/bin/fm-check-sort-lib.sh b/bin/fm-check-sort-lib.sh index 8a4b27d0efa..a50ddfdf628 100644 --- a/bin/fm-check-sort-lib.sh +++ b/bin/fm-check-sort-lib.sh @@ -105,7 +105,7 @@ EOF <(printf '%s' "$evidence" | tail -c "$FM_CHECK_SORT_TAIL_CHARS" | jq -Rsc --arg check "$name" '{check: $check, log_tail: .}') \ <(printf '%s' "$FM_CHECK_SORT_CRITERIA") || { echo unknown; return 0; } jq -r --argjson floor "$FM_JEV_CONFIDENCE_FLOOR" --argjson away "$FM_CHECK_SORT_AWAY_FLOOR" ' - if .choice != "unclear" and .confidence >= (if .choice == "code_bug" then $floor else $away end) + if (.choice | IN("code_bug", "flaky", "environment")) and .confidence >= (if .choice == "code_bug" then $floor else $away end) then "\(.choice | sub("_"; " ")) (jev, confidence \(.confidence))" else "unknown" end' \ <<<"$FM_JEV_ANSWER" 2>/dev/null || echo unknown } diff --git a/docs/scripts.md b/docs/scripts.md index 6136a5fdbc7..8e3042b5ee6 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -142,7 +142,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-contributions.sh` | Observe owned publications, retain exact-head judgments, measure required actors, and wake on maintainer signals | | `fm-pr-check.sh` | Record validated `pr=` and `pr_head=` values, then atomically arm a static merge poll | | `fm-pr-merge.sh` | Record PR metadata, merge a task's canonical full GitHub or GitLab URL, then refuse an outcome it cannot prove landed or queued | -| `fm-pr-state.sh` | Read-only: print one line per GitHub pull-request blocker it can see, reporting on checks that have reported rather than verdicting merge-readiness | +| `fm-pr-state.sh` | Read-only: print one line per GitHub pull-request blocker it can see, reporting on checks that have reported rather than verdicting merge-readiness; `--sort-failed-checks` adds the advisory failed check sort line under each failed required check | | `fm-pr-reviewers.sh` | Read-only: suggest reviewers from GitHub's own author mapping of recent commits on a pull request's changed files, never requesting one | | `fm-merge-outcome-lib.sh` | Publish a confirmed merge's durable, role-routed supervision outcome | | `fm-merge-authority-lib.sh` | Resolve merge authority at the gate, persist it against the accepted canonical PR, and identity-check its later poll consumption | diff --git a/tests/fm-pr-state.test.sh b/tests/fm-pr-state.test.sh index f106aeea2d3..85549dc5b7b 100755 --- a/tests/fm-pr-state.test.sh +++ b/tests/fm-pr-state.test.sh @@ -355,6 +355,8 @@ test_failed_check_sort_asks_only_for_a_listed_project() { || fail "a flaky answer at the stricter floor labels the check, got: $out" out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_CHOICE=unclear run_sort) [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] || fail "an unclear answer is unknown, got: $out" + out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_CHOICE=$'banana\nREQUIRED CHECK: x' run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] || fail "a choice outside the fixed list is unknown, got: $out" out=$(FM_TEST_HEAD_RUNS=$FAILED_RUN FM_TEST_LOG=$log STUB_FAIL=1 run_sort) [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] || fail "a failed call is unknown, got: $out" out=$(FM_TEST_LOG=$log run_sort) From 1a31aca43366328b84789d9a4c22dd7fd29cd091 Mon Sep 17 00:00:00 2001 From: Tejas Shah Date: Tue, 6 Oct 2026 02:39:56 +0700 Subject: [PATCH 4/6] no-mistakes(review): require a different run attempt for the flaky rule --- bin/fm-check-sort-lib.sh | 35 +++++++++++++++++++------- docs/configuration.md | 2 +- docs/verification/failed-check-sort.md | 2 +- tests/fm-pr-state.test.sh | 13 +++++++++- 4 files changed, 40 insertions(+), 12 deletions(-) diff --git a/bin/fm-check-sort-lib.sh b/bin/fm-check-sort-lib.sh index a50ddfdf628..8b899289533 100644 --- a/bin/fm-check-sort-lib.sh +++ b/bin/fm-check-sort-lib.sh @@ -16,7 +16,9 @@ # 1. Another attempt of the same workflow run passed this check on the same # head commit, so the same code and the same trigger both passed and # failed -> flaky. A pass from a different run does not count, because a -# check triggered again by a pull request edit saw a different input. +# check triggered again by a pull request edit saw a different input, and +# neither does a pass from the same attempt, which is another job that +# shares the check's name. # 2. The failed steps' log shows a connection error (FM_CHECK_SORT_MARKERS) and # the same check is failing on the base branch, which does not carry this # pull request's change -> environment. @@ -63,26 +65,41 @@ FM_CHECK_SORT_CRITERIA='{"code_bug":"The log shows the project'"'"'s own code or # Prints " ()" or "unknown" for one failed check. _fm_check_sort_one() { # - local listed=$1 repo=$2 head=$3 url=$4 name=$5 runs passed job log evidence base base_state + local listed=$1 repo=$2 head=$3 url=$4 name=$5 runs run passed job log evidence base base_state runs=$(gh api -X GET "/repos/$repo/commits/$head/check-runs" \ -f check_name="$name" -f filter=all -f per_page=100 --jq ' def run: (.details_url // "") | split("/job/") | .[0] // ""; ([.check_runs[] | select(.app.slug == "github-actions" and .status == "completed" and .conclusion != "success")] | max_by(.id)) as $failed | (($failed // {}) | run) as $run - | ($run != "" and any(.check_runs[]; .conclusion == "success" and run == $run) | tostring), - (($failed.id // "") | tostring)' 2>/dev/null) || { echo unknown; return 0; } + | ($run | split("/") | last // ""), + (($failed.id // "") | tostring), + ([.check_runs[] | select($run != "" and .conclusion == "success" and run == $run) | .id | tostring] | join(" "))' \ + 2>/dev/null) || { echo unknown; return 0; } { - IFS= read -r passed + IFS= read -r run IFS= read -r job + IFS= read -r passed } </dev/null \ + | awk -v job="$job" -v passed=" $passed " ' + { attempt[$1] = $2 } + END { + if (attempt[job] == "") exit 1 + for (id in attempt) if (index(passed, " " id " ") && attempt[id] != "" && attempt[id] != attempt[job]) exit 0 + exit 1 + }'; then + echo 'flaky (rule: another attempt of the same run passed)' + return 0 + fi ;; + esac # Each line is "\t\t "; only the text is kept. log=$(gh run view --job "$job" --log-failed -R "$repo" 2>/dev/null \ | tail -n "$FM_CHECK_SORT_LOG_LINES" | cut -f 3- | sed -E 's/^[0-9]{4}-[0-9T:.-]+Z //') || true diff --git a/docs/configuration.md b/docs/configuration.md index e11b7b81709..107e4f8265b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -333,7 +333,7 @@ The sort runs only when the caller passes that option: firstmate passes it when With the option it is still off unless `TYPESAFE_API_KEY` is available under the same environment-then-`.env` contract as [typed dispatch resolution](#typed-dispatch-resolution-env-typesafe_api_key); with the key absent the command reads and prints exactly what it does without the feature. The label never re-runs a check, never changes what a check reported, and never changes the command's blocker lines or exit status, so nothing merges, blocks, or discards on it. -Fixed rules decide first from GitHub reads alone: a check that another attempt of the same workflow run passed is flaky, and a check whose failed steps log a connection error while the same check also fails on the base branch is environment. +Fixed rules decide first from GitHub reads alone: a check that another attempt of the same workflow run passed is flaky, which a same-named job passing in the same attempt is not, and a check whose failed steps log a connection error while the same check also fails on the base branch is environment. Only a failure neither rule decides is put to typesafe.ai's System One model (Jev) as one fixed-choice question, `code_bug`, `flaky`, `environment`, or `unclear`. A `code_bug` answer needs the shared 0.6 confidence floor, and a `flaky` or `environment` answer needs 0.8, because those two point a worker away from the failure. A timeout, a transport or API error, a malformed answer, an `unclear` choice, an answer under its floor, a check with no GitHub Actions job log, and an unreadable GitHub read all print `unknown`, and the worker investigates as it does without the label. diff --git a/docs/verification/failed-check-sort.md b/docs/verification/failed-check-sort.md index 865e6478880..73d20f8f85a 100644 --- a/docs/verification/failed-check-sort.md +++ b/docs/verification/failed-check-sort.md @@ -45,7 +45,7 @@ gh 2.100.0 refuses to print a job's raw log from `gh api /repos///a No open pull request had a failing required check on the run date, so these were not observed live. `tests/fm-pr-state.test.sh` covers them with a fake `gh` that answers in GitHub's JSON shapes and `fm_jev_choice` stubbed at the library boundary, against fixture homes with no key in the environment: -- the `flaky` rule firing on a pass from another attempt of the same run, and not on a pass from a different run; +- the `flaky` rule firing on a pass from another attempt of the same run, and not on a pass from a different run or from a same-named job of the same attempt; - the `environment` rule firing on a connection error whose check also fails on the base branch; - `bin/fm-pr-state.sh` itself printing the label beside its unchanged blocker lines, with a fake `curl` that fails, and the key reaching neither `curl`'s argv or environment nor any `gh` call; - a call without `--sort-failed-checks`, key and listed repository present, making no check-run read, no log download, and no model call; diff --git a/tests/fm-pr-state.test.sh b/tests/fm-pr-state.test.sh index 85549dc5b7b..2560c57fb78 100755 --- a/tests/fm-pr-state.test.sh +++ b/tests/fm-pr-state.test.sh @@ -53,6 +53,9 @@ serve() { "api -X GET /repos/o/r/commits/$head/check-runs -f check_name="*" -f filter=all -f per_page=100 --jq "*) printf '%s\n' "${FM_TEST_HEAD_RUNS:-{\"check_runs\":[]\}}" ;; + "api -X GET /repos/o/r/actions/runs/5/jobs -f filter=all -f per_page=100 --jq "*) + printf '%s\n' "${FM_TEST_RUN_JOBS:-{\"jobs\":[]\}}" + ;; "run view --job 41 --log-failed -R o/r") printf '%s\n' "${FM_TEST_LOG-}" ;; @@ -287,10 +290,18 @@ test_failed_check_sort_is_off_without_a_key() { test_failed_check_sort_rules_decide_first() { local out - out=$(FM_TEST_HEAD_RUNS=$(passed_too 5) run_sort) + out=$(FM_TEST_HEAD_RUNS=$(passed_too 5) \ + FM_TEST_RUN_JOBS='{"jobs":[{"id":40,"run_attempt":1},{"id":41,"run_attempt":2}]}' run_sort) [ "$out" = 'FAILED CHECK SORT: CI Status: flaky (rule: another attempt of the same run passed)' ] \ || fail "a check that passed on another attempt of its run is flaky, got: $out" assert_no_grep '--log-failed' "$GH_CALLS" "a flaky verdict needs no log" + out=$(FM_TEST_HEAD_RUNS=$(passed_too 5) FM_TEST_LOG='boom' \ + FM_TEST_RUN_JOBS='{"jobs":[{"id":40,"run_attempt":1},{"id":41,"run_attempt":1}]}' run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ + || fail "a same-named job that passed in the same attempt is not a retry, got: $out" + out=$(FM_TEST_HEAD_RUNS=$(passed_too 5) FM_TEST_LOG='boom' run_sort) + [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ + || fail "a pass whose attempt cannot be read is not a retry, got: $out" out=$(FM_TEST_HEAD_RUNS=$(passed_too 6) FM_TEST_LOG='boom' run_sort) [ "$out" = 'FAILED CHECK SORT: CI Status: unknown' ] \ || fail "a pass from a different run, such as one a pull request edit triggered, is not a retry, got: $out" From 1b759c54932d0e0236ef25d68c0b075f775dc0c3 Mon Sep 17 00:00:00 2001 From: Tejas Shah Date: Tue, 6 Oct 2026 02:49:01 +0700 Subject: [PATCH 5/6] no-mistakes(document): docs: list failed check sort among Jev key consumers --- bin/fm-check-sort-lib.sh | 2 +- docs/configuration.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/bin/fm-check-sort-lib.sh b/bin/fm-check-sort-lib.sh index 8b899289533..66ea2e4db64 100644 --- a/bin/fm-check-sort-lib.sh +++ b/bin/fm-check-sort-lib.sh @@ -38,7 +38,7 @@ # shared FM_JEV_CONFIDENCE_FLOOR. `flaky` and `environment` need the stricter # FM_CHECK_SORT_AWAY_FLOOR, because those two tell a worker the failure is not # in its code, and a live call labelled a pull request process failure -# `environment` at 0.66 (docs/verification/failed-check-sort.md). +# `environment` above the shared floor (docs/verification/failed-check-sort.md). # # ponytail: only the first FM_CHECK_SORT_MAX failed checks are sorted, so a # pull request with many red checks costs a bounded number of reads; each of diff --git a/docs/configuration.md b/docs/configuration.md index 107e4f8265b..d70766d4eac 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -652,7 +652,7 @@ By accepted design, a `clear` result does not enforce catalog/authentication, re Firstmate passes its profile line unless it states a reason to override, such as the brief's reasoning class or an eligible-unranked-candidate note; every non-clear result returns to the full existing intake. The resolver, the watcher, the house-rules check, the finished check, and bootstrap copy an environment-provided key into a non-exported private variable and unset `TYPESAFE_API_KEY` before launching child processes, so the secret is absent from child environments. -`bin/fm-jev-lib.sh` is the single owner of the Jev request, the answer validation, and that key handling for the resolver, for [routine-wake triage](#routine-wake-triage-env-typesafe_api_key), for the [house-rules check](#house-rules-check-env-typesafe_api_key-confighouse-rulesjson), and for the [finished check](#finished-check-env-typesafe_api_key); it sends the key to `curl` only as a header read from a file descriptor, never on argv, and nothing prints, logs, or writes it. +`bin/fm-jev-lib.sh` is the single owner of the Jev request, the answer validation, and that key handling for the resolver, for [routine-wake triage](#routine-wake-triage-env-typesafe_api_key), for the [house-rules check](#house-rules-check-env-typesafe_api_key-confighouse-rulesjson), for the [finished check](#finished-check-env-typesafe_api_key), and for the [failed check sort](#failed-check-sort-env-typesafe_api_key-configjev-code-projects); it sends the key to `curl` only as a header read from a file descriptor, never on argv, and nothing prints, logs, or writes it. It fixes the endpoint at `https://api.typesafe.ai`, model at `jev-latest`, confidence floor at 0.6, and request timeout at 5 seconds; `TYPESAFE_API_KEY` is the resolver's only resolver-specific environment setting. The live rule-match evidence is recorded in [`verification/dispatch-resolve.md`](verification/dispatch-resolve.md). @@ -1243,7 +1243,7 @@ FMX_RELAY_URL=https://myfirstmate.io # optional Relay endpoint override, mainl FMX_ENV_FILE= # optional alternate .env file for direct Relay client invocations; bootstrap still checks $FM_HOME/.env FMX_DRY_RUN= # truthy previews Relay replies and dismissals to state/x-outbox/ without posting or requiring a token FMX_X_REPLY_MAX_CHARS=280 # X reply per-message split budget; values below 50 clamp to 50 -TYPESAFE_API_KEY= # typed dispatch resolution opt-in, from the environment or .env; absent means bin/fm-dispatch-resolve.sh is off (docs/configuration.md "Typed dispatch resolution"), and so are routine-wake triage, the finished check, and the house-rules check, which also needs a project opted in in config/house-rules.json +TYPESAFE_API_KEY= # typed dispatch resolution opt-in, from the environment or .env; absent means bin/fm-dispatch-resolve.sh is off (docs/configuration.md "Typed dispatch resolution"), and so are routine-wake triage, the finished check, the failed check sort, and the house-rules check, which also needs a project opted in in config/house-rules.json FMX_DISCORD_REPLY_MAX_CHARS=1900 # Discord reply per-message split budget; values below 50 clamp to 50, values above 2000 reset to 1900 FMX_X_THREAD_MAX=25 # maximum messages in one auto-split reply thread FMX_FOLLOWUP_MAX_AGE_SECS=604800 # local window for posting Relay completion follow-ups (7 days) From 7a37b55fde223e0589783bbb0fa1af36d2f8b917 Mon Sep 17 00:00:00 2001 From: Tejas Shah Date: Tue, 6 Oct 2026 02:49:38 +0700 Subject: [PATCH 6/6] no-mistakes(lint): silence SC2016 on literal jq filter in check sort --- bin/fm-check-sort-lib.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/fm-check-sort-lib.sh b/bin/fm-check-sort-lib.sh index 66ea2e4db64..a2e6bc3caa0 100644 --- a/bin/fm-check-sort-lib.sh +++ b/bin/fm-check-sort-lib.sh @@ -66,6 +66,7 @@ FM_CHECK_SORT_CRITERIA='{"code_bug":"The log shows the project'"'"'s own code or # Prints " ()" or "unknown" for one failed check. _fm_check_sort_one() { # local listed=$1 repo=$2 head=$3 url=$4 name=$5 runs run passed job log evidence base base_state + # shellcheck disable=SC2016 # jq variables are literal filter syntax. runs=$(gh api -X GET "/repos/$repo/commits/$head/check-runs" \ -f check_name="$name" -f filter=all -f per_page=100 --jq ' def run: (.details_url // "") | split("/job/") | .[0] // "";