feat(bin): sort a failed required check as code bug, flaky, or environment, opt-in

AGENTS.md

@@ -74,6 +74,7 @@ config/crew-harness  crewmate harness override; LOCAL, gitignored; absent or "de
 config/claude-permission-mode  optional one-token permission posture for every Claude worker launch: absent or "bypass" keeps --dangerously-skip-permissions, "auto" launches with --permission-mode auto; LOCAL, gitignored; inherited by secondmate homes; see docs/configuration.md "Claude permission mode"
 config/crew-dispatch.json  optional crewmate dispatch profiles; LOCAL, gitignored; firstmate-maintained but human-editable natural-language rules that choose a per-task harness/model/effort profile (section 4). Inherited by secondmate homes
 config/house-rules.json  optional opt-in list of project names, and optional rules, for the advisory house-rules check: a ship brief asks its worker to run the check only for a project named in the file's "projects" list, and only when TYPESAFE_API_KEY is present; absent means off for every project; LOCAL, gitignored, and not inherited; see docs/configuration.md "House-rules check"
+config/jev-code-projects  optional list, one entry per line, of the projects whose code or failure logs a Jev check may send to typesafe.ai: `<owner>/<repo>` for a check keyed by repository, matched exactly, or a firstmate project name for a check keyed by project; absent means none; LOCAL, gitignored, and not inherited; see docs/configuration.md "Failed check sort"
 config/secondmate-harness  harness the PRIMARY uses to launch SECONDMATE agents, optionally followed by a model and effort token on the same line ("<harness> [<model>] [<effort>]"; section 4); LOCAL, gitignored; absent or "default" harness falls back to config/crew-harness then firstmate's own. The primary's own setting; NOT inherited into secondmate homes (secondmates do not spawn secondmates)
 config/backlog-backend  backlog backend override; LOCAL, gitignored; absent or "tasks-axi" = the configured tasks-axi backend, "manual" = force routine backlog updates to hand-editing; inherited by secondmate homes (section 10)
 config/backend  runtime session-provider backend override for new tasks; LOCAL, gitignored; absent = falls through to runtime auto-detection (the runtime firstmate itself is executing inside), then tmux; tmux is the verified reference backend (docs/tmux-backend.md), herdr has its own required CI lane (docs/herdr-backend.md), while zellij, orca, and cmux remain experimental with no dedicated real-backend CI lane (docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md) - herdr and cmux can also be selected by runtime auto-detection, zellij and orca never are (always explicit), and codex-app is not accepted; see docs/codex-app-backend.md; inherited by secondmate homes under the primary-authoritative contract in secondmate-provisioning

bin/fm-check-sort-lib.sh

@@ -0,0 +1,150 @@
+# shellcheck shell=bash
+# Advisory sort of a failed required check: code bug, flaky, or environment.
+# Usage: . bin/fm-check-sort-lib.sh   (source it before the caller runs any child)
+#
+# bin/fm-pr-state.sh is the one caller, and only under its
+# --sort-failed-checks option. It hands over the required checks it already
+# read as failed, and this file prints one extra line per check:
+#   FAILED CHECK SORT: <check>: <class> (<why>)
+# where <class> is `flaky`, `environment`, `code bug`, or `unknown`. The line is
+# a label for the reader. It never re-runs a check, never changes what a check
+# reported, and nothing merges, blocks, or discards on it.
+#
+# The sort runs only when TYPESAFE_API_KEY is available (bin/fm-jev-lib.sh owns
+# the key); without it nothing is read and nothing is printed. Fixed rules
+# decide first, from GitHub reads alone:
+#   1. Another attempt of the same workflow run passed this check on the same
+#      head commit, so the same code and the same trigger both passed and
+#      failed -> flaky. A pass from a different run does not count, because a
+#      check triggered again by a pull request edit saw a different input, and
+#      neither does a pass from the same attempt, which is another job that
+#      shares the check's name.
+#   2. The failed steps' log shows a connection error (FM_CHECK_SORT_MARKERS) and
+#      the same check is failing on the base branch, which does not carry this
+#      pull request's change -> environment.
+# A failure no rule decides goes to Jev as one fixed-choice question over the
+# check's name and the log of the job's failed steps. Only the last
+# FM_CHECK_SORT_LOG_LINES lines are read; of those, the lines that name a
+# failure (FM_CHECK_SORT_EVIDENCE) are sent, or every line when none does, cut
+# to the last FM_CHECK_SORT_TAIL_CHARS characters, because a job states its
+# failure last and ends in summary lines that say nothing about the cause. A failure log quotes the project's
+# code, so it is sent only for a repository fm_jev_code_allowed finds in
+# config/jev-code-projects as its exact `<owner>/<repo>`; a bare name there
+# never matches a repository. An unlisted repository's unclear failure is
+# `unknown` with no call. `unknown` is also the answer for a check with no
+# GitHub Actions job log, an unreadable read, a Jev timeout or error, a
+# malformed answer, an `unclear` choice, and a confidence under the floor; the
+# worker then investigates as it does without the label. `code bug` needs the
+# shared FM_JEV_CONFIDENCE_FLOOR. `flaky` and `environment` need the stricter
+# FM_CHECK_SORT_AWAY_FLOOR, because those two tell a worker the failure is not
+# in its code, and a live call labelled a pull request process failure
+# `environment` above the shared floor (docs/verification/failed-check-sort.md).
+#
+# ponytail: only the first FM_CHECK_SORT_MAX failed checks are sorted, so a
+# pull request with many red checks costs a bounded number of reads; each of
+# the rest is printed as `unknown (not sorted)` with nothing read.
+#
+# fm_check_sort <home> <owner/repo> <head-sha> <pr-url>
+#   Reads failed check names, one per line, on stdin. Always returns 0.
+
+_FM_CHECK_SORT_DIR=${BASH_SOURCE[0]%/*}
+[ "$_FM_CHECK_SORT_DIR" != "${BASH_SOURCE[0]}" ] || _FM_CHECK_SORT_DIR=.
+# shellcheck source=bin/fm-jev-lib.sh
+. "$_FM_CHECK_SORT_DIR/fm-jev-lib.sh"
+
+FM_CHECK_SORT_MAX=3
+FM_CHECK_SORT_LOG_LINES=2000
+FM_CHECK_SORT_TAIL_CHARS=4000
+# A live call labelled a process failure `environment` at 0.65 to 0.73 (docs/verification/failed-check-sort.md).
+FM_CHECK_SORT_AWAY_FLOOR=0.8
+FM_CHECK_SORT_EVIDENCE='error|fail|not ok|panic|exception|assert|timed out|timeout|refused|denied|killed|cannot|unable'
+FM_CHECK_SORT_MARKERS='ECONNREFUSED|ECONNRESET|ETIMEDOUT|EAI_AGAIN|ENOTFOUND|could not resolve host|temporary failure in name resolution|connection (refused|reset|timed out)|TLS handshake timeout|50[234] (bad gateway|service unavailable|gateway time-?out)'
+# shellcheck disable=SC2016 # The backticks are literal markup in the question text.
+FM_CHECK_SORT_INSTRUCTIONS='An automated check named in `check` failed on a pull request, and `log_tail` holds the last lines of the failed job'"'"'s log that name a failure. Decide from the words of the log why it failed. Choose `unclear` whenever the log does not plainly show one of the other three.'
+FM_CHECK_SORT_CRITERIA='{"code_bug":"The log shows the project'"'"'s own code or tests failing on their merits: a compile, type, or lint error, a failed assertion, or a test that fails for a reason in the code.","flaky":"The log shows a failure that depends on timing, ordering, or chance rather than on the code: a race, a timing-dependent assertion, or a test that timed out waiting with no error in the code.","environment":"The log shows the runner or something outside the project failing: a network or connection error, a failed download or dependency install, an unavailable service, a rate limit, a missing credential, or a runner out of disk, memory, or time.","unclear":"The log does not show enough to tell which of the others applies."}'
+
+# Prints "<class> (<why>)" or "unknown" for one failed check.
+_fm_check_sort_one() {  # <listed:0|1> <owner/repo> <head-sha> <pr-url> <check-name>
+  local listed=$1 repo=$2 head=$3 url=$4 name=$5 runs run passed job log evidence base base_state
+  # shellcheck disable=SC2016  # jq variables are literal filter syntax.
+  runs=$(gh api -X GET "/repos/$repo/commits/$head/check-runs" \
+    -f check_name="$name" -f filter=all -f per_page=100 --jq '
+    def run: (.details_url // "") | split("/job/") | .[0] // "";
+    ([.check_runs[] | select(.app.slug == "github-actions" and .status == "completed" and .conclusion != "success")]
+      | max_by(.id)) as $failed
+    | (($failed // {}) | run) as $run
+    | ($run | split("/") | last // ""),
+      (($failed.id // "") | tostring),
+      ([.check_runs[] | select($run != "" and .conclusion == "success" and run == $run) | .id | tostring] | join(" "))' \
+    2>/dev/null) || { echo unknown; return 0; }
+  {
+    IFS= read -r run
+    IFS= read -r job
+    IFS= read -r passed
+  } <<EOF
+$runs
+EOF
+  case "$job" in ''|*[!0-9]*) echo unknown; return 0 ;; esac
+  case "$run$passed" in
+    ''|*[!0-9\ ]*) ;;
+    *)
+      if [ -n "$passed" ] && gh api -X GET "/repos/$repo/actions/runs/$run/jobs" -f filter=all -f per_page=100 \
+        --jq '.jobs[] | "\(.id) \(.run_attempt)"' 2>/dev/null \
+        | awk -v job="$job" -v passed=" $passed " '
+          { attempt[$1] = $2 }
+          END {
+            if (attempt[job] == "") exit 1
+            for (id in attempt) if (index(passed, " " id " ") && attempt[id] != "" && attempt[id] != attempt[job]) exit 0
+            exit 1
+          }'; then
+        echo 'flaky (rule: another attempt of the same run passed)'
+        return 0
+      fi ;;
+  esac
+  # Each line is "<job>\t<step>\t<timestamp> <text>"; only the text is kept.
+  log=$(gh run view --job "$job" --log-failed -R "$repo" 2>/dev/null \
+    | tail -n "$FM_CHECK_SORT_LOG_LINES" | cut -f 3- | sed -E 's/^[0-9]{4}-[0-9T:.-]+Z //') || true
+  [ -n "$log" ] || { echo unknown; return 0; }
+  if printf '%s\n' "$log" | grep -Eiq -- "$FM_CHECK_SORT_MARKERS"; then
+    base=$(gh pr view "$url" --json baseRefName --jq .baseRefName 2>/dev/null) || base=
+    base_state=
+    [ -z "$base" ] || base_state=$(gh api -X GET "/repos/$repo/commits/$base/check-runs" \
+      -f check_name="$name" --jq '
+      [.check_runs[] | select(.status == "completed")] | max_by(.id) | .conclusion // ""' 2>/dev/null) || base_state=
+    case "$base_state" in
+      failure|timed_out)
+        echo 'environment (rule: connection error in its log, and it fails on the base branch too)'
+        return 0 ;;
+    esac
+  fi
+  [ "$listed" -eq 1 ] || { echo unknown; return 0; }
+  evidence=$(printf '%s\n' "$log" | grep -Ei -- "$FM_CHECK_SORT_EVIDENCE") || evidence=$log
+  fm_jev_choice failed_check "$FM_CHECK_SORT_INSTRUCTIONS" \
+    <(printf '%s' "$evidence" | tail -c "$FM_CHECK_SORT_TAIL_CHARS" | jq -Rsc --arg check "$name" '{check: $check, log_tail: .}') \
+    <(printf '%s' "$FM_CHECK_SORT_CRITERIA") || { echo unknown; return 0; }
+  jq -r --argjson floor "$FM_JEV_CONFIDENCE_FLOOR" --argjson away "$FM_CHECK_SORT_AWAY_FLOOR" '
+    if (.choice | IN("code_bug", "flaky", "environment")) and .confidence >= (if .choice == "code_bug" then $floor else $away end)
+    then "\(.choice | sub("_"; " ")) (jev, confidence \(.confidence))" else "unknown" end' \
+    <<<"$FM_JEV_ANSWER" 2>/dev/null || echo unknown
+}
+
+fm_check_sort() {  # <home> <owner/repo> <head-sha> <pr-url>; failed check names on stdin
+  local home=$1 repo=$2 head=$3 url=$4 listed=0 name names=() n=0
+  while IFS= read -r name; do
+    [ -z "$name" ] || names+=("$name")
+  done
+  [ "${#names[@]}" -gt 0 ] || return 0
+  fm_jev_key_load "$home" || return 0
+  command -v jq >/dev/null 2>&1 || return 0
+  ! FM_HOME=$home fm_jev_code_allowed "$repo" || listed=1
+  for name in "${names[@]}"; do
+    n=$((n + 1))
+    if [ "$n" -gt "$FM_CHECK_SORT_MAX" ]; then
+      printf 'FAILED CHECK SORT: %s: unknown (not sorted)\n' "$name"
+      continue
+    fi
+    printf 'FAILED CHECK SORT: %s: %s\n' "$name" \
+      "$(_fm_check_sort_one "$listed" "$repo" "$head" "$url" "$name" </dev/null)" || true
+  done
+  return 0
+}

bin/fm-jev-lib.sh

@@ -9,8 +9,9 @@
 # option, and a status; every decision
 # made from that answer (a confidence floor, an approval gate, an absorb) stays
 # in the caller's own code. bin/fm-dispatch-resolve.sh, the watcher's
-# routine-wake triage (bin/fm-watch.sh), bin/fm-house-rules-check.sh, and the
-# worker stop hook's finished check (bin/fm-finished-check.sh) are the callers.
+# routine-wake triage (bin/fm-watch.sh), bin/fm-house-rules-check.sh, the
+# worker stop hook's finished check (bin/fm-finished-check.sh), and the failed
+# check sort (bin/fm-check-sort-lib.sh) are the callers.
 #
 # Key handling: sourcing this file copies an environment-provided
 # TYPESAFE_API_KEY into one non-exported shell variable and unsets the exported
@@ -22,6 +23,16 @@
 # fm_jev_key_load <home>
 #   0 when a key is available, 1 when it is absent from both sources.
 #
+# fm_jev_code_allowed <project>
+#   0 only when <project> is a whole line of $FM_HOME/config/jev-code-projects
+#   (one entry per line; blank lines and lines starting with # are ignored).
+#   A caller keyed by repository passes `<owner>/<repo>`, and a caller keyed by
+#   a firstmate project passes that project's name; neither form matches the
+#   other, so a bare name never allows a repository by its basename.
+#   The key alone lets a caller send routing text and status lines; a caller
+#   that would send a project's code, file list, or pull request text asks this
+#   first. An absent or empty file allows no project.
+#
 # fm_jev_choice <question-key> <instructions> <state-json-file> <criteria-json-file>
 #   One POST to $FM_JEV_BASE/v1/systemone. <state-json-file> holds the JSON
 #   value sent as `state`; <criteria-json-file> holds one JSON object mapping
@@ -82,6 +93,11 @@ fm_jev_key_load() {  # <home>
   [ -n "$TYPESAFE_API_KEY_PRIVATE" ]
 }
 
+fm_jev_code_allowed() {  # <project>
+  [ -n "$1" ] && [ "${1#\#}" = "$1" ] \
+    && grep -qxF -- "$1" "${FM_HOME:-.}/config/jev-code-projects" 2>/dev/null
+}
+
 _fm_jev_fail() {  # <reason>
   FM_JEV_STATUS=error
   FM_JEV_ERROR=$1

bin/fm-pr-state.sh

@@ -17,15 +17,25 @@
 # STALE when it was left at a superseded head.
 # A closed or merged pull request reports that terminal state and nothing else.
 # Unresolved review-thread state is out of this command's scope.
+# With --sort-failed-checks and TYPESAFE_API_KEY available, each failed
+# required check also gets one advisory FAILED CHECK SORT line labelling it a
+# code bug, flaky, environment, or unknown; bin/fm-check-sort-lib.sh owns that
+# sort. The label reads more of GitHub and changes nothing a check reported.
+# Firstmate passes the option when it reads a pull request's failed checks by
+# hand. The watcher's routine-wake triage does not, so its read costs and
+# prints what it does without the sort.
 #
-# Usage: fm-pr-state.sh <pr-url>
+# Usage: fm-pr-state.sh [--sort-failed-checks] <pr-url>
 #   Prints one line per blocker it can see and nothing when it sees none.
 #   Blockers do not change the successful exit status; lookup or usage refusal
 #   exits non-zero.
 set -eu
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
+# Sourced before any child runs, so no gh call inherits the Jev key.
+# shellcheck source=bin/fm-check-sort-lib.sh
+. "$SCRIPT_DIR/fm-check-sort-lib.sh"
 # shellcheck source=bin/fm-pr-lib.sh
 . "$SCRIPT_DIR/fm-pr-lib.sh"
 
@@ -42,7 +52,12 @@ if [ "${1:-}" = --help ] || [ "${1:-}" = -h ]; then
   usage
   exit 0
 fi
-[ "$#" -eq 1 ] || die "usage: fm-pr-state.sh <pr-url>"
+SORT=0
+if [ "${1:-}" = --sort-failed-checks ]; then
+  SORT=1
+  shift
+fi
+[ "$#" -eq 1 ] || die "usage: fm-pr-state.sh [--sort-failed-checks] <pr-url>"
 command -v gh >/dev/null 2>&1 || die "gh is required"
 
 URL=$1
@@ -106,10 +121,15 @@ esac
 GH_STDERR=$(mktemp "${TMPDIR:-/tmp}/fm-pr-state.XXXXXX") \
   || die "could not create temporary file"
 trap 'rm -f "$GH_STDERR"' EXIT INT TERM
-if ! REQUIRED=$(gh pr checks "$URL" --required --json name,state,bucket --jq '
+FAILED=
+if REQUIRED=$(gh pr checks "$URL" --required --json name,state,bucket --jq '
   .[]
   | select(.bucket != "pass" and .bucket != "skipping")
-  | "REQUIRED CHECK: \(.name) (\(.state))"' 2>"$GH_STDERR"); then
+  | "REQUIRED CHECK: \(.name) (\(.state))",
+    (select(.bucket == "fail") | "FAILED CHECK NAME: \(.name)")' 2>"$GH_STDERR"); then
+  FAILED=$(printf '%s\n' "$REQUIRED" | sed -n 's/^FAILED CHECK NAME: //p')
+  REQUIRED=$(printf '%s\n' "$REQUIRED" | sed '/^FAILED CHECK NAME: /d')
+else
   # These two sentences are gh's own human-readable error text, verified against
   # gh 2.100.0 on 2026-09-12. gh reports "nothing reported" as an error rather
   # than as structured data, so matching its text is the only way to tell that
@@ -125,6 +145,11 @@ if ! REQUIRED=$(gh pr checks "$URL" --required --json name,state,bucket --jq '
   fi
 fi
 [ -z "$REQUIRED" ] || printf '%s\n' "$REQUIRED"
+if [ "$SORT" -eq 1 ]; then
+  fm_check_sort "${FM_HOME:-$SCRIPT_DIR/..}" "$PATH_PART" "$HEAD" "$URL" <<EOF_FAILED
+$FAILED
+EOF_FAILED
+fi
 
 if [ "$REVIEW_DECISION" = CHANGES_REQUESTED ]; then
   printf 'REVIEW DECISION: CHANGES_REQUESTED\n'

bin/fm-test-run.sh

@@ -1456,13 +1456,18 @@ families_for_changed_path() {
       ;;
     bin/fm-jev-lib.sh)
       # The one Jev caller, sourced by bin/fm-dispatch-resolve.sh, the watcher's
-      # routine-wake triage, bin/fm-house-rules-check.sh, and the worker stop
-      # hook's finished check.
+      # routine-wake triage, bin/fm-house-rules-check.sh, the worker stop
+      # hook's finished check, and the failed check sort.
+      printf '%s\n' "__script__:fm-pr-state.test.sh"
       printf '%s\n' "__script__:fm-dispatch-resolve.test.sh"
       printf '%s\n' "__script__:fm-jev-wake-triage.test.sh"
       printf '%s\n' "__script__:fm-house-rules-check.test.sh"
       printf '%s\n' "__script__:fm-finished-check.test.sh"
       ;;
+    bin/fm-check-sort-lib.sh)
+      # The failed check sort, sourced by bin/fm-pr-state.sh.
+      printf '%s\n' "__script__:fm-pr-state.test.sh"
+      ;;
     bin/fm-house-rules-check.sh)
       printf '%s\n' "__script__:fm-house-rules-check.test.sh"
       ;;