Repository navigation
Conversation
…onment bin/fm-pr-state.sh adds one advisory FAILED CHECK SORT line under each failed required check when TYPESAFE_API_KEY is available. Fixed rules decide first from GitHub reads (another attempt of the same run passed means flaky; a connection error on a check the base branch also fails means environment); only an undecided failure is put to Jev as one fixed-choice question through bin/fm-jev-lib.sh, and only for a repository listed in config/jev-code-projects. Every doubt prints unknown. Without the key the command reads and prints exactly what it did before. The label never re-runs a check and never changes a blocker line or the exit status.
Owner
Author
|
Superseded by #50, which carries this change's final content together with the other three Jev items on one branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
A research report on using Jev (TypeSafe's decision-only model) in the coding workflow proposed ten integrations in one table. The answer on row 9: "implement". Later, on merging: "merge all jev items".
Row 9 as it was shown:
Stated with the table: Jev only picks from a fixed list; in every row, if Jev is unsure or down, work carries on exactly as today.
Rulings given the same evening on earlier rows, which apply to every Jev item: nothing a worker typed in its shell is ever sent to TypeSafe; no project's source code or diff is sent unless that project is explicitly named for it; Jev advises and never becomes the only thing that blocks, merges, approves or discards anything; a fix or feature adds nothing beyond what the row describes.
What Changed
bin/fm-pr-state.shaccepts a new--sort-failed-checksoption. With it andTYPESAFE_API_KEYavailable, each failed required check gets one extra advisory line,FAILED CHECK SORT: <check>: <class> (<why>), where the class iscode bug,flaky,environment, orunknown. Without the option or the key, output is unchanged. The line is a label only: no check is re-run, no reported result changes, and the exit status is unaffected.bin/fm-check-sort-lib.shdoes the sort. Fixed rules decide first from GitHub reads: another attempt of the same workflow run passing the check on the same head commit meansflaky; a connection-error marker in the failed-step log plus the same check failing on the base branch meansenvironment. Anything else goes to Jev as one fixed-choice question over the check name and a trimmed tail of the failed-step log.flakyandenvironmentanswers need a stricter 0.8 confidence floor, andcode buguses the shared floor. Every other outcome isunknown, including a Jev error or timeout, anunclearchoice, and a check with no readable job log. Only the first 3 failed checks are sorted; the rest printunknown (not sorted).bin/fm-jev-lib.shaddsfm_jev_code_allowed, backed by a new localconfig/jev-code-projectslist. A failure log is sent to Jev only for a repository listed there as its exact<owner>/<repo>; an unlisted repository's unclear failure isunknownwith no call. The branch also addstests/fm-pr-state.test.shcases and test-family routing inbin/fm-test-run.sh, plus docs indocs/configuration.md,docs/scripts.md,AGENTS.md, and a newdocs/verification/failed-check-sort.md.🤖 Generated with Claude Code
Risk Assessment
✅ Low: The sort is advisory, off unless a caller passes --sort-failed-checks and a key is present, sends a log only for an exact owner/repo entry, labels only the three fixed choices, and every failure path falls back to
unknownwithout touching blocker lines or exit status; the earlier rounds' fixes hold in the current code and the one remaining issue is a rare job-selection edge.Testing
Drove
bin/fm-pr-state.shlive against real open pull requests on github.com (cli/cli#13788, vercel/next.js#99644, rust-lang/cargo#17518, kubernetes/kubernetes#142693) with realghbehind a logging pass-through andcurlreplaced by a shim that logs and refuses, so nothing left the machine for typesafe.ai. The default call matched the base commit's output and made the same two GitHub reads; with the option, each failed required check got oneFAILED CHECK SORTline while blocker lines and exit status stayed the same; unlisted and near-miss allow-list entries made no model call; an exactly listed repository made one call that was blocked and printedunknown; the key reached noghchild and was not on curl's argv or environment. The focused hermetic testtests/fm-pr-state.test.shalso passed and is the only coverage for the Jev-answered labels, the flaky and environment rules, and the bound of three. No full suite was run. No screenshot: this is a CLI-only change, so the evidence is a CLI transcript.defaultvsbaseon cli/cli#13788 anddefault-listedvsbase-nextjson vercel/next.js#99644: identical output, 2 gh calls each (pr view, pr checks), 0 curl calls. The first…sorted(cli/cli#13788, three lines),unlisted-nextjs,unlisted-cargo: output minus the SORT lines equals the default output, exit 0. Pending checks on kubernetes#142693 got n…nokey: output and gh calls identical to the default call, 2 gh calls, 0 curl callsunlisted-nextjsandunlisted-cargo: job log read (non-empty, 56 and 188 lines), labelunknown, 0 curl calls, using the real home whose config lists no projectnearmiss-liston vercel/next.js#99644 with the key in the environment: labelunknown, 0 curl callsexact-jev-down: exactly one POST attempt to https://api.typesafe.ai/v1/systemone, refused by the shim; outputFAILED CHECK SORT: thank you, next: unknown, blocker lines unchang…gh_children_with_key=0on every run including those with the key exported by the caller; the curl line showskey_in_env=no key_in_argv=nowith the header passed as `-H @/dev/f…nonactions-k8s: two prow checks eachunknown, one check-runs read per check, nogh run view, 0 curl callspr view,pr checks,api -X GET,run view --log-failed); no rerun or write call appearsEvidence: Live CLI transcript of every driven run, with per-run gh and curl call logs
Source: Live CLI transcript of every driven run, with per-run gh and curl call logs
Evidence: Output of the focused hermetic test tests/fm-pr-state.test.sh
Source: Output of the focused hermetic test tests/fm-pr-state.test.sh
Evidence: Sort on an unlisted repository with a readable log (vercel/next.js#99644)
Evidence: Same pull request, repository listed exactly, Jev unreachable
Evidence: Default call (no option), key present and repository listed
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
bin/fm-pr-state.sh:141- The only programmatic caller of fm-pr-state.sh is the watcher's routine-wake triage (bin/fm-watch.sh:1715), which throws the label away: any REQUIRED CHECK line already makes it return 1. With the key in .env, every declared-pause recheck of a task whose PR has a failed required check now runs the full sort inside the watcher's 30 s timeout: per check a check-runs read, a job-log download, possibly two base-branch reads and a 5 s Jev call, for up to three checks. For a listed project it also sends the failure log to typesafe.ai with no reader. The wake outcome is unchanged, but the watcher stalls longer and the same log is re-sent on every recheck. The remedy, not the defect, needs authorization: skipping the sort for that caller means a new opt-out switch or moving where the sort is invoked.bin/fm-check-sort-lib.sh:118- The allow-list is checked against${repo##*/}, the repository name without its owner. Listingfirstmate(or a common name such asapi) in config/jev-code-projects therefore allows the failure log of any<owner>/<that-name>pull request passed to fm-pr-state.sh to be sent to typesafe.ai, including a fork or an unrelated same-named repository. The ruling is "no project's source code or diff is sent unless that project is explicitly named for it". The docs state the basename comparison, so this is deliberate, but whether a bare repository name counts as explicitly naming the project is a policy decision; matchingowner/repowould be the narrower form.bin/fm-check-sort-lib.sh:54- Unrequired component: FM_CHECK_SORT_AWAY_FLOOR=0.8, a second confidence floor forflakyandenvironmentbeside the shared FM_JEV_CONFIDENCE_FLOOR. Row 9 only requires that an unsure answer becomesunknown, and "a fix or feature adds nothing beyond what the row describes". The floor is backed by the live evidence in docs/verification/failed-check-sort.md (a process failure labelledenvironmentat 0.65 to 0.73) and only makes labels more conservative, but it is a second definition of the floor. Confirm keeping it, or remove it and use the shared floor.bin/fm-check-sort-lib.sh:51- Unrequired component: FM_CHECK_SORT_MAX=3. Only the first three failed required checks are sorted; a fourth gets no FAILED CHECK SORT line at all, not evenunknown. Row 9 describes sorting a failed check and falling back tounknown, with no cap, and the docs say a line is added "under each failed required check". Either remove the cap, or confirm it and accept that later checks are silently unlabelled.bin/fm-check-sort-lib.sh:101- Unrequired component: the FM_CHECK_SORT_EVIDENCE line filter with its fall-back to the whole log. Only lines matchingerror|fail|...are sent, so the surrounding context of a failure is dropped before Jev sees it;0 failedstyle summary lines still match. Row 9 does not ask for a filtering stage; sending the last 4000 characters of the failed-step log is the narrower form. The verification record motivates the filter (summary lines crowd out the cause), so confirm or remove.bin/fm-check-sort-lib.sh:63- Row 9's "passed on retry means flaky" cannot be observed from this plug-in point in its usual order: a check that failed and then passed on retry is green and never reaches the sort. The rule as implemented fires only when an earlier attempt of the same run passed and the latest attempt failed. That is the only reachable form inside fm-pr-state.sh and is labelled correctly; noted so the narrow coverage is understood.🔧 Fix applied.
2 issues (1 warning, 1 info) still open:
bin/fm-check-sort-lib.sh:108- The verdict filter accepts any.choicethat is notunclear._fm_jev_askin bin/fm-jev-lib.sh checks thatchoiceis a string and that the probability keys match the criteria, but never thatchoiceis one of those keys. A reply such as{choice:"banana", confidence:0.9}with well-formed probabilities therefore printsFAILED CHECK SORT: <check>: banana (jev, confidence 0.9), and because the output isjq -r, a choice containing a newline would print an extra line in fm-pr-state.sh's report. The header and docs say the class is one of four and that a malformed answer isunknown. Fix: only label when.choiceiscode_bug,flakyorenvironment; everything else isunknown.docs/scripts.md:145- After the fix round the sort runs only when a caller passes--sort-failed-checks, and no code path passes it: the watcher deliberately does not, and firstmate is expected to by hand. The option is documented in docs/configuration.md and the script's own help, but the script index row forfm-pr-state.shin docs/scripts.md still describes only the blocker read, and AGENTS.md mentions only the config file. A firstmate that finds the command through the index will never produce the label. Add the option to that row.🔧 Fix applied.
1 warning still open:
bin/fm-check-sort-lib.sh:70- Rule 1 labels a checkflakywhenever any check run with the same name and the same/actions/runs/<id>URL succeeded, but it never checks that the passing run belongs to a different attempt. Check runs carry no attempt number, so two jobs of one attempt that share a name look identical to a retry. Concrete case: a matrix job with a staticname: testyields several check runs namedtestin one run; one leg passes and one fails on a real code bug,gh pr checksreportstestas failed, and the sort printsflaky (rule: another attempt of the same run passed)with no log read and no Jev call. That is the label that tells a worker the failure is not in its code, and it bypasses the stricter 0.8 floor meant to guard it. Fix: when a same-run pass is found, confirm the passing and failing jobs have differentrun_attemptvalues (gh api /repos/<repo>/actions/jobs/<id> --jq .run_attempt) before printingflaky; otherwise fall through to the log path. The testpassed_too 5fixture would still pass with the code wrong, because it never models two same-attempt jobs.🔧 Fix applied.
1 info still open:
bin/fm-check-sort-lib.sh:72- The job to sort is chosen as the highest-id GitHub Actions check run of that name whose conclusion is anything other thansuccess, soskipped,neutralandcancelledruns qualify. Concrete case: two workflows on the same head both have a job namedtest; workflow A'stestfails (id 20) and workflow B'stestis skipped by anif:(id 30).gh pr checksreportstestas failed, the sort picks job 30, its--log-failedoutput is empty, and the line isunknownalthough a failing job with a log exists. In the rarer variant where B'stestpassed on attempt 1 and was skipped on a re-run, rule 1 compares the skipped job with that pass and printsflaky (rule: ...)for A's real failure. Fix: select only conclusions that are failures (failure,timed_out,startup_failure,action_required,cancelled) rather than!= "success". Same root: when two workflows both report a failedtest, both names resolve to the same highest-id job and one failure is never looked at.✅ **Test** - passed
✅ No issues found.
defaultvsbaseon cli/cli#13788 anddefault-listedvsbase-nextjson vercel/next.js#99644: identical output, 2 gh calls each (pr view, pr checks), 0 curl calls. The first…sorted(cli/cli#13788, three lines),unlisted-nextjs,unlisted-cargo: output minus the SORT lines equals the default output, exit 0. Pending checks on kubernetes#142693 got n…nokey: output and gh calls identical to the default call, 2 gh calls, 0 curl callsunlisted-nextjsandunlisted-cargo: job log read (non-empty, 56 and 188 lines), labelunknown, 0 curl calls, using the real home whose config lists no projectnearmiss-liston vercel/next.js#99644 with the key in the environment: labelunknown, 0 curl callsexact-jev-down: exactly one POST attempt to https://api.typesafe.ai/v1/systemone, refused by the shim; outputFAILED CHECK SORT: thank you, next: unknown, blocker lines unchang…gh_children_with_key=0on every run including those with the key exported by the caller; the curl line showskey_in_env=no key_in_argv=nowith the header passed as `-H @/dev/f…nonactions-k8s: two prow checks eachunknown, one check-runs read per check, nogh run view, 0 curl callspr view,pr checks,api -X GET,run view --log-failed); no rerun or write call appearsFM_HOME=~/p/firstmate bin/fm-pr-state.sh https://github.com/cli/cli/pull/13788compared with the same call on the base commit's script (d0dd8b5), three roundsbin/fm-pr-state.sh --sort-failed-checks https://github.com/cli/cli/pull/13788with the real home's key and no listed projectbin/fm-pr-state.sh --sort-failed-checks https://github.com/cli/cli/pull/13788with an empty home (no key)bin/fm-pr-state.sh --sort-failed-checkson vercel/next.js#99644 and rust-lang/cargo#17518 (readable job logs, repository unlisted)bin/fm-pr-state.sh --sort-failed-checks https://github.com/kubernetes/kubernetes/pull/142693(failed required checks that are not GitHub Actions jobs)bin/fm-pr-state.sh --sort-failed-checkson vercel/next.js#99644 with a fixture allow-list of near misses:# vercel/next.js,next.js,someoneelse/next.js,vercel/next.js-fork,vercel/next.jsbin/fm-pr-state.sh(no option) on vercel/next.js#99644 with the key in the environment andvercel/next.jslistedbin/fm-pr-state.sh --sort-failed-checkson vercel/next.js#99644 withvercel/next.jslisted and curl refused (Jev unreachable)bash tests/fm-pr-state.test.sh✅ **Document** - passed
✅ No issues found.
🔧 **Lint** - 1 issue found → auto-fixed ✅
🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Push** - passed
✅ No issues found.