Repository navigation
Conversation
bin/fm-pr-check.sh prints one advisory risk line after a pull request is registered as ready, when TYPESAFE_API_KEY is present. Code sets the level from paths, deletions and size; for a project listed in config/jev-code-projects Jev answers up to three yes/no questions through bin/fm-jev-lib.sh and may only raise it. An unanswered question never yields low, an unlisted project sends nothing, and the registration bin/fm-pr-merge.sh performs skips the rating.
This was referenced Oct 5, 2026
Owner
Author
|
Superseded by #50, which carries this change's final content together with the other three Jev items on one branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
A research report on using Jev (TypeSafe's decision-only model) in the coding workflow proposed ten integrations in one table. The answer on row 4: "impement"
Row 4 as it was shown:
bin/fm-pr-check.sh, the step that already records a pull request when a worker reports it readyStated with the table: Jev only picks from a fixed list; in every row, if Jev is unsure or down, work carries on exactly as today.
What Changed
bin/fm-pr-check.shnow prints one advisoryrisk: low|medium|high - <why>orrisk: not rated - <why>line after a pull request is recorded and its merge poll armed, via the newbin/fm-pr-risk-lib.sh. It is off unlessTYPESAFE_API_KEYis present, and nothing reads the line to block, merge, approve, or discard anything.bin/fm-pr-merge.shre-registers with the new--no-riskflag, so the level is rated once at ready time and never during a merge.bin/fm-review-diff.shprints: database migration, login and permissions, or payments paths, or 1500+ changed lines, are high; a deleted file, 400+ changed lines, or 20+ files are medium. Jev is then asked up to three yes/no questions (behaviour changed with no test, description does not match the change, something hard to undo). An answer counts only at or above the shared confidence floor and can only raise the level.risk: lowneeds all three questions settled; otherwise, with no fact raising the level, the line isrisk: not ratedand names each unanswered question.fm_jev_code_allowedinbin/fm-jev-lib.shgates what is sent: only projects listed in the localconfig/jev-code-projectshave their title, description, file list, and capped diff excerpt sent to Jev. Unlisted projects are rated from the facts alone, with no forge read and no model call. Addstests/fm-pr-risk.test.sh(registered inbin/fm-test-run.sh), updates totests/fm-pr-merge.test.shandtests/fm-teardown.test.sh, a "Pull request risk level" section indocs/configuration.md, anddocs/verification/jev-pr-risk.md.Risk Assessment
✅ Low: The fix round correctly keeps the path intact for an
authori[sz]match while still excludingauthors/paths, and the feature as a whole is an opt-in advisory line printed after registration completes, which sends nothing for unlisted projects, only ever raises a level, and cannot change the registration or its exit status.Testing
Drove 35 real
bin/fm-pr-check.shregistrations in a throwaway home with real git, the realbin/fm-review-diff.sh, and real curl over HTTP to a local stand-in for the Jev endpoint; every registration exited 0, recorded the pull request, and armed the merge poll, and every printed line matched the intent. Also rantests/fm-pr-risk.test.sh(passed) and the fulltests/fm-pr-merge.test.sh, whosetest_merge_registration_never_rates_riskcase I confirmed passing by running it alone. Not exercised: the real typesafe.ai model's answers, and a real GitHub or GitLab forge. One observation, not a failure: a file deleted beside a similar new file is reported by git as a rename, so it carries no deletion fact (run 12). No visual evidence applies; the surface is one CLI output line, captured as a transcript.risk: lowis printed after thearmed:lineuntestedis settled in code and Jev is asked only the other two questionsrisk: high - login and permissionssrc/authors/path is not read as login and prints lowrisk: not rated, never low, and a low-confidence yes is not countedrisk: not rated, exit 0, PR recorded, merge poll armed, only o…mismatchis left unanswered and the line isnot rated, the other two questions are still asked--no-riskprints no risk line and makes no Jev requesttest_merge_registration_never_rates_riskpassedtests/fm-pr-risk.test.shchild-environment assertionsgh pr viewtitle and description)Evidence: Live transcript of all 35 registrations
Source: Live transcript of all 35 registrations
Evidence: Scenario label and printed risk line, one pair per run
Source: Scenario label and printed risk line, one pair per run
Evidence: Driver script that produced the transcript
Source: Driver script that produced the transcript
Evidence: Local stand-in for the Jev endpoint
Source: Local stand-in for the Jev endpoint
Evidence: Representative printed lines
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 2 issues found → auto-fixed ✅
bin/fm-pr-risk-lib.sh:80-gsub(/author/, "", lp)is meant to stopauthors/paths reading as login, but it also erases the stem of every authorization path before the login/permissions pattern runs. Confirmed with the same awk expression:src/authorization/policy.rb->src/ization/policy.rb,lib/authorize.js->lib/ize.js,app/unauthorized_handler.py->app/unized_handler.py, all auth=0. A pull request touching only such a file gets nologin and permissionsfact and can printrisk: low(ornot rated) instead ofhigh, silently, on exactly the category the intent names ("login and permissions"). The strippedlpis also reused for the migration, payments, test and docs patterns. Fix: keeplpintact and exclude only the author word, e.g. compute a separate copy withgsub(/authors?([^a-z]|$)/, "/")-style narrowing or testlp ~ /authori[sz]/before stripping; add a case forauthorization/authorizenext to the existingsrc/authors/list.jstest.bin/fm-jev-lib.sh:93- New component not named by the intent: the per-project allow-listconfig/jev-code-projectsread byfm_jev_code_allowed(used at bin/fm-pr-risk-lib.sh:158), a second per-project opt-in beside the existing one inconfig/house-rules.json. It ships absent, so with only the key set every project is unlisted: Jev is asked nothing and a change with no path/size fact always printsrisk: not rated ... (project not listed), neverlow. That narrows the intent's "Risk level on every merge request" / "Result: low, medium or high beside the merge request" and "Projects without the no-mistakes review pipeline get a risk level too" to projects someone lists by hand. The intent requires no such list, so the simplification remedy is to remove it and let the key alone enable the rating. The counter-consideration is a policy decision only the author can make: without the list, any home that already holds TYPESAFE_API_KEY for dispatch resolution would start sending each ready pull request's title, description, file list and up to 30000 bytes of diff to https://api.typesafe.ai. Please decide: key alone is sufficient consent to send project code (remove the list), or keep the list as the required privacy gate. If kept, note it matches on the directory basename ofproject=, so two projects with the same folder name under different parents are opted in together.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
risk: lowis printed after thearmed:lineuntestedis settled in code and Jev is asked only the other two questionsrisk: high - login and permissionssrc/authors/path is not read as login and prints lowrisk: not rated, never low, and a low-confidence yes is not countedrisk: not rated, exit 0, PR recorded, merge poll armed, only o…mismatchis left unanswered and the line isnot rated, the other two questions are still asked--no-riskprints no risk line and makes no Jev requesttest_merge_registration_never_rates_riskpassedtests/fm-pr-risk.test.shchild-environment assertionsgh pr viewtitle and description)bash ~/.no-mistakes/evidence/01M46QQZZMBH33BFP81N8CVVQC/drive.sh "$PWD"- 35 livebin/fm-pr-check.sh task-a <url> [--no-risk]registrations against a local Jev stand-inbash tests/fm-pr-risk.test.sh- all four groups passedbash tests/fm-pr-merge.test.sh- full file run; no failure lines in the filtered outputtest_merge_registration_never_rates_riskfromtests/fm-pr-merge.test.shrun alone via a temporary trimmed copy (removed afterwards) - passedManual check: the Jev stand-in's request log shows path/v1/systemone, options exactlyno/yes, one question per request, and the key only in the Authorization header (0 occurrences in curl argv and in the task record)✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.