feat: Jev client, semantic checks and calibration harness (P2) - #12
Merged
Merged
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
P2 of the Jev plan: the semantic half of the rulebook becomes executable. The CLI can now send the five semantic rules to TypeSafe Jev (
jev-1.13.0), decide per primitive with calibrated thresholds, and never block without fitted evidence.scripts/lib/jev-client.ts:fetch-based client with Zod-validated responses, model pin assertion (model-mismatch→ every outcomeuncalibrated), sha256 cache, retries on 429/5xx honouringRetry-After, timeout, file-backed circuit breaker, JSONL log with answer values only (never the state text, never the key),too-largeguard for the 32k/64k limits.scripts/lib/state-builder.ts: slicesfile | diff-window | declaration | exports-only | strings-only | jsx-onlywith a token cap; code is data instate, never part of the instructions.scripts/lib/decide.ts: per-primitive outcomesdeny | ask | advise | pass | uncertain | uncalibrated; fitted thresholds > rulebook > defaults; without fitted thresholds a rule can neverdeny.scripts/lib/semantic-engine.ts+check.ts --classes semantic: one request per (file, slice group) with all questions keyed by rule id; skipped with a notice whenTYPESAFE_API_KEYis absent;--fail-on-uncertain;--explainprints questions and slice.adviseonly, nodenyin the rulebook.calibration/run.ts(budget guard, values only),calibration/fit.ts(Wilson intervals, cuts restricted to ≥ the uncertain band,adviseat the lowest cut of the best-F1 plateau,ask/denyat the highest;denyonly with ≥ 30 good and 30 bad cases),calibration/fitted/jev-1.13.0.json,calibration/report.md, regression spec against the committed results.hex/entity-not-anemicat 75% recall. These numbers describe the synthetic sets only; the E1 experiment on real diffs (summarised incalibration/experiments/e1/README.md) is the reasondenystays off.pushtomaincalibrationjob with the repository secret, never onpull_request; artifacts only.1.3.0-dev.0; README "Semantic checks (Jev)" section documents what is sent, when, to whom and how to disable.222 tests offline (
bun test ./scripts ./calibration/__tests__),tsc --noEmitclean, examples pass--strict.How to test