Skip to content

feat: Jev client, semantic checks and calibration harness (P2) - #12

Merged
joao-oliveira-softtor merged 17 commits into
mainfrom
feat/jev-client-calibration
Sep 20, 2026
Merged

joao-oliveira-softtor merged 17 commits into
mainfrom
feat/jev-client-calibration

Conversation

@joao-oliveira-softtor

Copy link
Copy Markdown
Member

Summary

P2 of the Jev plan: the semantic half of the rulebook becomes executable. The CLI can now send the five semantic rules to TypeSafe Jev (jev-1.13.0), decide per primitive with calibrated thresholds, and never block without fitted evidence.

  • scripts/lib/jev-client.ts: fetch-based client with Zod-validated responses, model pin assertion (model-mismatch → every outcome uncalibrated), sha256 cache, retries on 429/5xx honouring Retry-After, timeout, file-backed circuit breaker, JSONL log with answer values only (never the state text, never the key), too-large guard for the 32k/64k limits.
  • scripts/lib/state-builder.ts: slices file | diff-window | declaration | exports-only | strings-only | jsx-only with a token cap; code is data in state, never part of the instructions.
  • scripts/lib/decide.ts: per-primitive outcomes deny | ask | advise | pass | uncertain | uncalibrated; fitted thresholds > rulebook > defaults; without fitted thresholds a rule can never deny.
  • scripts/lib/semantic-engine.ts + check.ts --classes semantic: one request per (file, slice group) with all questions keyed by rule id; skipped with a notice when TYPESAFE_API_KEY is absent; --fail-on-uncertain; --explain prints questions and slice.
  • Rulebook 1.3.0: the five semantic rules rewritten as atomic English questions with criteria per option and explicit "answer no" orchestration shapes (from the E1 false positives); advise only, no deny in the rulebook.
  • Synthetic golden sets (114 cases, 5 adversarial "steering comment" cases per rule) and the calibration harness: calibration/run.ts (budget guard, values only), calibration/fit.ts (Wilson intervals, cuts restricted to ≥ the uncertain band, advise at the lowest cut of the best-F1 plateau, ask/deny at the highest; deny only with ≥ 30 good and 30 bad cases), calibration/fitted/jev-1.13.0.json, calibration/report.md, regression spec against the committed results.
  • Calibration run of 2026-09-20 (114 requests, 0 errors, p50 ~310 ms): four rules at 100% precision/recall on the synthetic sets, hex/entity-not-anemic at 75% recall. These numbers describe the synthetic sets only; the E1 experiment on real diffs (summarised in calibration/experiments/e1/README.md) is the reason deny stays off.
  • CI: weekly/push to main calibration job with the repository secret, never on pull_request; artifacts only.
  • Version 1.3.0-dev.0; README "Semantic checks (Jev)" section documents what is sent, when, to whom and how to disable.

222 tests offline (bun test ./scripts ./calibration/__tests__), tsc --noEmit clean, examples pass --strict.

How to test

bun install --frozen-lockfile
bun test ./scripts ./calibration/__tests__
bun scripts/check.ts --rulebook hexagonal --files 'examples/**/*.ts' --classes static --strict
bun scripts/check.ts --rulebook hexagonal --files 'examples/**/*.ts' --classes semantic          # skip notice without a key
TYPESAFE_API_KEY=... bun scripts/check.ts --rulebook hexagonal --files 'calibration/golden/hex/controller-thin/bad/*.ts' --classes semantic --explain

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
🔒 Security Review ✅ Completed 2026-09-20T21:12:15.834235Z 6d6b958 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@joao-oliveira-softtor
joao-oliveira-softtor merged commit 0f1a1d2 into main Sep 20, 2026
3 checks passed
@joao-oliveira-softtor
joao-oliveira-softtor deleted the feat/jev-client-calibration branch September 20, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant