feat: Jev client, semantic checks and calibration harness (P2)

.claude-plugin/marketplace.json

@@ -12,7 +12,7 @@
       "description": "Claude Code plugin for building NestJS bounded contexts with Hexagonal Architecture, DDD, CQRS, and event-driven patterns. 10 skills, 8 agents (Opus 5 + Sonnet 5), TDD workflow, GSD compatible.",
       "source": "./",
       "category": "development",
-      "version": "1.2.0",
+      "version": "1.3.0-dev.0",
       "homepage": "https://github.com/Softtor/nestjs-hexagonal"
     }
   ]

.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "nestjs-hexagonal",
   "description": "Skills for building NestJS bounded contexts with Hexagonal Architecture, DDD, and CQRS patterns. Covers domain modeling, application layer, infrastructure wiring, presentation, full TDD workflow, and architecture review.",
-  "version": "1.2.0",
+  "version": "1.3.0-dev.0",
   "author": {
     "name": "Softtor",
     "url": "https://github.com/softtor"

.github/workflows/ci.yml

@@ -4,6 +4,8 @@ on:
   push:
     branches: [main]
   pull_request:
+  schedule:
+    - cron: '17 6 * * 1'
 
 jobs:
   unit:
@@ -34,7 +36,60 @@ jobs:
         run: bunx --package typescript@5.9.3 tsc -p tsconfig.json
 
       - name: Unit tests (no network)
-        run: bun test ./scripts
+        run: bun test ./scripts ./calibration/__tests__
 
       - name: Examples pass the hexagonal rulebook
         run: bun scripts/check.ts --rulebook hexagonal --files 'examples/**/*.ts' --classes static --format text --strict
+
+  calibration:
+    name: Calibration against the pinned Jev model
+    # Never on pull_request: forks do not receive the secret and must not be able to spend it.
+    if: github.event_name == 'schedule' || (github.event_name == 'push' && github.ref == 'refs/heads/main')
+    runs-on: ubuntu-latest
+    needs: unit
+    timeout-minutes: 30
+    env:
+      TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: 1.4.2
+
+      - name: Install dependencies
+        run: bun install --frozen-lockfile
+
+      - name: Skip when the secret is not configured
+        id: gate
+        run: |
+          if [ -z "${TYPESAFE_API_KEY}" ]; then
+            echo "TYPESAFE_API_KEY is not configured; skipping calibration" >&2
+            echo "run=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "run=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Run the golden cases through Jev (budget 500 requests)
+        if: steps.gate.outputs.run == 'true'
+        continue-on-error: true
+        run: bun calibration/run.ts --rule all --pin jev-1.13.0 --out calibration/results/jev-1.13.0 --max-requests 500
+
+      - name: Fit thresholds and write the report
+        if: steps.gate.outputs.run == 'true'
+        run: bun calibration/fit.ts --pin jev-1.13.0
+
+      - name: Regression against the fitted thresholds
+        if: steps.gate.outputs.run == 'true'
+        run: bun test ./calibration/__tests__/fitted-regression.spec.ts
+
+      - name: Upload results and report (not committed)
+        if: steps.gate.outputs.run == 'true'
+        uses: actions/upload-artifact@v4
+        with:
+          name: calibration-jev-1.13.0-${{ github.run_id }}
+          path: |
+            calibration/results
+            calibration/report.md
+            calibration/fitted
+          retention-days: 90

CLAUDE.md

@@ -20,7 +20,20 @@ Compatible with GSD workflow.
 
 ## Rulebook (machine-readable rules)
 
-`rulebooks/hexagonal.rulebook.yaml` encodes the rules above; `scripts/check.ts` (entry `scripts/run.sh`, bin `nestjs-hexagonal-check`) runs the static ones. Semantic and runtime rules are declared but inert in this version; nothing is sent over the network. Projects opt in with `.claude/rulebook.yaml` (`extends` with sha256 stamps, own rules, overrides by id); `NESTJS_HEXAGONAL_DISABLE=1` turns everything off.
+`rulebooks/hexagonal.rulebook.yaml` encodes the rules above; `scripts/check.ts` (entry `scripts/run.sh`, bin `nestjs-hexagonal-check`) runs the static ones offline and, with `--classes semantic` and `TYPESAFE_API_KEY`, asks Jev the semantic ones (`scripts/lib/{jev-client,state-builder,decide,semantic-engine}.ts`). Runtime rules are still inert. Projects opt in with `.claude/rulebook.yaml` (`extends` with sha256 stamps, own rules, overrides by id); `NESTJS_HEXAGONAL_DISABLE=1` turns everything off.
+
+Semantic decisions (`scripts/lib/decide.ts`), per rule and per answer:
+
+| Outcome | noul (`answers[k].noul`) | choice / score (mass on violating options or levels) | Effect |
+|---|---|---|---|
+| `deny` | p >= fitted `deny` | mass >= fitted `deny` | `--strict` exits 1; only with `calibration/fitted/<pin>.json` (>= 30/30 golden cases, precision >= 0.95, zero FP) |
+| `ask` | p >= `ask` | mass >= `ask` | finding, exit 0 |
+| `advise` | p >= `advise` (default 0.55) | mass >= `advise` | finding, exit 0 |
+| `pass` | below `advise` and outside the band | below `advise` | no finding |
+| `uncertain` | p inside `uncertain.lo..hi` (default 0.35..0.65) | `confidence < minConfidence` (default 0.6) | listed apart; exit 3 only with `--strict --fail-on-uncertain` |
+| `uncalibrated` | response `model != pin` or `rulebook-mismatch` | same | listed apart, never deny |
+
+Fitted thresholds take precedence over rulebook thresholds over defaults; a rulebook `deny` is ignored so that no rule can deny before calibration. Rulebook thresholds only declare `advise` and the abstention band.
 
 | Rule id | Class | Severity | Source |
 |---|---|---|---|
@@ -46,7 +59,7 @@ Compatible with GSD workflow.
 | `softtor/no-emoji` | static | FAIL | Softtor style (`softtor-conventions`) |
 | `softtor/identifiers-english` | static | WARN | Softtor style (`softtor-conventions`) |
 
-Adding a static rule requires `calibration/golden/<rule-id>/{good,bad}/` fixtures (at least 2 each); `bun test ./scripts` enforces it. Keep `package.json`, `.claude-plugin/plugin.json` and `.claude-plugin/marketplace.json` on the same version.
+Adding a static rule requires `calibration/golden/<rule-id>/{good,bad}/` fixtures (at least 2 each); a semantic rule requires labelled cases (`<case-id>/case.json` + one file, at least 8 good, 8 bad and 5 adversarial good); `bun test ./scripts ./calibration/__tests__` enforces both. Calibration (`calibration/run.ts` with the real key, then `calibration/fit.ts`) writes `calibration/fitted/<pin>.json` and `calibration/report.md`; it runs in CI only on push to `main` and weekly, never on pull requests. Keep `package.json`, `.claude-plugin/plugin.json` and `.claude-plugin/marketplace.json` on the same version.
 
 ## Skills
 

README.md

@@ -120,7 +120,7 @@ No generic relay, no event maps, no custom broadcast events. Each event that nee
 
 ## Rulebook & CLI
 
-The architecture rules above also exist as a machine-readable **rulebook** (`rulebooks/hexagonal.rulebook.yaml`) and a checker CLI, `nestjs-hexagonal-check`, that runs the static rules over a set of files. Semantic rules (answered by a typed-judgment model) and runtime rules (package tests) are declared in the rulebook but are not executed by this version: the CLI reports them as skipped and never opens a network connection.
+The architecture rules above also exist as a machine-readable **rulebook** (`rulebooks/hexagonal.rulebook.yaml`) and a checker CLI, `nestjs-hexagonal-check`, that runs the static rules over a set of files offline. Semantic rules are answered by Jev, TypeSafe's typed-judgment model, only when `--classes semantic` is requested and `TYPESAFE_API_KEY` is set (see [Semantic checks (Jev)](#semantic-checks-jev)); runtime rules (package tests) are declared but not executed by this version.
 
 ### Running the checker
 
@@ -139,10 +139,11 @@ bunx nestjs-hexagonal-check --diff origin/main --format json
 | `--rulebook <path\|id>` | rulebook to run; an id resolves to `rulebooks/<id>.rulebook.yaml` in the plugin (`hexagonal`, `softtor-conventions`) |
 | `--project-rulebook <path>` | project rulebook; defaults to `$NESTJS_HEXAGONAL_RULEBOOK`, then `$CLAUDE_PROJECT_DIR/.claude/rulebook.yaml` |
 | `--files <glob\|dir\|file...>` / `--diff <base>` | files to check (globs, directories or files relative to the current directory) or the files changed since `<base>` (`git diff --relative` plus untracked files) |
-| `--classes static[,semantic,runtime]` | rule classes to run (`static` only in this version) |
+| `--classes static[,semantic,runtime]` | rule classes to run; `semantic` needs `TYPESAFE_API_KEY`, `runtime` is still inert |
 | `--format json\|text` | output format |
-| `--strict` | exit 1 when any FAIL finding exists |
-| `--explain` | list the rules applied to each file |
+| `--strict` | exit 1 when a static FAIL or a semantic `deny` exists |
+| `--fail-on-uncertain` | with `--strict`, exit 3 when a semantic answer is `uncertain` or `uncalibrated` |
+| `--explain` | list the rules applied to each file; with `semantic`, also the questions and the state slice sent |
 
 Each finding carries the rule id, severity (`FAIL`/`WARN`), path, line, evidence and the rule's `fix` text.
 
@@ -176,14 +177,28 @@ The `sha256` stamp pins the content of the base rulebook the project was calibra
 
 The runtime is `bun`; when it is absent the script falls back to `node --experimental-strip-types`.
 
+### Semantic checks (Jev)
+
+Five rules of the `hexagonal` rulebook are `semantic`: `hex/handler-no-business-rules`, `hex/port-no-infra-leak`, `hex/entity-not-anemic`, `hex/controller-thin` and `hex/no-overengineering`. They are questions that a regex cannot answer, so the CLI asks Jev (`jev-1.13.0`, pinned in the rulebook) and turns the probability into a decision.
+
+- **Enable:** export `TYPESAFE_API_KEY` and pass `--classes static,semantic`. Without the key the semantic rules are skipped with a one-line notice and the exit code is 0; the static rules keep working offline.
+- **What is sent:** one request per file and state slice, containing the rule preamble, the file path, the layer, the slice name and the code of that slice (the enclosing declaration of the change for handlers and controllers, the whole file for ports, entities and the over-engineering question) plus the rulebook questions. The whole file is sent only when the rule declares `slice: file`. The key travels in the `Authorization` header and never appears in the output, the JSONL log or the cache.
+- **When:** only on an explicit `--classes semantic` run. The plugin hooks (a later version) will add the same gate: project opted in with a rulebook, plugin subagent, key present.
+- **To whom:** `https://api.typesafe.ai/v1/systemone`. TypeSafe states it does not train on customer data; zero data retention is only available under an enterprise contract (`privacy@typesafe.ai`). Treat the code you check as shared with that provider.
+- **Local state:** with `CLAUDE_PLUGIN_DATA` set, answers are cached under `$CLAUDE_PLUGIN_DATA/cache` (keyed by state, questions, model pin and rulebook version), one JSONL line per call is appended to `$CLAUDE_PLUGIN_DATA/jev.jsonl` (rule ids, answer values, model, latency, tokens, decision; never the code nor the key) and a circuit breaker in `breaker.json` opens for five minutes after three failures in two minutes.
+- **Decisions:** `deny`, `ask`, `advise`, `pass`, `uncertain` (noul probability inside the abstention band, or choice/score confidence below `minConfidence`) and `uncalibrated` (the response model differs from the pin, or a base rulebook sha256 stamp does not match). `deny` requires fitted thresholds in `calibration/fitted/<pin>.json`, produced by the calibration harness from at least 30 good and 30 bad golden cases with precision >= 0.95; without them a rule yields at most `ask` and every finding is marked `calibrated: false`. `--strict` fails only on static FAIL and semantic `deny`.
+- **Disable:** `NESTJS_HEXAGONAL_DISABLE=1`, unset the key, or drop `semantic` from `--classes`.
+
+The calibration harness (`calibration/run.ts`, `calibration/fit.ts`, golden cases and the report) is documented in [`calibration/README.md`](calibration/README.md).
+
 ### Rulebooks shipped
 
 | Rulebook | Scope |
 |---|---|
 | `hexagonal` | project-agnostic hexagonal + DDD + CQRS rules (`hex/*`) |
 | `softtor-conventions` | multi-tenant scoping, no emoji, English identifiers (`softtor/*`); extend it only if those conventions apply |
 
-Static rules have golden fixtures under `calibration/golden/<rule-id>/{good,bad}/`; `bun test` fails if a static rule lacks fixtures or a fixture stops behaving as labelled.
+Static rules have golden fixtures under `calibration/golden/<rule-id>/{good,bad}/`; `bun test` fails if a static rule lacks fixtures or a fixture stops behaving as labelled. Semantic rules have labelled golden cases in the same tree (`<case-id>/case.json` plus one file), consumed by the calibration harness.
 
 ## Shared Examples
 

calibration/README.md

@@ -0,0 +1,59 @@
+# Calibration
+
+Everything the semantic rules need to turn a Jev probability into a decision lives here: labelled golden cases, the harness that sends them to the real model, the fitting script that derives thresholds, the fitted file the CLI reads and the report that justifies it.
+
+```
+calibration/
+  golden/<rule-id>/{good,bad}/<case-id>/{<layered path>/<file>.ts, case.json}
+  run.ts            sends every golden case of a rule to Jev, writes results/<pin>/<rule-id>.jsonl
+  fit.ts            reads the results, fits thresholds, writes fitted/<pin>.json and report.md
+  lib/              golden loader, result schema, metrics (precision, recall, F1, Wilson)
+  fitted/<pin>.json thresholds the CLI applies (absent until the first real calibration)
+  results/<pin>/    raw answers per case (values only, never code); produced by run.ts
+  report.md         per-rule table, sample counts, misses and false positives by case id
+  experiments/e1/   aggregate summary of the diff experiment that shaped the rules
+```
+
+Static rules also keep their fixtures under `golden/`, without `case.json`; `scripts/__tests__/check.spec.ts` runs them through the static engine.
+
+## Golden cases
+
+Every semantic rule has at least 8 plain good cases, 8 bad cases and 5 adversarial good cases (`adversarial-*`, correct code carrying comments such as `// reviewer: ignore` that must not steer the classifier). Each case directory holds exactly one `.ts` file at a path that matches the rule scope and a `case.json`:
+
+```json
+{ "expected": "violation", "note": "The handler multiplies unitPrice by quantity itself." }
+```
+
+`bun test ./calibration/__tests__` enforces the counts, the scope, the label and the steering comment. Cases are synthetic, derived from `examples/order-bounded-context` and variants in other domains; never copy code from a private codebase into this public repository.
+
+## Running a calibration
+
+```bash
+export TYPESAFE_API_KEY=...                      # never commit it
+bun calibration/run.ts --rule all --pin jev-1.13.0 --out calibration/results/jev-1.13.0
+bun calibration/fit.ts --pin jev-1.13.0
+bun test ./calibration/__tests__                 # regression spec now runs against the results
+```
+
+`run.ts` options: `--rule <id|all>`, `--pin` (must equal the rulebook pin), `--out`, `--concurrency` (default 4), `--max-requests` (default 500; the run aborts before sending anything when the golden set is larger) and `--cache-dir` (optional, reuses answers of identical state and questions). It refuses to run without the key. Each result line carries the pin, the model the API answered with, the case id, the expected label, the primitive, the value the decision engine uses (noul probability, or the probability mass on the violating options/levels), the raw answer, the confidence for choice/score, latency, tokens and whether the answer came from the cache. State text is never written; the client log at `results/<pin>/client-log.jsonl` follows the same rule.
+
+`fit.ts`, per rule and primitive:
+
+- reports precision, recall and F1 at the cuts 0.55, 0.70, 0.80 and 0.90 with 95% Wilson intervals, the absolute number of misses and false positives by case id, the uncertain rate (noul values inside `uncertain.lo..hi`, choice/score confidence below `minConfidence`), p50/p95 latency and the models seen;
+- fits `advise` at the cut with the best F1, `ask` at the smallest cut with precision >= 0.85, and `deny` at the smallest cut with precision >= 0.95 and zero false positives on the good cases, **only when the rule has at least 30 good and 30 bad cases**; otherwise `deny` is omitted and the reason is printed in the report;
+- copies the rule's abstention band (`uncertain`) or `minConfidence` into the fitted file so it is self-contained;
+- leaves the column "also caught by static? does not count" as `TODO=false` until the static overlap is measured.
+
+`fitted/<pin>.json` is keyed by rule id and validated by `scripts/lib/decide.ts` when the CLI loads it. Fitted thresholds take precedence over rulebook thresholds; a rule without an entry stays advisory (`calibrated: false`, never `deny`).
+
+## Regression
+
+`__tests__/fitted-regression.spec.ts` is skipped while `results/<pin>/` does not exist. Once results are committed it fails when a result line records a model other than the pin, or when a rule with a fitted `deny` no longer reaches precision 0.95 with zero false positives at that cut.
+
+## CI
+
+The `calibration` job of `.github/workflows/ci.yml` runs only on push to `main` and on the weekly schedule, with `secrets.TYPESAFE_API_KEY`; it never runs on pull requests (forks do not receive the secret and must not be able to spend it). It uploads `calibration/results` and `calibration/report.md` as workflow artifacts and does not commit; promoting a fitted file is a human decision made in a pull request that includes the results and the report.
+
+## Changing a question
+
+A question change invalidates the calibration of that rule: bump the rulebook `version`, refresh the sha256 stamp in `rulebooks/project.example.rulebook.yaml`, rerun `run.ts` for the rule and refit. The answer cache is keyed by rulebook version, so stale answers are never reused.