Conversation
Measurement packages bind a baseline, record usage and replay Chat Completions and Responses transports offline so a result can be checked without repeating the live call.
…nest Chat outages and shared-budget timeouts can no longer land in an adopted measurement.
…ntity Keep failed retry attempts, recompute prompt/wire/rubric and expectedFailures, and fail closed if a Chat measurement changes concrete model.
Replay now hashes the full Chat request, so the checked-in control had to be re-recorded against the current source identity.
A transport failure before the first complete round never wrote a cassette, so the runner dropped that provider attempt. Adopt attempts from the live endpoint instead.
runner.ts is a measurement source anchor, so the checked-in control had to be re-recorded after the pre-round attempt fix.
A failed retry could leave a partial tape that later became canonical while results kept the last attempt. Replay Chat httpStatus, pin the symlink tests, and isolate the Responses fixtures.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0371f41e37
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4db3807250
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cd712e04c8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1e62fc6b27
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 135fbbad40
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8c69984e0e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9c265fc0cf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e519dfaa90
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 47b755c195
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b416ab71c1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ials correctly Three related correctness bugs in the measurement arithmetic that the normal compare/baseline workflow hits without any artifact editing: - weightedAxisTrials rounded the weighted mean into whole successes, so a 0.625 mean over two repetitions entered the Newcombe comparison as 0.5. Trials stay fractional now; the Wilson arithmetic holds for non-integer k. - A wholly indeterminate repetition-axis scored a placeholder 1 (the score runChecks reports when nothing was decided), which averaged into axisScores and bound baselines as a nominal pass. Such repetitions now leave the mean, and an axis with nothing decidable scores 0. - The report summary averaged the same placeholder into its axis rates, so a judge-abstained scenario's freshly generated report disagreed with its manifest and finalize/export failed its own AXIS_SCORE_MISMATCH check. Report and manifest now share one implementation (undecidedAwareAxisMean), and MeasurementCounts carries measuredRepetitions so comparisons count only repetitions that decided weighted checks — an unknown repetition is not an implicit success, and comparing a wholly unmeasured axis is refused with UNMEASURED_AXIS instead of reading zero trials as a rate.
Four attribution bugs in the judge pass, each reachable through the
documented record/judge/replay workflow:
- The judge transport identity stamped the agent SDK's default retry
policy (default/2) while askJudge issues one raw fetch per verdict:
adopted evidence claimed retries that never ran. The judge recorder
now stamps {0,0}, and the chat identity reconstruction uses the
RECORDED policy so those cassettes survive replay validation.
- Offline judge replay sent the literal model "cassette"; the
transport-bound request hash covers the model, so every round came
back stale and assertFresh() failed the replay against itself. The
replay now sends the model the judge cassette was recorded with.
- persistRepetition truncated turn.answer at MAX_FIELD_CHARS, so the
judge scored cut text while replay regenerates the full answer — the
verdict evaluated different text than the measured turn and the
strict replay hash diverged. The answer is retained in full.
- The README's bounded Responses example ran the CLI without loading
.env.workbench, aborting at requireLiveEnv() with no credentials
even when the documented setup was followed exactly.
The adoption privacy gate scans artifact bytes, but a Responses cassette stores the full request base64-encoded — the user prompt, tool inputs and file paths are invisible to the file-level scan. A normal recording whose request body carries an OS absolute path or credential could therefore be adopted and committed while the same content in plaintext would have been refused. Decode each round's request and apply the existing privacy rules to the parsed content (keys and string values unescaped: the serialized form's own escape sequences read as drive paths to the private-path rule). No new rule set, no framework — the same privacyFailure that every other artifact goes through.
…tree One regeneration after the scope rollback, per the owner's instruction: the control binds its source manifest to the frozen clean tree (the original twelve commits plus the three retained fixes). The public synthetic fixture (input-fixture.json) and the recorded check verdicts (recorded-checks.json) are byte-identical to the original getopenscreen#653 head; the cassette request bytes, manifest, report and review receipt follow the retained measurement-arithmetic changes (measuredRepetitions, fractional trials) and the final source identity. Recorded offline with the same scripted loopback control; no provider call was made; verify, replay, self-compare and the bound-baseline path all pass on this exact tree.
b416ab7 to
e651390
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e651390dde
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Fork-only rehearsal for upstream getopenscreen#653 (
78cc1eb2, 12 commits, issue getopenscreen#459) semantically rebased onto upstream/main8a51f517, scope-rolled back per the owner's audit. Do not merge as the upstream PR.History (16 commits = original 12 + 3 fixes + 1 evidence regeneration)
.gitattributeshunk unioned with main's newer Markdown-LF rules).4561c681— measurement arithmetic: fractional comparison trials (0.625 mean no longer reads as 0.5), wholly-indeterminate repetitions excluded from the axis mean on BOTH sides of report/manifest via one shared implementation (undecidedAwareAxisMean,measuredRepetitionsinMeasurementCounts), comparing a wholly unmeasured axis refused withUNMEASURED_AXIS.d3483ae6— judge evidence attribution: judge transport stamps the retry policy that actually ran ({0,0}; cassette reconstruction uses the recorded policy), offline judge replay sends the model the cassette was recorded with,persistRepetitionkeeps the full answer for the judge (no truncated-text verdicts / divergent replay hashes), README's Responses example loads.env.workbench.ddfe274f— adoption privacy: Responses request bodies are decoded and scanned with the existing privacy rules (base64 previously made the user prompt/tool inputs/file paths invisible to the file-level gate). Same rule set, no new framework.e651390d— single final evidence regeneration on the frozen tree: fixture and recorded-checks byte-identical to the original head; cassette/manifest/report/receipt follow the retained arithmetic and final source identity. Offline scripted loopback; no provider call.Scope rollback
All review-round hardening that did not pass the owner's executable normal-workflow gate was dropped (manifest-tamper defenses, receipt/turn digest bindings, sample dedup, case-alias handling, source-overlay verification, stale-bundle guards, etc. — 11 families). Only the four areas above had discriminating executable counterexamples inside the getopenscreen#459/getopenscreen#653 contract. Details and the full KEEP/DROP audit table are in the owner's records.
Validation (clean HEAD
e651390d, all re-run)tscapp/test/workbench: exit 0 · Biome on changed files: clean ·git diff --check: clean8a51f517(local real-fixture environment)wb:measurement verify/replay/compare/baselineonlocal-pr15-scripted-control-20260913-v1: all exit 0Known limits