feat(evals): compute cross-model fixture saturation from committed results

[tvna]

Summary

Adds evals/scripts/gitapex_compute_corpus_saturation.py, a report-only tool that computes what share of a committed eval run's fixtures every model already solves, and records the resulting figure for the one multi-model run this repository has. This repository measured corpus breadth and assertion well-formedness but never corpus difficulty, so battle-testing-a-skill's dimension 14 ("growth history (case count over time)") could be satisfied by a corpus that carries no information.

Facts

All figures below come from committed JSON only -- no model was invoked, no credential is required.

• Against the only committed run carrying more than one model_id (evals/evaluating-skill-quality/results/2026-07-28-issue-500-phase1/, 23 fixtures x claude-haiku-4-5-20251001 / claude-opus-5 / claude-sonnet-5):

  saturated (every model 1.0): 5 of 23 (21.7 percent)
  discriminating (at least one model below 1.0): 18 of 23 (78.3 percent)
  uniformly hard (subset of the discriminating set): 2
  

• The two uniformly-hard fixtures are evaluating-skill-quality-scoring-axis-uncontrolled-speed-claim (0.833333 on all three tiers) and evaluating-skill-quality-tool-capability-verification-selection (0.750000 on all three). Reported separately from the saturated list, as assertion-defect candidates rather than as difficulty.

• Against a single-model run (results/2026-08-26-issue-1347-structural-identifier-portability/), the tool reports NOT COMPUTABLE, names the 1 model found, and exits 0 rather than emitting a zero rate.

• New module at the current head: 100 percent line and branch coverage (154 statements, 44 branches, 0 missed), 37 tests.

• The full 6761 passed in 101.69s suite run was taken at the first head (970546f9). After the independent-review fixes below, the targeted 37 tests plus all 41 wired local gates were re-run and pass; the full suite has not been re-run locally since, and CI re-runs it on this head.

• uv run --frozen mypy clean on all three new files; ruff check and ruff format --check clean.

• .github/scripts/gitapex_gate_local_preflight.py: local preflight: all 41 wired gate(s) passed.

A defect the property layer found while being written, disclosed rather than quietly fixed: a result file with an empty scores[] array vanished from the model list entirely, because the model was registered lazily on its first entry. A fixture only one of two models had scored was then reported as saturated across "every model" -- unmeasured read as measured. Fixed by registering a model before reading its entries; pinned by test_a_model_that_scored_nothing_still_counts_as_a_model and test_a_model_contributing_only_before_arms_still_counts_as_a_model. No example test in the first draft generated that shape.

Assumptions

• Fact, verified against the real corpus: the participation rules were derived by reading every committed results/*/ directory, not assumed. manifest.json carries neither model_id nor scores; dispatch-trace-check.json carries a model_id but no scores; the before/after gate records' overlapping after entries agree exactly with their sibling -after.json aggregate (checked on the 2026-08-15 issue-1124 run: all five overlaps agree).
• Speculation: that the two uniformly-hard fixtures are assertion defects rather than genuinely hard cases. The tool flags them as candidates; neither was investigated here, and Non-goals in the issue keeps that out of scope.
• Speculation: that a 3-tier saturation count tracks item difficulty in the sense Vania et al. estimate it. It is a count over the tiers one run happened to include; the module's own docstring and its printed report both say so explicitly and refuse the IRT vocabulary.

Risk / blast radius

Very low. The module is new, imported by nothing, wired into no workflow, and registered in no .gitapex/ssot.json gate cluster -- it runs only when a human invokes it. It writes nothing and reads only committed JSON. The one existing file touched is evals/evaluating-skill-quality/eval-status.md, narrative bookkeeping that gates on nothing. No existing behavior changes.

Rollback

Revert the merge commit (git revert -m 1 against it). Nothing depends on the new module, so the revert is self-contained; the eval-status.md section reverts with it.

Verification

Issue #1461's Acceptance Criteria Map, restated in full with a Result column added.

Criterion	Interpretation	Planned ops	Proof method	Residual risk	Result
Cross-model saturation is computed from committed results	A fixture every available model scores 1.0 on carries no information; the rate of such fixtures is the corpus-level number	New report-only script evals/scripts/gitapex_compute_corpus_saturation.py, reading one results/RUN/ directory and grouping per-fixture scores by model_id	Run against results/2026-07-28-issue-500-phase1/; reproduces 5 of 23 saturated and 18 of 23 discriminating	Saturation over 3 tiers is a coarse proxy for item difficulty, not an IRT estimate; the script must not name it one	PASS -- exact figures reproduced; pinned by test_phase1_reproduces_the_saturation_figures_the_issue_cites. The risk is addressed in text: the module docstring and the printed report both refuse the IRT vocabulary explicitly
Single-model runs are reported as not computable	A run with one model_id has no cross-model comparison to make; a silent 0 would read as "no saturation"	Detect distinct model_id count; below 2, report not-computable with the run path and the count found	Run against results/2026-08-26-issue-1347-structural-identifier-portability/; reports not-computable, names 1 model, emits no rate	None identified	PASS -- pinned by test_single_model_committed_run_is_reported_as_not_computable; a property test also asserts no percentage is ever printed in the not-computable branch. The independent review found the stated reason could contradict the printed counts and it now names whichever condition actually failed, pinned by one test per branch
Uniformly-hard fixtures are flagged separately from saturated ones	Every model equal and below 1.0 is an assertion-defect candidate, not evidence of difficulty	Emit a distinct section listing such fixtures with their common score	scoring-axis-uncontrolled-speed-claim (0.833333 on all 3 models) appears in that section and not in the saturated list	The flag is a candidate, not a verdict; a genuinely hard fixture can also land here and needs a human read	PASS -- pinned by test_phase1_flags_uniformly_hard_fixtures_outside_the_saturated_list, which also asserts the set is a subset of the discriminating set. A second such fixture (tool-capability-verification-selection, 0.750000 on all 3) was found and is reported too. The report labels both as candidates, never as verdicts
The number is recorded, not merely computable	A metric nothing records is not a measurement	Add the computed figures to evals/evaluating-skill-quality/eval-status.md as a summary plus a pointer	That file states the saturation and discrimination figures and the run directory they came from	Manual recording drifts. A CI-enforced disclosure gate is out of scope here	PASS -- new "Corpus saturation, computed from the Phase 1 run (issue #1461)" section carries the figures, the reproduction command, the scope limits, and the not-computable disposition for single-model runs. The drift risk stands as disclosed, not closed
The script is tested like its siblings	Every evals/scripts/*.py module here carries pytest coverage	Add tests/test_gitapex_compute_corpus_saturation.py covering the multi-model, single-model, and uniformly-hard branches	The new tests pass in CI alongside the existing suite	None identified	PASS -- 37 tests across an example layer and a Hypothesis property layer; 100 percent line and branch coverage, above the 90 percent floor gitapex_gate_evals_scripts_coverage.py enforces for this directory
Exit code stays report-only	This issue's scope is disclosure; a non-zero exit would make it a gate by the back door	Always exit 0 on well-formed input; reserve non-zero for malformed or unreadable input	The script exits 0 on a fully saturated run and on a fully discriminating one alike	None identified	PASS -- pinned by test_main_prints_the_report_and_exits_zero and test_main_exits_zero_on_a_fully_discriminating_run; a property test asserts exit 0 for every generated well-formed table. Exit 2 is reserved for unparseable input

Checklist

• Tests pass locally
• Docs updated if behavior changed
• Issue number cited in every commit
• If this PR adds/modifies a skills/*/SKILL.md, a docs/superpowers/specs/*.md design doc, a security-relevant skill, or a deterministic checker script (skills/*/scripts/*.py, evals/scripts/*.py, .github/scripts/*.py), a ## Skill audit evidence section discloses the required verdicts/waivers (see .github/scripts/gitapex_gate_skill_audit_disclosure.py)
• If this PR adds a new Kept-edit-log entry to any evals/*/split.md, that entry discloses a Transfer check line (see .github/scripts/gitapex_gate_transfer_check_disclosure.py) -- not applicable, no split.md entry added
• If this PR adds or increases a skills/*/SKILL.md's Stop-boundary bullets or named dispatch branches, evals/SKILL_NAME/tasks/*.yaml gained at least as many new fixtures (see .github/scripts/gitapex_gate_skill_branch_fixture_coverage.py) -- not applicable, no SKILL.md touched

Skill audit evidence

• checker-script-adversarial-review: RAN
• defeat-test-disclosure: RAN

checker-script-adversarial-review is RAN. The round is the reviewing-an-artifact invocation recorded under Independent review verdict below: five adversarial personas (correctness, blast-radius, reuse-and-simplification, convention, security) against this checker script's own diff, each candidate then put through the FABRICATED pre-check, an independent verification against actual behavior, and a counterfactual check. It produced three confirmed findings, all fixed in this PR before this line changed from NOT-RUN to RAN.

defeat-test-disclosure is RAN, with the artifact to show for it. tests/test_gitapex_compute_corpus_saturation_properties.py is a Hypothesis layer built specifically to defeat the classification logic rather than exercise its happy path: two of its properties are model-based (they recompute the intended partition from the generated table, so the module's own output cannot define correctness), and it found the empty-scores[] model-dropping defect described under Facts. The example layer adds nine loud-failure cases -- score conflict, boolean-as-score, string-as-score, missing and blank fixture_id, non-object entry, unparseable JSON, non-directory path, and blank model_id.

No skills/*/SKILL.md is touched by this PR, so battle-testing-a-skill and evaluating-skill-quality verdicts are not applicable here. deterministic-gate-quality is likewise not applicable: this module is registered in no .gitapex/ssot.json gate cluster, wired into no workflow, and always exits 0 on well-formed input -- it is a report tool in the same class as gitapex_check_dimension_coverage.py, not a gate.

Independent review verdict

• Verdict: CLEAN
• Verified commit: 22b23bc

Outer layer (GitHub-native reviewer): did not produce a review; disclosed, not equated with a pass. Anthropic's "Claude Code Review" GitHub App is not confirmed installed on this repository, so request_copilot_review was called for copilot-pull-request-reviewer[bot] instead. It returned no output and, at the time of this write, pull_request_read reports zero reviews and zero review threads on this PR. Copilot's review is in any case Comment-only, with no pass/fail signal of its own -- materially weaker than the App's severity summary, not equivalent to it. The two third-party statuses that did report are not this layer and are recorded only for completeness: Devin Review posted "Full review skipped: trial expired and no credits remaining"; CodeRabbit posted "Review skipped: manual review required for this OSS repository". Neither reviewed the diff.

Inner layer (reviewing-an-artifact, low effort): ran to completion; 3 confirmed findings, all fixed; 0 remaining. Step 0 found no specialist deferral (this module is report-only and registered in no gate cluster, so evaluating-deterministic-gate-quality does not own it; no workflow file, no SKILL.md, no instruction channel, no secrets-scan request), no causal-diagnosis redirect, and the target in scope. Step 1 classified the target dangerous (a new module carrying real parsing and classification logic), so the Step 2 fan-out ran rather than the safe-path skip.

Isolation disclosure, per that skill's own Stop boundary: the five persona passes did NOT run in fresh, isolated dispatches. This session authored the target, so prompt-level adversarial framing is not equivalent to context isolation. An isolation mechanism does exist in this harness; it was not used here because this session operates under an instruction not to dispatch subagents unless explicitly asked. The verdict below should be read with that weaker guarantee in mind.

Confirmed findings, all three fixed in commit e39fa8d1 (rebased into 22b23bcb):

1. FixtureVerdict.spread (reuse-and-simplification, root-cause) -- production surface no production code called; verified by grep across the tree, whose only two hits were the definition and its own test. Removed, per CLAUDE.md section 4.
2. That property's test name contradicted its assertion (convention, root-cause) -- test_spread_is_zero_when_every_model_agreed asserted 0.4 on two models that disagreed, so the agreement case it named had no coverage at all. Removed with the property it covered.
3. The NOT COMPUTABLE reason clause could contradict the counts printed beside it (correctness, root-cause) -- with two models present but no shared fixture, it printed 2 model(s) ... a cross-model rate needs at least 2 models, naming a condition that was in fact satisfied. Reproduced live before fixing. The clause now names whichever condition actually failed, pinned by one test per branch.

Audit trail -- candidates raised and rejected, with reasons:

• Float-equality sensitivity in is_uniformly_hard / is_saturated (rejected at the counterfactual check): exact equality is the honest reading of "every model scored the same value", and a tolerance would make the classification less faithful, not more. Not a defect; a documented property of the definition.
• Redundant str() around an already-validated model_id (rejected below the confidence bar): no behavior difference, no failure scenario.

Zero unconfirmed-concern findings: no security-tier signal is present anywhere in the target (no credential handling, no injection sink, no auth path, no network, no subprocess -- the module reads local JSON and prints a report), and low effort produces no speculative class otherwise.

One CI check is red for a pre-existing reason this PR did not cause, disclosed rather than presented as green. eval-gate fails at its "Run touched skills' evals suites" step in 2 seconds. The same step fails identically, in the same 2 seconds, on unrelated PR #1417 (run 33273426408, job 99155895803), whose diff shares nothing with this one. This PR touches evals/evaluating-skill-quality/eval-status.md, which is what makes that gate select evaluating-skill-quality and attempt a live suite run at all; that recording is required by the issue's own fourth acceptance criterion. Nothing in this diff participates in that step's execution path. This is an owner-side escalation, not an agent-fixable defect.

Related Issue

Closes #1461

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 7015ba6e-855e-4705-9dba-8da4b4b623a7

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.57%. Comparing base (9ae5bfb) to head (22b23bc).
⚠️ Report is 12 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff            @@
##             main    #1465    +/-   ##
========================================
  Coverage   99.57%   99.57%            
========================================
  Files         127      128     +1     
  Lines       22539    22693   +154     
  Branches     2766     2788    +22     
========================================
+ Hits        22443    22597   +154     
  Misses         96       96            

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tvna]

Standing down on eval-gate: not this PR's failure

Recording this once, rather than leaving a red check unexplained.

What is failing. eval-gate, at its step 8 "Run touched skills' evals suites". It fails 2 seconds into that step (job 99157829723 on head 22b23bcb, 20:41:57 -> 20:42:20 overall, step 8 itself 2s). Two seconds is far too short for a real suite run of evaluating-skill-quality's corpus; the step's own claude CLI invocation errors out immediately.

Why it is not this PR's. The identical step fails in the identical 2 seconds on unrelated PR #1417 (run 33273426408, job 99155895803, step 8 at 20:25:45 -> 20:25:47), whose diff shares no file with this one. Reviewing the last 25 runs of skill-eval-gate.yml, every run that actually had a skill to execute failed the same way; the runs that pass are the ones where Determine touched skills reported has-changes=false and step 8 never ran at all.

Nothing in this diff participates in that step's execution path. The new module is imported by nothing, wired into no workflow, and registered in no .gitapex/ssot.json gate cluster. What makes the gate select evaluating-skill-quality here is the one line this PR adds to evals/evaluating-skill-quality/eval-status.md -- and recording that figure is issue #1461's own fourth acceptance criterion, so it is not something the PR can drop to dodge the gate.

No fix exists to port. The failure is inside the gate's own live model-invocation path, not in repository code this PR could carry a change for. It needs an owner-side look at that step's runtime (its ANTHROPIC_API_KEY secret is present -- the preflight step passes -- so the immediate error is downstream of that check).

Re-run spent. Failed jobs re-run once on run 33274155577. If it fails again, that is the real signal, not a flake.

Everything else on head 22b23bcb is green, including the three that matter most for this change: pytest, mypy, and detection-logic-property-coverage. Codecov reports all modified lines covered (project 99.57%, +154 lines and +22 branches, 0 misses). independent-review-pending is green as of the run following the verdict record in the PR body.

---

Generated by Claude Code