[napetrov]

Summary

Some gates read an empty or all-errored result as a pass. Each one now exits 1 with an incomplete line and the counts. The fix reuses the existing "not measured" and incomplete terms. Partial errors are reported as counts, and no threshold changes.

                                  input                    main                         now
repo tasks check                  no task                  "0/0 ok", exit 0             refused, exit 1
repo tasks check --allow-unsup.   every task unsupported   "0/1 ok", exit 0             incomplete (0 ok, N unsupported), exit 1
repo run                          no task                  IndexError in the probe      refused, exit 1
repo run                          every cell unsupported   "0 ran, 0 done, 0 crashed"   summary counts unsupported; incomplete, exit 1
eval judge-wording --strict       0 pairs / all errored    "No rewrite moves any …"     NOT MEASURED + errored-pair row, exit 1
eval cross-judge --strict         every call errored       NO DETECTABLE EFFECT, 0      NOT MEASURED, exit 1
eval cross-judge --strict         no arm to compare        (no arms), exit 0            incomplete, exit 1
run --strict                      0 docs, gate at 0/off    exit 0                       incomplete, exit 1

The checks are in repos/workspace.load_tasks (no task), commands/repo_run (tasks check and run), eval/judge_wording.WordingReport.summary (unmeasured_conditions, measured, n_errors), eval/cross_judge.label (NOT MEASURED when no voting row has n > 0) and commands/run.cmd_run.

Not fixed here: report-only sites that print a reassuring line on empty input but set no exit code. They are follow-ups:

• baseline compare prints "No notable changes" on empty or unscored input.
• compare (snapshots) prints "No regressions" on 0 docs.
• The run report reads "minimal impact" on empty input.
• The trust gate does not count unscored rows on partial input.
• The novelty gate reports PASS with every check not measured.
• tasks run exits 0 when every cell errored.
• run raises a KeyError when a discovered doc fails to load.

Evidence

Each fixed site has one test that feeds empty and all-errored (or all-unsupported) input. All of them fail with the source changes stashed and pass now:

• tests/test_repo_host.py::test_tasks_check_with_no_task_checked_is_not_a_pass: no task, and every task unsupported with --allow-unsupported.
• tests/test_repo_host.py::test_a_run_with_no_cell_result_is_not_a_pass: no task (IndexError on main), and every cell unsupported.
• tests/test_judge_reliability.py::test_judge_wording_strict_exits_1_when_a_condition_scored_no_pair: 0 pairs, and 4 pairs that all errored.
• tests/test_judge_reliability.py::test_cross_judge_strict_exits_1_when_no_arm_was_measured: a judge whose every call errors, and an arms file with no scored treatment arm.
• tests/test_gate_call_sites.py::test_strict_exit_1_when_no_document_was_measured: 0 docs with the hard gate at 0, and with no gate.

main:  2 failed (test_repo_host -k not_a_pass) · 2 failed (test_judge_reliability) · 1 failed (test_gate_call_sites)
now:   7073 passed, 22 skipped

I updated two existing tests that ran only unsupported cells or tasks. Those runs now exit 1, so the tests expect SystemExit. One assertion on the summary line now includes the unsupported count.

Local checks: ruff check . clean and mypy clean (222 files). The import-architecture test and audit_test_quality.py pass. The docs review triggers report no stale page, and the changelog fragment check passes. The CLI surface diff is empty: only help text changed. Changed-line coverage is 96%. The one uncovered line is the existing FRAGILE exit, which moved.

Merge Danger

Door: two-way. Every change is an exit code or a label added for input that measured nothing, and reverting it restores the old behaviour.

Blast Radius: gates

• A CI job that calls repo tasks check --allow-unsupported or repo run on a host where every task or cell is unsupported now fails. Before, it passed with nothing checked.
• eval cross-judge gains a label, NOT MEASURED. The cross_judge.v0 JSON label field can now hold it, so readers that switch on the label need a case for it.
• eval judge-wording JSON summary gains three optional keys: n_errors, unmeasured_conditions and measured. The existing keys are unchanged.
• The repo run summary line now reads …, N not started (batch budget), M unsupported; $X. Any scraper of that line needs updating.
• eval cross-judge --require-validated --strict with no passing validity report was NO DETECTABLE EFFECT and exit 0; it is now NOT MEASURED and exit 1.
• eval judge-wording no longer lists a variant as fragile (report text and JSON) when a condition scored no pair; --strict still exits 1 for the unmeasured condition.
• run --strict over a tree with no markdown under docs/ now exits 1.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 97.29730% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 92.02%. Comparing base (94e3eb1) to head (1983d7d).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/commands/judge_reliability.py	94.11%	0 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #176      +/-   ##
==========================================
+ Coverage   91.97%   92.02%   +0.04%     
==========================================
  Files         223      223              
  Lines       27709    27745      +36     
  Branches     4553     4564      +11     
==========================================
+ Hits        25486    25532      +46     
+ Misses       1574     1569       -5     
+ Partials      649      644       -5     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/repo_run.py	92.65% <100.00%> (+0.15%)	⬆️
dokimi/dokimi/commands/run.py	69.33% <100.00%> (+1.27%)	⬆️
dokimi/dokimi/eval/cross_judge.py	99.47% <100.00%> (+<0.01%)	⬆️
dokimi/dokimi/eval/judge_wording.py	91.66% <100.00%> (+2.28%)	⬆️
dokimi/dokimi/repos/workspace.py	87.62% <100.00%> (+0.06%)	⬆️
dokimi/dokimi/commands/judge_reliability.py	91.76% <94.11%> (+2.13%)	⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review at edcf045. The fixes read right: each gate now exits 1 with an incomplete line when nothing was measured, and no threshold moves. Two test gaps would let a plausible edit of the new conditions through, and one doc is missing.

What I ran, on a Linux review host at edcf045:

• make ci-local: 20 passed, 8 failed, 7 skipped. All 8 failures are exit 127: ruff, mypy, zizmor, pre-commit, bandit, deptry, mkdocs and pip-audit are missing from my venv, because make dev stops at the tools install on this base, which #179 fixed on main. Static checks, mypy and the strict docs build pass on GitHub's CI at this head. 7074 tests passed, 21 skipped; changed-line coverage 100%.
• Mutation runs on a copy of the head tree for the new conditions (details inline): 3 mutants of the new conditions, all of which pass the full test_judge_reliability.py.

Checked, no issues

• load_tasks: the only non-test callers are repo_run.py:145 and :193, and both catch PrepError, so the new "no task" refusal prints a ✗ line and exits 1.
• repo run / tasks check: removing each new guard (workspace.py no-task, repo_run.py incomplete exits, run.py zero docs) fails a test. A resumed batch (cells skipped) still exits 0.
• label(): Measure.versus always sets n, so the r.get("n", 1) default is never used for real rows.

[xaleryb]

• Ran: the full suite, ruff, mypy and bandit on this head merged with current main, plus the new tests against the pre-fix source and mutants.
• Growth ratchet on the merge with main: tests/test_judge_reliability.py 1830 lines > baseline 1695 and cmd_eval_cross_judge complexity 33 > 32, so Tests turns red on main. The PR should stay within the baseline.
• repo tasks check --allow-unsupported --task X with X unsupported on the host exits 1, but --help still promises exit 0 for that case. The help should state the exception.
• eval cross-judge --require-validated --strict with no passing validity report: was NO DETECTABLE EFFECT and exit 0, is now NOT MEASURED and exit 1. No test covers it, and Merge Danger does not mention it.
• Pushed:
  • 2cc66db: moves the strict exit into a helper and the new tests into tests/test_judge_not_measured.py, which brings both numbers back to the baseline.
  • 2d97d0e: pins the zero-voter NOT MEASURED case.
  • 9ab2184: fixes the --allow-unsupported help and the contract sentence.
• Left for the author: a Merge Danger line for the --require-validated --strict exit change.

[antonwolfy]

Review at 9ab2184. The three threads from my earlier review are addressed, and the gates now exit 1 when nothing was measured. Two things are worth fixing before merge, both inline: a judge-wording report that prints NOT MEASURED and FRAGILE together, and an untested --strict guard on the new wording exit.

@xaleryb's question on the Merge Danger line for eval cross-judge --require-validated --strict (was exit 0 with NO DETECTABLE EFFECT, now NOT MEASURED and exit 1) waits for the author.

What I ran, on a Linux review host at 9ab2184:

• make ci-local: 27 passed, 1 failed, 7 skipped. The failure is pip-audit on the lock files, which this PR does not touch. Tests, ruff, mypy, bandit, schema-compat and the strict docs build pass. Changed-line coverage 97%.
• Reproductions for the code findings, with the LLM transport mocked.
• A mutation run on the new conditions: 39 mutants, 8 of which pass the full test files; the two that matter are inline.

Checked, no issues

• repo tasks check / repo run: len(bad) == len(results) is reached only when every bad result is an allowed unsupported one. load_tasks refuses an empty export first. A resumed batch still exits 0, because done cells count as skipped.
• run --strict: the zero-document exit sits after the hard-gate and critical-band exits, and summary["docs"] is always set.
• cross_judge.label: every Measure.versus row carries n. One voter with n=0 does not hide another with n>0, and any to all now fails a test.
• Schemas and docs: no schema under dokimi/dokimi/schemas/ covers cross-judge or judge-wording output, so the new label and keys need no version bump. The flags and the summary line in the docs match the code.

[antonwolfy]

A condition with no scored pair prints NOT MEASURED and FRAGILE in the same report. fragile_variants is still computed from the errored pairs, and measured does not suppress it.

Case: 20 bases, original scored 20 pairs, persona errored on all 20 (ThrottlingException). summary() gives measured: False and fragile_variants: ['wrong_numbers'], and format_summary() ends with:

  NOT MEASURED — no pair scored under: persona
  FRAGILE — a rewrite moves these on the same pairs (paired sign test, Holm): wrong_numbers

The --output JSON carries the same contradiction. A throttled run reads as wording sensitivity. --strict exits 1 either way, so the exit code is right; the report text and the JSON are not. With 19 of 20 pairs errored the old code already printed FRAGILE; the new errored pairs row shows the count but the flag still reads as a finding.

Suggestion: compute fragile as false for a variant when its condition is in unmeasured, or print FRAGILE only when s["measured"]. Add a format_summary() assertion for the one-condition-errored case.

[napetrov]

Fixed in 1983d7d: fragile is now false for every variant unless the report is measured, so a condition with no scored pair prints NOT MEASURED without FRAGILE, in the text and in the --output JSON. Test: test_judge_wording_strict_exits_1_when_one_condition_scored_and_another_did_not (8 pairs, persona errored; asserts no FRAGILE in the text, empty fragile_variants; fails on the old code).

[antonwolfy]

Nothing pins the new judge-wording exit to --strict. Without the flag the new exit must not fire, and no test checks that.

Mutation check: replacing if args.strict and not summary["measured"]: with if not summary["measured"]: passes test_judge_reliability.py and test_judge_not_measured.py in full (135 tests). Dropping args.strict and from the FRAGILE line below passes too. Either edit makes a non-strict judge-wording run exit 1 on an errored condition or a fragile variant and breaks pipelines that rely on the report-only default.

This predates the PR for the FRAGILE line and for the JUDGE-DEPENDENT branch of _strict_cross_judge_exit (if any(...) replaced by if False: also passes all 135), but the PR moved or extended both.

Suggestion: in test_judge_wording_strict_exits_1_when_a_condition_scored_no_pair, run the same argv without --strict and assert exit 0 with the NOT MEASURED line printed. Add a cross-judge --strict case with a JUDGE-DEPENDENT arm that expects exit 1, and the same run without --strict that expects 0.

[napetrov]

Fixed in 1983d7d: tests in test_judge_not_measured.py. The errored-condition judge-wording run without --strict exits 0 and prints NOT MEASURED; test_judge_wording_fragile_variant_exits_1_only_with_strict pins the FRAGILE line to --strict; test_cross_judge_judge_dependent_exits_1_only_with_strict covers the JUDGE-DEPENDENT branch (0 without, 1 with --strict).

[napetrov]

Merge with intel/main in 55d2d08 (one conflict in the --allow-unsupported help in repo_run.py; kept both sides, and two main tests that checked a warm check with only an unsupported task now include an ok task, since that check exits 1 by this PR). xaleryb's open item: the Merge Danger section of the description now lists the eval cross-judge --require-validated --strict change (NO DETECTABLE EFFECT/exit 0 to NOT MEASURED/exit 1) and the judge-wording FRAGILE suppression.