fix(gates): nothing measured is not a pass

dokimi/README.md

@@ -202,7 +202,7 @@ The default static benchmark is configured in `benchmarks/spec.v1.yaml` and curr
 | `readability` | Estimates text readability with a Flesch-Kincaid-style score. |
 | `example_pass_rate` | Executes supported fenced code examples in isolation. |
 
-Weights are normalized across enabled metrics. The benchmark also supports soft gates, hard gates, critical bands, and regression thresholds. Hard gates and critical bands are enforced only when `--strict` is passed.
+Weights are normalized across enabled metrics. The benchmark also supports soft gates, hard gates, critical bands, and regression thresholds. Hard gates and critical bands are enforced only when `--strict` is passed. `--strict` also exits 1 when no document was measured.
 
 ## LLM and treatment evaluation workflow
 

dokimi/changelog.d/20261021_120000_napetrov_outcome-completeness.md

@@ -0,0 +1,3 @@
+### Fixed
+
+- **Nothing measured is not a pass** — gates that read an empty or all-errored result as green now exit 1 with an `incomplete` line and the counts. `repo tasks check` and `repo run` refuse an export with no task (`repo run` crashed with an IndexError on one); `repo tasks check` exits 1 when no task passed the self-check, even with `--allow-unsupported`; `repo run` exits 1 when no cell of the batch has a result, and its summary line now counts unsupported cells. `eval judge-wording` reports a condition with no scored pair as NOT MEASURED (with errored-pair counts) instead of "No rewrite moves any variant", and `--strict` exits 1 on it. `eval cross-judge` labels an arm `NOT MEASURED` when no voting measure compared a single question, and `--strict` exits 1 on that or when no arm was compared. `run --strict` exits 1 when no document was measured. Partial errors change no threshold.

dokimi/docs/benchmarking-and-comparison.md

@@ -445,7 +445,9 @@ to the baseline's; `--escalate-model` is the same cheap-then-expensive cascade.
 under several judges (absolute, and with `--grounded` against each question's
 `expected_topics`), takes `eval pairwise` and `eval checklist` reports as more
 measures, and labels each arm `CONSISTENT ±`, `LEANING ±`, `JUDGE-DEPENDENT` or
-`NO DETECTABLE EFFECT`, on Holm-adjusted p over the measures that vote. With
+`NO DETECTABLE EFFECT`, on Holm-adjusted p over the measures that vote, or
+`NOT MEASURED` when no voting measure compared a single question (every call
+errored, or every judge was gated out); `--strict` exits 1 on that too. With
 `--validity` reports, a judge that failed its gate is shown but does not vote
 (its grounded and checklist measures included, in any context);
 `--require-validated` leaves only judges that passed. The stored measure is
@@ -482,7 +484,9 @@ python cli.py eval cross-judge --answers run.json --questions golden.json --grou
 under meaning-preserving rewrites of it and flags a degradation FRAGILE when a
 rewrite moves it on the same pairs: a paired sign test against the original
 prompt, Holm-adjusted over every rewrite and variant. Re-draws of the original
-are shown as a noise floor but do not decide the flag. Use at least 30 bases; at
+are shown as a noise floor but do not decide the flag. A condition with no
+scored pair (every call errored) is reported NOT MEASURED, not as "no rewrite
+moves anything", and `--strict` exits 1 on it. Use at least 30 bases; at
 20 bases only a large wording effect is detectable.
 
 `eval checklist` asks, per expected topic and in both polarities, whether the

dokimi/docs/repo-evaluation-contract.md

@@ -462,13 +462,21 @@ For every task, before any paid run:
 A task whose `host` requirements (§3) the probe container does not meet is
 `unsupported` with the reason, like a task whose `egress` needs the missing
 proxy, and `tasks check` exits 1 for it. With `--allow-unsupported` it is
-still printed with its reason but does not make the exit code 1; broken tasks,
+still printed with its reason but does not make the exit code 1 unless no task
+is ok (below); broken tasks,
 tasks that need egress and tasks whose host probe failed still do. `repo run` does not run its cells:
 each holds an `unsupported.json` with the reason, and `repo report` lists them
 apart, neither failed nor missing. If the probe itself fails, a task with
 `host` requirements is not run either, and `repo run` exits 1 when one of its
 cells is left unrun; a cell that already has a `result.json` keeps it.
 
+Nothing checked is not a pass. Both commands refuse an export with no task, and
+exit 1 as `incomplete` when nothing was measured: `tasks check` when no task
+passed the self-check (every one unsupported, even with `--allow-unsupported`),
+`repo run` when no cell of the batch has a result (every one unsupported). The
+`repo run` summary line counts unsupported cells beside ran, done, crashed and
+not started.
+
 Each task runs with its own network (`image.network` plus its `egress`).
 `tasks check --warm` runs the same null and oracle steps with network and
 writable caches, to fill the caches before offline runs (§2); it judges task
@@ -960,7 +968,7 @@ prompts disabled`) and the fix; any other git failure shows its last `fatal:`
 line, else its last `error:` line, else its last line.
 
 ```bash
-# self-check: no LLM; exits 1 if any task is broken or unsupported
+# self-check: no LLM; exits 1 if any task is broken or unsupported, or none is ok
 python cli.py repo tasks check ~/oneDAL@<sha> --manifest dev/agent_evals/repo-eval.yaml \
     --task fix_finiteness_mut --build-network host --cache bazel-repo-cache=/cache/bazel-repo -j 3
 

dokimi/dokimi/commands/judge_reliability.py

@@ -9,7 +9,7 @@
 import json
 import sys
 from pathlib import Path
-from typing import Optional
+from typing import Any, Optional
 
 from dokimi.defaults import (
     DEFAULT_CONCURRENCY,
@@ -248,7 +248,12 @@ def progress(cond: str, done: int, total: int) -> None:
     print(report.format_summary())
     if args.output:
         print(f"\n✅ Wording report saved to {out}")
-    if args.strict and report.summary()["fragile_variants"]:
+    summary = report.summary()
+    if args.strict and not summary["measured"]:
+        print(f"✗ incomplete: no pair scored under {', '.join(summary['unmeasured_conditions']) or 'any condition'} "
+              f"({len(pairs)} pairs, errors per condition: {summary['n_errors']})", file=sys.stderr)
+        sys.exit(1)
+    if args.strict and summary["fragile_variants"]:
         sys.exit(1)
 
 
@@ -409,7 +414,19 @@ def score(library, question, answer, ground_truth, context):
     print()
     print(md)
     print(f"✅ Cross-judge report saved to {out} (+ .md)")
-    if args.strict and any(a["label"] == "JUDGE-DEPENDENT" for a in report.summary()["arms"].values()):
+    if args.strict:
+        _strict_cross_judge_exit(report.summary()["arms"], report.baseline_arm)
+
+
+def _strict_cross_judge_exit(arms: dict[str, dict[str, Any]], baseline_arm: str) -> None:
+    """Exit 1 when no arm was compared, an arm is NOT MEASURED, or an arm is JUDGE-DEPENDENT."""
+    unmeasured = sorted(a for a, r in arms.items() if r["label"] == "NOT MEASURED")
+    if not arms or unmeasured:
+        why = f"{len(unmeasured)} of {len(arms)} arm(s) NOT MEASURED: {', '.join(unmeasured)}" if arms else \
+            "no arm has a scored question to compare"
+        print(f"✗ incomplete against `{baseline_arm}`: {why}", file=sys.stderr)
+        sys.exit(1)
+    if any(a["label"] == "JUDGE-DEPENDENT" for a in arms.values()):
         sys.exit(1)
 
 
@@ -447,7 +464,8 @@ def register(eval_sub, positive_int) -> None:
                          "flag is a paired test against the original and does not use them (default: 1)")
     jw.add_argument("--seed", type=int, default=0)
     jw.add_argument("--min-accuracy", type=float, default=DEFAULT_MIN_ACCURACY, dest="min_accuracy")
-    jw.add_argument("--strict", action="store_true", help="Exit 1 when any variant is FRAGILE")
+    jw.add_argument("--strict", action="store_true",
+                    help="Exit 1 when any variant is FRAGILE or a condition has no scored pair")
     jw.add_argument("--judge-model", default=JUDGE_MODEL_FOR_PROVIDER, help=JUDGE_MODEL_HELP)
     jw.add_argument("--judge-provider", default=DEFAULT_JUDGE_PROVIDER, choices=JUDGE_PROVIDERS)
     jw.add_argument("--judge-temperature", type=float, default=None, dest="judge_temperature",
@@ -485,7 +503,8 @@ def register(eval_sub, positive_int) -> None:
     cj.add_argument("--judge-temperature", type=float, default=None, dest="judge_temperature",
                     help="Judge sampling temperature, recorded as judge_temperature in the output")
     cj.add_argument("--concurrency", type=positive_int, default=DEFAULT_CONCURRENCY)
-    cj.add_argument("--strict", action="store_true", help="Exit 1 when any arm is JUDGE-DEPENDENT")
+    cj.add_argument("--strict", action="store_true",
+                    help="Exit 1 when any arm is JUDGE-DEPENDENT or NOT MEASURED, or no arm was compared")
     cj.add_argument("--output", default=None,
                     help="Output JSON (default: <answers>_crossjudge.json); Markdown written next to it")
     cj.set_defaults(func=cmd_eval_cross_judge)

dokimi/dokimi/commands/repo_run.py

@@ -180,6 +180,9 @@ def cmd_repo_tasks_check(args: argparse.Namespace) -> None:
               + ", ".join(r["task"] for r in allowed))
     if len(bad) > len(allowed):
         sys.exit(1)
+    if len(bad) == len(results):  # --allow-unsupported passed every task by: none was checked
+        print(f"✗ incomplete: no task passed the self-check (0 ok, {len(allowed)} unsupported)", file=sys.stderr)
+        sys.exit(1)
 
 
 def cmd_repo_run(args: argparse.Namespace) -> None:
@@ -234,12 +237,17 @@ def _run_locked(args: argparse.Namespace, arms: list[str], batch_dir: Path, lock
         print(f"✗ {exc}", file=sys.stderr)
         sys.exit(1)
     print(f"{summary['ran']} ran, {summary['skipped']} already done, {len(summary['errors'])} crashed, "
-          f"{summary['budget_stopped']} not started (batch budget); ${summary['spent_usd']:.4f}")
+          f"{summary['budget_stopped']} not started (batch budget), {summary['unsupported']} unsupported; "
+          f"${summary['spent_usd']:.4f}")
     if summary["probe_failed"]:
         print(f"✗ the host probe failed, so these tasks have cells that were not run: {', '.join(summary['probe_failed'])}",
               file=sys.stderr)
     if summary["errors"] or summary["budget_stopped"] or summary["probe_failed"]:
         sys.exit(1)
+    if not summary["ran"] and not summary["skipped"]:  # every cell unsupported: the batch holds no result
+        print(f"✗ incomplete: no cell has a result ({summary['cells']} cells, {summary['unsupported']} unsupported)",
+              file=sys.stderr)
+        sys.exit(1)
 
 
 def _pin_arms(arms: list[Arm], git_dir: Path, source: str) -> list[str]:
@@ -354,7 +362,8 @@ def register(repo_sub, positive_int, source_help: str, cache_help: str) -> None:
                          help="Exit 0 when the only tasks that are not ok are unsupported because this host "
                          "misses their task.json `host` requirements (still printed with the reason; "
                          "with --warm the host is judged too). Broken tasks, tasks that need egress "
-                         "and tasks whose host probe failed still exit 1")
+                         "and tasks whose host probe failed still exit 1, and so does a check in "
+                         "which no task is ok")
     check_p.add_argument("--out", default="repo-check", help="Work directory (default: repo-check)")
     check_p.set_defaults(func=cmd_repo_tasks_check)
 

dokimi/dokimi/commands/run.py

@@ -61,6 +61,11 @@ def cmd_run(args: argparse.Namespace) -> None:
                     print(f"  - {v.condition}: {v.actual:.4f} < {v.threshold:.4f}", file=sys.stderr)
             sys.exit(1)
 
+        # A score over no document passes any gate set at 0 or switched off.
+        if not result["summary"].get("docs"):
+            print(f"\n❌ incomplete: no document was measured under {root} (0 docs)", file=sys.stderr)
+            sys.exit(1)
+
 
 def cmd_compare(args: argparse.Namespace) -> None:
     """Compare two snapshots and show regression analysis."""

dokimi/dokimi/eval/cross_judge.py

@@ -26,6 +26,9 @@
   other way, but some point estimates disagree.
 * ``JUDGE-DEPENDENT`` — significant in both directions. Do not ship on it.
 * ``NO DETECTABLE EFFECT`` — no measure significant.
+* ``NOT MEASURED`` — no measure that votes scored a single question of the arm
+  against baseline (every call errored, or every judge is gated out). It is not
+  "no effect": nothing was compared.
 
 A label whose significant measures all rest on one judge says so —
 ``CONSISTENT + (one judge)`` — because then it is one judge's verdict, not
@@ -132,6 +135,8 @@ def label(rows: Sequence[Dict[str, Any]], *, require_validated: bool = False) ->
     """Verdict over the rows allowed to vote (see :func:`votes`), on Holm-adjusted p."""
     apply_holm(rows, require_validated=require_validated)
     rows = [r for r in rows if r["votes"]]
+    if not any(r.get("n", 1) for r in rows):  # no voter compared a single question
+        return "NOT MEASURED"
     sig = {r["sign"] for r in rows if r["significant"] and r["sign"]}
     if not sig:
         return "NO DETECTABLE EFFECT"

dokimi/dokimi/eval/judge_wording.py

@@ -148,6 +148,11 @@ def summary(self) -> Dict[str, Any]:
         tests = [(v, c) for v in sorted(acc) for c in rewrites if c in acc[v] and "original" in acc[v]]
         raw = [self._paired_p(v, c) for v, c in tests]
         holm = dict(zip(tests, adjust_pvalues(raw, method="holm", rounded=False), strict=True))
+        # A condition with no scored pair (none built, or every call errored) was
+        # not measured: "no rewrite moves anything" would be read off zeros, and an
+        # errored pair counts 0 in the sign test, so nothing is flagged either.
+        unmeasured = [c for c, r in self.reports.items() if not r.scored]
+        measured = bool(self.reports) and not unmeasured
         rows = {}
         for variant, by in sorted(acc.items()):
             noise = [by[c] for c in floor if c in by]
@@ -165,13 +170,16 @@ def summary(self) -> Dict[str, Any]:
                 "spread": spread,
                 "worst_wording": worst,
                 "p_holm": None if p_min is None else round(p_min, 4),
-                "fragile": spread >= MIN_FLAGGED_SPREAD and p_min is not None and p_min < ALPHA,
+                "fragile": measured and spread >= MIN_FLAGGED_SPREAD and p_min is not None and p_min < ALPHA,
             }
         pooled = {c: r.summary()["accuracy"] for c, r in self.reports.items()}
         return {
             "judge": self.judge,
             "conditions": list(self.reports),
             "pooled_accuracy": pooled,
+            "n_errors": {c: len(r.errors) for c, r in self.reports.items()},
+            "unmeasured_conditions": unmeasured,
+            "measured": measured,
             "per_variant": rows,
             "fragile_variants": sorted(v for v, r in rows.items() if r["fragile"]),
         }
@@ -200,14 +208,23 @@ def format_summary(self) -> str:
         lines.append(
             f"  {'pooled (graded)':20s} " + " ".join(f"{s['pooled_accuracy'][c]:10.2f}" for c in conds)
         )
+        if any(s["n_errors"].values()):
+            lines.append(
+                f"  {'errored pairs':20s} " + " ".join(f"{s['n_errors'][c]:10d}" for c in conds)
+            )
         lines.append("")
         lines.append("  * neutral variant: column is unmoved rate, not accuracy")
+        if not s["measured"]:
+            lines.append(
+                "  NOT MEASURED — no pair scored under: "
+                + (", ".join(s["unmeasured_conditions"]) or "any condition")
+            )
         if s["fragile_variants"]:
             lines.append(
                 "  FRAGILE — a rewrite moves these on the same pairs (paired sign test, Holm): "
                 + ", ".join(s["fragile_variants"])
             )
-        else:
+        elif s["measured"]:
             lines.append("  No rewrite moves any variant on the same pairs (paired sign test, Holm).")
         return "\n".join(lines)
 

dokimi/dokimi/repos/workspace.py

@@ -187,10 +187,14 @@ def score(self) -> dict[str, Any] | None:
 
 
 def load_tasks(content_dir: Path, manifest: RepoManifest, ids: list[str] | None = None) -> list[Task]:
-    """Tasks of the exported eval dir; ``ids`` selects (and must name existing tasks)."""
+    """Tasks of the exported eval dir; ``ids`` selects (and must name existing tasks).
+
+    No task at all is refused: a check or run over none would report 0/0 as a pass."""
     eval_dir, tasks = manifest.run_paths()
     tasks_dir = content_dir / PurePosixPath(tasks).relative_to(eval_dir)
     found = {p.parent.name: p for p in sorted(tasks_dir.glob("*/task.json"))}
+    if not found:
+        raise PrepError(f"no task under {manifest.tasks} (no */task.json): nothing to check or run")
     wanted = ids or sorted(found)
     missing = [t for t in wanted if t not in found]
     if missing:

dokimi/tests/test_gate_call_sites.py

@@ -87,3 +87,20 @@ def test_runner_passes_soft_kind(tmp_path):
             for p in ps:
                 p.stop()
     assert spy.call_args.args[2] == "soft"
+
+
+def test_strict_exit_1_when_no_document_was_measured(tmp_path, capsys):
+    # A gate at 0 (or a disabled one) passes a score over nothing.
+    for future in ({"hard_gate": G(0.0)}, {}):
+        with patch("dokimi.runner.run.discover_markdown", return_value=[]):
+            sp = tmp_path / "spec.yaml"
+            sp.write_text(yaml.safe_dump(dict(SPEC, future=future)))
+            args = argparse.Namespace(root=str(tmp_path), spec=str(sp), out_json=str(tmp_path / "o.json"),
+                                      out_md=str(tmp_path / "o.md"), strict=True)
+            try:
+                cmd_run(args)
+                code = 0
+            except SystemExit as e:
+                code = e.code
+        assert code == 1
+        assert "incomplete: no document was measured" in capsys.readouterr().err