EvalPlus harness scores empty server responses as wrong answers

[CryptoJones]

The harness scores an empty completion identically to a wrong answer. Both are simply a
failed task, so a server that returns nothing silently costs the model points.

Measured on the two runs whose raw generations still exist:

Run	Empty completions
Qwythos-9B-v2 (MLX)	26 / 164 (15.9%)
Qwythos-9B-v2 (llama.cpp)	8 / 164 (4.9%)
Ternary-Bonsai-27B (MLX)	33 / 164 (20.1%)
Ternary-Bonsai-27B (llama.cpp)	33 / 164 (20.1%)

These are raw, pre-sanitisation, zero-length responses — not code-extraction failures.

Comparing which tasks went empty separates two causes:

• Model-deterministic — Ternary-Bonsai-27B produced the identical set of 33 empties on
  both runtimes. Fair to score as failures.
• Runtime flakiness — Qwythos overlapped on only 5, leaving 21 MLX-only empties where
  llama.cpp answered the same prompt fine. That is noise depressing the MLX column.

Ruled out: the 4096-token cap. It is set in evalplus/provider/base.py and passed through
provider/openai.py, so both runtimes inherit it, and it predates these runs. Surviving MLX
generations show zero unclosed <think> blocks and a longer median than llama.cpp's.

Suggested fixes

1. Retry on empty in the codegen loop — retry 2-3× on a zero-length completion before
   recording a failure.
2. Preserve *.raw.jsonl next to the .evalplus summaries. Scratch cleanup destroyed the
   evidence for 13 of 15 models, which is why most MLX cells can no longer be audited.
3. Report an empty count per cell so the artifact is visible rather than buried in pass@1.
4. Capture mlx_lm.server stderr — ternary-bonsai-27b.mlxserver.log is 0 bytes, so the
   root cause of the MLX-only empties is currently unknowable.

Two affected cells were withdrawn in the meantime (see METHODOLOGY.md): Qwen3.5-9B MLX (70.7)
and Qwythos-9B-v2 MLX (75.0).

[CryptoJones]

Correction to the original issue body. The claim that Ternary-Bonsai-27B produced "the
identical set of 33 empties on both runtimes" was wrong, and so was the conclusion drawn
from it (that there is a clean "model-deterministic vs runtime-flakiness" split).

The glob used to locate runs matched the llama.cpp directory twice — the MLX ternary run lives
under ~/showdown-scratch/mlx_runs/eval/, not ~/showdown-scratch/eval/ — so the llama.cpp
file was compared against itself.

Correct figures, resolved by explicit path:

Model	empty (llama.cpp)	empty (MLX)	overlap	runtime-only
Qwythos-9B-v2	8 / 164	26 / 164	5	21 MLX-only
Ternary-Bonsai-27B	33 / 164	0 / 164	0	33 llama.cpp-only

No task came back empty across all four runs.

Neither runtime is systematically at fault. Qwythos is much worse on MLX; Ternary is the
reverse — 33 empties on llama.cpp and none on MLX. The effect is per-model and affects both
runtimes, so empty counts must be published per cell rather than blamed on a runtime.

Consequence: Ternary-Bonsai-27B's llama.cpp score (72.0) is itself depressed by ~20 points
of non-responses while its MLX score has none — the opposite of the direction I originally
implied.

Since fixed

• empty (lcpp) / empty (MLX) columns added to the matrix.
• harness/archive_run.sh archives every run into results/raw/ with a provenance manifest in
  results/runs/ (server cmd, harness commit, model sha256, host, CPU idle, counts). Wired into
  both run_quality.sh and run_mlx.sh.
• AGENTS.md records these rules, including resolve run directories by explicit path, never
  by wildcard.

Still open: retry-on-empty in the codegen loop, and capturing mlx_lm.server stderr
(ternary-bonsai-27b.mlxserver.log is 0 bytes).