[napetrov]

Part of #121 (item 1).

Summary

Before: a run's completeness came from whatever rows ended up in its artifact. A run that finished 2 of its 6 planned cells could still pass a gate built on per-harness counts.

After: completeness is computed from what was planned.

plan.v1   cells: [arm x model x task x rep, stable cell_id] + subjects, suites, grader, varied_variable, budget
   │  plan_digest (sha256 of the canonical plan; generated_at is left out)
   ▼
result.v1 cells: [{cell_id, status ∈ pass|fail|inconclusive|invalid|infra_error|not_run, model}]
   │
   ▼
runner.plan.completeness(plan, result) -> complete | incomplete (missing ids) | invalid (problems)

• plan.v1 records subject SHA and fingerprint, suites (hash plus selection reasons), requested models and harnesses, grader id and digest, one varied_variable (a string, or a list for a 2×2 design), planned cells, and a budget estimate (cells, agent_calls, cost_usd, wall_sec).
• result.v1 has plan_digest, an optional execution_id, and one status per cell. A cell can also record the resolved model (the feat(repo): record the model that answered each cell #119 per-cell record).
• completeness only counts cells. A cell is measured only when its status is pass, fail or inconclusive; not_run, infra_error, invalid and absent cells are listed in missing. The result is invalid when its digest does not match the plan, when it has a cell the plan does not, or when it gives one cell two statuses. Whether a complete run passes is left to the gate (J11/J12: distinct inconclusive verdict and blocking policy (AC-J11-002, AC-J12-002) #99).
• scripts/check_smoke_runs.py declares its planned cells (both harnesses on the one workflow task) and runs them through completeness. A missing or errored cell now fails the gate even when the summary counts are clean.

Not in this PR:

• --plan on repo run / arms / run, which would write result.v1. The execution id is still open in feat(repo): --delegation and an execution id for repo run cells #107, and repo run would need its cell loop reworked. That wiring is a follow-up once feat(repo): --delegation and an execution id for repo run cells #107 lands.
• Compare and gate policy (J11/J12: distinct inconclusive verdict and blocking policy (AC-J11-002, AC-J12-002) #99).

Evidence

• make ci-local: 25 pass, 1 FAIL, 6 skip. The one failure is pip-audit (diskcache 5.6.3 PYSEC-2026-2447, ragas 0.3.9 PYSEC-2026-3046). It fails the same way on requirements.lock.txt with this change stashed, and this PR does not touch any lock file.
• pytest tests/test_run_plan.py tests/test_check_smoke_runs.py tests/test_schema_compat.py tests/test_import_architecture.py tests/test_audit_test_quality.py: 208 passed.
• New tests that fail on wrong behaviour:
  • 2 of 6 cells passing gives incomplete, with the 4 missing ids listed.
  • A not_run, infra_error or invalid cell keeps the run incomplete.
  • Editing the grader digest after the run gives invalid.
  • Unplanned or doubled cells give invalid.
  • An unknown status is rejected by the schema.
  • A smoke artifact that lacks the oracle row fails the gate with 1/2 measured (incomplete).
• The smoke script run directly on an empty artifact: cells: 0/2 measured (incomplete), both cells listed as not measured, rc=1.
• schema_compat vs the PR base: pass. The fixtures plan.v1.json and result.v1.json were written through save_artifact.

Merge danger

• Low. Two new artifact kinds, a pure module in dokimi.runner (it imports only the standard library), and a stricter smoke gate.
• The external-catalog-smoke job now fails when runs.json has a row for a task or rep outside its plan. The workflow runs one task once, so that is expected not to happen. The job needs Docker and was not run locally.
• plan_digest hashes the plan as written, including schema_version. A producer has to compute the digest from the saved (stamped) plan, not from the dict it passed to save_artifact.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.69%. Comparing base (ad1cbd0) to head (043179c).
⚠️ Report is 17 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #161      +/-   ##
==========================================
+ Coverage   91.40%   91.69%   +0.29%     
==========================================
  Files         216      224       +8     
  Lines       26741    27584     +843     
  Branches     4377     4537     +160     
==========================================
+ Hits        24442    25293     +851     
+ Misses       1635     1622      -13     
- Partials      664      669       +5     

Files with missing lines	Coverage Δ
dokimi/dokimi/artifacts.py	93.75% <ø> (ø)
dokimi/dokimi/runner/plan.py	100.00% <100.00%> (ø)

... and 40 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review at 67975e9. The completeness computation is sound for well-formed input, and the smoke gate maps real task_runs rows onto the plan correctly. Three things are worth fixing before merge, and there is one nit.

What I ran, on a Linux review host at 67975e9:

• make ci-local: 25 passed, 1 failed, 6 skipped. The failure is pip-audit; the PR touches no lock file, so it is not caused by this change. The Docker smoke job was skipped.
• Reproductions for the completeness findings, run directly against the module; the smoke-gate reproduction and a 27-mutant run against the new tests (6 survived; the three that matter are in the inline comments).

Checked, no issues

• Smoke gate mapping: a healthy artifact (repeat absent for one repetition, model=None, task name onetbb-histogram-local-aggregation) passes. Missing, extra, duplicated and wrong-rep rows each fail with a specific message.
• Schema versioning: both kinds are new, each has a fixture, SCHEMA_VERSIONS is updated, and schema_compat passes. Fixtures load through load_artifact, their cell_ids match cell_id() and the result digest matches the plan.
• Digest: generated_at and key order do not change it; a changed grader digest gives invalid.

[antonwolfy]

I pushed a commit per nit, up to 7852366, which fix the nit from my review:

• 7852366: dokimi/dokimi/schemas/plan.v1.json:16: added additionalProperties: false to subject, suite, grader and budget objects, and to result cells in result.v1.json.

make ci-local at 7852366: 25 passed, 1 failed (pip-audit, same as at the reviewed head), 6 skipped.

Not approving yet. Still open: the three risks from my review.

[xaleryb]

• Ran: the head merged with current main, full suite, ruff, mypy strict, plus mutation of the malformed-cell guards in completeness.
• Growth ratchet: merging this head into main -> plan.py::completeness: 13 complexity is over the 10 cap and test_growth_ratchet fails -> expected green. This PR's CI ran on a base older than the ratchet.
• Digest trap: save_artifact("plan", plan, p), then result["plan_digest"] = plan_digest(plan) with every cell pass, then reload both -> invalid (digest mismatch) -> expected complete. plan_digest hashes schema_version, which save_artifact adds only to the copy it writes.
• Commits pushed:
  • 266a0aa: moves the status loop into _statuses(), so completeness is under the cap. No behaviour change.
  • 043179c: puts bad statuses on a planned cell and asserts the problem text. Covers a non-object cell and cells: None. Reverting any guard now fails a test.
• Left for the author: rebase onto main; choose a digest fix (default schema_version before hashing, or leave it out and regenerate result.v1.json); add a save-then-digest test.

[antonwolfy]

The three threads from my earlier review are fixed, and the malformed-cell handling in completeness now holds up. Two test gaps are worth closing before merge, both inline.

What I ran, on a Linux review host at 043179c:

• make ci-local: 25 passed, 1 failed, 6 skipped. The failure is pip-audit; the PR touches no lock file. The Docker smoke job was skipped. 6,748 tests passed.
• A mutation run of the new tests (about 40 mutants; the two that matter are inline).
• Reproductions for the digest point below, with the LLM transport mocked.

I agree with @xaleryb on the digest trap. New evidence: save_artifact("plan", plan, p), then result["plan_digest"] = plan_digest(plan) with six pass cells, reload both gives invalid (result is for plan sha256:7cd6c1…, not sha256:9ffebb…), and no test digests the pre-save dict. The digest choice waits for the author.

Checked, no issues

• Smoke gate mapping: task, harness and model come from TaskRunResult.as_dict; repeat is written only above 1, so absent means 1. A healthy artifact gives 2/2 measured (complete); missing, extra, duplicated and errored rows each fail. The script imports when run by path as CI does.
• Schema versioning: both kinds are new, SCHEMA_VERSIONS is updated, fixtures load and their digests match.

[antonwolfy]

No test pins this continue, and without it a list-valued cell_id crashes completeness.

Mutation check: replacing continue # reported by completeness with pass passes the full test_run_plan.py and test_check_smoke_runs.py (39 passed). With that mutant, a result cell {"cell_id": ["x"], "status": "pass"} raises TypeError: unhashable type: 'list' at status[cid] = st. The parametrised malformed-cell test uses {"cell_id": 3, "status": "pass"}, which is hashable, so it does not reach the crash. This is the failure the eb69341 fix was meant to remove.

Suggestion: add {"cell_id": ["c-1"], "status": "pass"} to the bad parameters of test_a_malformed_result_cell_is_invalid_not_an_exception.

[antonwolfy]

The :443 half of the egress check (line 87, outside this diff) is untested.

Mutation check: changing {"example.com:443", "example.com:80"} <= denied to {"example.com:80"} <= denied passes all 39 tests in test_check_smoke_runs.py and test_run_plan.py. A probe that never denied the CONNECT path would then pass the gate. test_check_smoke_runs.py:68 covers only a missing :80.

Suggestion: add a test with _runs(denied=("example.com:80",)) that asserts smoke_ok(...) is False.