[napetrov]

Part of #121, J7 #97.

Summary

Draft of the dokimi-design-evals skill (item 6 of #121), built only on commands that exist on main today.

dokimi/data/skills/dokimi-design-evals/
  SKILL.md                         workflow: sources -> scenario -> 3 layers -> verifier -> self-test -> multi-step -> coverage -> status
  references/scenario-template.md  initial state, request, expected outcome, allowed actions, forbidden consequences, verifier, repro, source, status
  references/verifier-self-test.md validate, null/oracle/isolation, mutation controls (manual), judge check
  references/coverage-report.md    happy path, branches, recovery, ambiguity, constraints, negatives, neighbours
  references/example-scenario.md   fixed-failure Harbor task + verifier; a mutation the first verifier misses
dokimi/tests/test_bundled_skills.py  each bundled skill passes check_skill, name == dir, description <= listing cap, relative links resolve inside the skill
dokimi/tests/test_docs_code_refs.py  now also scans data/skills/**/*.md, so every command/flag the skill names is checked against the parser

Steps the skill marks as manual, because no command exists yet:

Need	Today
Mutation controls (skipped step, false test claim, deleted tests, weakened criterion, wrong tool)	manual: mutate a scratch copy's oracle (solve.sh, or --manifest-file for repo tasks) and expect the self-test to fail
Null check for Harbor tasks	manual docker run from contributing-terminal-bench-task §8; repo tasks check covers it only for repo tasks
Valid inaction	repo tasks check always requires the null to fail; such a task needs a manual null check
Scenario coverage report	written by hand; python -m dokimi.coverage is a product x awareness/work count
draft -> reviewed -> holdout status, scenario source	no field dokimi reads; kept in task.toml [metadata] / extra task.json keys
Multi-step checkpoints	Harbor steps; dokimi does not validate per-step dirs
Drafting tasks from code/issues (AC-J07-006)	presented as hypotheses with source, by hand

Evidence

• --help checked for every command the skill names: repo inspect, repo validate --manifest --fail-on-warning, repo tasks check --manifest --task --manifest-file, tasks list --tasks-root, tasks run --tasks-root --tasks --harnesses docker-oracle|docker-probe --out-json, tasks harnesses, eval judge-validity --answers --sequential, python -m dokimi.coverage --tasks-root.
• pytest tests/test_bundled_skills.py tests/test_docs_code_refs.py tests/test_agent_profiles.py tests/test_audit_test_quality.py: 186 passed. The docs-ref check caught one false path (tests/test_outputs.py in the example) before commit.
• make ci-local: 25 pass, 6 skip, 1 fail: pip-audit on virtualenv 21.14.7 (CVE-2026-113722, fix 21.14.9), already in the lock on main and not touched here. Full test run 6728 passed.
• Not done: no agent run with the skill, no Harbor/docker run of the example task (the example's self-test table is labelled illustrative).

Merge danger

• skill-catalog:data/skills arms now see two skills instead of one; a catalog run from this tree is not comparable with an earlier one.
• The new docs-ref coverage of data/skills/**/*.md fails future skill edits that name a renamed command, by design.
• Draft: wording and the manual steps should be reviewed against User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 items 3-4 (gate, validate) before they land, so the skill can switch to those commands.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.53%. Comparing base (a44b3d0) to head (294bf2e).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #162      +/-   ##
==========================================
- Coverage   91.54%   91.53%   -0.01%     
==========================================
  Files         222      222              
  Lines       27369    27369              
  Branches     4505     4505              
==========================================
- Hits        25054    25052       -2     
- Misses       1644     1645       +1     
- Partials      671      672       +1     

see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review at 0389033. The skill is well structured, and every command and flag it names exists in the parser. Two things are worth fixing before merge, plus two nits inline.

What I ran, on a Linux review host at 0389033:

• make ci-local: 25 passed, 1 failed, 6 skipped. The failure is pip-audit on virtualenv 21.14.7 (CVE-2026-113722), which is already in the lock at the PR's base and not touched here (fixed on main by #152). Changed-line coverage passed.
• Reproductions for the findings, with the LLM transport mocked; a mutation run of test_bundled_skills.py and the test_docs_code_refs.py change against broken copies of the new skill files.

This PR adds a second skill under dokimi/data/, which the repo rules reserve for fastcontext. AGENTS.md:94-97 says dokimi/data/ holds only the fixtures the repo's own tests and examples read (data/skills/fastcontext/, ...), and that product skills and tasks belong in their content repository. dokimi/data/README.md:22 calls fastcontext "The one committed skill". The new skill is not a fixture for any treatment arm or example; the only reader is the new lint test. Either update AGENTS.md and data/README.md to say why a second bundled skill is allowed, or move the skill to the content repository. The README line (outside this diff) is wrong as soon as this merges.

Checked, no issues

• Commands: repo inspect, repo validate --fail-on-warning, repo tasks check --manifest/--manifest-file/--task, tasks list|run --tasks-root --tasks --harnesses --out-json, eval judge-validity --answers --sequential and python -m dokimi.coverage all exist in --help. docker-oracle and docker-probe are in the registry. The claim that repo tasks check always requires the null to fail holds.
• Catalog and packaging: discover_skills ignores references/*.md. A skill-catalog treatment over both skills builds with no tool-name collision. The SKILL.md context is under the 12,000-character cap. data/ is outside the package, so no package-data glob changes.
• Changelog: the fragment follows the changelog.d form, and no schema or mkdocs.yml nav change is needed.

[antonwolfy]

tasks list is not a runnability check, so "discovery is the check" gives a false pass. cmd_tasks_list (commands/tasks.py:37-47) prints only name<TAB>difficulty<TAB>tags. It never prints an error row. discover_tasks reads only task.toml and instruction.md.

• A Harbor task with just those two files (no tests/test.sh, no solution/solve.sh, no environment/) is listed normally. In a scratch copy, discover_tasks returned ['t1'].
• A task without instruction.md raises FileNotFoundError out of discover_tasks. cmd_tasks_list catches only ValueError, so the agent sees a traceback and no list, not "listed with an error".
• The missing-tests//solution/ check lives in the harness refusal at tasks run time (harnesses/tasks.py harbor_task_problem), which tasks list never calls.

An agent following this step reports "schema and paths: pass" for a task that cannot run. That is the claim-1-as-claim-2 mix-up the skill warns about.

Suggestion: say that tasks list only shows the task was discovered, and that runnability is first checked by the docker-oracle run in the next step.

[napetrov]

Fixed in 294bf2e: the Harbor step now says tasks list only shows discovery (no error row, lists a task without tests/solution/environment, tracebacks on a missing instruction.md) and that the first runnability check is the docker-oracle run in step 2. I checked the listing behaviour on scratch task dirs.

[antonwolfy]

I pushed a commit per nit, up to 4a0f048, which fix the two nits from my review:

• e2aae98: dokimi/data/skills/dokimi-design-evals/references/verifier-self-test.md:92: added --strict so an inconclusive PASS exits 1.
• 4a0f048: dokimi/tests/test_bundled_skills.py:24: LINK now accepts an optional title and angle brackets (including spaces inside them), and the dangling-link test covers a titled and an angle-bracket link.

make ci-local at 4a0f048: 25 passed, 1 failed, 6 skipped.

Not approving yet. Still open: the two risks from my review.

[napetrov]

Second bundled skill under data/: kept in place and the rules updated in 294bf2e. AGENTS.md (data/ bullet) and dokimi/data/README.md now name dokimi-design-evals as guidance for using dokimi itself, not product content, and say tests/test_bundled_skills.py checks it; the README no longer calls fastcontext "the one committed skill". If you would rather move it to the content repository, say so and I will; that needs your call.

[napetrov]

Owner decision: dokimi-design-evals stays in dokimi/data/skills/. Skills about using dokimi itself live with the CLI and are versioned with it, like fastcontext, so it is not moving to the content repository. AGENTS.md (data/ bullet) and dokimi/data/README.md already say this (294bf2e).