feat(skills): draft dokimi-design-evals skill

AGENTS.md

@@ -96,8 +96,10 @@ Registries, bundled in the package (`dokimi/dokimi/`) so they ship in the wheel:
   `baselines/`. Logs, coverage files and tarballs stay local too.
 - `dokimi/data/` holds only the fixtures this repo's own tests and
   examples read: `data/skills/fastcontext/`, `data/agent_profiles/`,
-  `data/explore_tasks/`. Question sets, product skills and tasks belong in
-  their content repository.
+  `data/explore_tasks/`. The one other skill there, `data/skills/dokimi-design-evals/`,
+  is guidance for using dokimi itself, not a product skill; its links and
+  commands are checked by `tests/test_bundled_skills.py`. Question sets,
+  product skills and tasks belong in their content repository.
 - A file the package reads at run time goes inside `dokimi/dokimi/` and into
   `[tool.setuptools.package-data]`, or the installed wheel goes without it.
 - Design rationale, status and plans go in the PR description or an issue;

dokimi/changelog.d/20261021_120000_napetrov_skill-design-evals.md

@@ -0,0 +1,9 @@
+### Added
+
+- **`dokimi-design-evals` skill (draft)** — `data/skills/dokimi-design-evals/`
+  guides an agent through designing evals for a skill or repository: sources,
+  a per-scenario template, outcome/process/cost layers, verifier self-test
+  with the existing `repo validate`, `repo tasks check` and `docker-oracle` /
+  `docker-probe` commands, manual mutation controls, coverage report and
+  `draft → reviewed → holdout` statuses. Bundled skills are now covered by
+  the docs command check and a new frontmatter/link test.

dokimi/data/README.md

@@ -19,7 +19,8 @@ write to the git-ignored `results/`, `reports/`, and the default scratch dirs
 
 | Path | What it is | Produced by / consumed by |
 | --- | --- | --- |
-| `skills/fastcontext/SKILL.md` | The one committed skill: a product-neutral "explore before you act" skill, used by the explore track and as the skill fixture in examples | `skill:`/`skill-agent:` treatment arms |
+| `skills/fastcontext/SKILL.md` | A product-neutral "explore before you act" skill, used by the explore track and as the skill fixture in examples | `skill:`/`skill-agent:` treatment arms |
+| `skills/dokimi-design-evals/` | The one committed skill about dokimi itself: how to design and self-test evals with its commands (`SKILL.md` plus `references/`). It is not product content. | `skill:` arms; `tests/test_bundled_skills.py` checks its loading, links and the commands it names |
 | `agent_profiles/concise_expert.md` | An answer-style persona (terse, code-first) | `profile:` treatment arm |
 | `agent_profiles/generic-expert.md` | Negative control: an expert framing that names no product, to pair with a product persona | `profile:` treatment arm |
 | `explore_tasks/<name>.yaml` | Localization tasks over this repository's own code, with known-good references | `cli.py explore run` |

dokimi/data/skills/dokimi-design-evals/SKILL.md

@@ -0,0 +1,108 @@
+---
+name: dokimi-design-evals
+description: Design, review or repair evals (tasks, verifiers, negative controls, coverage report) for a skill or repository before a dokimi run. Use when asked to write evals, add tasks for a subject, check whether existing evals or graders can be trusted, or state what a suite does not cover. Not for running a benchmark or comparing arms.
+---
+# Design evals that measure the user's work
+
+An eval set makes four separate claims. Keep them apart in everything you write:
+
+1. **Valid**: it parses against its schema.
+2. **Correctly graded**: the verifier fails an untouched workspace, passes the
+   reference solution, and catches the mistakes it is meant to catch.
+3. **Relevant**: its tasks are the work users actually do.
+4. **Covering**: what it does not test is listed.
+
+A passing self-check proves claim 2 at most. Never report it as evidence of 3 or 4.
+
+## Workflow
+
+### 1. Collect sources (before writing any task)
+
+In this order of weight:
+
+- real user tasks and requests (traces, support threads, examples users wrote);
+- issues and PRs, especially **fixed failures**: each one is a ready regression
+  scenario with a known base commit and fix;
+- the supported workflows the docs promise, and the constraints they state
+  (platforms, versions, forbidden actions);
+- negative cases: requests the subject should decline, or should not change.
+
+The skill or guidance text under test is a **secondary** source. Tasks written
+only from it test whether the agent repeats the skill, not whether it helps.
+Record the source of every scenario. A scenario you inferred from code or docs
+is a **hypothesis** for the user to confirm, not confirmed user intent.
+
+`python cli.py repo inspect <source>` lists a tree's instruction files, build
+and check entry points and risks without a model call; use it to find the
+workflows a repository offers.
+
+### 2. Write each scenario
+
+Fill every field of [references/scenario-template.md](references/scenario-template.md):
+initial state, request, expected outcome, allowed actions, forbidden
+consequences, verifier, how to reproduce, source, status. A worked example is
+in [references/example-scenario.md](references/example-scenario.md).
+
+### 3. Check three layers, in this priority
+
+- **Outcome**: is the end state correct? This is the primary verdict.
+- **Process**: only steps that are truly mandatory (the user forbade a tool,
+  a test must be run before a claim, a file must not be touched). Do not
+  require a fixed tool sequence when it is only one valid way to solve the task.
+- **Cost**: tokens, wall time, tool calls, retries. Report; gate only when the
+  user declared a budget up front.
+
+### 4. Choose the verifier
+
+Prefer deterministic checks: tests, file and diff assertions, exit codes,
+parsed output. Use an LLM judge only for qualities no check can capture; then
+use a written rubric, calibrate it
+(`python cli.py eval judge-validity --answers <answers.json> --sequential --strict`),
+and keep a human review of a sample. The command calibrates only dokimi's own
+answer judge (rows with `question_id`), not a task's rubric judge; check that one
+by hand.
+A judge that cannot discriminate degraded answers is not evidence.
+
+### 5. Self-test every verifier before any paid run
+
+Run the checks in [references/verifier-self-test.md](references/verifier-self-test.md):
+schema validation, oracle passes, untouched workspace fails, isolation, then
+the task-appropriate mutations (skipped step, false test claim, deleted tests,
+weakened criterion, wrong tool). Pick only the mutations that fit the task; a
+task where doing nothing is a correct answer must not be forced to fail on
+inaction. Mutation controls have no command yet: they are a manual step, and
+the reference says how.
+
+### 6. Multi-step scenarios
+
+Give each step a checkpoint verifier and keep a final-state check. Use
+Harbor's multi-step `task.toml` (`steps`) and a top-level `instruction.md`.
+`docker-oracle` does not run the checkpoint verifiers, so self-test each step's verifier
+by hand ([references/verifier-self-test.md](references/verifier-self-test.md) §5). The steps of one session are **one**
+task in every count, pass rate and sample size; never report them as
+independent tasks.
+
+### 7. Write the coverage report
+
+Use [references/coverage-report.md](references/coverage-report.md): happy path,
+branches, recovery, ambiguity, constraints, negative tasks, neighbouring tasks,
+each with the scenario ids that cover it or `none`. Put the gaps in the PR
+that adds the suite.
+
+### 8. Set the status
+
+`draft` (written, self-test not passed or not reviewed) → `reviewed` (self-test
+passed, a person confirmed relevance and the hypotheses) → `holdout` (kept out
+of development; used only to confirm a change). Never tune a skill or prompt
+against a `holdout` scenario; once it was used for tuning, it is `reviewed`
+again. dokimi does not read this status yet; record it next to the task (see
+the template).
+
+## Rules
+
+- Do not weaken a criterion to make a verifier pass. Fix the task or the oracle.
+- Do not change a grader and the subject in one comparison.
+- Keep answers out of the agent's reach: no solution, hidden test or expected
+  output inside the workspace or the image the agent sees.
+- An egress or host requirement the run cannot meet is `unsupported`, not a pass.
+- Report what you did not check, by name.

dokimi/data/skills/dokimi-design-evals/references/coverage-report.md

@@ -0,0 +1,37 @@
+# Coverage report
+
+dokimi has no command that writes this report (`python -m dokimi.coverage`
+prints a product x awareness/work task count, not scenario coverage). Write
+it by hand, in the suite's README and in the PR that adds or changes the
+suite.
+
+## Template
+
+```markdown
+## Coverage: <subject> at <SHA>, suite <name>
+
+| Area | Scenarios | Status | Notes |
+|---|---|---|---|
+| Happy path | fix-null-deref | reviewed | |
+| Branches (each supported variant, platform, option) | none | | GPU path untested: no device in CI |
+| Recovery (agent hits an error and must recover) | build-fails-first | draft | |
+| Ambiguity (request needs a question or a stated assumption) | none | | |
+| Constraints (forbidden tools, files, versions) | no-network-install | reviewed | |
+| Negative tasks (subject should decline or not apply) | unrelated-css-fix | reviewed | valid inaction |
+| Neighbouring tasks (close to the subject, should not trigger it) | none | | |
+
+Hypotheses not yet confirmed by a user: <ids, with their source>.
+Not covered, by decision: <area: why>.
+Self-test: <commands run, result>. A pass shows the verifiers are sane on
+null, oracle and the listed mutations; it does not show relevance or coverage.
+```
+
+## Rules
+
+- Every row has scenario ids or `none`. `none` with no note is a gap nobody
+  looked at; `none` with a note is a decision.
+- Count a multi-step scenario once, however many checkpoints it has.
+- List `draft` scenarios separately from `reviewed` and `holdout` ones; a
+  draft does not cover anything yet.
+- Name each hypothesis (a scenario inferred from code, docs or issues and not
+  confirmed by a user) with its source.

dokimi/data/skills/dokimi-design-evals/references/example-scenario.md

@@ -0,0 +1,85 @@
+# Worked example: a fixed failure as a Harbor task
+
+Source: an issue reported that `parse_range("5-")` raised `IndexError`
+instead of `ValueError`; the fix PR added a check and a test. That is a real
+failure with a known fix, so it becomes a regression scenario.
+
+## Scenario
+
+```yaml
+id: rangeparse-open-end
+status: draft
+source:
+  kind: fixed-failure
+  ref: <issue URL>, fixed by <PR URL>
+  confirmed_by: none
+initial_state: >
+  /app holds the library at the parent of the fix commit: rangeparse.py
+  without the check, and tests/test_rangeparse.py without the new test.
+request: >
+  parse_range("5-") crashes with IndexError. It should reject an open-ended
+  range with a clear ValueError. Fix it and keep the existing tests passing.
+expected_outcome: >
+  parse_range("5-") raises ValueError; parse_range("1-3") still returns
+  [1, 2, 3]; the existing tests in /app/tests pass unchanged.
+allowed_actions: >
+  Any edit to rangeparse.py; adding tests. Any way of detecting the open end.
+forbidden_consequences: >
+  Existing tests deleted, skipped or edited; a bare `except Exception`
+  that turns every error into ValueError.
+verifier:
+  kind: deterministic
+  checks:
+    - outcome: hidden test - "5-" and "-" raise ValueError, "1-3" -> [1, 2, 3], "x-2" still raises ValueError
+    - outcome: the existing tests/test_rangeparse.py passes
+    - process: the workspace's tests/test_rangeparse.py is byte-identical to the original
+mutations: [deleted-tests, weakened-criterion]
+valid_inaction: false
+steps: none
+reproduce: >
+  python cli.py tasks run --tasks-root <tasks-root> --tasks rangeparse-open-end --harnesses docker-oracle --out-json results/oracle.json
+```
+
+Mutations not chosen: skipped step (one-step fix), false test claim (the
+request does not ask for a report), wrong tool (no tool is forbidden).
+
+## Verifier (`test_outputs.py` in the task's tests directory, run by its `test.sh`)
+
+```python
+import hashlib, importlib, subprocess, sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, "/app")
+rp = importlib.import_module("rangeparse")
+ORIGINAL_TEST_SHA256 = "<sha256 of the original tests/test_rangeparse.py>"
+
+@pytest.mark.parametrize("bad", ["5-", "-", "x-2"])
+def test_bad_ranges_raise_value_error(bad):
+    with pytest.raises(ValueError):
+        rp.parse_range(bad)
+
+def test_closed_range_still_works():
+    assert rp.parse_range("1-3") == [1, 2, 3]
+
+def test_existing_tests_untouched_and_passing():
+    path = Path("/app/tests/test_rangeparse.py")
+    assert hashlib.sha256(path.read_bytes()).hexdigest() == ORIGINAL_TEST_SHA256
+    assert subprocess.run([sys.executable, "-m", "pytest", "-q", str(path)], cwd="/app").returncode == 0
+```
+
+## Self-test, first round (illustrative, not a recorded run)
+
+| Run | Expected | Observed |
+|---|---|---|
+| oracle (`docker-oracle`) | pass | pass |
+| untouched (manual `docker run`) | fail | fail: `"5-"` raises `IndexError` |
+| mutation: deleted tests | fail | fail: hash check |
+| mutation: weakened criterion (`except Exception: raise ValueError`) | fail | **pass** |
+
+The last row is why mutation runs are worth doing: a blanket `except` raises
+`ValueError` for every bad input, so the checks above pass it. The verifier
+needs a case that tells a targeted fix from a blanket one, for example
+`parse_range(None)` must still raise `TypeError`. Add it, rerun the oracle and
+the mutation, then move the scenario to `reviewed` after a person confirms it.

dokimi/data/skills/dokimi-design-evals/references/scenario-template.md

@@ -0,0 +1,58 @@
+# Scenario template
+
+One scenario per task directory. Fill every field; write `none` rather than
+leaving one out, so a reviewer can tell "considered, empty" from "forgotten".
+
+```yaml
+id: <lower-kebab-case, equals the task directory name>
+status: draft            # draft | reviewed | holdout
+source:                  # where this scenario comes from; hypotheses say so
+  kind: issue            # user-task | issue | pr | fixed-failure | workflow | constraint | negative | inferred
+  ref: <URL, trace id, or doc path + section>
+  confirmed_by: none     # a person who confirmed it is real user work; `none` = hypothesis
+initial_state: >
+  The repository at <full SHA>, plus any setup patch (injected bug, missing
+  file). What exists, what is broken, what the agent can and cannot see.
+request: >
+  The exact prompt the agent gets, worded as a user would word it. No hint of
+  the grader, the expected diff or the skill under test.
+expected_outcome: >
+  The end state that counts as success, stated so a check can decide it.
+allowed_actions: >
+  What the agent may do. Name the alternatives when more than one valid way
+  exists, so none of them is graded as a failure.
+forbidden_consequences: >
+  End states that fail the task however the outcome looks: tests deleted or
+  skipped, a criterion weakened, files outside scope changed, a claim of
+  "tests pass" with no run, a forbidden tool used.
+verifier:
+  kind: deterministic    # deterministic | rubric (calibrated judge + human review)
+  checks:                # one line per check, each tagged with its layer
+    - outcome: <what the test asserts>
+    - process: <only if truly mandatory; else omit>
+    - cost: <reported metric; gated only with a declared budget>
+mutations:               # verifier self-test controls that fit this task (verifier-self-test.md)
+  - skipped-step
+valid_inaction: false    # true when declining or doing nothing is a correct answer
+steps: none              # multi-step: list checkpoints, each with its own verifier
+reproduce: >
+  The commands that build the image and run oracle and null, with the SHA.
+```
+
+## Where the fields go in each task format
+
+| Field | Harbor task (`tasks run`) | Repository task (`repo run`) |
+|---|---|---|
+| request | `instruction.md` | `task.json` `prompt` |
+| initial state | `environment/Dockerfile` + seed files | `task.json` `base`, `setup_patches` |
+| expected outcome, forbidden consequences | `tests/test.sh` + `tests/test_*.py` | the repository's `grade` entry point, `task.json` `gate` |
+| reference solution | `solution/solve.sh` | the `oracle` entry point |
+| multi-step checkpoints | `task.toml` `steps` (Harbor runs them; the per-step self-test is manual) | not supported |
+| status, source, allowed actions, mutations | not read by dokimi | extra `task.json` keys are passed to the grader unread |
+
+`status`, `source` and `mutations` have no field dokimi reads today. Keep them
+in the task's `task.toml` `[metadata]` table, or as extra `task.json` keys,
+and in the suite's README, so the record travels with the task.
+
+Format references, in the dokimi project: `docs/contributing-terminal-bench-task.md`
+(Harbor) and `docs/repo-evaluation-contract.md` §3 (`task.json`).