User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121

Goal

Make Dokimi a closed loop for users: understand the task → build valid evals → measure baseline → change the artifact → verify the effect → keep evidence → catch regressions.

The first user-facing kit is three skills (dokimi-improve-skill, dokimi-improve-repository, dokimi-design-evals) plus one GitHub Action with three workflow recipes (PR check, record, revalidate). These names are proposals. Nothing below exists yet unless it says so.

Architecture: one engine, thin wrappers

CLI (deterministic)        dokimi plan → validate → run → compare → gate → record
Skills (judgment)          design-evals / improve-skill / improve-repository: call the CLI, read its output
Action (policy + safety)   pr-check / record / revalidate: call the same CLI with a policy file

Rule: skills and workflow YAML never compute a verdict. They call dokimi gate --policy … and show its JSON output.

Build on existing code instead of adding a second engine:

• dokimi/gate/ (hard_gate, soft_gate, regression, trust_gate, novelty_gate, critical_bands)
• dokimi/runner/compare.py
• the comparability guards in the arms plugin-delta
• commands/baseline.py
• scripts/check_smoke_runs.py
• contracts.yml (Check user outcomes in tests; pin --questions-root to its golden set #74)
• repo inspect (repo inspect: static look at what a repository offers an agent (repo_environment.v1) #66)
• the null/oracle checks in section 4.3 of the repo contract

P0 work items (each can merge as its own PR)

1. plan.v1 / result.v1 schemas

• plan.v1 records:
  • subject SHAs and fingerprints
  • suite hash
  • model and harness IDs
  • grader digest
  • one declared varied_variable (a tuple is allowed for 2×2 designs)
  • the planned cell list (arm × model × task × rep)
  • a budget estimate
  • the reasons each suite was selected
• result.v1 lists completed cells. Each cell gets one status: pass | fail | inconclusive | invalid | infra_error | not_run.
• The engine computes completeness as planned minus completed. Today a run that completes 2 of 6 planned cells can still pass the specialized smoke check. With this change, check_smoke_runs.py only declares its planned list.
• Existing arms, run and repo run accept --plan and emit result.v1.

2. dokimi compare

• Move the arms comparability guards into runner/compare.py. These cases must return invalid, not "No notable changes":

  • suite hash mismatch
  • same IDs with different text
  • disjoint IDs
  • model mismatch
  • missing items

  The legacy baseline.py helper currently passes all five, and it also returns success on a large regression.

• Per-task paired deltas, with uncertainty that accounts for task-level clustering. Repetitions of one task are not independent samples.

• Report these metrics separately: user-task success, rule violations, artifact quality, routing, wall time, agent and judge tokens, tool calls, retries, cost.

• baseline.py becomes a reporter built on compare, or is deprecated.

3. dokimi gate --policy policy.yaml

• The policy is YAML that composes the existing gate/* modules. Example rules: require_complete, max_quality_drop, min_tasks.
• Efficiency mode: quality is a constraint declared up front, and savings are judged only within it. A cheap incomplete run is inconclusive, never pass.
• Lack of statistical significance is not evidence of no regression.
• Exit codes, to be checked against current CLI conventions: 0 pass, 1 fail, 2 inconclusive, 3 invalid, 4 infra_error.
• The same command runs locally and in CI.

4. dokimi validate

• Checks schemas, paths, tool availability, trust boundaries and answer leakage.

• Covers the existing null/oracle checks plus mutation controls that each task opts into:

  • skipped step
  • false claim about tests
  • deleted tests
  • weakened criterion
  • wrong tool

  Controls must fit the task. Valid inaction is not turned into a failure just to fit a template.

• Runs before any paid run.

5. GitHub Action (composite action in an in-repo action/ dir, plus three workflow recipes)

pr-check

• Impact analysis maps changed files to subjects and suites. Inputs: SKILL.md, references, scripts, shared helpers, manifest, harness and environment, grader.
• A frontmatter change triggers a catalog-wide routing check.
• An unknown dependency triggers a conservative run or an explicit inconclusive, never a silent skip.
• Stages: structure → validate → small runtime smoke → full benchmark (only by label or policy). A smoke pass does not prove quality is unchanged.
• Base and head run on the base suite. New or changed tests from the PR run as a separate block. A deleted or weakened grader never becomes the new yardstick.

record (on push to main)

• Stores immutable result.v1, a Markdown summary, traces, hashes, environment and model parameters.
• Raw traces go to artifact storage with retention and access controls, not to main.
• A durable JSONL index keeps history after CI artifacts expire.
• The approved baseline advances only when the run is complete and the gate passes.

revalidate (workflow_dispatch, optional cron)

• Same skill SHA, with a different model, harness or dependency. This is reported separately from the effect of a skill edit.
• The no-skill arm always reruns, to show whether the skill is still needed and whether it now gets in the way.

Security

• Evaluator job: pull_request, no secrets, no write token. PR code, skill scripts, Dockerfiles and instructions are all treated as untrusted. "No LLM" does not mean "no code execution".
• Model-calling job: same-repo PRs, or maintainer approval tied to a specific SHA through environment protection. A new push invalidates the approval. Use a scoped proxy or short-lived credentials where possible.
• Publisher job: workflow_run with comment and write permission. It reads only schema-validated result.v1 and never executes anything from artifacts.
• Never run fork head code in a pull_request_target job that has secrets.
• Grader and policy are always taken from the trusted base ref. Changes to them need separate review.
• The agent never gets a repo write token, production credentials, the host Docker socket, or hidden-test results.

6. Skills (in dokimi/data/skills/)

dokimi-design-evals

• Each scenario has: initial state, request, expected outcome, allowed actions, forbidden consequences, verifier, and how to reproduce.
• Sources: real user tasks, issues and PRs, fixed failures, supported workflows, constraints, negative cases. Skill text is a secondary source.
• Three layers are checked: outcome correctness, process requirements that are truly mandatory, and cost. Don't require a fixed tool sequence when it is only one valid way to solve the task.
• Prefer deterministic checks. Subjective quality uses a calibrated rubric with human review.
• Calls validate. Writes a coverage report: happy path, branches, recovery, ambiguity, constraints, negative tasks, neighboring tasks.
• Suite statuses: draft → reviewed → holdout.
• Multi-step scenarios get checkpoint verifiers plus a final-state check, using Harbor's multi-step format. Steps of one session don't count as independent tasks.

dokimi-improve-skill

• Read the whole bundle and reuse existing suites and results; don't regenerate.
• Declare the expected effect axes.
• Check loading and selection first, then quality of use.
• Arms: no-skill, approved, candidate. Only the declared variable changes.
• Diagnose the failure type: not triggered, wrong skill triggered, bad instruction, skipped step, broken script, incomplete task, bad grader.
• Make a small patch without moving the success criteria. Verify on the dev suite, then on the holdout. Log rejected attempts too.
• Output: accept, reject or inconclusive, with evidence and limits.
• Novelty is not utility. A "redundant" finding is a candidate for a removal experiment, not permission to delete.

dokimi-improve-repository

• Setup is separate from improvement.
• Setup:
  • Uses repo inspect, scan and reach plus existing repo entry points.
  • Produces a minimal manifest and image, a few real tasks, and independent graders.
  • Inspect mode works without Docker or model keys and lists what it did not check.
• Keeps three evidence types apart: L0 fact checks, reach, task outcomes.
• Prefers small fixes (a stable make test-fast, a wrapper, a small example) over a longer AGENTS.md.
• When code and guidance change together, the plan uses a 2×2 design so the guidance text doesn't get credit for the code change.

Acceptance

An external repo goes from install to a valid PR result without help from a Dokimi maintainer. Success is measured by that, not by the number of new CLI commands.

Pilots:

1. a knowledge-heavy skill
2. a workflow or efficiency skill
3. one hard repository workflow (oneDAL build and review)

Measure:

• time to the first valid comparison
• manual setup fixes
• whether a planted regression is caught
• false blocks
• cost
• whether a result can be replayed from the saved plan.v1

Numeric targets are set after the baseline is measured.

P1 (mostly modes of the existing skills, not new skills)

• failure → regression eval (design-evals mode)
• benchmark failure triage, with a trajectory view in the style of Inspect
• guidance drift gate (repo lint, pre-commit)
• controlled shortening (improve-skill efficiency mode)
• routing and catalog interference (skill-catalog, discover arm feat(harnesses): skill discovery arm (skill installed, no treatment notice) #110)
• model and harness revalidation matrix

P2

Release evidence and scorecards, dashboards, multi-repo tracking, and a GitHub App only if it proves necessary. No marketplace, big UI or autonomous self-improving bot.

Matrix size (sizing only, not a measurement)

• 2 arms × 1 model × 3 tasks × 1 rep = 6 trials
• 3 arms × 2 models × 8 tasks × 3 reps = 144 trials

Judge calls, setup and retries come on top.

Prior art to reuse, not copy

• Anthropic skill-creator: clean-context A/B, triggering tuned separately, capability uplift vs encoded preference.
• Promptfoo: small focused skills, deterministic assertions, known-good/known-bad controls, PR Action.
• Harbor: task, reference solution, verifier and oracle format, multi-step tasks.
• Inspect: per-trajectory log viewer, partial logs, retries.

Open decisions

1. Long-term results storage: results branch, GH releases, or an external bucket?
2. Fork PRs: model runs only after maintainer approval of a pinned SHA, or no model runs for forks at all?
3. Action location: in-repo action/ first (proposed), or a separate intel/dokimi-action?