Skill catalog integration (intel/skills): per-skill evals in catalog CI, results, promotion to validated #138

Goal

Let a skill catalog — first intel/skills, then the repos it imports from (intel/intel-performance-skills, intel/gpu-ai-skills) — run dokimi in its own CI and publish per-skill evidence, so a catalog entry can move from published to validated on measured lift, not on review alone.

This is the catalog consumer of #121. #121 owns the generic engine (plan.v1/result.v1, compare, gate --policy, the Action, the security model). This issue lists only what a catalog needs on top of that, plus the defects found while mapping it. Where an item is already covered elsewhere, it links there instead of restating it.

Target shape

intel/skills PR touches skills/<x>/**
 ├─ dokimi-static   (required, no LLM, GitHub-hosted)   lint, listing budget, task validate
 ├─ label "eval" → dokimi-live (Haiku, ~3 reps, changed skills only, budget cap)
 │     arms: no-skill · skill · skill-discover        → sticky PR comment
 └─ nightly/weekly → full matrix on a self-hosted Intel runner (XPU / perf tasks)

storage
  raw trajectories      → Actions artifacts (retention-limited)
  per-skill summaries   → results branch or separate repo (open decision #121 Q1)
  catalog status        → bot PR sets `validated` + evidence in skills.yaml
  display               → Pages badge per skill (lift, discover %, date)

What already exists

Gaps to close

P0 — blocks a first catalog pilot

1. Installable release. release.yml publishes to PyPI only when vars.PYPI_PUBLISH == 'true'; the 0.1.0 release is on hold (build: versioned releases from 0.1.0 — changelog fragments, working wheel, release workflow #39 follow-up). Until then catalogs pin git+https://…@<sha> with requirements.lock.txt constraints, as ci: reusable repo-eval workflow (validate, scan, tasks check) with dokimi pinned #113 does. Decide which one catalogs are told to use, and document it.
2. Catalog entry point. No single command today maps a diff to the skills and tasks to run. Proposed: dokimi catalog eval <root> --changed-from <base> --tier static|live|full --out <dir>:
   • changed files → skill (skills/<x>/**, skills.yaml entry) → tasks via [metadata] skill;
   • frontmatter / description change → also a catalog-wide routing check (as in User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 §5);
   • a skill with no tasks → explicit no_tasks row, not a silent skip;
   • emits plan.v1 and result.v1 once User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 P0.1 lands; a fixed exit code in every case, so the check can be required.
3. evaluated_here: false. intel/skills marks imported skills (linux-perf, performance-patterns, phoronix, the gpu-ai set) as evaluated upstream, and its validator rejects tasks on them. dokimi does not read the field (git grep evaluated_here on main: 0 hits). The catalog reader must skip them, or report "evaluated in ", so a run never credits evidence to the wrong repo.
4. Reusable workflow for skill evals. ci: reusable repo-eval workflow (validate, scan, tasks check) with dokimi pinned #113 covers repo-eval (no LLM). Catalogs need a sibling skill-eval.yml (workflow_call) with inputs: catalog-root, tier, skills (default: changed), models, reps, budget-usd, runs-on. Model jobs follow the User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 security split (no secrets on fork PRs, SHA-bound approval).
5. Scratch location. In the feat(harnesses): skill discovery arm (skill installed, no treatment notice) #110 smoke run, a /tmp/dokimi-env-* dir (container_agent.py:153, tempfile.mkdtemp) vanished mid-run → FileNotFoundError, one cell errored. The name is unique, so the cause is outside dokimi (tmp cleaner or another process on a shared host). Shared self-hosted runners will hit this. Honour --work-dir / $RUNNER_TEMP for all scratch, and report a vanished scratch dir as infra_error, not as an agent failure.

P1 — results a catalog can gate and publish on

6. Per-skill result record. Either a skill_eval_result.v1, or a defined projection of result.v1: one row per skill × model × harness. Keys: skill_sha (the skill dir tree hash, not the repo commit), task_set_hash, dokimi_version, harness + CLI version, model_id. Values: pass rate per arm with n, lift, discover rate, Skill-call rate, errored count, cost. subject_scorecard.v2 and matrix_rollup.v1 do not carry skill_sha or task_set_hash today.
7. Small-n statistics in the verdict. Live tier runs ~3 reps per arm. Report a task-clustered interval for lift (bootstrap or exact sign-flip, as repo compare does in feat(repo): repo compare, treatment arms against a baseline arm paired by task #42), and return inconclusive (J11/J12: distinct inconclusive verdict and blocking policy (AC-J11-002, AC-J12-002) #99) when it crosses zero. 3/3 vs 0/3 on one task is not a validated lift.
8. Promotion policy as a gate policy. Express published → validated as a gate --policy file (User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 P0.3), e.g. min_tasks, lift_lower_bound > 0, discover_rate > 0, max_errored. Open question below on whether the discover arm is required.
9. Baseline per skill. Compare a PR head against the stored record for the base skill_sha on the same task_set_hash. model-compare / baseline have no question-set/task-set hash guard today (open item from the Check user outcomes in tests; pin --questions-root to its golden set #74 audit; User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 P0.2 covers the guard).
10. Budget before spend. --max-usd / --max-cells hard caps and a cost estimate printed before the first paid call. This is required before the live tier can be triggered by a label.

P2 — publishing and reach

11. Output formats. report --format pr-comment (short, sticky, one block per skill) and a badge/site JSON that the catalog's Pages build can read.
12. Storage writer. dokimi writes files only. The workflow commits them to the results branch/repo, so dokimi never holds a write token (matches User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 "publisher job").
13. Static tier per skill. One command, machine-readable output: description length vs the 1536-char cap, the skill's share of the listing budget (8000 chars at 200k context), and trigger-quality lint. These pieces exist in feat(harnesses): skill discovery arm (skill installed, no treatment notice) #110 metrics and repo lint, but not as one per-skill check.
14. Hardware requirements for Harbor tasks. feat(repo): task host requirements, unsupported instead of a false pass (#41 D3) #112 adds host: requirements for repo tasks only. Catalog Harbor tasks need the same for xpu, perf_events and avx512, read from task.toml [metadata]/[sandbox]: the result is unsupported or routed to a labelled runner, never a fail. The perf tasks in evaluation: Harbor tasks and two-arm evals for performance-patterns and linux-perf intel-performance-skills#11 use recorded artifacts for now, so they don't block on this.
15. Batch backend (cost). A provider-level batch mode (Bedrock batch ≈ −50%) for the nightly full matrix. Prototype exists outside dokimi; blocked on IAM/S3 permissions.
16. Shared summary format with skill_search. isl-org/skill_search#41 plans to import dokimi summaries joined by behavior_digest. The catalog record (item 6) and that import should be one format, not two.

Pilot

1. intel/skills onetbb Harbor tasks (arm data already exists: no-skill 0/3, skill 3/3, discover 0/3 on Haiku).
2. evaluation: Harbor tasks and two-arm evals for performance-patterns and linux-perf intel-performance-skills#11 tasks: they replace its local run_arms.sh / report.py with the reusable workflow. This is the first upstream-evaluated catalog, and it exercises item 3.

Acceptance: a skill PR in the pilot catalog gets a static result on every PR and a live result on label, with no dokimi maintainer involvement. A planted regression (skill text that removes the key step) is reported as fail or inconclusive, never pass.

Open decisions

1. Does validated require discover-arm > 0, or only skill-arm lift? Today onetbb is 3/3 with notice and 0/3 without.
2. Results location: shared with User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 Q1, or a catalog-owned repo (e.g. intel/skills-evals)?
3. Who owns model credentials and the self-hosted runner for catalog CI?
4. Is catalog eval a new command, or a mode of the User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 plan with a catalog impact analyzer? This issue proposes the latter if User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121's plan lands first.