[napetrov]

Stacked on #211 (needs its precision in skill_selection_summary); the diff against main includes #211 until that merges. My commit is the top one.

Summary

The skill-catalog: arm over-triggers (recall 0.77, false-trigger 0.46 on 30 skill-reach queries). This PR adds a way to measure the real harness instead, plus an audit that checks what dokimi sends carries none of its own artifacts.

 dokimi/harnesses/
+├── trigger_probe.py   # fresh `claude -p` per query: new git repo, catalog in .claude/skills,
+│                      # scratch HOME/CLAUDE_CONFIG_DIR, query verbatim, no notice, bundled skills on
+└── payload_audit.py   # audit_payload / new_leaks: arm names, label keys, qids, grading words, paths
 dokimi/commands/
+└── catalog_probe.py   # `catalog trigger-probe`, scored by summarize_skill_selection, one arm per trial

Why the emulated arm over-triggers. I captured both request bodies through a logging proxy, then ablated them one at a time (haiku-4.5, 30 queries x 2, single turn):

variant	recall	precision	false-trigger
skill-catalog as sent	0.84	0.70	0.56
no sub-file enum	0.77	0.67	0.44
neutral tool wording	0.77	0.63	0.62
CC listing truncation	0.73	0.57	0.75
+ CC system prompt and tools	0.50	0.69	0.19
+ thinking	0.86	0.79	0.25
full CC replay	0.23	0.56	0.12

The main driver is that the arm offers only skill tools and no system prompt. The enum and the wording are within noise.

Evidence

• catalog trigger-probe, stock, CC 2.1.304, haiku-4.5, 30 queries x 3 trials: recall 0.23 / 0.27 / 0.27, false-trigger 0.00, misroute 0.09, cost $2.30. This is far below the emulated arm and in line with the full-replay ablation.
• The same run with --listing-budget-fraction 0.03 (skill-reach's setting), 2 trials: recall 0.45 / 0.36, false-trigger 0.00, cost $2.59. skill-reach on the same set gives recall 0.41–0.50 and false-trigger 0.19–0.31. Recall matches. False-trigger is lower, but there are only 8 negatives, so one false trigger moves the rate by 0.125.
• Leak audit: on the stock CC payload, the only hits are CC's own "golden" and "judge" text. The skill-catalog request is clean. The first probe draft leaked dokimi-run- through the working directory that CC shows the model. Fixed with a neutral project- prefix and a neutral "Initial commit".
• Tests: tests/test_trigger_probe.py and tests/test_payload_audit.py, 12 tests, FakeRunner, no network.
• make ci-local: all pass except pip-audit. pip-audit fails on filelock/setuptools CVEs in the lock that this PR does not change.

Merge Danger

Door: two-way

The command and modules are new. The only edit to existing code is two lines in catalog.py.

Blast Radius: none

Existing arms and outputs are unchanged.

🤖 Generated with Claude Code

[github-actions]

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi catalog formats`
    + option `--help/-h`
    + option `--json`
    + option `--root`
+ added command `dokimi catalog trigger-probe`
    + option `--catalog`
    + option `--claude`
    + option `--concurrency`
    + option `--help/-h`
    + option `--limit`
    + option `--listing-budget-fraction`
    + option `--max-turns`
    + option `--model`
    + option `--out`
    + option `--sealed`
    + option `--target`
    + option `--timeout`
    + option `--trials`
    + option `--trigger-set`
+ added command `dokimi tasks inspect`
    + argument `path`
    + option `--help/-h`
    + option `--json`
~ `dokimi catalog trigger-set` parameters changed:
    + added option `--trigger-set`
    ~ changed option `--questions`: required: True -> False

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 88.75639% with 132 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.88%. Comparing base (09be35b) to head (5de5199).

Files with missing lines	Patch %	Lines
dokimi/dokimi/commands/catalog_probe.py	48.00%	26 Missing ⚠️
dokimi/dokimi/external/trigger_set.py	81.91%	8 Missing and 9 partials ⚠️
dokimi/dokimi/harnesses/trigger_probe.py	88.79%	7 Missing and 6 partials ⚠️
dokimi/dokimi/task_formats/detect.py	87.61%	8 Missing and 5 partials ⚠️
dokimi/dokimi/task_formats/source.py	82.43%	8 Missing and 5 partials ⚠️
dokimi/dokimi/task_formats/datasets.py	94.54%	5 Missing and 7 partials ⚠️
dokimi/dokimi/skills/formats.py	92.90%	6 Missing and 4 partials ⚠️
dokimi/dokimi/task_formats/dirs.py	89.36%	4 Missing and 6 partials ⚠️
dokimi/dokimi/harnesses/payload_audit.py	89.83%	3 Missing and 3 partials ⚠️
dokimi/dokimi/task_formats/configs.py	90.32%	3 Missing and 3 partials ⚠️
... and 3 more
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #217      +/-   ##
==========================================
- Coverage   92.01%   91.88%   -0.14%     
==========================================
  Files         223      234      +11     
  Lines       27768    28928    +1160     
  Branches     4563     4753     +190     
==========================================
+ Hits        25552    26580    +1028     
- Misses       1570     1651      +81     
- Partials      646      697      +51     

Files with missing lines	Coverage Δ
dokimi/dokimi/__init__.py	100.00% <ø> (ø)
dokimi/dokimi/eval/arm_summary.py	97.41% <ø> (ø)
dokimi/dokimi/report/arms_report.py	96.95% <100.00%> (ø)
dokimi/dokimi/task_formats/__init__.py	100.00% <100.00%> (ø)
dokimi/dokimi/task_formats/model.py	100.00% <100.00%> (ø)
dokimi/dokimi/commands/tasks.py	92.40% <96.77%> (+1.06%)	⬆️
dokimi/dokimi/commands/arms.py	88.85% <83.33%> (-0.19%)	⬇️
dokimi/dokimi/commands/catalog.py	97.10% <95.52%> (-1.20%)	⬇️
dokimi/dokimi/harnesses/payload_audit.py	89.83% <89.83%> (ø)
dokimi/dokimi/task_formats/configs.py	90.32% <90.32%> (ø)
... and 8 more

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.