[napetrov]

Summary

New subsection in docs/evaluating-treatments.md, under Skill selection. It covers skills, MCP tools and function tools:

 ## Skill selection: does the agent load the right skill?
   ...
   Real harness: an external tool
+  ### How to measure tool choice
+  - Score the calls, not the judge (failed calls -> tool_usage.call_errors, on their own line)
+  - Compare against an equal alternative (tools preference twins + controls, order swapped)
+  - Repeat; do not pin (temperature=1 only; >= 3 reps; same_choice_rate / reach diff noise floor)
+  - Treat a proxy as an upper bound on effect size (-0.42 proxy vs -0.20 Claude Code)
 ## Comparing cells across runs

NEXT_STEPS item 5b. Each rule points to something that already exists: call_errors (#49), tool_preference.v1 (#53), the temperature recipe (#46), and the proxy-vs-harness numbers in tool-preference.md.

Evidence

• pytest -k "doc or link or markdown": 462 passed. This includes test_docs_links.py, which checks the #pinning-sampling and tool-preference.md links.
• Checked the claims against the code: agent_runner.py:134 says a failed call is counted only in call_errors.

Merge Danger

Door: two-way

Docs only.

Blast Radius: none

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.55%. Comparing base (ad1cbd0) to head (a91e6b5).
⚠️ Report is 12 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #158      +/-   ##
==========================================
+ Coverage   91.40%   91.55%   +0.14%     
==========================================
  Files         216      222       +6     
  Lines       26741    27404     +663     
  Branches     4377     4508     +131     
==========================================
+ Hits        24442    25089     +647     
- Misses       1635     1644       +9     
- Partials      664      671       +7     

see 34 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

LGTM. The new section is accurate against the code: the call_errors accounting, the twin and control swap in tools preference, same_choice_rate, the reach diff noise floor and the −0.42 / −0.20 figures all match. Two nits inline.

What I ran, on a Linux review host at fa9e5d2:

• make ci-local: 25 passed, 1 failed, 6 skipped; 6,715 tests passed. The one failure is pip-audit reporting CVE-2026-113722 in virtualenv 21.14.7 (a dev tool), fixed on main by #152; the PR touches no dependency.
• mkdocs build --strict: no warnings. test_docs_links.py and test_docs_code_refs.py: 50 passed. #pinning-sampling and tool-preference.md resolve.

Checked, no issues

• Claims: agent_runner.py keeps a malformed call with its error, counts it in call_errors, and leaves it out of called_tools, skill.loaded and the other counters.
• Numbers: −0.42 and −0.20 match tool-preference.md:129-130; the temperature=1 sentence matches benchmarking-and-comparison.md.
• Repo rules: the page is already in nav:, no schema or package change, so no changelog fragment is needed.

[antonwolfy]

LGTM. I pushed a commit per nit, up to a91e6b5, which fix the two nits from my review:

• a67d570: dokimi/docs/evaluating-treatments.md:595: the repeat flags are now tools preference --reps 3 and reach eval --attempts 3, and the other proxies are told to repeat the run.
• a91e6b5: dokimi/docs/evaluating-treatments.md:578: dropped "and in which turn" from the trace list and added where the turn is recorded.

make ci-local at a91e6b5: 25 passed, 1 failed (pip-audit, same as reviewed head), 6 skipped.