From fa9e5d2cc08b95b10c6e6c42903757e0a3b15632 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Wed, 21 Oct 2026 21:48:55 -0700 Subject: [PATCH 1/3] docs(treatments): how to measure tool choice Score the calls rather than the judge, count failed calls on their own, compare against an equal alternative with the order swapped, repeat at temperature=1 instead of pinning, and read a one-call proxy as an upper bound (about 2x the harness effect in the one case measured). Signed-off-by: Nikolay Petrov --- dokimi/docs/evaluating-treatments.md | 33 ++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/dokimi/docs/evaluating-treatments.md b/dokimi/docs/evaluating-treatments.md index cfe79297..2c2670c8 100644 --- a/dokimi/docs/evaluating-treatments.md +++ b/dokimi/docs/evaluating-treatments.md @@ -571,6 +571,39 @@ Two things decide whether such a number means anything: - **Repeats.** Selection varies run to run; use `--attempts 3` or more and read the noise floor `reach diff` reports before calling a description change an improvement. +### How to measure tool choice + +The same rules hold for skills, MCP tools and plain function tools. + +- **Score the calls, not the judge.** Which tool the agent called, and in which + turn, is in the trace: the *Skill selection* table, `tool_preference.v1` + `results`, a transcript's `tool_usage`. A judge score mixes two things: whether + the tool was picked and whether its content helped. A better answer can come + from a tool the agent should not have needed. A call with arguments that do not + fit the tool's schema is not run. It is kept with its `error` and counted in + `tool_usage.call_errors`, and the other counters leave it out. Report it on its + own line: the agent picked the tool and could not use it. +- **Compare against an equal alternative.** A choice tells you something only + when another offered tool would have done the job just as well. Otherwise it + measures whether the model can see that the task needs a tool. `tools + preference` builds such pairs (twins) and controls that need no tool, and it + swaps which twin is listed first. With both twins neutral, the first-listed one + is still not chosen exactly half the time, so a comparison that never swaps the + order mixes the position effect into the effect you are measuring. +- **Repeat; do not pin.** Current reasoning models accept only `temperature=1` + ([Pinning sampling](benchmarking-and-comparison.md#pinning-sampling)), so the + choice varies from run to run. Run 3 or more repeats (`--reps`, `--attempts`). + Read `same_choice_rate`, or the noise floor `reach diff` reports, before you read + an effect. A change smaller than the variation between repeats of one setting + is not an effect. +- **Treat a proxy as an upper bound on effect size.** A one-call proxy (the + `skill-catalog:` arm, `tools preference`, `novelty routing`) shows every + description in full and gives the model no reason to explore first. In one + measured case the effect kept its direction on Claude Code but was about half + as large (−0.42 on the proxy, −0.20 on the harness; + [Tool preference](tool-preference.md)). Use the proxy to find and rank changes, + then confirm the ones you ship on the real harness with a pinned listing budget. + ## Comparing cells across runs One treatment measured over several models is many artifacts, one per cell. Two tools read From a67d570c5b439133f31b4a144a0876b131952753 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Thu, 22 Oct 2026 16:54:36 +0000 Subject: [PATCH 2/3] docs(treatments): tie repeat flags to the commands that take them Signed-off-by: Anton Volkov --- dokimi/docs/evaluating-treatments.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dokimi/docs/evaluating-treatments.md b/dokimi/docs/evaluating-treatments.md index 2c2670c8..d473ded7 100644 --- a/dokimi/docs/evaluating-treatments.md +++ b/dokimi/docs/evaluating-treatments.md @@ -592,8 +592,8 @@ The same rules hold for skills, MCP tools and plain function tools. order mixes the position effect into the effect you are measuring. - **Repeat; do not pin.** Current reasoning models accept only `temperature=1` ([Pinning sampling](benchmarking-and-comparison.md#pinning-sampling)), so the - choice varies from run to run. Run 3 or more repeats (`--reps`, `--attempts`). - Read `same_choice_rate`, or the noise floor `reach diff` reports, before you read + choice varies from run to run. Run 3 or more repeats (`tools preference --reps 3`, + `reach eval --attempts 3`); for the other proxies, repeat the run. Read `same_choice_rate`, or the noise floor `reach diff` reports, before you read an effect. A change smaller than the variation between repeats of one setting is not an effect. - **Treat a proxy as an upper bound on effect size.** A one-call proxy (the From a91e6b5f789a58ad2277b49d51f97e81355bcfe2 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Thu, 22 Oct 2026 16:54:58 +0000 Subject: [PATCH 3/3] docs(treatments): say where the turn of a tool call is recorded Signed-off-by: Anton Volkov --- dokimi/docs/evaluating-treatments.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/dokimi/docs/evaluating-treatments.md b/dokimi/docs/evaluating-treatments.md index d473ded7..a7c32041 100644 --- a/dokimi/docs/evaluating-treatments.md +++ b/dokimi/docs/evaluating-treatments.md @@ -575,9 +575,10 @@ Two things decide whether such a number means anything: The same rules hold for skills, MCP tools and plain function tools. -- **Score the calls, not the judge.** Which tool the agent called, and in which - turn, is in the trace: the *Skill selection* table, `tool_preference.v1` - `results`, a transcript's `tool_usage`. A judge score mixes two things: whether +- **Score the calls, not the judge.** Which tool the agent called is in the + trace: the *Skill selection* table, `tool_preference.v1` `results`, a + transcript's `tool_usage` (the turn is in `first_call_iteration_by_name`, and + for skill tools in `skill.first_call_iteration`). A judge score mixes two things: whether the tool was picked and whether its content helped. A better answer can come from a tool the agent should not have needed. A call with arguments that do not fit the tool's schema is not run. It is kept with its `error` and counted in