[napetrov]

Part of #4.

Summary

novelty routing --model gets two changes: a --repeats N loop for a noise floor, and the model's raw reply kept whenever it declines to pick.

 model_routing(candidates, queries, client, gold, seed)
+  model_routing(..., repeats=1)           # --repeats N, needs --model
   for each query
     shuffle cards (seeded, one order per query)
+    for repeat in range(repeats)
-      client.text(prompt)
+      client.text(prompt, draw=repeat)    # a new sample, not a cache replay
       abstain or unparseable pick -> row.raw_reply (first 2000 chars)
+  _noise(groups)                          # when repeats > 1
+    per query: agreement across repeats, modal pick
+    per repeat: selection rate; half_range = half the spread

Repeats keep one card order per query, so the spread between repeats is sampling noise and not a change of layout. LLMClient.text takes an optional draw: without it, under DOKIMI_RESPONSE_CACHE the repeats would replay one reply and report a noise floor of zero. Draw 0 is the default and adds nothing to the cache key, so existing entries still hit.

Evidence

• Before: the new tests fail on the intel/main sources (12 of 60 in tests/test_novelty_routing.py): TypeError for repeats, KeyError on raw_reply and noise, and the --repeats CLI is unknown. With the old client.py alone, test_with_the_response_cache_on_each_repeat_is_asked_again fails (n_failed 3, not 0), so the draw forwarding is load-bearing.
• After: tests/test_novelty_routing.py 60 passed; full pytest -n auto 7080 passed, 22 skipped. Changed lines are at 100% coverage (diff-cover against intel/main).

model    selection 0.0  (None among answered; 0 abstained, 0 failed)  [m]
noise    2 repeats per query: mean agreement 1.0, 0 queries picked differently across repeats; selection by repeat [0.0, 0.0], half range 0.0 (a crude sampling floor)

Checks: ruff check passed, mypy dokimi clean (222 files), schema_check and config_check ok.

Merge Danger

Door: two-way. Every change is additive: new flag, new keys, and an optional keyword on LLMClient.text. Removing the flag restores the old output shape.

Blast Radius: routing command only.

• Default runs (no --repeats) make the same calls with the same cache keys. per_query rows gain repeat: 0, and abstention rows gain raw_reply; routing.json has no versioned schema, so readers of the old keys are unaffected.
• --repeats N multiplies model calls by N, so it counts against --max-usd. It is refused without --model.
• LLMClient.text(..., draw=) is a new optional keyword; injected transports do not receive it.
• No dependency on other open PRs.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — docs/skill-novelty.md routing section; changelog fragment added
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml) — the changelog fragment is exempt

🤖 Generated with Claude Code

[antonwolfy]

Review at 6a6631c. The design holds: one card order per query, draw forwarded so repeats are not cache replays, and draw 0 keeps existing cache keys. Two things are worth fixing before merge, and one nit.

What I ran, on a Linux review host at 6a6631c:

• make ci-local did not run: make dev stops at the tools install on this base, which #179 fixed on main. GitHub's CI passes at this head (tests on 3.11-3.14, static checks, mypy, the strict docs build).
• tests/test_novelty_routing.py: 60 passed.
• Reproductions for the code findings, with the LLM transport mocked.
• A mutation run: 24 mutants, 19 killed, 5 survive the full suite (listed under Test gap).

Test gap. Five mutants pass all 60 tests: dropping the "(abstained)" label in _noise (routing.py:350), changing the len(known) > 1 guard to > 0 (routing.py:376, a single usable repeat would then report half_range 0.0 instead of None), keeping the tail instead of the head of raw_reply (routing.py:294), dividing agreement by the whole group instead of the non-errored rows (routing.py:355), and the abstention case in _noise generally. Add a _noise test with one abstention, one errored repeat and one usable repeat, and make the raw_reply test assert reply[:2000], not only the length.

Checked, no issues

• Draw forwarding: dropping draw=repeat or the pass-through in client.py fails the tests.
• Card order: identical across repeats of a query.
• Spend cap: SpendCapReached is not swallowed by the per-repeat except Exception.
• Docs and changelog: match the flag and keys in the code.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.99%. Comparing base (9a93635) to head (56897ba).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #175   +/-   ##
=======================================
  Coverage   91.98%   91.99%           
=======================================
  Files         223      223           
  Lines       27743    27778   +35     
  Branches     4560     4568    +8     
=======================================
+ Hits        25520    25553   +33     
- Misses       1574     1575    +1     
- Partials      649      650    +1     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/novelty.py	95.92% <100.00%> (+0.08%)	⬆️
dokimi/dokimi/novelty/client.py	100.00% <100.00%> (ø)
dokimi/dokimi/novelty/routing.py	99.00% <100.00%> (+0.14%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

I pushed a commit per nit, up to 85765e2, which fix the nit from my review:

• 85765e2: dokimi/dokimi/novelty/routing.py:313: added an n_calls key (queries x repeats) and documented that n_abstained and n_failed count calls; the CLI abstain line now reads "4 of 4 calls abstained or unparseable" when repeats > 1.

make ci-local at 85765e2: 19 passed, 9 failed (all host-caused), 7 skipped.

Not approving yet. Still open: the two risks from my review.

[napetrov]

Test-gap items from the review, in 2b9a28f: abstained label in _noise (test_noise_counts_an_abstention_as_a_pick_and_an_error_as_no_answer, test_an_abstention_can_be_the_modal_pick); len(known) > 1 guard (test_a_single_usable_repeat_has_no_half_range); raw_reply head vs tail (test_a_long_declined_reply_keeps_its_head_not_its_tail, asserts reply[:2000]); agreement denominator (the abstention/error test, 1/2 not 1/3).

[antonwolfy]

The thread on half_range is fixed, and the earlier test gaps are closed. One test gap is left to fix before merge, and one nit.

What I ran, on a Linux review host at f21dcf6:

• make ci-local: 28 passed, 1 failed, 7 skipped. The failure is pip-audit on the lock files, which this PR does not touch. Changed-line coverage passes.
• tests/test_novelty_routing.py: 65 passed.
• A mutation run: 37 mutants, 30 killed. The five survivors from the last review are killed now. The ones that survive are in the inline comment.
• Reproductions for the code paths, with the LLM transport mocked: errors and abstentions mixed across repeats, every call failing, non-numeric picks, SpendCapReached mid-repeats. All behave as documented.

Checked, no issues

• Draw forwarding: draw=repeat reaches llm_call_with_usage; draw 0 adds nothing to the cache key (caching.draw_extra), so existing entries still hit.
• Guards: --repeats without --model exits before any work; --repeats 0 is rejected.
• Spend cap: each repeat is counted, and SpendCapReached is not swallowed by the per-repeat except Exception.
• Repo rules: no versioned schema covers the routing output; the changelog fragment and growth baseline are in order.

[antonwolfy]

Three new output fields can be changed without a test failing. Each mutation below passes all 65 tests in tests/test_novelty_routing.py:

• "n_calls": len(queries) instead of len(picks) (routing.py:315). The CLI prints n_calls as "of N calls" when repeats > 1.
• / len(groups) instead of / len(complete) in the per-repeat rate (routing.py:375). test_a_failed_repeat_is_not_reported_as_sampling_noise has only wrong answers among the complete queries, so the numerator is 0 and the denominator does not matter. A query that is right in a complete group plus one query with a failed call would tell the two apart.
• round(max(agreements), 4) instead of the mean (routing.py:380). No test asserts mean_agreement with unequal agreements.

Suggestion: assert n_calls == 6 for 3 queries x 2 repeats; in the failed-repeat test make q1 right in both repeats and assert selection_rate_by_repeat == [1.0, 1.0] with n_queries_compared == 1; assert mean_agreement in a test with agreements 2/3 and 1.0 (0.8333).

[napetrov]

Fixed in bebf891 (tests only; the code was right). test_the_output_counts_calls_and_the_noise_means_each_agreement asserts n_calls == 6 for 2 queries x 3 repeats, and mean_agreement ~ 5/6 for agreements 2/3 and 1.0. test_a_failed_call_elsewhere_does_not_change_the_rate_of_the_complete_queries has q1 right in both repeats and q2 failing once, and asserts selection_rate_by_repeat == [1.0, 1.0] with n_queries_compared == 1. I ran your three mutants against the new tests; each now fails one. mean_agreement is compared with approx, since the mean of the rounded per-query values is 0.8334, not 0.8333.

[antonwolfy]

I pushed a commit per nit, up to 881d5d2, which fix the nit from my review:

• 881d5d2: dokimi/docs/skill-novelty.md:619: added that with repeats n_calls is queries x repeats, n_abstained and n_failed count calls, and each per_query row carries its repeat index.

make ci-local at 881d5d2: 28 passed, 1 failed (pip-audit, as at the reviewed head), 7 skipped.

Not approving yet. Still open: the risk from my review.

[napetrov]

Merge with intel/main in ab6e6a7 (conflicts in routing.py, commands/novelty.py, test_novelty_routing.py, growth_baseline.json: kept both sides; the routing summary now carries main's n_queries and triage tag plus the repeat/noise lines; baseline raised to 884 lines / complexity 13 with the rationale naming W8). Review item (test gap, three mutants) is in bebf891.

[github-actions]

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi novelty routing` parameters changed:
    + added option `--repeats`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.