[napetrov]

Part of #4.

Summary

novelty analyze --estimate-only now projects the calls the run will make and what they will cost, not only the calls the estimator could count before.

 analyze --estimate-only
   atoms     = max(spans // 2, tokens // 23), capped at max_atoms
+            (spans alone undercount: short table rows each yield an atom)
   by_role   per role: extractor, answerer, judge
   expected_calls
+            = stage calls + verify/shorten/carrier/classify
+            + ESCALATION_SHARE (0.5) of the oracle pairs
   total_calls                          unchanged
   total_calls_if_every_pair_escalates  unchanged
+  expected_usd
+            = sum over roles of calls x list price of a token profile
+            null when a role's model has no list price

expected_calls sits between the existing no-escalation and full-escalation totals. expected_usd is the run's own price map applied to a token profile fitted per tier: a triage extraction packs the capped atom set into each call, and a standard extraction does not.

Evidence

The constants were fitted on 13 of the 25 measured skill-scan runs. The other 12 are held out, and the regression tests check every projection against them. Each run's figures are in MEASURED_RUNS in tests/test_novelty_tiers.py as numbers only.

held-out (12 runs), projection vs measured
  calls    within 13%
  atoms    within 11%
  dollars  within 33%   (high on 5 runs, up to +29%; low on 7, down to -33%)
  tolerance 35%  -> test_held_out_{calls,atoms,dollars,triage_dollars}_within_35_percent

Fail on intel/main:

commands/novelty.py reverted to intel/main, rest of this branch kept
  FAILED test_estimate_only_prints_the_expected_calls_and_a_dollar_figure
  1 failed, 79 passed   (test_novelty_tiers.py + test_novelty_settings_parity.py)

The other new tests import names that do not exist on intel/main (estimate_usd, call_tokens, TOKENS_PER_ATOM, CALL_TOKENS), so against intel/main they cannot run at all. The tier-level behaviour is covered by those tests rather than by the CLI test alone.

Local checks on the rebased head:

uv run ruff check .                              All checks passed!
uv run mypy dokimi                               Success: no issues found in 222 source files
uv run python -m dokimi.schema_check             exit 0
uv run python -m dokimi.config_check             exit 0
uv run pytest -n auto -q                         6889 passed, 22 skipped

Merge Danger

Door: two-way. The change is to a projection, not to a run: --estimate-only still makes no client and spends nothing. The output keys are added, and the existing total_calls figures are unchanged.

Blast Radius: low for runs, but the numbers people read change.

• New numbers in the estimate output. atoms and expected_calls now reflect the token rule, so an estimate for a long artifact with short spans is higher than before. Scripts that read the estimate will see tokens, by_role, expected_calls and expected_usd.
• Dollar figure is a projection. expected_usd is list price applied to fitted token profiles. It ran high on 5 held-out runs and low on 7 (worst -33%), so it is not a ceiling. It is not a bill, and it is null when any role's model has no list price in the litellm map.
• Thin standard-tier data. Only three of the 12 held-out runs are standard runs, so the standard profile rests on little data. The docs say so.
• Constants are fitted, not configured. TOKENS_PER_ATOM, ESCALATION_SHARE, CALL_TOKENS and EXTRACTOR_TOKENS live in novelty/tiers.py. Changing them changes every estimate.
• No flags, no persisted-format change beyond the additive estimate keys.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated: docs/skill-novelty.md (--estimate-only section)
• No ad-hoc run artifacts committed
• New code and test files carry the SPDX license header (the changelog fragment follows the existing fragments, which carry none)

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.01%. Comparing base (9a93635) to head (5450df5).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #166      +/-   ##
==========================================
+ Coverage   91.98%   92.01%   +0.02%     
==========================================
  Files         223      224       +1     
  Lines       27743    27810      +67     
  Branches     4560     4567       +7     
==========================================
+ Hits        25520    25589      +69     
+ Misses       1574     1572       -2     
  Partials      649      649              

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/novelty.py	96.06% <100.00%> (+0.21%)	⬆️
dokimi/dokimi/llm.py	95.78% <100.00%> (+<0.01%)	⬆️
dokimi/dokimi/llm_pricing.py	100.00% <100.00%> (ø)
dokimi/dokimi/novelty/tiers.py	100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review at 6e01b78. The projection math holds up and the CLI handles unpriced and unresolvable models cleanly, but one published claim is false and two behaviours the PR adds are not pinned by its tests. Four things are worth fixing before merge, plus two nits inline.

What I ran, on a Linux review host at 6e01b78:

• make ci-local: 20 passed, 7 skipped, 8 failed. The 8 failures are steps whose tool is missing from my venv (exit 127): make dev stops at the tools install on this base, which #179 fixed on main. Static checks, mypy and the strict docs build pass on GitHub's CI at this head. The test step and the changed-line coverage step pass.
• Recomputed the held-out errors from MEASURED_RUNS with the PR's own estimate() and estimate_usd() and the pinned prices.
• A mutation run of the new tests (test_novelty_tiers.py, test_novelty_settings_parity.py): 14 mutants, 2 of which pass both files.
• CLI repros with the LLM transport blocked: unknown extractor/judge model, bad provider, zero-token artifact, --all-atoms, --max-atoms 1.

Findings that cannot be anchored inline:

1. docs/skill-novelty-walkthrough.md still describes the old estimate. Lines 65-71 name only total_calls and total_calls_if_every_pair_escalates, and line 71 says "The bill is in calls, not dollars". The estimate now prints expected_calls and expected_usd. The docs-review-triggers step flags the page: it depends on commands/novelty.py and novelty/tiers.py, which changed, and it was not updated. Suggestion: describe expected_calls, by_role and expected_usd there, and keep the --max-usd advice.

Checked, no issues

• Calls and atoms: held-out errors are +13.3% max for calls (held-01, 111 vs 98) and -11.1% max for atoms (held-02, 16 vs 18), as the PR states. Three of the 12 runs are standard.
• Dollar magnitude: the worst held-out dollar error is 33.0% (held-12), inside the 35% tolerance. Only its sign claim is wrong, see inline.
• Models and prices: an unknown extractor or judge model gives expected_usd: null with a stderr note naming only the unpriced roles. A judge provider that default_judge rejects gives null plus a note instead of aborting. Free local models price to 0.0, unknown ones to None.
• Edge inputs: a zero-token artifact, --all-atoms (200 atoms) and --max-atoms 1 give sane figures, and --arms paired is refused as before.
• Consumers: estimate() and estimate_calls() are used only by commands/novelty.py and tests, and no schema describes the payload. The added keys are additive. The _clients refactor behaves as before.

[napetrov]

Review-body item 1 (walkthrough still describes the old estimate): fixed in 88d697a. docs/skill-novelty-walkthrough.md now describes expected_calls, by_role and expected_usd, keeps the --max-usd advice, and says expected_usd is a projection, not a ceiling.

[antonwolfy]

I pushed a commit per nit, up to de6c34f, which fix the two nits from my review:

• 5dd7057: dokimi/docs/skill-novelty.md:794: re-wrapped the override sentence so the paragraph lines stay near 92 characters.
• de6c34f: dokimi/docs/skill-novelty.md:787: noted that pricing imports litellm, which tries a price-map fetch before using the bundled copy, and that LITELLM_LOCAL_MODEL_COST_MAP=True skips it.

make ci-local at de6c34f: 20 passed, 8 failed (host: tool missing, exit 127), 7 skipped.

Not approving yet. Still open: the four risks from my review.

[antonwolfy]

Review at 88d697a. The three threads from my earlier review are fixed: the dollar-direction claim, the CLI tier pin and the no-price-entry branch each have the right text or test now. One new bug: --estimate-only can print non-JSON on stdout for a model litellm cannot route. Two smaller points inline.

What I ran, on a Linux review host at 88d697a:

• make ci-local: 27 passed, 1 failed, 7 skipped. The failure is pip-audit (a new gitpython 3.2.0 advisory, CVE-2026-114145); the PR does not touch the lock files, so I do not count it against the PR.
• Recomputed the 12 held-out rows with estimate() and estimate_usd() at the pinned prices.
• A mutation run of test_novelty_tiers.py and test_novelty_settings_parity.py: 13 mutants, 1 of which passes both files. Reproductions for the code finding, with the LLM transport mocked.

Checked, no issues

• Dollar claim: held-out dollar errors are +28.7% to -33.0%, high on 5 runs and low on 7, as the docs and changelog now say. The PR description still says every dollar projection came out high. Calls (max 13.3%) and atoms (max 11.1%) match.
• New tests: replacing settings.tier with None fails only the new triage CLI test. Changing the return None for a missing price entry to return 0.0 fails only the new list_cost_usd test. Eight other mutants of estimate_usd and call_tokens are caught.
• Existing keys: total_calls and total_calls_if_every_pair_escalates are unchanged.

[xaleryb]

• Ran: full suite, ruff, mypy and bandit at 500282f plus the commits below; differential --estimate-only runs on main vs head; mutants of the new guards.
• Large artifacts: the walkthrough's 19k-token pair (~$1.80 triage, ~$14.50 study) -> no one model mix matches both; every mix that matches triage projects study 86-229% high -> state the fitted token range (up to 6,723) and flag extrapolation, or refit the answerer and extractor profiles.
• --arms no_artifact,full,shuffled, triage, ~20k tokens -> $2.79 -> about $3.58 by the PR's own fit. The answerer is priced at the two-arm mean, and the --placebo-text length is never read -> scale the answerer's per-token term by the number of context-carrying arms (factor 1 at the default arms).
• PR body: the table and Merge Danger still say total_calls is unchanged; it moves with the atom count (fastcontext, standard: 129 -> 229).
• Commits: 96daabd changelog states that the totals move; bdf8d85 test for the null-USD branch when no judge resolves; 76a567c test that by_role scales with --arms.

[antonwolfy]

Review at 76a567c. The fixes for my three threads hold: the banner stays off stdout, the max(0, ...) floor is pinned, and the walkthrough paragraph is re-wrapped. Two test gaps inline are worth closing before merge, and @xaleryb's two points are still open.

I agree with @xaleryb on the three-arm answerer pricing and the large-artifact extrapolation. New evidence: with --arms no_artifact,full,shuffled at 6,723 tokens the answerer term is about 21% low, and at 19k tokens no model mix lands both triage and study near the walkthrough's $1.80 and $14.50 (where triage matches within 1%, study is +86%).

What I ran, on a Linux review host at 76a567c:

• make ci-local: 28 passed, 1 failed, 7 skipped. The failure is pip-audit (gitpython 3.2.0 advisory); the PR does not touch the lock files, so I do not count it. The test step passes.
• A mutation run of test_novelty_tiers.py and test_novelty_settings_parity.py: 37 mutants, 9 of which pass both files. Two are inline.
• CLI runs with the LLM transport mocked and the network blocked: unroutable, unpriced and unresolvable models, all three tiers, --arms variants, --all-atoms, --max-atoms 1, empty and tiny artifacts.

Checked, no issues

• Stdout: json.loads(stdout) succeeds in every case that produces an estimate, including --model foo-bar-9 --provider openai; the litellm banner goes to stderr.
• _has_price_entry refactor: 14 x 15 (routed name, model) pairs give the same answer as the base.
• Held-out figures: recomputed with the pinned prices: calls max 13.3%, atoms max 11.1%, dollars +28.7% to -33.0%, as the docs say.
• Import architecture: the cost_per_token wrapper keeps litellm inside dokimi.llm; the architecture tests pass.

[antonwolfy]

The reversals term in by_role for the answerer and judge is not pinned by any test.

Mutation check: replacing (paired + reversals) * n_arms with paired * n_arms for both answerer and judge passes test_novelty_tiers.py and test_novelty_settings_parity.py in full (86 passed). Every fixture in MEASURED_RUNS is a triage or standard run, and both have reversals off, so the term is always 0 in the tests. A study run (reversals on) would then lose n_atoms * n_arms calls per role from expected_calls and expected_usd with nothing failing.

Suggestion: add a test with resolve_settings("study", {}) that asserts by_role["answerer"] == by_role["judge"] == (paired_probes + reversal_probes) * n_arms and reversal_probes > 0.

[napetrov]

Added in ee809ec: test_a_study_run_counts_its_reversal_probes_in_the_answerer_and_judge_calls asserts reversal_probes > 0 and by_role answerer == judge == (paired + reversal) * n_arms for the study tier. Verified it fails with the reversals term dropped.

[antonwolfy]

Nothing pins that the CLI passes the artifact's token count to estimate_usd.

Mutation check: replacing n_tokens with 0 here passes both new-test files in full (86 passed). The CLI tests compare expected_usd only to > 0, or to another estimate_usd call that uses the same tier. A refactor that drops the count would make the figure stop scaling with the artifact, which is the main input to the dollar projection, and no test would notice.

Suggestion: in the triage CLI test, assert expected_usd == estimate_usd(by_role, payload["tokens"], models, "triage") with payload["tokens"] > 0, and that it differs from the figure at n_tokens=0.

[napetrov]

Fixed in ee809ec: test_the_cli_prices_a_triage_estimate_with_the_triage_extractor_profile now uses an artifact large enough that the max(0, ...) floor does not hide the token count, asserts payload tokens > 100 and that expected_usd differs from estimate_usd at n_tokens=0. Verified it fails with n_tokens replaced by 0 in the CLI.

[napetrov]

Review-body items from @xaleryb's comment, in ee809ec: (1) large artifacts: the payload now sets usd_extrapolated and warns on stderr above 6,723 tokens (the largest measured artifact), and the docs state the range and that the triage projection ran low as the artifact grew. I did not refit the profiles. (2) three-arm answerer: the answerer's per-token term now scales with the share of arms that carry the artifact (factor 1 at the default two arms, 1.5 at four); a --placebo-text of another length is still not read, and the docs say so (test_the_answerer_artifact_term_scales_with_the_arms_that_carry_it, test_the_cli_prices_three_context_arms_above_two). (3) PR body claims: deferred, I did not edit the description; it still says every dollar projection came out high and that total_calls is unchanged.