feat(novelty): estimate-only projects the calls and dollars a run will spend

dokimi/changelog.d/20261021_120000_napetrov_estimate-expected-usd.md

@@ -0,0 +1,17 @@
+### Changed
+
+- **`novelty analyze --estimate-only` projects the bill it will actually run.** The atom
+  count is the larger of one per two spans and one per 23 artifact tokens, capped as
+  before; the new `expected_calls` counts the verify, shorten, carrier and classify calls
+  and half the retention-oracle pairs that escalate to a model, and `by_role` splits it
+  into extractor, answerer and judge. `expected_usd` prices those calls at list prices,
+  with the extractor's output per call set per tier, and is `null` when a model has no
+  price. The constants were fitted on 13 of 25 measured runs; on the other 12 (held out)
+  calls landed within 13%, atoms within 11% and dollars within 33%, all within the 35%
+  tolerance. Dollars ran high on 5 runs and low on 7 (worst -33%), so `expected_usd` is not a
+  ceiling. `total_calls` and `total_calls_if_every_pair_escalates` keep their formula but
+  follow the new atom count, so they rise wherever the span rule left the count under the
+  cap and the token rule gives more (a 588-token skill at `--tier standard`: 129 -> 229).
+  The answerer's artifact term is priced per arm that carries the artifact, so
+  `--arms no_artifact,full,shuffled` projects higher than the two-arm default; an artifact
+  above the largest one measured (6,723 tokens) sets `usd_extrapolated: true`.

dokimi/docs/skill-novelty-walkthrough.md

@@ -64,17 +64,20 @@ python cli.py novelty analyze --artifact skill:skills/my-skill --tier study --es
 
 Pass the same `--include-resources` (and `--tier`) you will run with: the bill covers only
 the files the flags select, and a thin `SKILL.md` over `references/` is priced many times
-lower without it. It prints the projected model-call bill and exits before building a client. The bill is
-`probes × arms × (answer + grade)` plus the retention oracle, which escalates only its
-ambiguous band — so the output carries both ends, `total_calls` and
-`total_calls_if_every_pair_escalates`. The probe term was validated to within 1–3% of
+lower without it. It prints the projected model-call bill and exits before building a client. The
+bill is `probes × arms × (answer + grade)` plus the retention oracle, which escalates only its
+ambiguous band. The output carries `expected_calls`, the point figure, with `by_role` splitting
+it into extractor, answerer and judge calls; `total_calls` and
+`total_calls_if_every_pair_escalates` are the two ends. `expected_usd` prices `by_role` at list
+prices, and is `null` when a model has no price. The probe term was validated to within 1–3% of
 counted calls at nine (skill, tier) points, on content that did not escalate.
 
-The bill is in calls, not dollars: a call's price follows the artifact's length and the
-models. Pass `--max-usd <n>` to every step that calls a model: the run stops once its calls
-across all roles have cost that much (calls already in flight can overshoot it), and a call
-with no known price stops it too, since the cap cannot see that spend. One measured pair, a
-19k-token skill: triage 103 calls, about $1.80; `study` 1,901 calls, about $14.50.
+`expected_usd` is a projection, not a ceiling (held-out runs ran up to 33% low): a call's price
+follows the artifact's length and the models. Pass `--max-usd <n>` to every step that calls a
+model: the run stops once its calls across all roles have cost that much (calls already in
+flight can overshoot it), and a call with no known price stops it too,
+since the cap cannot see that spend. One measured pair, a 19k-token skill: triage 103 calls,
+about $1.80; `study` 1,901 calls, about $14.50.
 
 ## 3. Triage: is there anything in here? (~100 calls)
 

dokimi/docs/skill-novelty.md

@@ -792,9 +792,26 @@ python cli.py novelty analyze --artifact skill:s --tier study --estimate-only
 ```
 
 `--estimate-only` prints the projected bill and exits without constructing a client, so
-the decision costs nothing. An explicit flag always beats the tier, and the override is
-recorded — a `--tier triage --max-atoms 90` run is not a stock triage run and does not get
-to be read as one.
+the decision costs nothing. Pricing imports litellm, which tries to fetch its price map
+from the network before it falls back to the copy bundled with the package; on a host
+with no network that is a pause of a few seconds and a warning. Setting
+`LITELLM_LOCAL_MODEL_COST_MAP=True` skips the fetch and uses the bundled prices.
+
+The estimate payload has `expected_calls`, the point figure between
+the two totals (stages, every probe, and half the oracle pairs, split by `by_role`), and
+`expected_usd`: those calls priced at list prices by role, `null` when a model has no
+price. The constants were fitted on 13 of the 25 measured runs; the other 12 are held out.
+On those 12 every projection landed within 35%: calls within 13%, atoms within 11%, dollars
+within 33%. The dollar projections ran high on 5 runs (up to +29%) and low on 7 (down to
+-33%), so `expected_usd` is not a ceiling; the triage projection ran lower the larger the
+artifact was (-4% at 1,153 tokens, -33% at 6,723). The largest artifact measured has 6,723
+tokens: above that the dollar figure extrapolates, and the payload says so
+(`usd_extrapolated: true`, with a note on stderr). The answerer's artifact term is priced per
+arm that carries the artifact (every arm but `no_artifact`), so a three-arm run costs more
+per answerer call than a two-arm one; a `--placebo-text` of another length is not read. Only
+three of the 12 are standard runs, so the standard profile rests on little data. An explicit flag always beats the tier,
+and the override is recorded — a `--tier triage --max-atoms 90` run is not a stock triage
+run and does not get to be read as one.
 
 Each tier carries the **claims it forfeits**, and those strings are written into
 `run_config.tier` *and* appended to the artifact's warnings. They follow the settings the

dokimi/dokimi/commands/novelty.py

@@ -49,6 +49,7 @@ def cmd_novelty_analyze(args: argparse.Namespace) -> None:
     )
 
     from dokimi.novelty.spans import segment
+    from dokimi.novelty.tiers import FITTED_TOKENS_MAX
 
     artifact_bundle = _bundle_from_args(args)
     settings = _settings(args)
@@ -62,16 +63,35 @@ def cmd_novelty_analyze(args: argparse.Namespace) -> None:
             print(f"  overridden by flags: {sorted(record['overrides'])}", file=sys.stderr)
 
     if args.estimate_only:
-        estimate = settings.estimate(len(segment(artifact_bundle.text)),
-                                     budgets=tuple(args.budgets), n_arms=len(_arms(args.arms)))
+        spans = segment(artifact_bundle.text)
+        n_tokens = sum(span.tokens for span in spans)
+        estimate = settings.estimate(
+            len(spans), n_tokens=n_tokens, budgets=tuple(args.budgets), n_arms=len(_arms(args.arms))
+        )
+        n_arms = len(_arms(args.arms))
+        estimate["expected_usd"] = _estimate_usd(
+            args, n_tokens, estimate["estimate"]["by_role"], settings.tier,
+            context_share=(n_arms - 1) / n_arms,
+        )
+        # Every arm but the baseline is priced as carrying the artifact; a placebo of another
+        # length is not read.
+        if n_tokens > FITTED_TOKENS_MAX:
+            estimate["usd_extrapolated"] = True
+            print(f"expected_usd extrapolates: the artifact has {n_tokens} tokens and the "
+                  f"profiles were fitted on artifacts up to {FITTED_TOKENS_MAX}; the projection "
+                  "ran low as the artifact grew", file=sys.stderr)
         print(json.dumps(estimate, indent=2))
         print("Estimate only, nothing was measured. ~4 calls per paired probe is the "
               "measured shape; batch sizes and retries move the real figure.",
               file=sys.stderr)
         if args.max_usd is None:
-            print("The estimate counts calls, not dollars: what a call costs depends on the "
-                  "artifact's length and the models' prices. Pass --max-usd to stop the run "
-                  "at a dollar figure.", file=sys.stderr)
+            print(
+                "expected_usd prices the calls by role at list prices, from a token profile "
+                "fitted on measured runs: a longer artifact or longer answers cost more, and "
+                "a model with no price gives null. Pass --max-usd to stop the run at a "
+                "dollar figure.",
+                file=sys.stderr,
+            )
         return
 
     extractor, answerer, judge = _clients(args)
@@ -751,8 +771,8 @@ def _add_model_args(parser: argparse.ArgumentParser, positive_int) -> None:
     _add_spend_cap(parser)
 
 
-def _clients(args: argparse.Namespace):
-    """Build the three role clients, defaulting so roles never silently collapse."""
+def _role_models(args: argparse.Namespace) -> dict[str, tuple[str, str]]:
+    """The (model, provider) of each role, defaulting so roles never silently collapse."""
     from dokimi.novelty.client import default_judge
 
     if args.judge_model:
@@ -764,10 +784,52 @@ def _clients(args: argparse.Namespace):
             raise SystemExit(str(exc)) from None
     extract_model = args.extract_model or judge_model
     extract_provider = args.extract_provider or judge_provider
+    return {
+        "extractor": (extract_model, extract_provider),
+        "answerer": (args.model, args.provider),
+        "judge": (judge_model, judge_provider),
+    }
+
+
+def _estimate_usd(
+    args: argparse.Namespace, n_tokens: int, by_role: dict[str, int], tier: str | None,
+    context_share: float = 0.5,
+) -> float | None:
+    """The estimate's dollar figure; ``None`` when a role's model cannot be resolved or priced.
+
+    The reason goes to stderr: a role whose model does not resolve, or a role with no list
+    price, makes the figure unavailable, and the note names it.
+    """
+    from dokimi.llm_pricing import list_cost_usd
+    from dokimi.novelty.tiers import estimate_usd
+
+    try:
+        models = _role_models(args)
+    except SystemExit as exc:
+        print(f"The USD estimate is unavailable: {exc}", file=sys.stderr)
+        return None
+    usd = estimate_usd(by_role, n_tokens, models, tier, context_share)
+    if usd is None:
+        unpriced = sorted(
+            f"{role} ({model})"
+            for role, (model, provider) in models.items()
+            if list_cost_usd(model, provider, 1, 1) is None
+        )
+        print(
+            "The USD estimate is unavailable: no list price in the litellm price map for "
+            + ", ".join(unpriced),
+            file=sys.stderr,
+        )
+    return usd
+
+
+def _clients(args: argparse.Namespace):
+    """Build the three role clients, defaulting so roles never silently collapse."""
+    models = _role_models(args)
     return (
-        _client(extract_model, extract_provider, "extractor", args),
-        _client(args.model, args.provider, "answerer", args),
-        _client(judge_model, judge_provider, "judge", args),
+        _client(*models["extractor"], "extractor", args),
+        _client(*models["answerer"], "answerer", args),
+        _client(*models["judge"], "judge", args),
     )
 
 

dokimi/dokimi/llm.py

@@ -285,20 +285,20 @@ def is_retryable(exc: BaseException) -> bool:
 
 
 def _has_price_entry(litellm_model: str, model: str) -> bool:
-    """Whether litellm's price map has an entry for this model.
-
-    Looks up the routed name (``bedrock/us.anthropic.x``), the name without its
-    route prefixes (``us.anthropic.x``; ``bedrock/converse/openai.x`` ->
-    ``openai.x``) and the caller's bare model name.
-    """
+    """Whether litellm's price map has an entry for the routed name, the name without its
+    route prefixes (``bedrock/converse/openai.x`` -> ``openai.x``) or the bare model name."""
     from litellm import model_cost
 
-    candidates = {litellm_model, model, litellm_model.rsplit("/", 1)[-1]}
-    if "/" in litellm_model:
-        candidates.add(litellm_model.split("/", 1)[1])
+    candidates = {litellm_model, model, *litellm_model.split("/", 1)[1:],
+                  litellm_model.rsplit("/", 1)[-1]}
     return any(c in model_cost for c in candidates)
 
 
+def cost_per_token(**kwargs: Any) -> tuple[float, float]:  # litellm's, for llm_pricing
+    from litellm import cost_per_token as price
+    return price(**kwargs)
+
+
 def _safe_completion_cost(resp: Any, model: str, litellm_model: Optional[str] = None) -> Optional[float]:
     """Dollar cost of a completion via litellm, or ``None`` if no pricing.
 

dokimi/dokimi/llm_pricing.py

@@ -0,0 +1,33 @@
+# Copyright (C) 2026 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+"""List-price estimates for a call that has not been made (kept out of ``dokimi.llm``)."""
+
+from __future__ import annotations
+
+import contextlib
+import sys
+
+
+def list_cost_usd(model: str, provider: str, prompt_tokens: int,
+                  completion_tokens: int) -> float | None:
+    """Dollar cost of one call of these token counts at list price, or ``None``.
+
+    The price map :func:`dokimi.llm._safe_completion_cost` reads, for a call that has not
+    been made: an estimate prices a token profile rather than a response. ``None`` when the
+    model has no entry, for the same reason as there.
+    """
+    from dokimi.llm import _has_price_entry, cost_per_token, resolve_model
+
+    litellm_model = resolve_model(model, provider)
+    # litellm prints a "Provider List" help banner to stdout before it raises for a model
+    # it cannot route. ``--estimate-only`` prints JSON on stdout, so keep that out of it.
+    with contextlib.redirect_stdout(sys.stderr):
+        try:
+            prompt, completion = cost_per_token(model=litellm_model, prompt_tokens=prompt_tokens,
+                                                completion_tokens=completion_tokens)
+        except Exception:
+            return None
+        if prompt + completion == 0.0 and not _has_price_entry(litellm_model, model):
+            return None
+    return float(prompt + completion)