Skip to content

docs(cost): the card cost fallback is always unknown in-tree, and require_cost defers on it (#1085) - #1102

Merged
Birfy merged 2 commits into
mainfrom
agent/1085-cost-card-fallback
Nov 15, 2026
Merged

Birfy merged 2 commits into
mainfrom
agent/1085-cost-card-fallback

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

Fixes #1085

What changed

docs/cost.md told a reader that when the caller passes no
MergeContext.diff_cost, the gate prices the diff from Cost.from_cards —
"the usage the loop already records per working diff". No producer in the
package writes that usage onto a card, so the fallback is always Cost():
unknown, not a free diff. The page and the module now say so.

  • docs/cost.md §2 — the two sentences at the fallback rewrite: in-tree, no
    producer fills EvidenceCard.cost_tokens/cost_wallclock, so the fallback is
    unknown, and a caller that wants the gate to price a diff passes
    MergeContext.diff_cost = Cost.from_usage(usage). A warning follows:
    require_cost=True defers every candidate the inner gate accepted, with
    category="cost-unknown", so that run cannot commit until the caller prices
    the diff; reference_quality_per_cost and min_gain_per_1k_tokens only read
    a measured cost, so neither fires on the card fallback.
  • agentdescent/cost.py:155-161 (Cost.from_cards docstring) — the sentence
    "The loop already records one proposing rollout's usage on each card" is
    replaced by what the method actually returns in-tree and by the caller's
    route (MergeContext.diff_cost = Cost.from_usage(usage)).
  • agentdescent/cost.py:35-39 (module docstring) — the first sentence, that the
    fallback is "what the loop already collects", is the false one and is
    rewritten; the second sentence, that aggregator._decide does not fill
    diff_cost so every in-tree caller takes the card fallback, is correct and
    stays.
  • docs/cost.md §1 — the one-sentence summary (Cost.from_cards "sums what the
    contributing evidence cards already record") now says the sum is empty in-tree
    and points at §2, so the page cannot contradict itself two sections apart.
  • changelog.d/1085-cost-card-fallback.md — one Changed fragment.

No behaviour change and no test surface: the only code touched is two
docstrings — no branch, no signature, no new import.

The claim, re-measured at base 09492927

CostPolicy's fallback reads EvidenceCard.cost_tokens/cost_wallclock
(cost.py:167-168), and neither producer site sets either field
(evolution.py:4362-4381, async_evolve.py:998-1012; evolvable.py:237-238
are defaults). A probe at 09492927f614b5303e54cc3f87b0bb809929e336, with a
card built the way those two sites build it, reproduces every row of the issue:

Shape Measured at 09492927
producer-shaped card cost_tokens/cost_wallclock 0/0.0; from_cards().known = False; cost_efficiency(0.10) = None
require_cost=True accept=False, defer=True, category='cost-unknown'
reference_quality_per_cost=0.05 accept=True, status='unknown'
min_gain_per_1k_tokens=1.0 accept=True, status='unknown'
control, cost_tokens=900 status='improved', 0.111 per 1k
control, cost_tokens=5000 accept=False, category='cost-regression'

Acceptance greps, both at the tip: grep -n "already records one proposing rollout" agentdescent/cost.py → no match (exit 1); grep -n "already record for one working diff" docs/cost.md → no match (exit 1).

Verification at the tip

Tip SHA: 72b33d0

Ran, at this tip:

  • pytest -o addopts="" -q tests/test_cost_per_accepted.py tests/test_cost_aware_selection.py tests/test_site_navigation.py tests/test_repo_hygiene.py → 46 passed
  • pytest -o addopts="" -q tests/test_docs_links.py → 22 passed
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • python -m tools.changelog_fragments check → 44 changelog fragment(s) OK

Not run:

  • mkdocs build --strict — mkdocs is not installed in this clone's venv;
    CI's build job is the line

Size: 3 files, +46/−12 against base 09492927. T2 by
.github/agent/risk-rules.json (agentdescent/** is a T2 path; the docs alone
would be T0).

The real fix — recording the proposing rollout's tokens on the card at the two
producer sites — stays out of scope: the Usage is one shared Meter across
workers, so per-rollout attribution is racy, and that is a T3 change in
evolution.py.

Claim-Id: #1085

@HugoXK

HugoXK commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

Label state on this PR: agent-pr, documentation, area:docs. Declared before the write, as the repo asks.

  • documentation is the type rather than the issue's enhancement: nothing user-visible changes behaviour — the code half is two docstrings with no branch, signature or import change.
  • area:docs is the primary area; the agentdescent/cost.py half is docstring text that belongs to area:evolve, and it is named here rather than double-labelled.
  • risk:T* is deliberately not set: that field belongs to the controller. The tier of this diff is T2 under .github/agent/risk-rules.json (agentdescent/** is a T2 path; the docs alone would be T0), and the PR body states it for the record.
  • maintainer_can_modify reads false structurally — same-repo head, not a fork — so it cannot be enabled and is not chased here.

@HugoXK HugoXK added agent-pr Pull request opened by an agent area:docs Documentation site (docs/, mkdocs.yml) and README documentation Improvements or additions to documentation labels Nov 15, 2026
No producer writes cost_tokens/cost_wallclock onto an EvidenceCard, so
Cost.from_cards sums nothing and returns Cost(): the page and the two
docstrings said the loop already records that usage, which a reader could
only take as "priced". They now say the fallback is unknown, name
Cost.from_usage as the caller's route, and warn that require_cost=True
defers every candidate on a run whose cards carry no cost.

No behaviour change: two docstrings and the page. The producers stay out
of scope (per-rollout attribution is a T3 change in evolution.py).
@HugoXK

HugoXK commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator Author

Birfy — one request: a counting verdict at 72b33d0f4c82a3a394b79d4300d9b4737f5a00ca, the current tip of this PR.

State at that tip: all five test legs green (3.9–3.13), build green, risk tier green, deploy skipped; mergeable_state: blocked on the missing review alone, base 09492927, 3 files +46/−12, labels agent-pr, documentation, area:docs, no auto-merge.

The read is cheap on purpose: two agentdescent/cost.py docstrings, docs/cost.md §1 and §2, and one changelog fragment — no branch, signature or import change, so there is no behaviour to re-derive. The body carries the six-row probe taken at the base, the red-before/green-after greps, and the tip-level gate output.

No further commit is planned on this tip, so a PASS here will not be voided; if your verdict asks for changes, they land on top and the request repeats at the new tip.

Tier T2 by .github/agent/risk-rules.json (agentdescent/** is a T2 path; the docs alone would be T0). risk:T2 itself stays with the controller, and the diff changes no shipped text other than the two pages above.

The §1 summary said from_cards "sums what the cards already record", which
reads as a measured sum a page away from §2 saying the fallback is unknown.
Say the sum is empty in-tree and point at §2.
@github-actions github-actions Bot added the risk:T2 Risk tier T2 (package code or an unnamed path), set by the controller from the diff label Nov 15, 2026

@Birfy Birfy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (Birfy)

Tip SHA: 72b33d0
Ran:

  • pytest -o addopts="" -q tests/test_cost_per_accepted.py tests/test_cost_aware_selection.py tests/test_site_navigation.py tests/test_repo_hygiene.py tests/test_docs_links.py tests/test_changelog_fragments.py tests/test_api_reference.py → 140 passed, 2 skipped
  • python -m tools.changelog_fragments check → 44 changelog fragment(s) OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • mkdocs build --strict → pass
  • greps for the two false phrases ("already records one proposing rollout", "already record for one working diff") → no match
  • producer sites on this base: EvidenceCard construction in evolution.py and async_evolve.py still omits cost_tokens/cost_wallclock (those fields are written on Trajectory, not on the card)
  • CI at this tip → required checks green (build, test 3.9–3.13); review-gate was cancelled as report-only (not a required check)
    Not run:
  • full local pytest -q — no behaviour change (docstrings and docs only)
  • Ark real-model run — no model path changed
    Findings:
  • none
    Verdict: PASS

@Birfy
Birfy merged commit 914c626 into main Nov 15, 2026
9 of 10 checks passed
@Birfy
Birfy deleted the agent/1085-cost-card-fallback branch November 15, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent area:docs Documentation site (docs/, mkdocs.yml) and README documentation Improvements or additions to documentation risk:T2 Risk tier T2 (package code or an unnamed path), set by the controller from the diff

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs(cost): the card cost fallback is always unknown in-tree — say so and warn about require_cost

2 participants