Repository navigation
docs(cost): the card cost fallback is always unknown in-tree, and require_cost defers on it (#1085) - #1102
Conversation
|
Label state on this PR:
|
No producer writes cost_tokens/cost_wallclock onto an EvidenceCard, so Cost.from_cards sums nothing and returns Cost(): the page and the two docstrings said the loop already records that usage, which a reader could only take as "priced". They now say the fallback is unknown, name Cost.from_usage as the caller's route, and warn that require_cost=True defers every candidate on a run whose cards carry no cost. No behaviour change: two docstrings and the page. The producers stay out of scope (per-rollout attribution is a T3 change in evolution.py).
|
Birfy — one request: a counting verdict at State at that tip: all five The read is cheap on purpose: two No further commit is planned on this tip, so a PASS here will not be voided; if your verdict asks for changes, they land on top and the request repeats at the new tip. Tier T2 by |
The §1 summary said from_cards "sums what the cards already record", which reads as a measured sum a page away from §2 saying the fallback is unknown. Say the sum is empty in-tree and point at §2.
Birfy
left a comment
There was a problem hiding this comment.
Review (Birfy)
Tip SHA: 72b33d0
Ran:
- pytest -o addopts="" -q tests/test_cost_per_accepted.py tests/test_cost_aware_selection.py tests/test_site_navigation.py tests/test_repo_hygiene.py tests/test_docs_links.py tests/test_changelog_fragments.py tests/test_api_reference.py → 140 passed, 2 skipped
- python -m tools.changelog_fragments check → 44 changelog fragment(s) OK
- python -m tools.gen_api_docs --check → docs/api.md is up to date
- mkdocs build --strict → pass
- greps for the two false phrases ("already records one proposing rollout", "already record for one working diff") → no match
- producer sites on this base: EvidenceCard construction in evolution.py and async_evolve.py still omits cost_tokens/cost_wallclock (those fields are written on Trajectory, not on the card)
- CI at this tip → required checks green (build, test 3.9–3.13); review-gate was cancelled as report-only (not a required check)
Not run: - full local pytest -q — no behaviour change (docstrings and docs only)
- Ark real-model run — no model path changed
Findings: - none
Verdict: PASS
Fixes #1085
What changed
docs/cost.mdtold a reader that when the caller passes noMergeContext.diff_cost, the gate prices the diff fromCost.from_cards—"the usage the loop already records per working diff". No producer in the
package writes that usage onto a card, so the fallback is always
Cost():unknown, not a free diff. The page and the module now say so.
docs/cost.md§2 — the two sentences at the fallback rewrite: in-tree, noproducer fills
EvidenceCard.cost_tokens/cost_wallclock, so the fallback isunknown, and a caller that wants the gate to price a diff passes
MergeContext.diff_cost = Cost.from_usage(usage). A warning follows:require_cost=Truedefers every candidate the inner gate accepted, withcategory="cost-unknown", so that run cannot commit until the caller pricesthe diff;
reference_quality_per_costandmin_gain_per_1k_tokensonly reada measured cost, so neither fires on the card fallback.
agentdescent/cost.py:155-161(Cost.from_cardsdocstring) — the sentence"The loop already records one proposing rollout's usage on each card" is
replaced by what the method actually returns in-tree and by the caller's
route (
MergeContext.diff_cost = Cost.from_usage(usage)).agentdescent/cost.py:35-39(module docstring) — the first sentence, that thefallback is "what the loop already collects", is the false one and is
rewritten; the second sentence, that
aggregator._decidedoes not filldiff_costso every in-tree caller takes the card fallback, is correct andstays.
docs/cost.md§1 — the one-sentence summary (Cost.from_cards"sums what thecontributing evidence cards already record") now says the sum is empty in-tree
and points at §2, so the page cannot contradict itself two sections apart.
changelog.d/1085-cost-card-fallback.md— oneChangedfragment.No behaviour change and no test surface: the only code touched is two
docstrings — no branch, no signature, no new import.
The claim, re-measured at base
09492927CostPolicy's fallback readsEvidenceCard.cost_tokens/cost_wallclock(
cost.py:167-168), and neither producer site sets either field(
evolution.py:4362-4381,async_evolve.py:998-1012;evolvable.py:237-238are defaults). A probe at
09492927f614b5303e54cc3f87b0bb809929e336, with acard built the way those two sites build it, reproduces every row of the issue:
09492927cost_tokens/cost_wallclock0/0.0;from_cards().known = False;cost_efficiency(0.10) = Nonerequire_cost=Trueaccept=False, defer=True, category='cost-unknown'reference_quality_per_cost=0.05accept=True, status='unknown'min_gain_per_1k_tokens=1.0accept=True, status='unknown'cost_tokens=900status='improved',0.111per 1kcost_tokens=5000accept=False, category='cost-regression'Acceptance greps, both at the tip:
grep -n "already records one proposing rollout" agentdescent/cost.py→ no match (exit 1);grep -n "already record for one working diff" docs/cost.md→ no match (exit 1).Verification at the tip
Tip SHA: 72b33d0
Ran, at this tip:
pytest -o addopts="" -q tests/test_cost_per_accepted.py tests/test_cost_aware_selection.py tests/test_site_navigation.py tests/test_repo_hygiene.py→ 46 passedpytest -o addopts="" -q tests/test_docs_links.py→ 22 passedpython -m tools.gen_api_docs --check→docs/api.md is up to datepython -m tools.changelog_fragments check→44 changelog fragment(s) OKNot run:
mkdocs build --strict— mkdocs is not installed in this clone's venv;CI's
buildjob is the lineSize: 3 files,
+46/−12against base09492927. T2 by.github/agent/risk-rules.json(agentdescent/**is a T2 path; the docs alonewould be T0).
The real fix — recording the proposing rollout's tokens on the card at the two
producer sites — stays out of scope: the
Usageis one sharedMeteracrossworkers, so per-rollout attribution is racy, and that is a T3 change in
evolution.py.Claim-Id: #1085