Repository navigation
Conversation
Encode the checks that catch a change which is correct in its diff but wrong in its claims, its issue links, or its blast radius: a tree-wide grep for the identifiers it touches, a mapping from every PR-body claim to the line or command that proves it, and a match between `Fixes #N` and the issue's acceptance criteria. Add one benchmark rule as well: do not publish a delta whose spread is wider than the effect being claimed. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
There was a problem hiding this comment.
Thanks!
I don't want to block it but I never had any of those problems with my Google models (Gemini 4 argon, carbon hydro).
Anyone has those problems? @nicolastakashi @saswatamcode
Do we have evidences contributors did this mistake at least once in this repo?
Run the tree, claims, and issue audits before opening or updating a PR. Search old and new identifiers and relevant issue references, verify claims in PR descriptions, commits, and specs, and link issues accurately for both complete and partial work. Remove the proposed benchmark rule. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
|
@bwplotka I took a critical look at this PR, and revised it. Are the points clearer now? |
The added/revised points came out of reviewing PRs to this repo, and concrete issues found. I can provide specific evidence if that's desired. |
|
Yes please I am curious! What model, what's setup? Your agents? |
|
@bwplotka added to the PR description. Agent-wise, I used DeepSeek v4.1 Flash 1017 in Pi for the PR review, but after your initial review, cross-checked my AGENTS.md changes with Codex/Claude for non-biased takes. The latter helped me par down (remove subjectively weak benchmark advice) and clarify. |
bwplotka
left a comment
There was a problem hiding this comment.
Ok, then can you add those rules to your coding rules? They are not specific to the rust repo are those?
I feel AGENTs.md should have repo specific rule not generic coding rules, wdyt?
@bwplotka I think for that to work, every potential contributor would have to have them. The point is to steer agents better to avoid the pitfalls made in a PR I was reviewing. The idea is to save reviewers' energy. I don't need them in my own AGENTS.md :) I also think that the current advice of "Link the GitHub issue the PR closes" ought to change at the very least, because not every PR closes an issue. Even if it relates to an issue, it may not actually close it. |
Add three checks to
AGENTS.mdto run before opening or updating a PR. Also clarify the instruction about closing any issue, because not every PR will close one.Refs #Nfor partial work and keep the remaining work tracked.The revisions stem from reviewing #52. AFAICT, its claim of fixing #33 isn't correct, it only solves part of it. I propose these revisions, to hopefully steer future agents to avoid falling into the same trap.
Added checks
Tree audit
Check what a change affects outside its diff. This turns up the
scrape-managerTODO that says to wireScrapeConfig::honor_timestampsonceDecodeOptions::with_honor_timestampslands. #52 adds that method, so it makes the wiring actionable. The PR author cites the TODO as out of scope however.Claims audit
#52 says there are “no intentional differences from Go,” and its commit message broadly describes the timestamp behavior as matching Go. However; AFAICT, for an exemplar without a timestamp,
honor_timestamps=true, a sample timestamp of 10 seconds, and a configured fallback of 20 seconds, Go gives the exemplar 10 seconds because it uses the resolved sample time. Rust gives it 20 seconds because it uses the configured fallback. This seems to be a difference predating #52 though.Issue audit
PR #52 says it fixes #33, but explicitly defers
MetricsVieweven though #33 calls for it. The PR should instead say e.g. “Part of #33".