Skip to content

docs: add a pre-PR consistency audit to AGENTS.md - #58

Open
aknuds1 wants to merge 2 commits into
prometheus:mainfrom
aknuds1:docs/agents-consistency-audit
Open

aknuds1 wants to merge 2 commits into
prometheus:mainfrom
aknuds1:docs/agents-consistency-audit

Conversation

@aknuds1

@aknuds1 aknuds1 commented Nov 14, 2026 •

Copy link
Copy Markdown

Add three checks to AGENTS.md to run before opening or updating a PR. Also clarify the instruction about closing any issue, because not every PR will close one.

  • Search the tree for changed identifiers and related issue references. For renames, search both the old and new names.
  • Verify behavior and compatibility claims in the PR description, commit messages, and changed spec text.
  • Check closing keywords against the issue's acceptance criteria. Use Refs #N for partial work and keep the remaining work tracked.

The revisions stem from reviewing #52. AFAICT, its claim of fixing #33 isn't correct, it only solves part of it. I propose these revisions, to hopefully steer future agents to avoid falling into the same trap.

Added checks

Tree audit

Check what a change affects outside its diff. This turns up the scrape-manager TODO that says to wire ScrapeConfig::honor_timestamps once DecodeOptions::with_honor_timestamps lands. #52 adds that method, so it makes the wiring actionable. The PR author cites the TODO as out of scope however.

Claims audit

#52 says there are “no intentional differences from Go,” and its commit message broadly describes the timestamp behavior as matching Go. However; AFAICT, for an exemplar without a timestamp, honor_timestamps=true, a sample timestamp of 10 seconds, and a configured fallback of 20 seconds, Go gives the exemplar 10 seconds because it uses the resolved sample time. Rust gives it 20 seconds because it uses the configured fallback.  This seems to be a difference predating #52 though.

Issue audit

PR #52 says it fixes #33, but explicitly defers MetricsView even though #33 calls for it. The PR should instead say e.g. “Part of #33".

Encode the checks that catch a change which is correct in its diff but wrong in its
claims, its issue links, or its blast radius: a tree-wide grep for the identifiers it
touches, a mapping from every PR-body claim to the line or command that proves it, and a
match between `Fixes #N` and the issue's acceptance criteria.

Add one benchmark rule as well: do not publish a delta whose spread is wider than the
effect being claimed.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
@aknuds1
aknuds1 marked this pull request as ready for review November 14, 2026 17:42

@bwplotka bwplotka left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

I don't want to block it but I never had any of those problems with my Google models (Gemini 4 argon, carbon hydro).

Anyone has those problems? @nicolastakashi @saswatamcode

Do we have evidences contributors did this mistake at least once in this repo?

Run the tree, claims, and issue audits before opening or updating a PR. Search old and new identifiers and relevant issue references, verify claims in PR descriptions, commits, and specs, and link issues accurately for both complete and partial work.

Remove the proposed benchmark rule.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
@aknuds1
aknuds1 requested a review from bwplotka November 18, 2026 08:23
@aknuds1

aknuds1 commented Nov 18, 2026

Copy link
Copy Markdown
Author

@bwplotka I took a critical look at this PR, and revised it. Are the points clearer now?

@aknuds1

aknuds1 commented Nov 18, 2026 •

Copy link
Copy Markdown
Author

Do we have evidences contributors did this mistake at least once in this repo?

The added/revised points came out of reviewing PRs to this repo, and concrete issues found. I can provide specific evidence if that's desired.

@bwplotka

Copy link
Copy Markdown
Member

Yes please I am curious! What model, what's setup? Your agents?

@aknuds1

aknuds1 commented Nov 18, 2026

Copy link
Copy Markdown
Author

@bwplotka added to the PR description. Agent-wise, I used DeepSeek v4.1 Flash 1017 in Pi for the PR review, but after your initial review, cross-checked my AGENTS.md changes with Codex/Claude for non-biased takes. The latter helped me par down (remove subjectively weak benchmark advice) and clarify.

@bwplotka bwplotka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, then can you add those rules to your coding rules? They are not specific to the rust repo are those?

I feel AGENTs.md should have repo specific rule not generic coding rules, wdyt?

@aknuds1

aknuds1 commented Nov 19, 2026 •

Copy link
Copy Markdown
Author

Ok, then can you add those rules to your coding rules? They are not specific to the rust repo are those?

@bwplotka I think for that to work, every potential contributor would have to have them. The point is to steer agents better to avoid the pitfalls made in a PR I was reviewing. The idea is to save reviewers' energy. I don't need them in my own AGENTS.md :)

I also think that the current advice of "Link the GitHub issue the PR closes" ought to change at the very least, because not every PR closes an issue. Even if it relates to an issue, it may not actually close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

arrow: add honor_timestamps option to DecodeOptions

2 participants