Skip to content

Simplify PR review output and reviewer guidance selection - #69467

Closed
PureWeen wants to merge 6 commits into
dotnet:mainfrom
PureWeen:pureween-simplify-pr-reviewer
Closed

PureWeen wants to merge 6 commits into
dotnet:mainfrom
PureWeen:pureween-simplify-pr-reviewer

Conversation

@PureWeen

@PureWeen PureWeen commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Return concise source-review findings or an honest no-findings/incomplete result instead of routine bookkeeping. Preserve internal evidence, independent topic accounting, finding validation, and hosted publication conditions.
  • Read reviewer-owned guides and delegated policies from the current checkout by default, with truthful provenance. An explicit caller-supplied repo@sha still uses coordinator-supplied complete exact excerpts through the existing GitHub retrieval path. Target implementation and binding repository contracts retain their separate frozen revisions.
  • Prepare hosted guidance at the immutable workflow revision using non-cone Markdown-only checkout, the selected skill directory, and the required Copilot settings file. Verify repository/revision and remove the unused current-job activation backup. This retains repository-relative policy links without exposing the workflow revision's product files through ordinary local browsing.
  • Give local/prepared-checkout workers complete source-referenced principles, assigned topics, delegated clauses, and source restrictions through bounded original-text reads. Preserve the original complete briefing for a retry of the same assignment; missing required input remains incomplete, not LGTM.
  • Retain the compiler's default 500-request allowance and restore the main reviewer's 1,500-credit cap and budget steering. The 90-minute timeout, 200-credit detector cap, model, worker topology, and COMMENT-only publication remain unchanged.

Only the skill, workflow source, and generated lock are changed. This PR replaces the approach in closed #69422; that preserved work is not modified.

Validation and revision mapping

Role Revision
Retained local candidate used for the latest hosted test 7b79b3b35dc1b65383df012f093720057664bdff
Hosted workflow/guidance deployment 19fdb9b84cde449d6273aea030b18377b6eb0776
Fresh historical fixture head 8cecf5489f43c76073dd914ad63c70343121a4bf
Fixture base 13909f3b7ddbfeab04aac1c290f8abf4b75fe42c
This PR update, including the restored finite budget ef7ee316070543cfb9d323dca6f806dc0dd27b86

Local preparation checks exercised the actual pinned checkout action, skill installation, and subsequent framework preparation. Required guidance survived; 17,379 non-Markdown product/build/resource files were excluded from ordinary workspace paths. Explicit remote-excerpt delivery was retained.

The latest hosted test was run 35937232460 on PureWeen/aspnetcore#77. It activated the candidate skill, delivered the complete technical principles/topics/delegated clauses to all 27 final workers, and retrieved their distinct results before publication. This includes the CONTRIBUTING test exceptions and Router-present/absent and declared-exclusions clauses.

The coordinator consumed the correct frozen formatter's null-return body and published the supported nullable-rendering regression: inline finding and COMMENT review, both at the exact frozen fixture head. No PR product code or tests were executed.

The final update preserves the tested skill byte-for-byte. Its workflow differs from that candidate only by restoring max-ai-credits: 1500; pinned gh-aw v0.88.7 reproduced the generated lock twice. Configuration comparison confirmed that the generated budget delta only restores credit enforcement/steering and budget reporting, retaining maxRuns: 500 and the remaining runtime settings. The restored-budget revision has not had another agent or hosted run.

Remaining limitations

  • Source selection: one final worker fetched BindConverter.cs at the workflow revision through GitHub. The wrong-revision response was saved, but only its metadata preview—not implementation content—reached the recorded model input. No materially wrong conclusion or incorrect published feedback was demonstrated in this run. Remote source binding is not guaranteed by removing local product files.
  • Whole-section delivery: all final workers omitted the caller-publication exception paragraphs from the required entire Hard prohibitions section. No technical review criterion was missing, and no resulting worker-behavior impact was demonstrated; the whole-section requirement was nevertheless not fully met.
  • Assignment correction versus retry: the coordinator discovered incorrect initial Components topic assignments and rebuilt 13 briefings with the correct headings and complete technical criteria. Correcting invalid assignments is distinct from retrying the same valid assignment. The one explicit same-assignment retry preserved its original prompt plus the failure reason.
  • Evidence strength and runtime: only the coordinator's trace establishes consumption of the decisive formatter body. Oversized encoded source responses, GitHub rate limits, and circuit-breaker failures remain observed limitations. Fragment-requesting searches returned no source, and attempted shell calls were denied.

This is a tested advisory-review improvement, not a claim of full acceptance or regression-free review quality. Green Actions and 27 returned results alone do not establish that. No controlled comparison against the released reviewer or new completed-no-findings hosted case is claimed.

Fork restoration

The temporary deployment was restored by forward commit dfe4e92e0de42f24723bbfb278b328066b75905b; its entire tree matches pre-deployment baseline 6b76a8aa79029b01eca97a2a0289663d9b8a7040. The candidate is no longer active on the fork's deployment branch. Fixtures, publication, history, tested snapshots, and evidence are retained for the separate comparison experiment; prior fixtures were left unchanged.

Copilot AI added 4 commits September 23, 2026 11:49
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Allow hosted callers to select the workflow revision through one existing-job output.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@PureWeen
PureWeen marked this pull request as ready for review September 23, 2026 18:05
@PureWeen
PureWeen requested review from a team and wtgodbe as code owners September 23, 2026 18:05
Copilot AI lite review requested due to automatic review settings September 23, 2026 18:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

A moderate workflow-revision pinning issue remains, and hosted acceptance is incomplete.

Review effort: Lite
Findings: None

What changed in this PR

Simplifies automated PR review output and guidance selection while preserving evidence accounting and publication safeguards.

Changes:

  • Produces concise findings or explicit incomplete-review results.
  • Loads guidance from the current checkout or a pinned revision.
  • Raises the call limit to 500 and removes the main reviewer credit cap.
File Summary
.github/​workflows/​pull-request-review.md Updates orchestration, guidance loading, limits, and publication gates; workflow revision pinning needs correction.
.github/​workflows/​pull-request-review.lock.yml Regenerated compiled workflow.
.github/​skills/​review-pull-request/​SKILL.md Simplifies guidance selection and review output.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

PureWeen pushed a commit to PureWeen/aspnetcore that referenced this pull request Sep 23, 2026
Temporary fork-only live-publication test. Includes the complete 4539ceb candidate plus local snapshot 7d8b2a6c9952e86c63440511094395e8e43ef32e. Preserve fork guard and scope; restore recorded original files after one fixture run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
PureWeen pushed a commit to PureWeen/aspnetcore that referenced this pull request Sep 23, 2026
…e-reference handoff

Temporary fork validation of upstream 4539ceb and local snapshot b566e6a59610e24335a8fca1378e0db06784ca56. Restore all deployment paths after the single authorized run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
PureWeen pushed a commit to PureWeen/aspnetcore that referenced this pull request Sep 24, 2026
…nd preserved worker inputs

Temporary fork validation of upstream 4539ceb and local snapshot 7b79b3b35dc1b65383df012f093720057664bdff. Restore every deployment path after this bounded test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI added 2 commits September 24, 2026 06:00
Keep the tested guidance-only preparation and source-reference criteria delivery. Restore the 1,500-credit reviewer budget while retaining the default request allowance and regenerate with gh-aw v0.88.7.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Local invocations of the review-pull-request skill now run a packaged
scripts/prepare-review.mjs entry point. It exports the PR head,
merge-base and current base tip as immutable .source files, plus a
snapshot of the selected guidance, outside the invocation checkout.
Coordinator and workers read product evidence from those prepared roots,
so the user never switches branches, dirty guidance is honored without
leaking into target evidence, and servicing targets need no reviewer
checklists. A failed or missing preparation is BLOCKED; a local run never
falls back.

Hosted callers that supply no manifest and no shell keep the existing
read-only GitHub path (hosted GitHub mode); the hosted workflow and its
lock file are unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
PureWeen pushed a commit to PureWeen/aspnetcore that referenced this pull request Sep 24, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
PureWeen pushed a commit to PureWeen/aspnetcore that referenced this pull request Sep 24, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
PureWeen pushed a commit to PureWeen/aspnetcore that referenced this pull request Sep 24, 2026
@PureWeen

Copy link
Copy Markdown
Member Author

Superseded by #69502, which replaces this approach with a frozen review bundle.

@PureWeen PureWeen closed this Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants