Skip to content

.NET: Improve resilience of verify-samples by building separately and improving evaluation instructions#5151

Open
westey-m wants to merge 2 commits intomicrosoft:mainfrom
westey-m:verify-samples-improve-resilience
Open

.NET: Improve resilience of verify-samples by building separately and improving evaluation instructions#5151
westey-m wants to merge 2 commits intomicrosoft:mainfrom
westey-m:verify-samples-improve-resilience

Conversation

@westey-m
Copy link
Copy Markdown
Contributor

@westey-m westey-m commented Apr 7, 2026

Motivation and Context

  • Moving to a newer model resulted in poorer structured output compliance
  • Running sample builds in parallel causes failures that are not sample related.

Description

  • Default to building samples separately
  • Improve instructions and structured output model to improve output compliance

Contribution Checklist

  • The code builds clean without any errors or warnings
  • The PR follows the Contribution Guidelines
  • All unit tests pass, and I have added new tests where possible
  • Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

Copilot AI review requested due to automatic review settings April 7, 2026 16:14
@moonbox3 moonbox3 added documentation Improvements or additions to documentation .NET labels Apr 7, 2026
@github-actions github-actions bot changed the title Improve resilience of verify-samples by building separately and improving evaluation instructions .NET: Improve resilience of verify-samples by building separately and improving evaluation instructions Apr 7, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the .NET verify-samples tool to reduce flakiness from parallel sample builds by defaulting to running samples with dotnet run --no-build (assuming a prior build step), and refines the AI-verifier prompt/schema to improve structured output compliance.

Changes:

  • Add a --build option to control whether sample projects are built during execution (default: do not build).
  • Update sample execution to pass --no-build by default, plus documentation/workflow updates to build the solution first.
  • Improve AI verification instructions and structured response schema (including stderr in the prompt).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
dotnet/eng/verify-samples/VerifyOptions.cs Adds CLI flag parsing for --build and exposes BuildSamples option.
dotnet/eng/verify-samples/VerificationOrchestrator.cs Threads buildSamples through orchestration into sample runs.
dotnet/eng/verify-samples/SampleVerifier.cs Enhances AI verification prompt/schema and includes stderr in the verification prompt.
dotnet/eng/verify-samples/SampleRunner.cs Implements --no-build default behavior and adds a build parameter.
dotnet/eng/verify-samples/Program.cs Wires BuildSamples into the orchestrator and updates usage comment.
dotnet/.github/skills/verify-samples-tool/SKILL.md Documents the new default pre-build requirement and the --build escape hatch.
.github/workflows/dotnet-verify-samples.yml Builds the solution before running verify-samples in CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation .NET

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants