Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/backlog/ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -742,6 +742,8 @@ Depends on: #79 and #90.

Labels: `stage: 3`, `area: cli`, `area: docs`, `help wanted`, `need help`, `agent-friendly`, `status: ready`

Status: completed by [Markdown PR Summary Contract](../../docs/markdown-pr-summary-contract.md).

Acceptance criteria:

- Define required sections for findings, reproduction commands, confidence, and provider context.
Expand Down
2 changes: 2 additions & 0 deletions docs/deterministic-output.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ For the same issue in the same project, deterministic text should keep:

The text format is meant for humans, so wording may improve over time. When wording changes intentionally, golden tests should be updated in the same pull request and the change should be documented as user-facing output.

Future Markdown PR summary output has a separate Stage 3 contract in [Markdown PR Summary Contract](markdown-pr-summary-contract.md).

## Deterministic JSON

Deterministic JSON is the structured output printed by commands such as:
Expand Down
293 changes: 293 additions & 0 deletions docs/markdown-pr-summary-contract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,293 @@
# Markdown PR Summary Contract

This document defines the Stage 3 Markdown PR summary output contract.

The contract exists before `-format markdown` or PR-comment output is implemented so contributors can agree on content, ordering, deterministic behavior, and provider-context boundaries first.

Use this document with [Deterministic Output](deterministic-output.md), [JSON Contract](json-contract.md), [PR Context JSON Extension Contract](pr-context-json-extension.md), [Confidence](confidence.md), and [Provider Errors And Offline Fallbacks](provider-errors-and-fallbacks.md).

## Goals

- Produce a compact Markdown summary suitable for pull request comments.
- Preserve deterministic section order and finding order.
- Show Maven findings, reproduction commands, confidence reasons, and optional provider context.
- Avoid relying on remote provider data to create Maven findings.
- Keep no-finding output clear and quiet.

## Required Sections

Markdown PR summaries should use these sections in this order:

1. Title.
2. Summary.
3. Findings.
4. Reproduction.
5. Confidence.
6. Provider Context.

Sections may be omitted only when they are not applicable and the omission is deterministic. For example, `Provider Context` may be omitted when provider context was not requested.

## Deterministic Rules

For the same report data, Markdown output must keep:

- the same section order;
- the same finding order as JSON output;
- the same stable finding IDs;
- the same module paths and report paths;
- the same reproduction commands;
- the same confidence values and reasons;
- slash-separated paths across platforms;
- bounded wording for provider diagnostics.

Markdown output may improve wording over time, but wording changes should be treated as user-facing output changes and covered by tests or golden fixtures once implementation begins.

## PR Comment Constraints

The output should be suitable for GitHub pull request comments and similar review surfaces:

- avoid excessively long raw logs;
- avoid raw stack traces unless a future issue scopes truncation rules;
- avoid secrets, tokens, signed URLs, actor emails, or private account IDs;
- prefer fenced code blocks for shell commands;
- keep one finding block concise enough to scan;
- include provider context as supporting metadata, not as the main evidence.

## Finding Block

Each finding should include:

- stable finding ID;
- module path;
- report kind;
- Maven plugin and phase;
- test class, test name, source file, rule, bug type, or threshold when available;
- one-line message when available;
- report path;
- reproduction command;
- confidence value;
- confidence reasons.

Recommended finding shape:

```md
### Finding 1: payment-core

- ID: `payment-core-paymentroundingtest-shouldrejectinvalidscale-failure`
- Source: `surefire`
- Plugin: `maven-surefire-plugin`
- Phase: `test`
- Test: `dev.prmaven.demo.PaymentRoundingTest#shouldRejectInvalidScale`
- Report: `payment-core/target/surefire-reports/TEST-dev.prmaven.demo.PaymentRoundingTest.xml`
- Message: expected:<2.50> but was:<2.49>
```

## Reproduction Section

Reproduction commands should be copyable and deterministic.

Use one fenced block per unique command:

````md
## Reproduction

```bash
mvn -pl payment-core -am -Dtest=PaymentRoundingTest test
```
````

When multiple findings share the same command, print it once and reference the related finding IDs.

## Confidence Section

Confidence should explain why the finding is trusted.

Recommended shape:

```md
## Confidence

- `payment-core-paymentroundingtest-shouldrejectinvalidscale-failure`: `high`
- failure was found in a Maven Surefire JUnit XML report
- report path maps to Maven module payment-core
```

Provider relevance, changed files, or check-run metadata may appear as provider-context reasons, but they must not replace local Maven confidence reasons.

## Provider Context Section

Provider context is optional and additive.

When available, include concise provider context:

- provider name;
- repository visibility when known;
- pull request number when known;
- changed-file relevance;
- related check-run names and states;
- provider diagnostics for partial or unavailable context.

Recommended shape:

```md
## Provider Context

- Provider: `github`
- Pull request: `#123`
- Context state: `available`
- Changed files: module `payment-core` changed
- Related checks: `Maven test` completed with `failure`
```

If provider context is unavailable after being requested:

```md
## Provider Context

- Context state: `unavailable`
- Diagnostic: `missing_token`
- Local Maven findings were preserved.
```

## Example: One Finding

````md
# PR Maven CLI Summary

## Summary

- Findings: 1
- Modules with findings: 1
- Reports analyzed: 1

## Findings

### Finding 1: payment-core

- ID: `payment-core-paymentroundingtest-shouldrejectinvalidscale-failure`
- Source: `surefire`
- Plugin: `maven-surefire-plugin`
- Phase: `test`
- Test: `dev.prmaven.demo.PaymentRoundingTest#shouldRejectInvalidScale`
- Report: `payment-core/target/surefire-reports/TEST-dev.prmaven.demo.PaymentRoundingTest.xml`
- Message: expected:<2.50> but was:<2.49>

## Reproduction

```bash
mvn -pl payment-core -am -Dtest=PaymentRoundingTest test
```

## Confidence

- `payment-core-paymentroundingtest-shouldrejectinvalidscale-failure`: `high`
- failure was found in a Maven Surefire JUnit XML report
- report path maps to Maven module payment-core

## Provider Context

- Context state: `not_requested`
````

## Example: Multiple Findings

````md
# PR Maven CLI Summary

## Summary

- Findings: 2
- Modules with findings: 2
- Reports analyzed: 2

## Findings

### Finding 1: payment-api

- ID: `payment-api-paymentapiit-shouldexposepaymentsummary-error`
- Source: `failsafe`
- Plugin: `maven-failsafe-plugin`
- Phase: `verify`
- Test: `dev.prmaven.demo.PaymentApiIT#shouldExposePaymentSummary`
- Report: `payment-api/target/failsafe-reports/TEST-dev.prmaven.demo.PaymentApiIT.xml`
- Message: java.net.ConnectException: connection refused

### Finding 2: payment-core

- ID: `payment-core-paymentroundingtest-shouldrejectinvalidscale-failure`
- Source: `surefire`
- Plugin: `maven-surefire-plugin`
- Phase: `test`
- Test: `dev.prmaven.demo.PaymentRoundingTest#shouldRejectInvalidScale`
- Report: `payment-core/target/surefire-reports/TEST-dev.prmaven.demo.PaymentRoundingTest.xml`
- Message: expected:<2.50> but was:<2.49>

## Reproduction

```bash
mvn -pl payment-api -am -Dit.test=PaymentApiIT verify
```

```bash
mvn -pl payment-core -am -Dtest=PaymentRoundingTest test
```

## Confidence

- `payment-api-paymentapiit-shouldexposepaymentsummary-error`: `high`
- failure was found in a Maven Failsafe JUnit XML report
- report path maps to Maven module payment-api
- `payment-core-paymentroundingtest-shouldrejectinvalidscale-failure`: `high`
- failure was found in a Maven Surefire JUnit XML report
- report path maps to Maven module payment-core

## Provider Context

- Provider: `github`
- Pull request: `#123`
- Context state: `available`
- Changed modules: `payment-api`, `payment-core`
- Related checks: `Maven test` completed with `failure`
````

## Example: No Findings

```md
# PR Maven CLI Summary

## Summary

- Findings: 0
- Modules discovered: 3
- Reports analyzed: 2

## Findings

No supported Maven failures, violations, bugs, rules, or thresholds were found in the analyzed report artifacts.

## Reproduction

No reproduction command is available because there are no findings.

## Confidence

No findings were emitted.

## Provider Context

- Context state: `not_requested`
```

## Non-Goals

The Markdown summary contract does not define:

- automatic pull request commenting;
- comment update or deduplication behavior;
- GitHub annotation output;
- SARIF output;
- raw CI log rendering;
- live provider calls;
- provider-token resolution.

Those behaviors should be handled by separate issues.
1 change: 1 addition & 0 deletions docs/provider-context-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,3 +74,4 @@ GitHub read-token expectations live in [GitHub Provider Permissions](github-prov
Changed-file fixture expectations live in [Changed-Files Fixture Contract](changed-files-fixture-contract.md).
Check-run fixture expectations live in [Check-Runs Fixture Contract](check-runs-fixture-contract.md).
PR context JSON extension expectations live in [PR Context JSON Extension Contract](pr-context-json-extension.md).
Markdown PR summary expectations live in [Markdown PR Summary Contract](markdown-pr-summary-contract.md).
Loading