Skip to content

Improve conformance baseline reporting - #105

Merged
lucarlig merged 3 commits into
mainfrom
user/luca/conformance-reporting
Aug 18, 2026
Merged

Improve conformance baseline reporting#105
lucarlig merged 3 commits into
mainfrom
user/luca/conformance-reporting

Conversation

@lucarlig

@lucarlig lucarlig commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

What changed

  • use one cargo-nextest-style colored report for local make conformance and CI
  • distinguish XFAIL (expected failure reproduced), FAIL (expected pass failed), and XPASS (expected failure passed)
  • classify pinned fixture findings as informational UPSTREAM results so they do not fail the dataplane job or appear in dataplane failure details
  • add make conformance-bless to refresh dataplane-owned expected failures from the current run
  • isolate each local run's result directory and add focused reporter regression coverage

Why

The official runner's raw status combined dataplane baseline mismatches with known upstream fixture findings, while the local target did not run the detailed reporter. That made it hard to see which expectation changed and could fail CI for an upstream issue.

The shared reporter is now authoritative in both environments. Missing results remain an error, but pinned upstream findings are reported without being counted as dataplane failures.

Usage

Run the conformance suite and compare its results with the checked-in expectations:

make conformance

After an intentional behavior change, run the suite and refresh the expected-failure baseline:

make conformance-bless

Review and commit the resulting change to tests/conformance/expected-failures.yml.

Verification

  • tests/conformance/report-baseline-diff-test.sh
  • shellcheck tests/conformance/report-baseline-diff.sh tests/conformance/report-baseline-diff-test.sh tests/conformance/run-conformance.sh tests/conformance/run-local.sh
  • actionlint .github/workflows/mcp_conformance.yml
  • pinned full make conformance: 59 passed, 44 expected failures, 0 failed, 0 unexpected passes
  • make pre-commit

Signed-off-by: lucarlig <luca.carlig@ibm.com>
@lucarlig
lucarlig requested a review from cafalchio August 18, 2026 14:19
@lucarlig
lucarlig marked this pull request as ready for review August 18, 2026 14:22
Signed-off-by: lucarlig <luca.carlig@ibm.com>
@lucarlig
lucarlig marked this pull request as draft August 18, 2026 14:31
Signed-off-by: lucarlig <luca.carlig@ibm.com>
@lucarlig
lucarlig marked this pull request as ready for review August 18, 2026 14:35

@cafalchio cafalchio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure id make conformance bless is working.
But make conformance is more human readable.

@lucarlig
lucarlig merged commit 94f97c4 into main Aug 18, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants