Skip to content

Route conformance backend through Host proxy - #94

Merged
lucarlig merged 4 commits into
mainfrom
user/luca/fix-stateless-conformance-backends
Aug 18, 2026
Merged

Route conformance backend through Host proxy#94
lucarlig merged 4 commits into
mainfrom
user/luca/fix-stateless-conformance-backends

Conversation

@lucarlig

@lucarlig lucarlig commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Route the conformance fixture through a small HTTP proxy so every request reaches the fixture backend with the expected Host value, including stateless tools/call requests.
  • Keep the pinned JavaScript fixture unchanged.
  • Separate dataplane-owned expected failures from failures reproduced directly against the pinned fixture.
  • Make conformance failures actionable: CI now prints the exact actual-versus-baseline difference, includes each unexpected result's status and error message, emits GitHub error annotations, and writes the comparison to the job summary.

Pinned fixture baseline

Running the official conformance runner directly against the pinned alpha.11 TypeScript fixture produces:

  • 129 passed and 54 failed, plus one scored warning.
  • 47 failures are extension or pending scenarios and are already unscored by the requirements file.
  • The remaining seven scored failures are tools/call wire-schema checks.
  • The scored warning is input-required-result-validate-input:sep-2322-validate-input-responses.

Those eight fixture-owned findings are documented separately in tests/conformance/upstream-fixture-failures.yml; they do not hide dataplane regressions.

Validation

  • Full signed-commit hooks passed.
  • Focused retry of the transient gateway_list_tools test passed (1/1).
  • shellcheck tests/conformance/report-baseline-diff.sh
  • actionlint
  • mdbook build _context/wiki
  • mdbook test _context/wiki
  • Reporter validated against both the direct fixture results (8/8 matched) and the dataplane results (44/44 matched).
  • No Rust changes and no fixture patches.

@lucarlig
lucarlig requested a review from cafalchio August 17, 2026 09:47
@lucarlig
lucarlig marked this pull request as ready for review August 17, 2026 11:09
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
@lucarlig
lucarlig force-pushed the user/luca/fix-stateless-conformance-backends branch from 5002201 to ea007a7 Compare August 17, 2026 11:17
@lucarlig lucarlig self-assigned this Aug 17, 2026

@cafalchio cafalchio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lucarlig
lucarlig merged commit 52eea91 into main Aug 18, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants