Skip to content

Expect resolved_model in hosted platform e2e test responses - #3031

Merged
PawelPeczek-Roboflow merged 3 commits into
mainfrom
fix/e2e-tests
Sep 19, 2026
Merged

PawelPeczek-Roboflow merged 3 commits into
mainfrom
fix/e2e-tests

Conversation

@PawelPeczek-Roboflow

@PawelPeczek-Roboflow PawelPeczek-Roboflow commented Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator

What

The hosted inference e2e suite fails on staging with 20 failures (run 35393354896). Every failure is the same assertion: the response carries one extra key, resolved_model.

Extra items in the left set:
'resolved_model'

#2999 added resolved_model as a top-level field of HTTP inference responses. The tests in tests/inference/hosted_platform_tests compare the exact key set of the response.

Fix

Add "resolved_model" to the expected key set in the 10 assertions that receive it: 4 in test_core_models.py (OCR, EasyOCR, CLIP, YOLO-World) and 6 in test_roboflow_models.py (object detection, instance segmentation, classification; plain and with-visualisation-and-payload). Net diff: 10 added lines.

Visualisation-only responses do not carry the field (those tests passed in the failing run) and are unchanged.

The assertion is strict on purpose: the suite also fails if the field disappears.

Note for production runs

Production serverless is on 1.6.0-post1, which does not return the field. The production e2e workflow will fail from main until 1.6.1 is deployed there. It is dispatch-only and is run after production deploys, so this is expected ordering, not a defect.

Verification

  • pytest --collect-only on both files: 82 tests collected. black clean.
  • Not run against staging locally: HOSTED_PLATFORM_TESTS_API_KEY is not available on my machine. The staging e2e workflow needs to be dispatched on this branch to confirm.

🤖 Generated with Claude Code

#2999 added an optional top-level `resolved_model` field to HTTP inference
responses. The hosted platform e2e tests assert exact response key sets, so 20
tests failed on staging once the new build was deployed there.

The field is optional by design: some models do not report it, and staging and
production are deployed independently, so the suite must pass against servers
both with and without it. `required_response_keys()` in conftest strips the
optional keys before the exact-set comparison and checks the shape of
`resolved_model` when it is present. Unknown extra keys still fail the tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Claude review started at commit 19827b01973b4cc7d10c5d26645181537af32824.

New commits are not auto-reviewed. Add the claude-review label to request a re-review — the label is consumed when the review starts, so just add it again next time.

@github-actions

Copy link
Copy Markdown
Contributor

👋 Thanks for the pull request! Here is how automated Claude review works here, so you spend credits (and reviewer time) wisely.

🚦 This PR is marked Ready for review, so automated Claude review will run — and every pass spends real credits.

Warning

💸 The Claude reviewer bills in credits, not vibes

Automated review spins up a real agent that reads real code and spends real credits on every pass. It is glad to help — but it is not a rubber duck, a linter you poke in a loop, or a substitute for reading the contributing guide. Treat it like an expensive senior reviewer whose time you booked, and show up prepared.

Draft when unsure, Ready when you mean it:

  • 🌱 Not sure the PR is in good shape yet? Keep it (or set it back) as a draft — drafts pause review, so you can push and iterate without burning credits on a moving target.
  • 💪 Feel strong about the contents? Mark it Ready for review and the reviewer will take a look.

However you get there, arrive prepared:

  • 🧱 Bring a SOLID, thorough PR. Point your local agent at our skills/ to tune it to our guidelines first — or, if you are one of those fabled carbon-based contributors, read them yourself. A half-baked diff costs exactly the same to review as a finished one.
  • ✅ Resolve every comment before you re-request review. Re-requesting with threads still open means paying twice for the same conversation.
  • 🔁 Do not use CI review as an inner loop for a local agent. The reviewer is not a step-by-step debugger — do the unfolding locally and arrive with the answer, not the search.
  • 🙋 If something looks off, ask a human. One question to a maintainer is cheaper and faster than three rounds of agent re-review chasing a misread.

Reviews are not free. A draft costs nothing to review; a Ready PR is a promise that it is worth reviewing.

  • Prefer to skip automated review entirely? Add the skip-claude-review label.

@github-actions

Copy link
Copy Markdown
Contributor

Skills: review-http-api-server, review-topic-test-hygiene

Reviewed the test-only change that makes the hosted-platform e2e assertions tolerant of the optional resolved_model field. Verified against the code:

  • Dict/Any are imported in conftest.py, so required_response_keys type-annotates cleanly.
  • resolved_model is Optional[ResolvedModel] with a required model_id: str, and is serialized into the response dict only when non-None (_is_response_dc_to_dict, inference/core/entities/responses/inference.py:288), matching the helper's present/absent handling.
  • The helper validates shape when present, tolerates absence/null, and unknown extra keys still fail the exact-set comparison — the schema tripwire is preserved.

No production behavior changes, so no changelog/version coordination is needed.

Reviewed at HEAD: 19827b0

@github-actions

Copy link
Copy Markdown
Contributor

😎 PR passes the vibe-check and trust-me-bro verification.

…olerating it

Drop the optional-key helper. The 10 response assertions that receive
`resolved_model` since #2999 now list it in the expected key set, so the suite
also fails if the field disappears. Visualisation-only responses do not carry
the field and stay unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@PawelPeczek-Roboflow PawelPeczek-Roboflow changed the title Accept optional resolved_model field in hosted platform e2e tests Expect resolved_model in hosted platform e2e test responses Sep 18, 2026
@PawelPeczek-Roboflow
PawelPeczek-Roboflow merged commit 11049bd into main Sep 19, 2026
56 checks passed
@PawelPeczek-Roboflow
PawelPeczek-Roboflow deleted the fix/e2e-tests branch September 19, 2026 00:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants