fix(translation): return HTTP 502 for buffered Chat abort and error finish reasons

[colinmcnamara]

What

A non-streaming OpenAI Chat response now fails when choices[0].finish_reason is "abort" or "error". Decode returns UpstreamFailure, the same path #703 uses for a Responses body with status: "failed". The server answers HTTP 502 with the finish reason in the message, and a route with another target tries it.

Fixes #876

Why

These reasons map to StopReason::Unknown today, which Anthropic clients receive as end_turn and Responses clients as completed. A cut-off or failed answer looks complete, and fallback never runs. Details and captured bodies are in the issue.

Notes for reviewers

Start at OpenAiChatCodec::decode_response in openai_chat/buffered.rs: one guard after the body is parsed. Tests: one translation test, and one server test next to #703's failed_responses_return_errors_and_try_fallback_across_endpoints.

Upstream finish_reason	Client	Before	After
abort / error	Anthropic	200, end_turn	502 with the reason
abort / error	Responses	200, completed	502 with the reason
abort / error	Chat	200, reason passed through	502 with the reason
abort / error, route with another target	all	200 from the failing target	200 from the next target
anything else, and all streams	all	unchanged	unchanged

• Chat clients get a 502 too. This follows fix(translation): return HTTP 502 when Responses generation fails #703, whose server test returns 502 for Responses-to-Responses. It drops any partial text in an abort body for Chat clients; the issue asks whether that trade is wanted.
• Streams are unchanged. Stream completion checks belong in the stream decoder (fix(translation): reject incomplete upstream streams #425), so they are a separate change.
• Client disconnect. vLLM also aborts a request when its own client disconnects, but then no one reads the reply. The cases that reach a client are server-side aborts.
• First choice only. The check reads choices[0]. When a request asks for more than one choice, an abort or error on a later choice still looks successful.
• fix(translation): return HTTP 502 for Chat and Anthropic error bodies #855. My open fix(translation): return HTTP 502 for Chat and Anthropic error bodies #855 fails the turn when a Chat body carries an error object, with a guard at the same spot. Whichever lands second needs a small rebase, and the combined guard should check fix(translation): return HTTP 502 for Chat and Anthropic error bodies #855's error object first, so the provider's message wins.
• Out of scope: vLLM's opt-in repetition still maps to Unknown.

Checks run on this commit, with CI's commands: cargo fmt --all --check; cargo clippy --workspace --all-targets --locked -- -D warnings (pinned 1.96.1); cargo clippy -p switchyard-server --all-targets --features prefill-router --locked -- -D warnings; cargo test --workspace --locked (893 passed, 0 failed); cargo test -p switchyard-runner --features prefill-router --locked (66 passed). The two new tests fail on abf90180.

Test run

$ git log --oneline -1
5b238491 fix(translation): return HTTP 502 for buffered Chat abort and error finish reasons
$ cargo test -p switchyard-translation --test response_translation abort_and_error
     Running tests/response_translation.rs (target/debug/deps/response_translation-96d3aada475219da)

running 1 test
test abort_and_error_finish_reasons_return_upstream_failure ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 23 filtered out; finished in 0.00s

$ cargo test -p switchyard-server --test server chat_abort_and_error
     Running tests/server.rs (target/debug/deps/server-ee7fc13a9d2c5304)

running 1 test
test chat_abort_and_error_finish_reasons_return_errors_and_try_fallback ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 56 filtered out; finished in 0.03s

AI tools helped me trace the code, run the tests and review this change.

Summary by CodeRabbit

• Bug Fixes
  • Upstream responses marked as aborted or errored now return a clear API error instead of being treated as successful completions.
  • When fallback routing is enabled, requests can continue to a fallback model after an upstream abort or error.
  • These outcomes are consistent across Chat Completions, Messages, and Responses APIs.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA-NeMo/Switchyard/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 84a6dc9d-e0e1-4777-946a-1834d99e5f02

📥 Commits

Reviewing files that changed from the base of the PR and between abf9018 and 5b23849.

📒 Files selected for processing (3)

• crates/switchyard-server/tests/server.rs
• crates/switchyard-translation/src/codecs/openai_chat/buffered.rs
• crates/switchyard-translation/tests/response_translation.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The OpenAI Chat decoder now treats abort and error finish reasons as upstream failures. Translation and server tests cover the resulting errors and fallback routing across Chat Completions, Messages, and Responses.

Changes

OpenAI Chat Finish-Reason Handling

Layer / File(s)	Summary
Reject failed Chat completions crates/switchyard-translation/src/codecs/openai_chat/buffered.rs, crates/switchyard-translation/tests/response_translation.rs	The decoder returns UpstreamFailure when the first choice has abort or error as its finish reason. Tests check that translation to Anthropic Messages returns an error containing that reason.
Verify endpoint errors and fallback crates/switchyard-server/tests/server.rs	The mock upstream can return either finish reason. Tests check for 502 responses without fallback and successful responses from model/fallback when fallback is enabled, across all three inference endpoints.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 5b238

The change implements the intended failure and fallback behavior for buffered Chat responses. Discarding partial abort text is intentional; no actionable merge-blocking risk remains in the supplied evidence.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 3 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: returning HTTP 502 for buffered Chat responses with abort or error finish reasons.
Linked Issues check	✅ Passed	The changes satisfy the coding requirements in #876. OpenAiChatCodec::decode_response returns TranslationError::UpstreamFailure for first-choice finish_reason values abort and error, and inc…
Out of Scope Changes check	✅ Passed	The changes remain within #876. The mock upstream branch supplies the required non-streaming test responses. The translation and server tests verify the requested failure and fallback behavior. No unr…

• Fix all pre-merge checks with AI

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

_{A rabbit checks the finish sign,
“Abort” and “error” end the line.
A fallback hop brings hope anew,
The next model answers through.
I nibble greens and test once more!}

_{Comment @coderabbitai help to get the list of available commands.}