Skip to content

test(e2e): retry a cold-start chat request and report why it failed - #155

Open
rominf wants to merge 1 commit into
mainfrom
test/e2e-chat-cold-start-retry
Open

test(e2e): retry a cold-start chat request and report why it failed#155
rominf wants to merge 1 commit into
mainfrom
test/e2e-chat-cold-start-retry

Conversation

@rominf

@rominf rominf commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator
  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No xfail change: this is a harness flake, not a product bug — chat-tool-definitions-accepted is expected to pass on this host and should stay that way.)

Summary

  • Give a chat inference request one retry, for expect-pass scenarios only, so the first (cold) inference after a serve doesn't fail the run on a busy runner.
  • Report why an inference request failed: classify timeout vs. connect failure, print elapsed time per attempt and the full error source chain.
  • Route both chat steps (plain and tool-definitions) through one shared helper instead of two near-identical copies.

Why: the Strix-Windows lane failed chat-tool-definitions-accepted on run 30614673685. Root cause is a harness flake, not a product regression: the first inference after a serve pays a cold start (weights paged in on demand), and it overran the flat 10s inference cap. Evidence — the step failed at exactly 10.000s, the very next scenario's chat on the same host answered in 1.4s, and the same scenario took 1.0–3.1s in the four preceding Windows runs.

The retry is scoped the same way the existing serve relaunch is: attempts = if world.expect_xfail { 1 } else { 2 }. A known-bug scenario keeps its single fail-fast attempt, so the hang detection that inference_timeout_secs documents stays as prompt as before. The aborted attempt leaves the model resident, so the retry runs warm.

The diagnostics change is what made this failure expensive to triage: reqwest::Error's Display prints only the error kind, so a client timeout and a refused connection both read as error sending request for url (...), with the real cause reachable only through source(). The panic now says which one it was.

Risk: low — test harness only, no product code. Worst case for a genuinely failing expect-pass scenario is one extra bounded attempt (10s).

Test plan

  • cargo xtask e2e (mock lane): chat scenarios 5/6/7 pass. Two diagnose-* failures on my box are pre-existing — verified identical with the change stashed; they're a local host-capability quirk, not related to this diff.
  • Retry and diagnostics exercised directly by temporarily pointing the POST at a dead port (reverted before commit): both attempts logged with [connect failure — nothing accepted the connection] and the full Connection refused (os error 111) chain.
  • cargo fmt --check and cargo clippy --tests clean.

Not verified here: that the retry actually rescues a real GPU cold start — that needs the Strix-Windows runner. The failure reproduces roughly 1 run in 5, so confirmation comes from the lane staying green.

The Strix-Windows lane failed 'chat-tool-definitions-accepted' at exactly
the 10s inference cap while the very next scenario's chat answered in
1.4s: the first inference after a serve pays a cold start, and on a busy
runner it can outrun the flat cap. Give an expect-pass scenario a second,
now-warm attempt — a known-bug scenario keeps its single fail-fast
attempt, so hang detection stays as prompt as before.

The panic message was also unactionable: reqwest's Display prints only
the error kind, so a client timeout and a refused connection both read as
"error sending request". Classify the failure, print elapsed time and the
source chain, and route both chat steps through one shared helper instead
of two near-identical copies.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf requested a review from a team as a code owner July 31, 2026 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant