test(e2e): retry a cold-start chat request and report why it failed - #155
Open
rominf wants to merge 1 commit into
Open
test(e2e): retry a cold-start chat request and report why it failed#155rominf wants to merge 1 commit into
rominf wants to merge 1 commit into
Conversation
The Strix-Windows lane failed 'chat-tool-definitions-accepted' at exactly the 10s inference cap while the very next scenario's chat answered in 1.4s: the first inference after a serve pays a cold start, and on a busy runner it can outrun the flat cap. Give an expect-pass scenario a second, now-warm attempt — a known-bug scenario keeps its single fail-fast attempt, so hang detection stays as prompt as before. The panic message was also unactionable: reqwest's Display prints only the error kind, so a client timeout and a refused connection both read as "error sending request". Classify the failure, print elapsed time and the source chain, and route both chat steps through one shared helper instead of two near-identical copies. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No xfail change: this is a harness flake, not a product bug —chat-tool-definitions-acceptedis expected to pass on this host and should stay that way.)Summary
Why: the Strix-Windows lane failed
chat-tool-definitions-acceptedon run 30614673685. Root cause is a harness flake, not a product regression: the first inference after a serve pays a cold start (weights paged in on demand), and it overran the flat 10s inference cap. Evidence — the step failed at exactly 10.000s, the very next scenario's chat on the same host answered in 1.4s, and the same scenario took 1.0–3.1s in the four preceding Windows runs.The retry is scoped the same way the existing serve relaunch is:
attempts = if world.expect_xfail { 1 } else { 2 }. A known-bug scenario keeps its single fail-fast attempt, so the hang detection thatinference_timeout_secsdocuments stays as prompt as before. The aborted attempt leaves the model resident, so the retry runs warm.The diagnostics change is what made this failure expensive to triage:
reqwest::Error'sDisplayprints only the error kind, so a client timeout and a refused connection both read aserror sending request for url (...), with the real cause reachable only throughsource(). The panic now says which one it was.Risk: low — test harness only, no product code. Worst case for a genuinely failing expect-pass scenario is one extra bounded attempt (10s).
Test plan
cargo xtask e2e(mock lane): chat scenarios 5/6/7 pass. Twodiagnose-*failures on my box are pre-existing — verified identical with the change stashed; they're a local host-capability quirk, not related to this diff.[connect failure — nothing accepted the connection]and the fullConnection refused (os error 111)chain.cargo fmt --checkandcargo clippy --testsclean.Not verified here: that the retry actually rescues a real GPU cold start — that needs the Strix-Windows runner. The failure reproduces roughly 1 run in 5, so confirmation comes from the lane staying green.