Remove short workflow deadlines from openai_agents tests - #1841
Closed
DABH wants to merge 1 commit into
Closed
Conversation
Several openai_agents tests started workflows with a 5-10s execution timeout nested inside pytest-timeout's 60s bound. On a loaded macOS CI runner the first workflow task was occasionally delayed past the default 10s workflow task timeout; the retried task then ran after the workflow's own execution timeout had already fired, so test_run_config_models failed with 'TimeoutError: Workflow timed out' and test_exception_handling found no activity in history ('assert found'). Drop those deadlines, as #1816 did for test_replay_tool_loop; pytest-timeout still bounds real hangs. Also assert the failure cause in assert_status_retry_behavior so an unexpected failure mode is reported directly.
Contributor
Author
|
Folded into #1835, which now removes the redundant deadlines from both the google_genai and openai_agents tests under one rationale. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
execution_timeoutvalues intests/contrib/openai_agents/test_openai.pyare removed (14 call sites), following [SDK Sentinel] Remove flaky Gemini replay workflow deadline #1816.assert_status_retry_behaviorasserts the workflow failure cause chain (ActivityError->ApplicationErrorof typeAPIStatusError) before inspecting history.Why
On loaded macOS CI runners the first workflow task is occasionally delayed past the default 10s workflow task timeout; the failing jobs log the agent starting 13s after the workflow was started, immediately followed by
Evicting workflow ... Error reporting WFT to server. The server retries the task, but by then the workflow's own 10s execution timeout has fired, sotest_run_config_modelsfails withTimeoutError: Workflow timed out(run 34253408987 attempt 1, 3.14/macos-arm) andtest_exception_handlinggets aWorkflowFailureErrorwith no activity in history and fails onassert found(run 33848127955 attempt 1, 3.10/macos-arm).These deadlines are nested inside pytest-timeout's 60s bound and serve no other purpose in these tests, so a stalled runner can only turn a healthy workflow into a timed-out one. Dropping them matches #1816; the remaining 30-120s deadlines in the file were not part of the failures and are left alone. Asserting the cause in
assert_status_retry_behaviormakes an unexpected failure mode show up directly rather than as a missing history event.Testing
TestModel(no API key); nothing here touches the live-model paths.TimeoutError('Workflow timed out'),assert found) and pass after it (the retried task runs, the model activity fails withAPIStatusErroron attempt 2, cause chain as asserted).pytest --flake-finder --flake-runs=30of both tests alongside a concurrent-n 8pytest session: all passes (30 runs each; the run covered the eight tests from the same investigation, 240/240 in total, load average 45-53). The unmodified tests also pass 240/240 on the same machine, so the CI failures only reproduce through the stall emulation above.poe lintclean.