Skip to content

Replace traced readiness query polls with untraced waits - #1848

Open
DABH wants to merge 4 commits into
mainfrom
flake/untraced-readiness-probes
Open

Replace traced readiness query polls with untraced waits#1848
DABH wants to merge 4 commits into
mainfrom
flake/untraced-readiness-probes

Conversation

@DABH

@DABH DABH commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

What was changed

  • Add tests.helpers.wait_for_workflow_idle(handle): polls untraced describe() until the running workflow has no pending workflow task, activity, child or Nexus operation, and fails fast if the workflow is no longer running. The docstring records two limits: a timer-only wait counts as idle, and the time-skipping test server never reports pending_workflow_task.
  • langgraph test_summary_fn.py: the ran/invoked query polls become wait_for_workflow_idle.
  • openai_agents test_openai_tracing.py: the four ready query polls become an untraced history poll (plain client) for ACTIVITY_TASK_COMPLETED then WORKFLOW_TASK_COMPLETED.
  • assert_eventually: a cancelled RPC now retries on the normal interval and gives up at the deadline instead of spinning.
  • Remove the now-unused queries and flags.

Why

A readiness query is attached to every attempt of a pending workflow task, so it can run twice or time out client-side, and in tracing tests each poll runs through the instrumented client. The OpenAI tests use history because CI also runs them on the time-skipping server, which never reports pending_workflow_task and returns no events for wait_new_event polls on a running workflow. #1830 does the same for LangSmith and can adopt the shared helper once both land.

Testing

--flake-finder --flake-runs=20 per converted test under CPU load: 120/120 on the dev server, 80/80 for the OpenAI tests on time-skipping (the originals also passed, so this is hardening, not a reproduced failure). After the follow-up commits: both files pass on the dev server and under time-skipping (the converted langgraph tests skip there), 12/12 repeated runs with four workers, lint and type checkers clean.

Several contrib tests polled a workflow query (ready, ran, invoked) until
the workflow reached its signal wait. A query issued while a workflow
task is pending is buffered by the server and attached to every attempt
of that task, so the poll can run twice or time out client-side before
an attempt completes, and in the tracing tests each poll also runs
through the instrumented client and worker interceptors.

Add tests.helpers.wait_for_workflow_idle, which waits via describe()
until the workflow is RUNNING with no pending workflow task, activity,
child or Nexus operation, and use it in the langgraph summary tests,
whose workflow parks on a signal wait right after its first task.

The OpenAI Agents tracing tests also run on the time-skipping test
server in CI, which never reports pending_workflow_task and returns no
events for wait_new_event history polls on a running workflow. Those
tests instead poll history through the plain client for a workflow task
completion after the activity completion (assert_event_subsequence),
which is exact on both servers.

Remove the now-unused ready/ran/invoked queries and their backing flags.
@DABH DABH added the skip-changelog PR changes do not require changelog updates label Sep 10, 2026
@DABH
DABH marked this pull request as ready for review September 11, 2026 05:39
@DABH
DABH requested review from a team as code owners September 11, 2026 05:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The test synchronization changes are consistent, targeted, and account for time-skipping server limitations.

Pull request overview

Replaces traced readiness queries with untraced synchronization to reduce tracing-test flakiness.

Changes:

  • Adds a shared workflow-idle polling helper.
  • Uses idle polling in LangGraph tests.
  • Uses workflow-history polling in OpenAI tracing tests.
File summaries
File Description
tests/helpers/__init__.py Adds wait_for_workflow_idle.
tests/contrib/openai_agents/test_openai_tracing.py Replaces readiness queries with history polling.
tests/contrib/langgraph/test_summary_fn.py Replaces query polling with idle waits.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 0
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Build wait_for_workflow_idle on assert_eventually so it shares the
timeout, interval and cancelled-RPC retry of the other helpers, while
still failing fast when the workflow is no longer running. Name the
pending work in the timeout message and print the status name instead
of its integer value.

Document that timer-only waits count as idle, that the time-skipping
test server never reports a pending workflow task, and that the OpenAI
history probe assumes the first completed activity precedes the park.
A cancelled RPC used to retry immediately with no sleep and no deadline
check, so a persistently cancelled probe spun until pytest's timeout and
hid the helper's own diagnostic. Retry on the normal interval and give
up at the deadline instead. Also name the elapsed timeout in the
wait_for_workflow_idle message.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog PR changes do not require changelog updates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants