Skip to content

fix(workflow): stop a re-emitted output echo from reordering the replay sequence - #7028

Open
chelsealong wants to merge 1 commit into
google:mainfrom
chelsealong:fix-7027-hitl-replay-sequence-deadlock
Open

fix(workflow): stop a re-emitted output echo from reordering the replay sequence#7028
chelsealong wants to merge 1 commit into
google:mainfrom
chelsealong:fix-7027-hitl-replay-sequence-deadlock

Conversation

@chelsealong

Copy link
Copy Markdown
Contributor

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

Problem:

On a resumable Workflow running the documented review/revise HITL loop
(review → revise → review → ... → approve), a third resume can deadlock with:

RuntimeError: Replay divergence detected: Timed out waiting for sequence key 'HITL@1' to be unblocked.

Root cause: Workflow._maybe_reemit_replayed_output re-surfaces a
fast-forwarded node's output again on every turn it is replayed through, so a
resumable event stream stays complete for that turn. That re-emitted event
carries the same node path / run id (HITL@1) as the node's original,
genuine completion.

ReplayManager._scan_sequence builds the replay barrier's expected
completion order by scanning terminal events and, for a segment it has
already seen, removing the old position and appending the new one — logic
meant to update a node's position when it moves from "interrupted" to
"completed". Applied to the re-emitted echo, it instead moves HITL@1 past
Revise@1, even though Revise@1 can only ever run after HITL@1
completes. On the next resume this produces a corrupted barrier sequence
([Revise@1, HITL@1, ...]) that creates a real cycle: HITL@1's
fast-forward path waits for Revise@1's turn, but Revise@1 can't even be
scheduled until HITL@1's (fast-forwarded) task returns — a deadlock that
only times out after 15s.

Solution:

_scan_sequence now tracks which child run ids have already reached a
genuine terminal completion (an event carrying output, route, or
error_code). Once a run id is marked completed, any further terminal event
for that same id is ignored for ordering purposes, since a completed run id
cannot complete a second time — a repeat is always the re-emitted echo, not
new information. This preserves the legitimate "interrupted → completed"
reposition (the first time a node moves from pending to done) while making
the re-emitted echo a no-op for sequencing.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

Added test_scan_workflow_events_sequence_ignores_reemitted_completion_echo
in tests/unittests/workflow/utils/test_replay_manager.py, which reproduces
the exact echo pattern from the issue (hitl@1 completes, revise@1
completes, then hitl@1's output is echoed again) and asserts the sequence
stays in the causally-correct order ["hitl@1", "revise@1"].

Verified the new test fails without the fix (git checkout HEAD~1 -- src/google/adk/workflow/utils/_replay_manager.py, rerun, restore):

$ python -m pytest tests/unittests/workflow/utils/test_replay_manager.py -k reemit -q
FAILED ...::test_scan_workflow_events_sequence_ignores_reemitted_completion_echo
AssertionError: assert ['revise@1', 'hitl@1'] == ['hitl@1', 'revise@1']

With the fix:

$ python -m pytest tests/unittests/workflow/utils/test_replay_manager.py -q
....................
20 passed in 1.10s

Full workflow suite:

$ python -m pytest tests/unittests/workflow -q
797 passed, 1 skipped, 5 xfailed, 44 warnings in 12.77s

Full unit test suite (unaffected):

$ python -m pytest tests/unittests -q -n auto
13940 passed, 85 skipped, 27 xfailed, 2 xpassed, 2176 warnings, 24 subtests passed in 220.78s

Manual End-to-End (E2E) Test:

Ran the standalone repro script from the issue (unmodified, google-adk from
this checkout) end to end:

Before the fix — third resume raises the reported RuntimeError and
Publish never runs:

{"phase": 0, "requests": ["review_0"]}
{"phase": 1, "requests": ["review_1"]}
{"phase": 2, "requests": ["review_2"]}
{"exception_type": "RuntimeError", "exception": "Replay divergence detected: Timed out waiting for sequence key 'HITL@1' to be unblocked.", "published": 0}

After the fix — all three reviews are processed and Publish runs exactly
once:

{"phase": 0, "requests": ["review_0"]}
{"phase": 1, "requests": ["review_1"]}
{"phase": 2, "requests": ["review_2"]}
{"phase": 3, "requests": []}

(The script's final assert published == [True] passes; no exception is
raised.)

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

This PR was prepared with AI assistance (Claude Code), including root-cause
analysis via source inspection, the fix, the added test, and this
description. All changes were verified locally as described above before
submission.

…ay sequence

A resumable HITL loop (review -> revise -> review -> ...) can deadlock on a
later resume with "Replay divergence detected: Timed out waiting for
sequence key '<node>@1' to be unblocked."

Workflow._maybe_reemit_replayed_output resurfaces a fast-forwarded node's
output again on each turn so a resumable stream stays complete. That
resurfaced event carries the same node path/run id as the node's original
completion. ReplayManager._scan_sequence treated it as a brand-new
completion and moved the node past whatever ran in between (e.g. its own
downstream child), corrupting the barrier's expected order into a cycle
the next resume can never satisfy.

_scan_sequence now tracks which child run ids have already reached a
genuine terminal completion (output, route, or error) and ignores further
terminal events for that same id, since a completed run id cannot complete
again — a repeat is always this re-emitted echo.

Fixes google#7027
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] ADK 2.8.0: repeated HITL deadlocks on replay within one invocation (no LLM)

2 participants