fix(workflow): stop a re-emitted output echo from reordering the replay sequence - #7028
Open
chelsealong wants to merge 1 commit into
Open
fix(workflow): stop a re-emitted output echo from reordering the replay sequence#7028chelsealong wants to merge 1 commit into
chelsealong wants to merge 1 commit into
Conversation
…ay sequence A resumable HITL loop (review -> revise -> review -> ...) can deadlock on a later resume with "Replay divergence detected: Timed out waiting for sequence key '<node>@1' to be unblocked." Workflow._maybe_reemit_replayed_output resurfaces a fast-forwarded node's output again on each turn so a resumable stream stays complete. That resurfaced event carries the same node path/run id as the node's original completion. ReplayManager._scan_sequence treated it as a brand-new completion and moved the node past whatever ran in between (e.g. its own downstream child), corrupting the barrier's expected order into a cycle the next resume can never satisfy. _scan_sequence now tracks which child run ids have already reached a genuine terminal completion (output, route, or error) and ignores further terminal events for that same id, since a completed run id cannot complete again — a repeat is always this re-emitted echo. Fixes google#7027
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
Problem:
On a resumable
Workflowrunning the documented review/revise HITL loop(review → revise → review → ... → approve), a third resume can deadlock with:
Root cause:
Workflow._maybe_reemit_replayed_outputre-surfaces afast-forwarded node's output again on every turn it is replayed through, so a
resumable event stream stays complete for that turn. That re-emitted event
carries the same node path / run id (
HITL@1) as the node's original,genuine completion.
ReplayManager._scan_sequencebuilds the replay barrier's expectedcompletion order by scanning terminal events and, for a segment it has
already seen, removing the old position and appending the new one — logic
meant to update a node's position when it moves from "interrupted" to
"completed". Applied to the re-emitted echo, it instead moves
HITL@1pastRevise@1, even thoughRevise@1can only ever run afterHITL@1completes. On the next resume this produces a corrupted barrier sequence
(
[Revise@1, HITL@1, ...]) that creates a real cycle:HITL@1'sfast-forward path waits for
Revise@1's turn, butRevise@1can't even bescheduled until
HITL@1's (fast-forwarded) task returns — a deadlock thatonly times out after 15s.
Solution:
_scan_sequencenow tracks which child run ids have already reached agenuine terminal completion (an event carrying
output,route, orerror_code). Once a run id is marked completed, any further terminal eventfor that same id is ignored for ordering purposes, since a completed run id
cannot complete a second time — a repeat is always the re-emitted echo, not
new information. This preserves the legitimate "interrupted → completed"
reposition (the first time a node moves from pending to done) while making
the re-emitted echo a no-op for sequencing.
Testing Plan
Unit Tests:
Added
test_scan_workflow_events_sequence_ignores_reemitted_completion_echoin
tests/unittests/workflow/utils/test_replay_manager.py, which reproducesthe exact echo pattern from the issue (
hitl@1completes,revise@1completes, then
hitl@1's output is echoed again) and asserts the sequencestays in the causally-correct order
["hitl@1", "revise@1"].Verified the new test fails without the fix (
git checkout HEAD~1 -- src/google/adk/workflow/utils/_replay_manager.py, rerun, restore):With the fix:
Full workflow suite:
Full unit test suite (unaffected):
Manual End-to-End (E2E) Test:
Ran the standalone repro script from the issue (unmodified,
google-adkfromthis checkout) end to end:
Before the fix — third resume raises the reported
RuntimeErrorandPublishnever runs:After the fix — all three reviews are processed and
Publishruns exactlyonce:
(The script's final
assert published == [True]passes; no exception israised.)
Checklist
Additional context
This PR was prepared with AI assistance (Claude Code), including root-cause
analysis via source inspection, the fix, the added test, and this
description. All changes were verified locally as described above before
submission.