[Data] Release dead actors from the actor pool so they can be replaced - #65543
[Data] Release dead actors from the actor pool so they can be replaced#65543lonexreb wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Code Review
This pull request ensures that dead actors are properly released from the actor pool and replaced by the autoscaler, preventing pipelines from silently stalling. It updates refresh_actor_state to track and release dead actors, handles already-released actors in on_task_completed, and adds an end-to-end test to verify this behavior. There are no review comments, so I have no feedback to provide.
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
When an actor died mid-pipeline (e.g. sys.exit(0) inside the UDF), refresh_actor_state detected it as DEAD and removed it from the scheduling heaps, but the entry stayed in _running_actors. The pool's current size therefore never dropped, so for a fixed-size pool the autoscaler never created a replacement. Combined with max_errored_blocks=-1, the pipeline hung silently with zero schedulable actors. Release definitively-DEAD actors from the pool during refresh_actor_state; the autoscaler then scales back up to the pool's min size. Actors whose state is unknown (None) are left in place. on_task_completed now tolerates actors already released with tasks in flight, whose counters were reconciled by _release_running_actor. Fixes ray-project#62746 Signed-off-by: lonexreb <reach2shubhankar@gmail.com>
4809226 to
09d3627
Compare
|
Not stale — rebased onto current master; the regression test ( |
Why are these changes needed?
With a fixed-size actor pool and
max_errored_blocks = -1, an actor that dies mid-pipeline (e.g.sys.exit(0)inside the UDF) makes the pipeline hang silently forever (repro from #62746, verified on master: hangs indefinitely; with this PR it completes).Root cause, as diagnosed in the issue:
refresh_actor_statedetects the actor asDEADand_update_rankremoves it from the scheduling heaps, so no new tasks are dispatched to it — but the entry stays in_running_actors.current_size()therefore never drops, so for a fixed-size pool (min_size == max_size) the autoscaler never creates a replacement (default_actor_autoscaleronly upscales whencurrent_size() < min_size()).max_errored_blocks = -1, the failed block is ignored instead of aborting, leaving 0 schedulable actors, a non-empty queue, and no exception.The fix:
refresh_actor_statenow releases definitively-DEADactors from the pool (with a warning log); the autoscaler then scales the pool back up to its min size. Actors whose local state isNone(unknown, possibly transient) are left in place, as before.on_task_completedtolerates actors already released with tasks in flight —_release_running_actorhas already reconciled the pool's counters for them.Related issue number
Fixes #62746
Checks
black==22.10.0.setup-dev.pysymlinks):materialize()finishes with the errored blocks dropped.test_dead_actor_released_and_replaced_e2e: size-1 pool, UDF exits once viasys.exit(0), asserts the pipeline completes. Passes in ~7s (pre-fix it would hang into the 180s pytest timeout).pytest python/ray/data/tests/test_actor_pool_map_operator.py→ 54 passed.Notes for reviewers / AI-assistance disclosure