Skip to content

[Data] Release dead actors from the actor pool so they can be replaced - #65543

Open
lonexreb wants to merge 1 commit into
ray-project:masterfrom
lonexreb:fix/62746-actor-pool-dead-actor
Open

[Data] Release dead actors from the actor pool so they can be replaced#65543
lonexreb wants to merge 1 commit into
ray-project:masterfrom
lonexreb:fix/62746-actor-pool-dead-actor

Conversation

@lonexreb

Copy link
Copy Markdown
Contributor

Why are these changes needed?

With a fixed-size actor pool and max_errored_blocks = -1, an actor that dies mid-pipeline (e.g. sys.exit(0) inside the UDF) makes the pipeline hang silently forever (repro from #62746, verified on master: hangs indefinitely; with this PR it completes).

Root cause, as diagnosed in the issue:

  • refresh_actor_state detects the actor as DEAD and _update_rank removes it from the scheduling heaps, so no new tasks are dispatched to it — but the entry stays in _running_actors.
  • The pool's current_size() therefore never drops, so for a fixed-size pool (min_size == max_size) the autoscaler never creates a replacement (default_actor_autoscaler only upscales when current_size() < min_size()).
  • With max_errored_blocks = -1, the failed block is ignored instead of aborting, leaving 0 schedulable actors, a non-empty queue, and no exception.

The fix:

  • refresh_actor_state now releases definitively-DEAD actors from the pool (with a warning log); the autoscaler then scales the pool back up to its min size. Actors whose local state is None (unknown, possibly transient) are left in place, as before.
  • on_task_completed tolerates actors already released with tasks in flight — _release_running_actor has already reconciled the pool's counters for them.

Related issue number

Fixes #62746

Checks

  • I've signed off every commit (DCO).
  • Formatting with the repo-pinned black==22.10.0.
  • Testing strategy (run locally against master via Ray nightly wheel + setup-dev.py symlinks):
    • Issue's repro script (bounded by a watchdog alarm): hangs on master, completes with this PR — dead actors are released ("actor ... is dead; releasing it from the actor pool so it can be replaced") and replaced, and materialize() finishes with the errored blocks dropped.
    • Added test_dead_actor_released_and_replaced_e2e: size-1 pool, UDF exits once via sys.exit(0), asserts the pipeline completes. Passes in ~7s (pre-fix it would hang into the 180s pytest timeout).
    • pytest python/ray/data/tests/test_actor_pool_map_operator.py → 54 passed.

Notes for reviewers / AI-assistance disclosure

@lonexreb
lonexreb requested a review from a team as a code owner August 17, 2026 19:43

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request ensures that dead actors are properly released from the actor pool and replaced by the autoscaler, preventing pipelines from silently stalling. It updates refresh_actor_state to track and release dead actors, handles already-released actors in on_task_completed, and adds an end-to-end test to verify this behavior. There are no review comments, so I have no feedback to provide.

@ray-gardener ray-gardener Bot added data Ray Data-related issues community-contribution Contributed by the community labels Aug 18, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@github-actions github-actions Bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Sep 1, 2026
When an actor died mid-pipeline (e.g. sys.exit(0) inside the UDF),
refresh_actor_state detected it as DEAD and removed it from the
scheduling heaps, but the entry stayed in _running_actors. The pool's
current size therefore never dropped, so for a fixed-size pool the
autoscaler never created a replacement. Combined with
max_errored_blocks=-1, the pipeline hung silently with zero schedulable
actors.

Release definitively-DEAD actors from the pool during
refresh_actor_state; the autoscaler then scales back up to the pool's
min size. Actors whose state is unknown (None) are left in place.
on_task_completed now tolerates actors already released with tasks in
flight, whose counters were reconciled by _release_running_actor.

Fixes ray-project#62746

Signed-off-by: lonexreb <reach2shubhankar@gmail.com>
@lonexreb
lonexreb force-pushed the fix/62746-actor-pool-dead-actor branch from 4809226 to 09d3627 Compare September 4, 2026 06:56
@lonexreb

lonexreb commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Not stale — rebased onto current master; the regression test (test_dead_actor_released_and_replaced_e2e) passes locally in ~7s. Ready for review whenever a maintainer has bandwidth.

@github-actions github-actions Bot added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community data Ray Data-related issues unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Data] Dead Actor occupies actor pool slots which severely effects resource usage

1 participant