fix(server): report why a tokenizer/detokenizer worker died during load - #269
Open
sime2408 wants to merge 1 commit into
Open
fix(server): report why a tokenizer/detokenizer worker died during load#269sime2408 wants to merge 1 commit into
sime2408 wants to merge 1 commit into
Conversation
_run_scheduler wraps startup and pushes an ("error", reason) ack before dying,
so the supervisor can report the real cause. _run_tokenize_worker did not, so
every tokenizer-side startup failure reached the parent only as a dead process
and surfaced as the generic "backend worker freetoken-detokenizer-0 exited
during load" — the actionable reason (unreadable tokenizer.json, missing
checkpoint file, ZMQ bind clash) was lost.
Wrap tokenize_worker in the same try/report/re-raise the scheduler already uses.
The traceback still prints and the process still exits non-zero; only the ack is
added. No ack_queue (direct call) stays a plain propagate.
Adds tests/server/test_launch_worker_errors.py, which fails before this change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_run_scheduler wraps startup and pushes an ("error", reason) ack before dying, so the supervisor can report the real cause. _run_tokenize_worker did not, so every tokenizer-side startup failure reached the parent only as a dead process and surfaced as the generic "backend worker freetoken-detokenizer-0 exited during load" — the actionable reason (unreadable tokenizer.json, missing checkpoint file, ZMQ bind clash) was lost.
Wrap tokenize_worker in the same try/report/re-raise the scheduler already uses. The traceback still prints and the process still exits non-zero; only the ack is added. No ack_queue (direct call) stays a plain propagate.
Adds tests/server/test_launch_worker_errors.py, which fails before this change.