feat(worker): Add metrics for tasks lost during shutdown - #791
Open
evanh wants to merge 1 commit into
Open
Conversation
Shutdown drops tasks in three places and none of them are measurable today: tasks queued in `child_tasks` are discarded, children still running at the join timeout are SIGKILLed without producing a result, and the final result drain drops a batch on its first RPC error rather than requeueing it. Adds counters for each, a joined/timeout outcome for the result thread, and per-stage durations emitted as each stage completes so a pod killed part way through still reports how far it got. No behaviour change beyond emptying `child_tasks` at the end of shutdown, which was already discarded when the process exited. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Worker shutdown drops tasks in three places and none of them are measurable today:
child_tasksare discarded — nothing drains that queueWORKER_CHILD_JOIN_TIMEOUT_SECare SIGKILLed without producing a resultAnything lost this way stays
processingon the broker until its deadline lapses, which burns a processing attempt;at_most_oncetasks are failed outright rather than retried.This adds instrumentation only, so the loss can be sized before any behaviour changes.
Metrics
taskworker.worker.shutdown.child_tasks_discardedtaskworker.worker.shutdown.children_killedtaskworker.worker.shutdown.results_drainedtaskworker.worker.shutdown.results_droppedtaskworker.worker.shutdown.result_threadjoinedvstimeouttaskworker.worker.shutdown.stage_durationstagetaskworker.worker.shutdown.durationAll tagged with
processing_pool. Counters are emitted even at zero so dashboards can tell "nothing lost" from "no data".Stage durations are emitted as each stage completes rather than once at the end, so a pod that is SIGKILLed part way through still reports how far it got. They also give the numbers needed to size the pod termination grace period.
Behaviour change
One, and it is a no-op in effect:
shutdown()now emptieschild_tasksin order to count it. Nothing consumed that queue once the children were gone, so those tasks were already being discarded when the process exited.child_tasks_discardedis a lower bound —mp.Queue.get_nowait()also raisesEmptywhile items sit unflushed in the pipe.Testing
8 new tests in
tests/worker/test_worker.pycovering each counter, the drop-vs-requeue split between draining and normal operation, stage ordering, and the result-thread outcome. Full client suite passes (tests/test_metrics.py::test_track_memory_usagefails identically with and without this change — a pre-existing RSS-delta flake under full-suite conditions).🤖 Generated with Claude Code