GH-46421: [C++][Acero] Asofjoin respect PauseProducing from downstream. - #46140
GH-46421: [C++][Acero] Asofjoin respect PauseProducing from downstream.#46140gitmodimo wants to merge 21 commits into
Conversation
|
Thanks for opening a pull request! If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or See also: |
|
|
1ffdb3d to
cb14b54
Compare
|
I merged @mroz45 tests for this use case. |
…into asof_join_pause
|
Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer. |
Rationale for this change
Current implementation of asof join does not respect backpressure from downstream nodes. PauseProducing implementation is empty. This leads to unbound memory consumption with slow sinks.
What changes are included in this PR?
With this change PauseProducing causes ProcessThread to pause when backpressure is applied. Pause logic stolen from source_node and seems to work.
Inputs do not require additional handling sice each input has its own backpressure queue.
Are these changes tested?
Tested in my application. Still deciding how to test it. Either by checking memory consumption or by checking whether the pause is propagating upstream? The latter seems easier.Yes
Are there any user-facing changes?
No
PS
Also I did not modify the single threaded version. Can someone using it chime in?