Skip to content

GH-46421: [C++][Acero] Asofjoin respect PauseProducing from downstream. - #46140

Draft
gitmodimo wants to merge 21 commits into
apache:mainfrom
gitmodimo:asof_join_pause
Draft

GH-46421: [C++][Acero] Asofjoin respect PauseProducing from downstream.#46140
gitmodimo wants to merge 21 commits into
apache:mainfrom
gitmodimo:asof_join_pause

Conversation

@gitmodimo

@gitmodimo gitmodimo commented Apr 14, 2025

Copy link
Copy Markdown
Contributor

Rationale for this change

Current implementation of asof join does not respect backpressure from downstream nodes. PauseProducing implementation is empty. This leads to unbound memory consumption with slow sinks.

What changes are included in this PR?

With this change PauseProducing causes ProcessThread to pause when backpressure is applied. Pause logic stolen from source_node and seems to work.

Inputs do not require additional handling sice each input has its own backpressure queue.

Are these changes tested?

Tested in my application. Still deciding how to test it. Either by checking memory consumption or by checking whether the pause is propagating upstream? The latter seems easier.
Yes

Are there any user-facing changes?

No

PS
Also I did not modify the single threaded version. Can someone using it chime in?

@gitmodimo
gitmodimo requested a review from westonpace as a code owner April 14, 2025 18:57
@github-actions

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@gitmodimo gitmodimo changed the title [C++][Acero] Asofjoin respect PauseProducing from downstream. GH-46421: [C++][Acero] Asofjoin respect PauseProducing from downstream. May 13, 2025
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #46421 has been automatically assigned in GitHub to PR creator.

@gitmodimo

Copy link
Copy Markdown
Contributor Author

I merged @mroz45 tests for this use case.

Rafał Hibner added 2 commits May 27, 2025 10:23
@gitmodimo
gitmodimo marked this pull request as draft May 27, 2025 10:26
@gitmodimo
gitmodimo marked this pull request as ready for review July 4, 2025 09:27
@github-actions

Copy link
Copy Markdown

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.

@github-actions github-actions Bot added the Status: stale-warning Issues and PRs flagged as stale which are due to be closed if no indication otherwise label Aug 26, 2026
@github-actions github-actions Bot removed the Status: stale-warning Issues and PRs flagged as stale which are due to be closed if no indication otherwise label Aug 27, 2026
@gitmodimo
gitmodimo marked this pull request as draft August 29, 2026 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants