Skip to content

GH-51547: [C++][R] Reopen IPC dataset reader asynchronously - #51603

Open
gitedmond wants to merge 2 commits into
apache:mainfrom
gitedmond:fix-ipc-dataset-async-reopen
Open

gitedmond wants to merge 2 commits into
apache:mainfrom
gitedmond:fix-ipc-dataset-async-reopen

Conversation

@gitedmond

@gitedmond gitedmond commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Fixes #51547. The IPC dataset scan opens a reader asynchronously, then synchronously reopens it to apply projected fields. The synchronous reopen can wait for footer IO while a generator lock is held, creating the blocking cycle described in the issue.

What changes are included in this PR?

  • Use the existing OpenReaderAsync path for the projected reader reopen.
  • Add a controlled C++ regression test that gates the second reader’s footer read and checks that scan setup and a CPU worker can make progress before that read completes.

Are these changes tested?

  • Before the fix, the new regression failed at the expected blocking assertion in 10/10 runs.
  • After the fix, the identical regression passed in 100/100 runs.
  • The other 34 tests in arrow-dataset-file-ipc-test passed.
  • git diff --check and formatting of changed lines passed.

The reporter’s intermittent R workload was not reproduced here; this test isolates the blocking reader reopen that their stacks identified.

Are there any user-facing changes?

IPC dataset scans no longer block during the projected reader reopen. There is no public API change.

Was AI used for this PR?

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

AI assistance implemented the C++ change and regression test, inspected the code path, and ran the before/after tests. This draft is available for human review.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51547 has been automatically assigned in GitHub to PR creator.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51547 has been automatically assigned in GitHub to PR creator.

@gitedmond
gitedmond marked this pull request as ready for review September 28, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++][R] Deadlock in IPC dataset scan with a small IO thread pool: synchronous reader re-open inside MergedGenerator lock

1 participant