Skip to content

Make the loop iterating over multiple parquet files parallel#5486

Merged
jabraham17 merged 1 commit intoBears-R-Us:mainfrom
jabraham17:make-parquet-para
Apr 17, 2026
Merged

Make the loop iterating over multiple parquet files parallel#5486
jabraham17 merged 1 commit intoBears-R-Us:mainfrom
jabraham17:make-parquet-para

Conversation

@jabraham17
Copy link
Copy Markdown
Contributor

This makes the loop over multiple parquet files parallel

This PR is the same as #4952, except it fixes the bug exposed by that PR. CPtrsToData and CPtrsToWhereNulls need to be local for each parallel task, so they must be declared inside of the forall loop. Technically, they will be instantiated more than they really need to be, but it is correct.

Resolves #4952

Signed-off-by: Jade Abraham <jade.abraham@hpe.com>
@bradcray
Copy link
Copy Markdown
Contributor

@jabraham17 : Belated thought: Can/should this be propagated to the standalone Parquet package? (Am I correct in remembering that it lives here: https://github.com/e-kayrakli/Parquet ?).

@jabraham17
Copy link
Copy Markdown
Contributor Author

Belated thought: Can/should this be propagated to the standalone Parquet package?

No idea. ParquetMsg and the standalone Parquet module have diverged enough that its not immediately obvious. Note the Parquet mason package does not even use c_readAllCols, so may not suffer from this issue at all.

@bradcray
Copy link
Copy Markdown
Contributor

OK, thanks. Tagging @e-kayrakli and @benharsh to see if they know whether it has alternate/equivalent functionality or whether it's focused only on features of use to FastOTF2 so is missing other features we'd want for Arkouda.

Merged via the queue into Bears-R-Us:main with commit dce396d Apr 17, 2026
19 checks passed
@benharsh
Copy link
Copy Markdown
Collaborator

Right now the package is focused on features to be used in FastOTF2, and is not at a point where we could replace the Arkouda implementation.

@jabraham17 jabraham17 deleted the make-parquet-para branch April 17, 2026 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants