Make the loop iterating over multiple parquet files parallel#5486
Make the loop iterating over multiple parquet files parallel#5486jabraham17 merged 1 commit intoBears-R-Us:mainfrom
Conversation
Signed-off-by: Jade Abraham <jade.abraham@hpe.com>
|
@jabraham17 : Belated thought: Can/should this be propagated to the standalone Parquet package? (Am I correct in remembering that it lives here: https://github.com/e-kayrakli/Parquet ?). |
No idea. ParquetMsg and the standalone Parquet module have diverged enough that its not immediately obvious. Note the Parquet mason package does not even use c_readAllCols, so may not suffer from this issue at all. |
|
OK, thanks. Tagging @e-kayrakli and @benharsh to see if they know whether it has alternate/equivalent functionality or whether it's focused only on features of use to FastOTF2 so is missing other features we'd want for Arkouda. |
|
Right now the package is focused on features to be used in FastOTF2, and is not at a point where we could replace the Arkouda implementation. |
This makes the loop over multiple parquet files parallel
This PR is the same as #4952, except it fixes the bug exposed by that PR.
CPtrsToDataandCPtrsToWhereNullsneed to be local for each parallel task, so they must be declared inside of the forall loop. Technically, they will be instantiated more than they really need to be, but it is correct.Resolves #4952