Repository navigation
[CI][C++][Parquet] Failure on parquet-reader-test with simdjson 5.x #51638
Description
Describe the bug, including details regarding any error messages, version, and platform.
Failing jobs: AMD64 Windows MinGW MINGW64 C++, AMD64 Windows MinGW CLANG64 C++, AMD64 Conda C++ AVX2 and
C++ Extra / AMD64 Ubuntu Meson
With simdjson 5.x failing on parquet-reader-test
[ FAILED ] 4 tests, listed below:
[ FAILED ] TestJSONWithLocalFile.JSONOutputWithStatistics
[ FAILED ] TestJSONWithLocalFile.JSONOutput
[ FAILED ] TestJSONWithLocalFile.JSONOutputFLBA
[ FAILED ] TestJSONWithLocalFile.JSONOutputSortColumns
...
The following tests FAILED:
92 - parquet-reader-test (Failed) parquet-tests unittestFirst MinGW job failure on main: https://github.com/apache/arrow/actions/runs/36565311423/job/109395674762#step:13:1885
Because of MSYS2 simdjson update to 5.0.1
mingw-w64-x86_64-simdjson-5.0.1-1-any downloading...
( -- ARROW_SIMDJSON_BUILD_VERSION: v4.6.4 later in the log is probably misleading )
Last successful MinGW job log on main still had
mingw-w64-x86_64-simdjson-4.6.11-1-any downloading...
Component(s)
Continuous Integration, C++
Activity
aryan9948 commentedon Sep 29, 2026
Hi @tadeja, I'd like to investigate this issue. I can look into the MinGW
parquet-reader-testfailures and whether the MSYS2 simdjson 5.0.1 update is causing the regression. Would you be okay with assigning this issue to me?kou commentedon Sep 29, 2026
You can just do it and share what you found here. We can collaborate with it. See also the findings in #51640.
tadeja commentedon Sep 29, 2026
PR #50877 introduced
simdjson::fractured_json_stringand testing exact outputs inreader_test.ccNow simdjson 5.0 changes FracturedJson (simdjson/simdjson#2870, simdjson/simdjson#2878 and simdjson/simdjson#2879 change whitespace, line breaks, key order in tables),
which already causes
parquet-reader-testfailures for MinGW jobs and soon other jobs (probablyARM64 macOS 14 C++andAMD64 Conda C++ AVX2when Docker layer cache gets invalidated.)I have a workaround in #51640 to get MinGW two jobs from 5.0.1 to bundled 4.6.4 simdjson and passing CI for the time being.
Likely this needs a more permanent fix follow-up in
parquet/reader_test.ccto accommodate for both test outputs, 4.6 and 5.0(I have a working demo by 🤖 on my fork tadeja#2)
Fyi, @kou @Reranko05
kou commentedon Sep 30, 2026
How about testing
TestJSONWithLocalFile.JSONOutput*only with simdjson 5 and updating the expected JSONs only for simdjson 5?raulcd commentedon Sep 30, 2026
I think the approach @kou is proposing is reasonable. Either only test and validate the output (as I understand is slightly different in format) on simdjson 5 or update the check so it matches the expected output based on the simdjson version.
@tadeja I am marking this as 26.0.0 even though it does not feel like a blocker is good to have a green CI on the release.
tadeja commentedon Sep 30, 2026
@kou, @raulcd so what would you say about both in #51661 - new expected output for
>= 5(else existing 4.x test output)tadeja commentedon Oct 1, 2026
Two more jobs fail on main:
C++ / AMD64 Conda C++ AVX2
C++ Extra / AMD64 Ubuntu Meson
[-][CI][C++][Parquet] MinGW jobs fail on `parquet-reader-test` with MSYS2 simdjson 5.0.1[/-][+][CI][C++][Parquet] Failure on `parquet-reader-test` with simdjson 5.x[/+]GH-51638: [CI][C++][Parquet] Update expected JSON output in parquet-r…
kou commentedon Oct 1, 2026
Issue resolved by pull request 51661
#51661
GH-51638: [CI][C++][Parquet] Update expected JSON output in parquet-r…