Skip to content

[C++] Sporadic test errors in parallel mode #51095

Closed
Closed
[C++] Sporadic test errors in parallel mode#51095
Assignees
hiroyuki-sato
Labels
Component: C++Type: bug
Milestone
26.0.0

Description

@pitrou
pitrou
opened on Aug 31, 2026

Describe the bug, including details regarding any error messages, version, and platform.

I run C++ tests in parallel (26 threads) and I have started to get random sporadic errors like this:

76/97 Test #13: arrow-gtest-util-test ........................***Failed    0.15 sec
Running arrow-gtest-util-test, redirecting output into /build/build-test/build/test-logs/arrow-gtest-util-test.txt (attempt 1/1)
Running main() from gmock_main.cc
[==========] Running 8 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from TestAssertContainsNaN
[ RUN      ] TestAssertContainsNaN.BatchesEqual
[       OK ] TestAssertContainsNaN.BatchesEqual (1 ms)
[ RUN      ] TestAssertContainsNaN.TableEqual
[       OK ] TestAssertContainsNaN.TableEqual (0 ms)
[ RUN      ] TestAssertContainsNaN.ArrayEqual
[       OK ] TestAssertContainsNaN.ArrayEqual (0 ms)
[ RUN      ] TestAssertContainsNaN.ChunkedEqual
[       OK ] TestAssertContainsNaN.ChunkedEqual (0 ms)
[ RUN      ] TestAssertContainsNaN.DatumEqual
[       OK ] TestAssertContainsNaN.DatumEqual (0 ms)
[----------] 5 tests from TestAssertContainsNaN (2 ms total)

[----------] 2 tests from TestTensorFromJSON
[ RUN      ] TestTensorFromJSON.FromJSONAndArray
[       OK ] TestTensorFromJSON.FromJSONAndArray (0 ms)
[ RUN      ] TestTensorFromJSON.FromJSON
[       OK ] TestTensorFromJSON.FromJSON (0 ms)
[----------] 2 tests from TestTensorFromJSON (0 ms total)

[----------] 1 test from AssertTestWithinUlp
[ RUN      ] AssertTestWithinUlp.Basics
[       OK ] AssertTestWithinUlp.Basics (0 ms)
[----------] 1 test from AssertTestWithinUlp (0 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 3 test suites ran. (3 ms total)
[  PASSED  ] 8 tests.
find: ‘/tmp/s3fs-test-7cm44otu’: Aucun fichier ou dossier de ce nom

Component(s)

C++

Activity

  1. pitrou
    added
    Type: bug
    on Aug 31, 2026
  2. github-actions
    added
    Component: C++
    on Aug 31, 2026

  3. pitrou commented on Aug 31, 2026

    @pitrou
    pitrou
    on Aug 31, 2026
    MemberAuthor

    I think this is due to the use of GNU find in #50935 . @kou @hiroyuki-sato

  4. pitrou commented on Aug 31, 2026

    @pitrou
    pitrou
    on Aug 31, 2026
    MemberAuthor

    It seems to happen in print_coredumps. I've tried passing -ignore_readdir_race but it doesn't seem to change everything.

    Edit: the patch I've tried:

    diff --git a/cpp/build-support/run-test.sh b/cpp/build-support/run-test.sh
index 2dc18860a1..9f2828a9d9 100755
--- a/cpp/build-support/run-test.sh
+++ b/cpp/build-support/run-test.sh
@@ -141,7 +141,7 @@ function print_coredumps() {
   FILENAME=$(basename "${TEST_EXECUTABLE}")
   FILENAME=$(echo "${FILENAME}" | cut -c-15)
 
-  COREFILES=$(find /tmp -maxdepth 1 -type f -name "core.${FILENAME}*" -exec basename {} \;)
+  COREFILES=$(find /tmp -ignore_readdir_race -maxdepth 1 -type f -name "core.${FILENAME}*" -exec basename {} \;)
   if [ -n "$COREFILES" ]; then
     for COREFILE in $COREFILES; do
       COREPATH="/tmp/${COREFILE}"
@@ -199,12 +199,12 @@ for ATTEMPT_NUMBER in $(seq 1 "$TEST_EXECUTION_ATTEMPTS") ; do
     # contents before and after the test runs, and deleting anything new.
     #
     # The comm program requires that its two inputs be sorted.
-    TEST_TMPDIR_BEFORE=$(find "$TEST_TMPDIR" -maxdepth 1 -type d | sort)
+    TEST_TMPDIR_BEFORE=$(find "$TEST_TMPDIR" -ignore_readdir_race -maxdepth 1 -type d | sort)
   fi
 
   if [ "$ATTEMPT_NUMBER" -lt "$TEST_EXECUTION_ATTEMPTS" ]; then
     # Now delete any new test output.
-    TEST_TMPDIR_AFTER=$(find "$TEST_TMPDIR" -maxdepth 1 -type d | sort)
+    TEST_TMPDIR_AFTER=$(find "$TEST_TMPDIR" -ignore_readdir_race -maxdepth 1 -type d | sort)
     DIFF=$(comm -13 <(echo "$TEST_TMPDIR_BEFORE") \
                     <(echo "$TEST_TMPDIR_AFTER"))
     for DIR in $DIFF; do

  5. pitrou commented on Aug 31, 2026

    @pitrou
    pitrou
    on Aug 31, 2026
    MemberAuthor

    What we could do perhaps is only print coredumps on error. This would not eliminate a potential race condition in find, but would make it much less likely:

    diff --git a/cpp/build-support/run-test.sh b/cpp/build-support/run-test.sh
index 2dc18860a1..c618e90c98 100755
--- a/cpp/build-support/run-test.sh
+++ b/cpp/build-support/run-test.sh
@@ -229,18 +229,16 @@ for ATTEMPT_NUMBER in $(seq 1 "$TEST_EXECUTION_ATTEMPTS") ; do
   fi
   if [ "$STATUS" -eq "0" ]; then
     break
-  elif [ "$ATTEMPT_NUMBER" -lt "$TEST_EXECUTION_ATTEMPTS" ]; then
-    echo Test failed attempt number "$ATTEMPT_NUMBER"
-    echo Will retry...
   fi
+  print_coredumps
+  echo Test failed attempt number "$ATTEMPT_NUMBER"
+  echo Will retry...
 done
 
 if [ "$RUN_TYPE" = "test" ]; then
   post_process_tests
 fi
 
-print_coredumps
-
 popd
 rm -Rf "$TEST_WORKDIR"

  6. kou commented on Sep 1, 2026

    @kou
    kou
    on Sep 1, 2026
    Member

    Can we close this because we merged #51110 ?

  7. pitrou commented on Sep 1, 2026

    @pitrou
    pitrou
    on Sep 1, 2026
    MemberAuthor

    Can we close this because we merged #51110 ?

    That PR doesn't fix the issue, I still see error occasionally.

    [----------] Global test environment tear-down
[==========] 21 tests from 7 test suites ran. (127 ms total)
[  PASSED  ] 21 tests.
find: ‘/tmp/buffered-test-0gzee19x’: Aucun fichier ou dossier de ce nom

    The problem is in print_coredumps, which is still there.

  8. pitrou commented on Sep 1, 2026

    @pitrou
    pitrou
    on Sep 1, 2026
    MemberAuthor

    This patch avoids the problem if all tests pass successfully:

    diff --git a/cpp/build-support/run-test.sh b/cpp/build-support/run-test.sh
index ae39bef671..e8e1629104 100755
--- a/cpp/build-support/run-test.sh
+++ b/cpp/build-support/run-test.sh
@@ -200,7 +200,9 @@ if [ "$RUN_TYPE" = "test" ]; then
   post_process_tests
 fi
 
-print_coredumps
+if [ "$STATUS" -ne "0" ]; then
+  print_coredumps
+fi
 
 popd
 rm -Rf "$TEST_WORKDIR"

    Would you be ok with this @kou ?

  9. github-actions
    assigned
    hiroyuki-sato
    on Sep 1, 2026

  10. hiroyuki-sato commented on Sep 1, 2026

    @hiroyuki-sato
    hiroyuki-sato
    on Sep 1, 2026
    Collaborator

    @pitrou

    In #50934, find was used instead of (ls /tmp | grep $PATTERN) to find core files in a ShellCheck-safe way, but the fix was incomplete. Could you try #51121?

  11. hiroyuki-sato commented on Sep 1, 2026

    @hiroyuki-sato
    hiroyuki-sato
    on Sep 1, 2026
    Collaborator

    @pitrou
    By the way, is this error from a script before #51110 was merged?
    After #51110 was merged, the only place in run-test.sh that uses find is the part that looks for core files.
    #51095 (comment)

  12. pitrou commented on Sep 1, 2026

    @pitrou
    pitrou
    on Sep 1, 2026
    MemberAuthor

    @hiroyuki-sato Have you read #51095 (comment) ?

  13. hiroyuki-sato commented on Sep 1, 2026

    @hiroyuki-sato
    hiroyuki-sato
    on Sep 1, 2026
    Collaborator

    @pitrou Thank you for your comment.

    In the latest run-test.sh at commit a769c29, this is the only place where find is used:

    grep find cpp/build-support/run-test.sh
  COREFILES=$(find /tmp -maxdepth 1 -type f -name "core.${FILENAME}*" -exec basename {} \;)

    Since the error below appears to come from the find command, I asked because I still haven't been able to identify exactly where the error is coming from.

    find: ‘/tmp/buffered-test-0gzee19x’: Aucun fichier ou dossier de ce nom

    For example, this is the kind of error you get when running a command like this:

    find /tmp/buffered-test-0gzee19x -type d -print
    find: /tmp/buffered-test-0gzee19x: No such file or directory

  14. pitrou commented on Sep 1, 2026

    @pitrou
    pitrou
    on Sep 1, 2026
    MemberAuthor

    Since the error below appears to come from the find command, I asked because I still haven't been able to identify exactly where the error is coming from.

    As I said, it comes from the find in print_coredumps. This command iterates over directory entries in /tmp and then filters them using -type f -name "core.${FILENAME}*". Perhaps this involves a stat call that fails because the directory entry has vanished in the meantime.

  15. hiroyuki-sato commented on Sep 1, 2026

    @hiroyuki-sato
    hiroyuki-sato
    on Sep 1, 2026
    Collaborator

    @pitrou Thank you. I understand the issue now. It seems better to use globbing instead of find, as shown below, so I’ll update it accordingly.

    COREFILES=(/tmp/"core.${FILENAME}"*)

  16. hiroyuki-sato
    added 4 commits that reference this issue on Sep 2, 2026

    Revert "apacheGH-51095: [CI][C++] Fix core file detection in run-test…

    e69e3e0

    apacheGH-51095: [CI][C++] Fix core file detection in run-test.sh

    067cac6

    apacheGH-51095: [CI][C++] Fix core file detection in run-test.sh

    ce8a351

    apacheGH-51095: [CI][C++] Fix core file detection in run-test.sh

    997a160
  17. kou
    added a commit that references this issue on Sep 2, 2026

    GH-51095: [CI][C++] Fix core file detection in run-test.sh (#51121)

    a62870c
  18. kou
    added this to the 26.0.0 milestone on Sep 2, 2026

  19. kou commented on Sep 2, 2026

    @kou
    kou
    on Sep 2, 2026
    Member

    Issue resolved by pull request 51121
    #51121

  20. kou
    closed this as completedon Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Component: C++Type: bug

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions