Android: inproc stress test deadlocks (API 24-27)

[jpnurmi]

Random CI hangs occurred while setting up the Android API 26 CI job in #1574 / #1572. I was able to reproduce the deadlock locally using API 24-27 emulators.

test:

$ ANDROID_API=26 pytest -v -s tests/test_inproc_stress.py
[...]
tests/test_inproc_stress.py::test_inproc_concurrent_crash_repeated[1]
[...]
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /private/var/folders/91/ctzy99m94bb6v2k1_37l5x140000gn/T/pytest-of-jpnurmi/pytest-390/cmake0/inproc_stress_build
[100%] Built target inproc_stress_test

process status:

$ adb shell ps -A | grep inproc
shell        14164 14162   34032   3628 futex_wait_queue_me 70d30bd7f0 S inproc_stress_test

stacktrace:

     * thread #1
       * frame #0: 0x00000070d30bd7ec libc.so`syscall + 28
         frame #1: 0x00000070d3107d1c libc.so`pthread_join + 212
         frame #2: 0x00000070d33b3078 inproc_stress_test`run_concurrent_crash at concurrent_crash.c:114:9
         frame #3: 0x00000070d33b28c4 inproc_stress_test`test_concurrent_crash(database_path="/data/local/tmp/.sentry-native-1") at main.c:236:5
         frame #4: 0x00000070d33b27a0 inproc_stress_test`main(argc=3, argv=0x0000007ff56a74a8) at main.c:475:16
         frame #5: 0x00000070d3141440 libc.so`__libc_init + 92
         frame #6: 0x00000070d33b25ec inproc_stress_test`_start_main + 88
       thread #2
         frame #0: 0x00000070d310a810 libc.so`read + 4
         frame #1: 0x00000070d2f15e48 libsentry.so`handler_thread_main [inlined] read(fd=<unavailable>, buf=0x00000070d29b4444, (null)=1, count=1) at unistd.h:190:12
         frame #2: 0x00000070d2f15e44 libsentry.so`handler_thread_main(UNUSED_data=<unavailable>) at sentry_backend_inproc.c:1214:22
         frame #3: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #4: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #3
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #4
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #5
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #6
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #7
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #8
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #9
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #10
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #11
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #12
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #13
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #14
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #15
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #16
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #17
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #18
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #19
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d3242690
         frame #4: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #5: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #6: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #7: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #8: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #20
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #21
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72
       thread #22
         frame #0: 0x00000070d32e0f3c linker64`__dl_syscall + 28
         frame #1: 0x00000070d32efe88 linker64`__dl__ZL33__pthread_mutex_lock_with_timeoutP24pthread_mutex_internal_tbPK8timespec + 268
         frame #2: 0x00000070d326bb78 linker64`__dl__ZL24debuggerd_signal_handleriP7siginfoPv + 332
         frame #3: 0x00000070d2f15d60 libsentry.so`handle_signal(signum=<unavailable>, info=<unavailable>, user_context=<unavailable>) at sentry_backend_inproc.c:1661:5
         frame #4: 0x00000070d3242690
         frame #5: 0x00000070d33b2f68 inproc_stress_test`do_crash [inlined] memset(s=0x0000000000000001, (null)=<unavailable>, c=<unavailable>, n=<unavailable>) at string.h:138:12
         frame #6: 0x00000070d33b2f54 inproc_stress_test`do_crash at concurrent_crash.c:43:5
         frame #7: 0x00000070d33b30cc inproc_stress_test`crash_thread(param=0x0000000000000000) at concurrent_crash.c:90:5
         frame #8: 0x00000070d3107754 libc.so`__pthread_start(void*) + 40
         frame #9: 0x00000070d30bee38 libc.so`__start_thread + 72

[linear-code]

NATIVE-173

[jpnurmi]

AI-assisted analysis:

Summary

When multiple threads crash simultaneously on Android API 24–27, the process hangs indefinitely instead of terminating. The deadlock occurs in bionic's debuggerd_signal_handler, which uses a non-reentrant mutex that cannot handle concurrent entry from multiple threads.

Root Cause

After the inproc backend processes a crash, all crashing threads fall through to cleanup: in process_ucontext() (src/backends/sentry_backend_inproc.c), where they call reset_signal_handlers() and invoke_signal_handler(). This causes all threads to enter bionic's debuggerd_signal_handler simultaneously, deadlocking on its internal crash_mutex.

The deadlock has two contributing factors:

1. Mutex contention: debuggerd_signal_handler acquires crash_mutex and intentionally never releases it for fatal signals. Other threads block on pthread_mutex_lock forever.

2. Race condition: The winning thread's debuggerd_signal_handler calls signal(SIGSEGV, SIG_DFL) before re-raising, but a losing thread's reset_signal_handlers() can re-install debuggerd_signal_handler via sigaction() in between, causing the winner to re-enter and deadlock on its own mutex.

Our SA_NODEFER flag and empty sa_mask exacerbate the issue — debuggerd_signal_handler assumes all signals are blocked during its execution (bionic registers it with sigfillset(&action.sa_mask)), but our handler replaces that mask.

Fix Proposal

Only the winning thread (the one that won the CAS IDLE→HANDLING) should call reset_signal_handlers() and invoke_signal_handler(). Losing threads should set signal(signum, SIG_DFL) and return, so their re-fault hits the default handler instead of debuggerd_signal_handler.

Notes

It's unclear why the deadlock doesn't reproduce on API 28+. The debuggerd_signal_handler source code between API 27 and 28 shows no obvious fix for this issue — both versions use the same crash_mutex with the same "don't unlock for fatal signals" strategy. strace comparison of successful runs shows identical rt_sigaction sequences, with the only visible difference being one fewer signal(SIGSEGV, SIG_DFL) call on API 28. The race window may simply be narrower on API 28 rather than eliminated.

[supervacuus]

Thanks for raising @jpnurmi. The analysis is extremely long-winded for essentially a clean-up bypass. The interactions vs the debuggerd_signal_handler sound a lot like LSD consumption. But yeah, no reason to re-raise specifically from within our handler (however, it is absolutely possible that those crashes hit again, after the return, which is also why I do not buy that sketched interaction model).