Skip to content

[C++] Hang in ThreadPool::Shutdown() after failure to spawn a worker thread #48137

Closed
Closed
[C++] Hang in ThreadPool::Shutdown() after failure to spawn a worker thread#48137
Assignees
advitrocks9
Labels
Component: C++Type: bug
Milestone
26.0.0

Description

@Pear0
Pear0
opened on Nov 14, 2025

Describe the bug, including details regarding any error messages, version, and platform.

I have a case where my C++ application that uses arrow hangs on shutdown after failing to construct a thread for the IOThreadPool.

Expected behavior: clean shutdown after catching the exception or some kind of fatal condition
Actual behavior: hang on shutdown after catching the exception
Platform: Linux x86_64
Arrow Version: 19.0.1 (but based on reading code I believe this to still be an issue on main)

It is pretty easy to trigger std::thread::thread() to throw by setting RLIMIT_NPROC to something extreme like 1.

If std::thread::thread() throws in ThreadPool::LaunchWorkersUnlocked() there is a state_->workers entry even though the thread was never spawned. It is the thread's responsibility to remove the list entry when it exits so ThreadPool::Shutdown() never reaches its state_->workers_.empty() condition.

This is usually not noticeable because an abnormal shutdown will not call the IOThreadPool's destructor. In my case, I have a C++ binary that catches the exception then exits "normally" with an error message and exit code. This does call the ThreadPool destructor leading to the hang.

Hopefully this is clear. If it is not, I could put together a repro example.

Component(s)

C++

Activity

  1. Pear0
    added
    Type: bug
    on Nov 14, 2025
  2. github-actions
    added
    Component: C++
    on Nov 14, 2025
  3. advitrocks9
    added a commit that references this issue on Sep 2, 2026

    apacheGH-48137: [C++] Return a Status when a worker thread fails to s…

    0fa803a
  4. pitrou
    added a commit that references this issue on Sep 3, 2026

    GH-48137: [C++] Restore ThreadPool state when a worker fails to start (…

    157575a
  5. pitrou
    added this to the 26.0.0 milestone on Sep 3, 2026

  6. pitrou commented on Sep 3, 2026

    @pitrou
    pitrou
    on Sep 3, 2026
    Member

    Issue resolved by pull request 51107
    #51107

  7. pitrou
    closed this as completedon Sep 3, 2026

  8. pitrou commented on Sep 3, 2026

    @pitrou
    pitrou
    on Sep 3, 2026
    Member

    @advitrocks9 Can you post here so that we assign this issue to you?

  9. advitrocks9 commented on Sep 3, 2026

    @advitrocks9
    advitrocks9
    on Sep 3, 2026
    Contributor

    Fixed in #51107

  10. pitrou
    assigned
    advitrocks9
    on Sep 3, 2026
  11. whr778
    added a commit that references this issue on Sep 30, 2026

    fetch_fineweb: exit without C++ teardown; pyarrow 25.0.1 deadlocks th…

    1e6f33b

  12. whr778 commented on Sep 30, 2026

    @whr778
    whr778
    on Sep 30, 2026

    We hit this on pyarrow 25.0.1 (macOS arm64, Python 3.12) through Hugging Face datasets streaming parquet: ~176 configs read concurrently from 8 Python threads. Every file is written and main() returns, but the process never exits.

    A native sample of the stuck process shows:

    exit (libsystem_c)
  __cxa_finalize_ranges
    std::shared_ptr<arrow::internal::ThreadPool>::~shared_ptr()
      arrow::internal::ThreadPool::~ThreadPool()
        arrow::internal::ThreadPool::Shutdown(bool)
          std::condition_variable::wait

    with 8 pool workers (ThreadPool::LaunchWorkersUnlocked threads) idle in condition_variable::wait, at 0% CPU, indefinitely. It reproduced 3/3 on 25.0.1 and does not reproduce on 26.0.0.dev323, which includes GH-48137's fix. We could not bisect, since every available nightly postdates the fix, so this is consistent with rather than proven to be GH-48137.

    Would a backport to a 25.0.x patch release be possible? Many datasets users on 25.x would hit this at the end of long streaming jobs.

  13. pitrou commented on Sep 30, 2026

    @pitrou
    pitrou
    on Sep 30, 2026
    Member

    Hi @whr778 , I think it's unlikely that we'll make a 25.0.x patch release since we plan to release 26.0.0 in the next month. @raulcd might elaborate.

  14. raulcd commented on Sep 30, 2026

    @raulcd
    raulcd
    on Sep 30, 2026
    Member

    The feature freeze for 26.0.0 is scheduled for tomorrow. This will be part of it.
    Currently due to our extensive CI heavy release process we do not backport patches to previous releases unless the main release is still far away.

    More details on why we don't do it: We require many third party libraries and CI jobs to build correctly in order to be able to release and it is not only backporting this commit but the many commits that have been done during the last 3 months to keep CI green to fix the various jobs and dependency issues. Honestly we do not have the resources to do that at the moment so we release bug fixes on the new major release.

    The only issues that would grant a patch release like that would be critical security fixes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Component: C++Type: bug

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions