[Python] Interpreter hangs at exit after pq.read_table(python_file_object) - thread pool teardown deadlock #50482

Activity

1. imanhabib

   added

   Type: bug

   on Jul 12, 2026
2. raulcd commented on Jul 13, 2026

   

   raulcd

   on Jul 13, 2026

   Member

   More actions

   Thank you very much for creating the issue.
   I am not sure I understand the Version matrix section above, can you help me understand clearly whether this:

   • Is a new issue given the latest 25.0.0 release
   • Has been happening since 22.0.0

   Is this only happening with a specific parquet file or is it reproducible with a minimal generated parquet file?
3. mikamikasuki commented on Oct 2, 2026

   

   mikamikasuki

   on Oct 2, 2026

   More actions

   I have now independently reproduced this with a generated five-row Parquet file on macOS 15 (arm64), CPython 3.14.7, and the PyArrow 25.0.0 wheel. With 24 CPU threads, the file-object case timed out in 3 of 30 fresh processes; path reads exited cleanly in 12 of 12, and file-object reads with use_threads=False exited cleanly in 12 of 12.

   I captured native stacks from three hung processes. The main thread was waiting in Arrow's CPU ThreadPool::Shutdown; a worker was destroying a ParquetFileFragment / PyReadableFile, then blocked in PyGILState_Ensure → CPython thread-state attachment during finalization. This matches CPython's documented behavior that non-finalization threads attempting to attach during late interpreter finalization can be blocked permanently: https://docs.python.org/3.14/c-api/interp-lifecycle.html

   The source path leads to OwnedRefNoGIL in python/pyarrow/src/arrow/python/common.h, which reacquires the GIL to decref a retained Python object. I have a local candidate guard for the finalization path, but it still needs a source build and review of the check-to-GIL-acquire race, so I have not opened a PR.

   AI assistance was used to help trace the source and draft the candidate change; I ran the reproductions and examined the captured stacks. I would like to work on this issue.
4. deepkilo commented on Oct 18, 2026

   

   deepkilo

   on Oct 18, 2026

   More actions

   Additional Linux evidence from compatibility testing of a quantitative-research runtime. This was an observed process-exit failure investigated back to native cleanup, not a finding from source review alone. Our reproducer uses generated files and a Python filesystem wrapper; it is related evidence, not a claim that every file-object case has exactly the same ownership path.

   With official PyArrow 26, a malformed fragment raises the expected ArrowInvalid, but the process can still fail during interpreter shutdown. On Linux x86_64 our reduced case aborted on CPython 3.12.15 (12/12 four-fragment runs and 12/12 eight-fragment runs), timed out on 3.13.16 (4/4), and also timed out on unmodified 26 with 3.15.0 (3/3). Upgrading Python alone did not repair it.

   Self-contained reduced case; run in fresh processes with a timeout, since the symptom is at exit:

   import tempfile
   from pathlib import Path
   import fsspec
   import pyarrow as pa
   import pyarrow.dataset as ds
   import pyarrow.fs as fs
   import pyarrow.parquet as pq
   
   root = Path(tempfile.mkdtemp())
   pq.write_table(pa.table({'value': [1, 2, 3]}), root / 'valid.parquet')
   (root / 'bad.parquet').write_bytes(b'not a parquet file')
   filesystem = fs.PyFileSystem(fs.FSSpecHandler(fsspec.filesystem('file')))
   paths = [str(root / name) for name in
            ['valid.parquet', 'bad.parquet', 'valid.parquet', 'valid.parquet']]
   dataset = ds.dataset(paths, filesystem=filesystem, format='parquet',
                        schema=pa.schema([('value', pa.int64())]),
                        exclude_invalid_files=False)
   try:
       dataset.to_table(use_threads=True, fragment_readahead=4)
   except pa.ArrowInvalid:
       pass
   else:
       raise AssertionError('Expected the malformed-fragment error')

   Our captured native stack passes through callback-capture destruction, MergedGenerator / SerialReadahead, the Parquet reader / ReadRangeCache, and PyBuffer::~PyBuffer, then PyGILState_Ensure during Py_FinalizeEx. On 3.12 the forced unwind crosses a noexcept boundary and terminates; newer Python parks the late worker while Arrow's pool shutdown joins it.

   This also relates to #46945: checking Py_IsInitialized or finalization before acquiring the GIL leaves a check-to-acquire race. A local candidate uses Python 3.15's PEP 788 view/token admission API for PyBuffer, NumPyBuffer, and OwnedRefNoGIL cleanup. The complete Python-native wheel was rebuilt, with the eight computational Arrow/Parquet DSOs retained byte-for-byte. Eighteen bounded subprocess checks then exited normally, including malformed-input and valid-input controls; the malformed fragment still raises rather than being suppressed.

   That candidate is qualified only for main-interpreter, GIL CPython 3.15, not older Python, free-threaded builds or subinterpreters. A failed admission can also mean allocation failure, so the candidate's skipped cleanup may leak a reference in that case. I am sharing the mechanism and verification boundary for discussion, not claiming a universally ready upstream fix or a newly introduced 26 regression.

   Disclosure: OpenAI Codex (GPT-6 family; exact variant/effort unavailable in this session) assisted with investigation, tool-driven reproduction, candidate work and this text. The stated runs and native stacks are recorded execution evidence, not a claim that the operator manually performed or reviewed every step. No maintainer ping or competing PR is intended.
5. mikamikasuki

   added a commit that references this issue on Oct 18, 2026

   apacheGH-50482: [Python] Wait for CPU tasks before interpreter finali…

   ...

   1050454
6. github-actions

   assigned

   mikamikasuki

   on Oct 18, 2026
7. MagicLex commented on Oct 22, 2026

   

   MagicLex

   on Oct 22, 2026

   More actions

   The same exit hang also happens when the scan goes through a pyarrow.fs.PyFileSystem instead of a Python file object (macOS arm64, CPython 3.12.12).

   # child.py DIR, where DIR is an absolute path holding 4 Parquet files of 1000 int64 rows
   import sys, fsspec
   import pyarrow.dataset as ds, pyarrow.fs as pafs
   fs = pafs.PyFileSystem(pafs.FSSpecHandler(fsspec.filesystem("file")))
   # control: fs = pafs.LocalFileSystem()
   print(ds.dataset(sys.argv[1], filesystem=fs, format="parquet").to_table().num_rows, flush=True)

   Result on pyarrow 26.0.0, fresh processes, 8 in parallel, hung = printed the count and still alive after 20 s: 15/200 hung with PyFileSystem, 0/200 with LocalFileSystem.

   Main thread of a hung process (sample):

   exit
   std::shared_ptr<arrow::internal::ThreadPool>::~shared_ptr
   arrow::internal::ThreadPool::~ThreadPool()
   arrow::internal::ThreadPool::Shutdown(bool)
   

   PR #54467 likely covers this path; not tested.

   Workaround: scan with a native filesystem (LocalFileSystem, or SubTreeFileSystem over it) instead of a Python-implemented one.