Manage CPython thread states automatically across Julia tasks by cjdoris · Pull Request #824 · JuliaPy/PythonCall.jl

cjdoris · 2026-09-19T18:47:06Z

Motivation

Replace the old assumption that Julia permanently holds Python's GIL with a task-safe thread-state model so PythonCall APIs work correctly from any Julia task/thread.
Provide user-facing, nestable hints (@pyregion/@pyregionbreak) that amortize attach/detach transitions without exposing GIL details.
Preserve compatibility with existing callback paths (JuliaCall/JlWrap) and keep finalizers non-blocking.

Description

Added a new internal module Region that implements per-OS-thread persistent PythonCall-owned PyThreadState + cooperative semaphore, and per-task dynamic TaskState with stickiness and token-based nested enter/exit semantics; exported macros @pyregion and @pyregionbreak use this machinery. (src/Region/Region.jl, src/API/macros.jl, src/API/exports.jl)
Hooked startup/finalization so standalone PythonCall detaches after init and finalization re-attaches a known interpreter state before calling Py_FinalizeEx(). (src/C/context.jl, src/Region/Region.jl, src/PythonCall.jl)
Replaced the old PyGILState_* usage internally for region semantics and exposed macros; provided a compatibility layer for the existing PythonCall.GIL API by routing it through the new region enter/exit functions. (src/GIL/GIL.jl, src/C/pointers.jl)
Applied @pyregion / @pyregionbreak throughout the core C-API call sites to ensure correct attach/detach behavior while keeping regions tight; updated @py lowering so generated code has one outer @pyregion and each @jl splice becomes a @pyregionbreak. (bulk edits under src/Core/*, src/PyMacro/PyMacro.jl, src/JlWrap/*, src/GC/GC.jl, src/Core/builtins.jl)
Revised finalizer/decref logic to avoid attaching a Python state in finalizers; finalizers enqueue pointers if no attached state exists and PythonCall.GC.gc() can be used to drain the queue inside a region. (src/GC/GC.jl, test/GC.jl)
Adjusted low-level C pointer handling to expose a safe unchecked thread-state getter (PyThreadState_GetUnchecked / _PyThreadState_UncheckedGet) and PyThreadState_New usage for persistent per-thread states. (src/C/pointers.jl, src/C/C.jl)
Added task/region tests that exercise nested region/break combinations, detach-yield-resume, multi-threaded interleaving, and Python->Julia callbacks. (test/Region.jl)
Bumped the package Julia requirement to 1.12 to rely on OncePerThread / OncePerTask primitives and updated julia/juliapkg metadata where appropriate.

Testing

Ran the Julia test suite with multiple threads: JULIA_NUM_THREADS=4 julia --project -e 'using Pkg; Pkg.test()' — all tests passed (4,095 tests passed).
Ran coverage-enabled tests: JULIA_NUM_THREADS=4 julia --project -e 'using Pkg; Pkg.test(coverage=true)' — suite passed (4,095 tests passed); uncovered lines are limited to defensive rollback/impossible-token branches and the Python-originated root-break path that requires JuliaCall-specific scenarios.
Performed a threaded smoke test invoking PythonCall from Threads.@spawn to verify detached initialization and cross-thread Python calls — succeeded (simple runtime checks demonstrated PyThreadState_GetUnchecked() is NULL outside regions and that @pyregion amortizes transitions).
Attempted the Python-side tests (uv run pytest -s --nbval ./pytest) but they failed to run in this environment due to an external constraint: the environment's Python 3.14 OpenSSL/juliapkg interaction restricts Julia to ≤1.11 and therefore conflicts with the new Julia ≥1.12 requirement; the Python tests are otherwise intended to run in a matching environment and were not failing due to region logic.
Documentation build attempted with docs/make.jl; Documenter ran but could not infer repository remote source links because this checkout lacks a valid origin remote (non-test blocking).

If you want, I can (1) open follow-ups to address remaining minor uncovered branches and defensive assertions, (2) add CI matrix notes to run the Python tests under a Python/Julia combination that satisfies OpenSSL/juliapkg constraints, or (3) narrow/expand any @pyregion placements based on preferred granularity.

Codex Task

cjdoris · 2026-09-19T21:38:27Z


 using Preferences: @load_preference

+# Minimal package-local equivalents of Base.OncePerThread and Base.OncePerTask. The Base


Only define these on old Julia versions. Use the actual types when available from Base.

cjdoris · 2026-09-19T21:39:15Z

+    return
+end
+
+function enter_break()


The enter/exit functions should take the current task and task state as args.

cjdoris · 2026-09-19T21:40:21Z

+    end
+end
+
+function __init__()


This init and the start runtime code could happen in the init function in context.jl.

cjdoris · 2026-09-19T21:40:48Z

+
+const THREAD_STATE = Utils.OncePerThread{ThreadState}(ThreadState)
+const TASK_STATE = Utils.OncePerTask{TaskState}(TaskState)
+const INTERP = Ref{Ptr{Cvoid}}(C_NULL)


The interpreter pointer can go in C.CTX.

And be initialised in the init context function.

cjdoris · 2026-09-19T21:41:08Z

+    return ts
+end
+
+check_thread(s::TaskState) = (@assert current_task().sticky && Threads.threadid() == s.tid)


Make it 2 separate asserts.

cjdoris · 2026-10-03T13:20:54Z

-        pyisnull(exc._b) && pycopy!(exc._b, pybuiltins.None)
-        pyisnone(exc._v) || (exc._v.__traceback__ = exc._b)
-        exc._isnormalized = true
+    @pyregion begin


Prob only need a pyregion inside the first branch?

cjdoris · 2026-10-03T13:23:13Z

        a = @pyconvert Union{Int,Nothing} k.start begin
-            errset(pybuiltins.TypeError, "slice components must be integers")
-            pythrow()
+            @pyregion errset(pybuiltins.TypeError, "slice components must be integers")


An errset and pythrow should be in the same region as they need the same threadstate.

Similarly any error checks should occur in the same region as the code that set the error, as the error state is on the threadstate. Check this throughout the codebase.

cjdoris · 2026-10-03T13:26:17Z

-                    return pyjl_handle_error(f, self, exc)
-                else
-                    errset(pyJuliaError, pytuple((pyjl(exc), pyjl(catch_backtrace()))))
+    @pyregion begin


This pyregion shouldn't be needed as you can only reach this by calling the method from python which will already have the threadstate.

cjdoris · 2026-10-03T13:30:30Z

 """
 macro py(ex)
-    esc(py_macro(ex, __module__, __source__))
+    lowered = py_macro(ex, __module__, __source__)


For now revert this and don't bother putting a pyregion around the py macro. It messes with the semantics of py.

cjdoris · 2026-10-03T13:32:09Z

 include("JlWrap/JlWrap.jl")
 include("Compat/Compat.jl")

+function __init__()


Shouldn't be needed, any initialisation can occur in the init context function.

cjdoris · 2026-10-03T13:35:30Z

@codex work on all the pr comments

chatgpt-codex-connector · 2026-10-03T13:53:48Z

Summary

Moved task-safe CPython thread-state and region management into the C module. Region transitions now receive the current task and task state explicitly, preserve task stickiness, and use separate thread-safety assertions. src/C/regions.jlL3-L19 src/C/regions.jlL32-L46 src/C/regions.jlL96-L123
Moved interpreter ownership and initialization into C.CTX, with dynamic finalization that attaches a thread state only when necessary. src/C/context.jlL6-L24 src/C/context.jlL282-L297 src/C/context.jlL325-L331
Updated @pyregion and @pyregionbreak to carry the exact task and task state through matching enter/exit operations. src/C/regions.jlL184-L223
Used Julia’s native Base.OncePerThread and Base.OncePerTask implementations when available, retaining compatibility implementations for older supported Julia versions. src/Utils/Utils.jlL5-L42
Removed the legacy PythonCall.GIL compatibility module and documentation, leaving the region macros as the documented thread-state interface. docs/src/pythoncall-reference.mdL223-L235
Allowed user-defined conversion rules to yield outside an enclosing Python region while keeping built-in integer conversion and its Python error checks in one region. src/Convert/pyconvert.jlL345-L383 src/Convert/rules.jlL61-L70
Narrowed regions around reference-counting and exception-related operations, inlined propertynames, and kept paired Python error creation/throwing operations on the same thread state. src/Core/Py.jlL1-L2 src/Core/Py.jlL87-L110 src/Core/Py.jlL275-L294 src/JlWrap/array.jlL5-L21
Committed the changes as 0f86231 and created the follow-up pull request titled “Address review feedback for automatic CPython thread states.”

Testing

✅ JULIA_NUM_THREADS=4 julia --project -e 'using Pkg; Pkg.test()' — 4,102 tests passed.
✅ uv run pytest -s --nbval ./pytest — 15 tests passed.
✅ git diff --check
✅ git status --short — clean after commit.

View task →

Manage Python thread states per Julia task

3369142

cjdoris added the codex label Sep 19, 2026 — with ChatGPT Codex Connector

cjdoris added 2 commits September 19, 2026 22:08

Restore Julia 1.10 thread state compatibility

d35fa84

Keep Python error handling within regions

8f82c46

cjdoris commented Oct 3, 2026

View reviewed changes

cjdoris added 3 commits October 3, 2026 14:57

Address thread-state review feedback

61e51b1

Document automatic thread-state migration

6625707

Remove jl_call_nogil and benchmark regions

f82658e


		using Preferences: @load_preference

		# Minimal package-local equivalents of Base.OncePerThread and Base.OncePerTask. The Base

Conversation

cjdoris commented Sep 19, 2026

Motivation

Description

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cjdoris commented Oct 3, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 3, 2026

Summary

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant