Isolate Julia's bundled OpenSSL from CPython on Linux - #97
Open
MilesCranmerBot wants to merge 2 commits into
Open
MilesCranmerBot wants to merge 2 commits into
MilesCranmerBot wants to merge 2 commits into
Conversation
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
Contributor
|
Hey @cjdoris - The story of this PR is I was getting frustrated by Julia's OpenSSL being incomptaible with some Python versions, which leads Google Colab to download an older Julia even though it already has Julia 1.12 installed (which causes a really slow startup time, given Colab's small CPUs). This led me down a long rabbit hole investigating whether there was any way to simultaneously allow for mismatched OpenSSL versions, which might be able to solve this incompatibility at a more fundamental level. Then, after many back-and-forths with various coding agents, and an embarassing number of API credits later, this patch finally popped out. It seems to work in some of the contexts which were incompatible before. Let me know what you think! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Allow Julia 1.12 and newer to run inside Python processes whose OpenSSL is older than Julia's bundled OpenSSL.
On Linux, this PR:
OpenSSL_jllat those private names;OpenSSL_jll = "<=python"bound only when the selected Julia is safe;No new runtime dependency is added.
Background
CPython and embedded Julia share one glibc link map. Python OpenSSL 3.0 and Julia OpenSSL 3.5 both use
libcrypto.so.3andlibssl.so.3, despite requiring different symbol versions.If CPython has already loaded its copy, glibc may substitute it for Julia's copy. This causes errors such as:
Julia 1.12 also loads OpenSSL through its stdlib
OpenSSL_jll. A bare-soname request there can silently resolve to Python's library even after Julia's ELF dependencies have been isolated.The current
<=pythonbehavior avoids this collision by restricting Python linked against OpenSSL older than 3.5 to Julia 1.11 or earlier. That prevents otherwise compatible projects from using Julia 1.12.Implementation
For juliapkg-owned Linux installations,
openssl.py:libcrypto.so.3andlibssl.so.3;libcrypto.jl.3andlibssl.jl.3;OpenSSL_jll.jlto the private names.The equal-length replacement preserves offsets shared by
DT_SONAME,DT_NEEDED, and.gnu.version_rentries. The stdlib source update uses an atomic temporary-file replacement and preserves its original mode.The resolver then applies these rules:
<=pythonbound.The metadata version is incremented so existing projects resolve once and migrate their owned Julia installation.
Verification
The final Linux acceptance gate passed 69 assertions, including two controls that deliberately disable one part of the fix.
Target pairs, using juliacall 0.9.35 with no thread environment override:
Additional checks:
Pkg,Downloads, andOpenSSL_jllwork in the embedded process.OpenSSL_jllresolves to Julia-owned storage.OpenSSL_jllselecting the host library.The GitHub workflow adds the two target Python/Julia pairs across Ubuntu, macOS, and Windows, with both Python TLS-context modes.
Fixes #75.
Related to JuliaPy/PythonCall.jl#612 and JuliaPy/PythonCall.jl#682. This replaces the Julia 1.11 restriction introduced in #74 when OpenSSL isolation succeeds.