Skip to content

Isolate Julia's bundled OpenSSL from CPython on Linux - #97

Open
MilesCranmerBot wants to merge 2 commits into
JuliaPy:mainfrom
MilesCranmerBot:openssl-shield
Open

MilesCranmerBot wants to merge 2 commits into
JuliaPy:mainfrom
MilesCranmerBot:openssl-shield

Conversation

@MilesCranmerBot

Copy link
Copy Markdown
Contributor

Summary

Allow Julia 1.12 and newer to run inside Python processes whose OpenSSL is older than Julia's bundled OpenSSL.

On Linux, this PR:

  • gives OpenSSL in juliapkg-owned Julia installations private library names;
  • points Julia 1.12's stdlib OpenSSL_jll at those private names;
  • drops juliacall's generated OpenSSL_jll = "<=python" bound only when the selected Julia is safe;
  • retains the existing Julia 1.11 fallback when isolation cannot be applied;
  • leaves system and JuliaUp installations unchanged.

No new runtime dependency is added.

Background

CPython and embedded Julia share one glibc link map. Python OpenSSL 3.0 and Julia OpenSSL 3.5 both use libcrypto.so.3 and libssl.so.3, despite requiring different symbol versions.

If CPython has already loaded its copy, glibc may substitute it for Julia's copy. This causes errors such as:

version `OPENSSL_3.3.0' not found

Julia 1.12 also loads OpenSSL through its stdlib OpenSSL_jll. A bare-soname request there can silently resolve to Python's library even after Julia's ELF dependencies have been isolated.

The current <=python behavior avoids this collision by restricting Python linked against OpenSSL older than 3.5 to Julia 1.11 or earlier. That prevents otherwise compatible projects from using Julia 1.12.

Implementation

For juliapkg-owned Linux installations, openssl.py:

  1. scans regular ELF files for the exact NUL-delimited names libcrypto.so.3 and libssl.so.3;
  2. replaces them in place with the equal-length names libcrypto.jl.3 and libssl.jl.3;
  3. creates private-name symlinks beside each provider;
  4. repoints the selected Julia stdlib's OpenSSL_jll.jl to the private names.

The equal-length replacement preserves offsets shared by DT_SONAME, DT_NEEDED, and .gnu.version_r entries. The stdlib source update uses an atomic temporary-file replacement and preserves its original mode.

The resolver then applies these rules:

  • Julia 1.11 and earlier retain the existing OpenSSL bound.
  • A safe Julia 1.12 or newer drops only the generated <=python bound.
  • An incompatible system Julia is left unchanged and replaced with a juliapkg-owned installation.
  • A forced incompatible executable raises an actionable error.
  • Failed or unsupported isolation restores the safe Julia 1.11 restriction.
  • macOS and Windows require no binary rewrite.

The metadata version is incremented so existing projects resolve once and migrate their owned Julia installation.

Verification

The final Linux acceptance gate passed 69 assertions, including two controls that deliberately disable one part of the fix.

Target pairs, using juliacall 0.9.35 with no thread environment override:

Python Python OpenSSL Julia Result
3.10 3.0.20 1.12.7 20/20 exit 0, no segfaults, no OpenSSL load failures
3.14 3.5.6 1.10.12 20/20 exit 0, no segfaults, no OpenSSL load failures

Additional checks:

  • Julia Pkg, Downloads, and OpenSSL_jll work in the embedded process.
  • Julia and Python both complete real HTTPS operations.
  • Every OpenSSL provider shipped with Julia loads by absolute path.
  • OpenSSL_jll resolves to Julia-owned storage.
  • System Julia installations remain byte-identical.
  • A rename-disabled control reproduces the original load failure.
  • A bare-soname control detects OpenSSL_jll selecting the host library.
  • 157 tests pass with 5 skipped on the Linux gate.
  • 162 tests pass locally.
  • Ruff formatting and lint checks pass.

The GitHub workflow adds the two target Python/Julia pairs across Ubuntu, macOS, and Windows, with both Python TLS-context modes.

Fixes #75.

Related to JuliaPy/PythonCall.jl#612 and JuliaPy/PythonCall.jl#682. This replaces the Julia 1.11 restriction introduced in #74 when OpenSSL isolation succeeds.

MilesCranmerBot and others added 2 commits August 31, 2026 01:22
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
@MilesCranmer

Copy link
Copy Markdown
Contributor

Hey @cjdoris - The story of this PR is I was getting frustrated by Julia's OpenSSL being incomptaible with some Python versions, which leads Google Colab to download an older Julia even though it already has Julia 1.12 installed (which causes a really slow startup time, given Colab's small CPUs).

This led me down a long rabbit hole investigating whether there was any way to simultaneously allow for mismatched OpenSSL versions, which might be able to solve this incompatibility at a more fundamental level.

Then, after many back-and-forths with various coding agents, and an embarassing number of API credits later, this patch finally popped out. It seems to work in some of the contexts which were incompatible before.

Let me know what you think!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Empty compat intersection after Julia v1.12 upgrade (OpenSSL_jll)

2 participants