Reproduction for sentry-python#7014

sentry-python/7014/README.md

@@ -0,0 +1,84 @@
+# Reproduction for sentry-python#7014
+
+**Issue:** https://github.com/getsentry/sentry-python/issues/7014
+
+## Description
+
+The `AI (3.10)` CI job fails now and then in `tests/integrations/litellm/test_litellm.py`. The test that fails changes from run to run. Recent example: `test_binary_content_encoding_mixed_content[False-False]`, which failed on master 7 times between Sep 30 and Oct 13, 2026 ([job](https://github.com/getsentry/sentry-python/actions/runs/38237427670/job/114771446502), [job](https://github.com/getsentry/sentry-python/actions/runs/38248860372/job/114809620395)). Each failure shows the same two errors:
+
+```
+FAILED ...::test_binary_content_encoding_mixed_content[False-False] - assert 0 == 1   (len(chat_spans))
+ERROR  ...::test_binary_content_encoding_mixed_content[False-False] - PytestUnraisableExceptionWarning:
+  Exception ignored in thread started by: <bound method Thread._bootstrap of <Thread(ThreadPoolExecutor-76_0, ...)>>
+/usr/lib/python3.10/threading.py:1011: in _bootstrap_inner
+    _sys.settrace(_trace_hook)
+E   RuntimeError: Cannot install a trace function while another trace function is being installed
+```
+
+How the failure happens:
+
+1. **The CPython bug.** On CPython < 3.12, `sys.settrace()` is protected by one `static int reentrant` flag shared by the whole process ([python/cpython#98257](https://github.com/python/cpython/issues/98257), fixed in 3.12). That flag stays set while audit hooks run.
+2. **The audit hook.** [filelock](https://pypi.org/project/filelock/) 3.30.0 through 4.0.2 registers a Python-level audit hook (`sys.addaudithook(_audit_fork_safety)`) when it is imported. In the litellm test env, `huggingface_hub` imports filelock during the first test.
+3. **The tracer.** coverage / pytest-cov calls `threading.settrace(...)`. As a result, every new thread calls `sys.settrace()` inside `Thread._bootstrap_inner()`.
+4. **The race.** Thread A can be inside filelock's hook while holding the flag, and the GIL can switch to thread B. When thread B then calls `settrace()`, it raises `RuntimeError`. Thread B dies before it runs its target.
+5. **The lost span.** LiteLLM finishes the Sentry `gen_ai.chat` span in a success callback that runs on a `ThreadPoolExecutor`. The tests create a new executor for each test (`reset_litellm_executor`), so it has exactly one worker. If that worker dies, the callback never runs, the span is never finished, and `chat_spans` is empty.
+
+`repro.py` does the same thing without litellm or pytest. In each iteration it creates a transaction and a burst of new single-worker executors. Each executor's only job is `span.finish`. The script then counts callbacks that never ran.
+
+## Steps to Reproduce
+
+Requires [uv](https://docs.astral.sh/uv/). No DSN is needed because events go to an in-memory transport. If you want to send events to a real project, `export SENTRY_DSN=` and change `sentry_sdk.init`.
+
+1. Bug: Python 3.10 with filelock 3.32.4, the version pinned in CI. `repro.py` installs `threading.settrace` itself:
+   ```bash
+   uv run --python 3.10 --with sentry-sdk==2.72.0 --with filelock==3.32.4 python repro.py
+   ```
+   Same check under real coverage, like CI does:
+   ```bash
+   uv run --python 3.10 --with sentry-sdk==2.72.0 --with filelock==3.32.4 --with coverage==7.16.2 \
+     python -m coverage run repro.py --no-tracer
+   ```
+2. Controls, which should exit 0:
+   ```bash
+   # filelock 4.0.3+ skips the audit hook on CPython < 3.12
+   uv run --python 3.10 --with sentry-sdk==2.72.0 --with filelock==4.0.3 python repro.py
+   # CPython 3.12 fixed the shared flag
+   uv run --python 3.12 --with sentry-sdk==2.72.0 --with filelock==3.32.4 python repro.py
+   # no audit hook / no tracer
+   uv run --python 3.10 --with sentry-sdk==2.72.0 --with filelock==3.32.4 python repro.py --no-filelock
+   uv run --python 3.10 --with sentry-sdk==2.72.0 --with filelock==3.32.4 python repro.py --no-tracer
+   ```
+3. To measure how often it happens, run it several times:
+   ```bash
+   for i in $(seq 5); do uv run --python 3.10 --with sentry-sdk==2.72.0 --with filelock==3.32.4 python repro.py --iterations 2000 | grep "never ran"; done
+   ```
+
+The script exits 1 and prints `BUG REPRODUCED` when at least one callback was lost.
+
+## Expected Behavior
+
+Every callback submitted to an executor runs, and every `gen_ai.chat` span is finished and sent.
+
+## Actual Behavior
+
+With Python 3.10 and filelock 3.32.4, a small share of executor worker threads die while starting. Their callbacks never run and those spans are lost:
+
+```
+filelock 3.32.4
+python 3.10.22, executors=32000, 67.3s
+worker threads that died in Thread._bootstrap: 14
+  e.g. RuntimeError('Cannot install a trace function while another trace function is being installed')
+callbacks that never ran (span never finished): 14
+gen_ai.chat spans received by transport: 31986/32000
+BUG REPRODUCED
+```
+
+The same run with filelock 4.0.3 on 3.10, or with filelock 3.32.4 on 3.12, lost **0 of 32000** callbacks.
+
+## Environment
+
+- Python: 3.10.22 (bug); 3.12.15 (control)
+- sentry-sdk: 2.72.0
+- filelock: 3.32.4 (bug), 4.0.3 (fixed)
+- coverage: 7.16.2 (optional; `threading.settrace` stands in for it by default)
+- OS: Linux x86_64

sentry-python/7014/repro.py

@@ -0,0 +1,117 @@
+"""
+Reproduction for getsentry/sentry-python#7014
+
+On CPython < 3.12, `sys.settrace()` is guarded by ONE process-wide `reentrant`
+flag that stays set while audit hooks run (python/cpython#98257). filelock
+3.30.0 - 4.0.2 registers a Python-level audit hook at import time. When a
+tracer is installed for all threads (coverage/pytest-cov does
+`threading.settrace(...)`), every new thread calls `sys.settrace()` from
+`Thread._bootstrap_inner()`. If the GIL switches to another thread while the
+first one is inside filelock's audit hook, the second thread's `settrace()`
+raises:
+
+    RuntimeError: Cannot install a trace function while another trace function
+    is being installed
+
+The thread dies before running its target. For a fresh ThreadPoolExecutor
+that means the submitted work item never runs. LiteLLM finishes Sentry's
+`gen_ai.chat` span in a success callback submitted to such an executor, so
+the span is silently lost -> `assert len(chat_spans) == 1` fails with 0.
+
+Expected: every submitted callback runs, every transaction has its child span.
+Actual (py3.10 + filelock<4.0.3 + tracer): some callbacks never run and the
+span is missing; the worker thread dies with the RuntimeError above.
+"""
+
+import argparse
+import sys
+import threading
+import time
+from concurrent.futures import ThreadPoolExecutor
+
+import sentry_sdk
+from sentry_sdk.transport import Transport
+
+
+class MemoryTransport(Transport):
+    """Keep envelopes in memory so no DSN / network is needed."""
+
+    def __init__(self, options=None):
+        super().__init__(options)
+        self.chat_spans = 0
+
+    def capture_envelope(self, envelope):
+        for item in envelope.items:
+            payload = item.payload.json or {}
+            if item.type == "transaction":
+                spans = payload.get("spans", [])
+                self.chat_spans += sum(1 for s in spans if s.get("op") == "gen_ai.chat")
+            elif item.type == "span":
+                for s in payload.get("items", []):
+                    op = s.get("attributes", {}).get("sentry.op", {}).get("value")
+                    self.chat_spans += op == "gen_ai.chat"
+
+
+def tracer(frame, event, arg):
+    # Stand-in for coverage's per-thread tracer.
+    return None
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--iterations", type=int, default=5000)
+    parser.add_argument("--burst", type=int, default=16, help="executors started concurrently per iteration")
+    parser.add_argument("--no-filelock", action="store_true", help="control: don't import filelock")
+    parser.add_argument("--no-tracer", action="store_true", help="control: don't install a thread tracer")
+    args = parser.parse_args()
+
+    if not args.no_filelock:
+        import filelock  # registers sys.addaudithook() on import (3.30.0 - 4.0.2)
+
+        print(f"filelock {filelock.__version__}")
+    if not args.no_tracer:
+        threading.settrace(tracer)  # what coverage / pytest-cov does
+
+    # Make GIL hand-offs (and therefore the race) more frequent.
+    sys.setswitchinterval(1e-6)
+
+    transport = MemoryTransport()
+    sentry_sdk.init(transport=transport, traces_sample_rate=1.0)
+
+    dead_threads = []
+    sys.unraisablehook = lambda u: dead_threads.append(u.exc_value)
+
+    started = time.monotonic()
+    lost = 0
+    for i in range(args.iterations):
+        with sentry_sdk.start_transaction(name=f"txn-{i}"):
+            pending = []
+            for _ in range(args.burst):
+                span = sentry_sdk.start_span(op="gen_ai.chat", name="chat")
+                # Fresh executor, like the litellm tests' `reset_litellm_executor`.
+                executor = ThreadPoolExecutor(max_workers=1)
+                # LiteLLM finishes the gen_ai.chat span in a success callback run in a thread pool.
+                executor.submit(span.finish)
+                pending.append((span, executor))
+            for span, executor in pending:
+                executor.shutdown(wait=True)  # the tests do exactly this before asserting
+                if span.timestamp is None:
+                    lost += 1  # the callback never ran
+
+    sentry_sdk.flush()
+    total = args.iterations * args.burst
+
+    print(f"python {sys.version.split()[0]}, executors={total}, {time.monotonic() - started:.1f}s")
+    print(f"worker threads that died in Thread._bootstrap: {len(dead_threads)}")
+    if dead_threads:
+        print(f"  e.g. {dead_threads[0]!r}")
+    print(f"callbacks that never ran (span never finished): {lost}")
+    print(f"gen_ai.chat spans received by transport: {transport.chat_spans}/{total}")
+    if lost or dead_threads:
+        print("BUG REPRODUCED")
+        sys.exit(1)
+    print("OK: every callback ran")
+
+
+if __name__ == "__main__":
+    main()

sentry-python/7014/requirements.txt

@@ -0,0 +1,4 @@
+# Versions from the failing CI env py3.10-litellm-v1.83.14 (see tox.ini in getsentry/sentry-python)
+sentry-sdk==2.72.0
+filelock==3.32.4
+coverage==7.16.2