From 103e6c80eb06e33ba7fd549d0f2a2d0897c5f0e9 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Wed, 14 Oct 2026 08:50:19 +0000 Subject: [PATCH] Add reproduction for sentry-python#7014 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Johannes Daxböck --- sentry-python/7014/README.md | 84 ++++++++++++++++++++ sentry-python/7014/repro.py | 117 ++++++++++++++++++++++++++++ sentry-python/7014/requirements.txt | 4 + 3 files changed, 205 insertions(+) create mode 100644 sentry-python/7014/README.md create mode 100644 sentry-python/7014/repro.py create mode 100644 sentry-python/7014/requirements.txt diff --git a/sentry-python/7014/README.md b/sentry-python/7014/README.md new file mode 100644 index 0000000..7151af4 --- /dev/null +++ b/sentry-python/7014/README.md @@ -0,0 +1,84 @@ +# Reproduction for sentry-python#7014 + +**Issue:** https://github.com/getsentry/sentry-python/issues/7014 + +## Description + +The `AI (3.10)` CI job fails now and then in `tests/integrations/litellm/test_litellm.py`. The test that fails changes from run to run. Recent example: `test_binary_content_encoding_mixed_content[False-False]`, which failed on master 7 times between Sep 30 and Oct 13, 2026 ([job](https://github.com/getsentry/sentry-python/actions/runs/38237427670/job/114771446502), [job](https://github.com/getsentry/sentry-python/actions/runs/38248860372/job/114809620395)). Each failure shows the same two errors: + +``` +FAILED ...::test_binary_content_encoding_mixed_content[False-False] - assert 0 == 1 (len(chat_spans)) +ERROR ...::test_binary_content_encoding_mixed_content[False-False] - PytestUnraisableExceptionWarning: + Exception ignored in thread started by: > +/usr/lib/python3.10/threading.py:1011: in _bootstrap_inner + _sys.settrace(_trace_hook) +E RuntimeError: Cannot install a trace function while another trace function is being installed +``` + +How the failure happens: + +1. **The CPython bug.** On CPython < 3.12, `sys.settrace()` is protected by one `static int reentrant` flag shared by the whole process ([python/cpython#98257](https://github.com/python/cpython/issues/98257), fixed in 3.12). That flag stays set while audit hooks run. +2. **The audit hook.** [filelock](https://pypi.org/project/filelock/) 3.30.0 through 4.0.2 registers a Python-level audit hook (`sys.addaudithook(_audit_fork_safety)`) when it is imported. In the litellm test env, `huggingface_hub` imports filelock during the first test. +3. **The tracer.** coverage / pytest-cov calls `threading.settrace(...)`. As a result, every new thread calls `sys.settrace()` inside `Thread._bootstrap_inner()`. +4. **The race.** Thread A can be inside filelock's hook while holding the flag, and the GIL can switch to thread B. When thread B then calls `settrace()`, it raises `RuntimeError`. Thread B dies before it runs its target. +5. **The lost span.** LiteLLM finishes the Sentry `gen_ai.chat` span in a success callback that runs on a `ThreadPoolExecutor`. The tests create a new executor for each test (`reset_litellm_executor`), so it has exactly one worker. If that worker dies, the callback never runs, the span is never finished, and `chat_spans` is empty. + +`repro.py` does the same thing without litellm or pytest. In each iteration it creates a transaction and a burst of new single-worker executors. Each executor's only job is `span.finish`. The script then counts callbacks that never ran. + +## Steps to Reproduce + +Requires [uv](https://docs.astral.sh/uv/). No DSN is needed because events go to an in-memory transport. If you want to send events to a real project, `export SENTRY_DSN=` and change `sentry_sdk.init`. + +1. Bug: Python 3.10 with filelock 3.32.4, the version pinned in CI. `repro.py` installs `threading.settrace` itself: + ```bash + uv run --python 3.10 --with sentry-sdk==2.72.0 --with filelock==3.32.4 python repro.py + ``` + Same check under real coverage, like CI does: + ```bash + uv run --python 3.10 --with sentry-sdk==2.72.0 --with filelock==3.32.4 --with coverage==7.16.2 \ + python -m coverage run repro.py --no-tracer + ``` +2. Controls, which should exit 0: + ```bash + # filelock 4.0.3+ skips the audit hook on CPython < 3.12 + uv run --python 3.10 --with sentry-sdk==2.72.0 --with filelock==4.0.3 python repro.py + # CPython 3.12 fixed the shared flag + uv run --python 3.12 --with sentry-sdk==2.72.0 --with filelock==3.32.4 python repro.py + # no audit hook / no tracer + uv run --python 3.10 --with sentry-sdk==2.72.0 --with filelock==3.32.4 python repro.py --no-filelock + uv run --python 3.10 --with sentry-sdk==2.72.0 --with filelock==3.32.4 python repro.py --no-tracer + ``` +3. To measure how often it happens, run it several times: + ```bash + for i in $(seq 5); do uv run --python 3.10 --with sentry-sdk==2.72.0 --with filelock==3.32.4 python repro.py --iterations 2000 | grep "never ran"; done + ``` + +The script exits 1 and prints `BUG REPRODUCED` when at least one callback was lost. + +## Expected Behavior + +Every callback submitted to an executor runs, and every `gen_ai.chat` span is finished and sent. + +## Actual Behavior + +With Python 3.10 and filelock 3.32.4, a small share of executor worker threads die while starting. Their callbacks never run and those spans are lost: + +``` +filelock 3.32.4 +python 3.10.22, executors=32000, 67.3s +worker threads that died in Thread._bootstrap: 14 + e.g. RuntimeError('Cannot install a trace function while another trace function is being installed') +callbacks that never ran (span never finished): 14 +gen_ai.chat spans received by transport: 31986/32000 +BUG REPRODUCED +``` + +The same run with filelock 4.0.3 on 3.10, or with filelock 3.32.4 on 3.12, lost **0 of 32000** callbacks. + +## Environment + +- Python: 3.10.22 (bug); 3.12.15 (control) +- sentry-sdk: 2.72.0 +- filelock: 3.32.4 (bug), 4.0.3 (fixed) +- coverage: 7.16.2 (optional; `threading.settrace` stands in for it by default) +- OS: Linux x86_64 diff --git a/sentry-python/7014/repro.py b/sentry-python/7014/repro.py new file mode 100644 index 0000000..46bce5d --- /dev/null +++ b/sentry-python/7014/repro.py @@ -0,0 +1,117 @@ +""" +Reproduction for getsentry/sentry-python#7014 + +On CPython < 3.12, `sys.settrace()` is guarded by ONE process-wide `reentrant` +flag that stays set while audit hooks run (python/cpython#98257). filelock +3.30.0 - 4.0.2 registers a Python-level audit hook at import time. When a +tracer is installed for all threads (coverage/pytest-cov does +`threading.settrace(...)`), every new thread calls `sys.settrace()` from +`Thread._bootstrap_inner()`. If the GIL switches to another thread while the +first one is inside filelock's audit hook, the second thread's `settrace()` +raises: + + RuntimeError: Cannot install a trace function while another trace function + is being installed + +The thread dies before running its target. For a fresh ThreadPoolExecutor +that means the submitted work item never runs. LiteLLM finishes Sentry's +`gen_ai.chat` span in a success callback submitted to such an executor, so +the span is silently lost -> `assert len(chat_spans) == 1` fails with 0. + +Expected: every submitted callback runs, every transaction has its child span. +Actual (py3.10 + filelock<4.0.3 + tracer): some callbacks never run and the +span is missing; the worker thread dies with the RuntimeError above. +""" + +import argparse +import sys +import threading +import time +from concurrent.futures import ThreadPoolExecutor + +import sentry_sdk +from sentry_sdk.transport import Transport + + +class MemoryTransport(Transport): + """Keep envelopes in memory so no DSN / network is needed.""" + + def __init__(self, options=None): + super().__init__(options) + self.chat_spans = 0 + + def capture_envelope(self, envelope): + for item in envelope.items: + payload = item.payload.json or {} + if item.type == "transaction": + spans = payload.get("spans", []) + self.chat_spans += sum(1 for s in spans if s.get("op") == "gen_ai.chat") + elif item.type == "span": + for s in payload.get("items", []): + op = s.get("attributes", {}).get("sentry.op", {}).get("value") + self.chat_spans += op == "gen_ai.chat" + + +def tracer(frame, event, arg): + # Stand-in for coverage's per-thread tracer. + return None + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--iterations", type=int, default=5000) + parser.add_argument("--burst", type=int, default=16, help="executors started concurrently per iteration") + parser.add_argument("--no-filelock", action="store_true", help="control: don't import filelock") + parser.add_argument("--no-tracer", action="store_true", help="control: don't install a thread tracer") + args = parser.parse_args() + + if not args.no_filelock: + import filelock # registers sys.addaudithook() on import (3.30.0 - 4.0.2) + + print(f"filelock {filelock.__version__}") + if not args.no_tracer: + threading.settrace(tracer) # what coverage / pytest-cov does + + # Make GIL hand-offs (and therefore the race) more frequent. + sys.setswitchinterval(1e-6) + + transport = MemoryTransport() + sentry_sdk.init(transport=transport, traces_sample_rate=1.0) + + dead_threads = [] + sys.unraisablehook = lambda u: dead_threads.append(u.exc_value) + + started = time.monotonic() + lost = 0 + for i in range(args.iterations): + with sentry_sdk.start_transaction(name=f"txn-{i}"): + pending = [] + for _ in range(args.burst): + span = sentry_sdk.start_span(op="gen_ai.chat", name="chat") + # Fresh executor, like the litellm tests' `reset_litellm_executor`. + executor = ThreadPoolExecutor(max_workers=1) + # LiteLLM finishes the gen_ai.chat span in a success callback run in a thread pool. + executor.submit(span.finish) + pending.append((span, executor)) + for span, executor in pending: + executor.shutdown(wait=True) # the tests do exactly this before asserting + if span.timestamp is None: + lost += 1 # the callback never ran + + sentry_sdk.flush() + total = args.iterations * args.burst + + print(f"python {sys.version.split()[0]}, executors={total}, {time.monotonic() - started:.1f}s") + print(f"worker threads that died in Thread._bootstrap: {len(dead_threads)}") + if dead_threads: + print(f" e.g. {dead_threads[0]!r}") + print(f"callbacks that never ran (span never finished): {lost}") + print(f"gen_ai.chat spans received by transport: {transport.chat_spans}/{total}") + if lost or dead_threads: + print("BUG REPRODUCED") + sys.exit(1) + print("OK: every callback ran") + + +if __name__ == "__main__": + main() diff --git a/sentry-python/7014/requirements.txt b/sentry-python/7014/requirements.txt new file mode 100644 index 0000000..7706644 --- /dev/null +++ b/sentry-python/7014/requirements.txt @@ -0,0 +1,4 @@ +# Versions from the failing CI env py3.10-litellm-v1.83.14 (see tox.ini in getsentry/sentry-python) +sentry-sdk==2.72.0 +filelock==3.32.4 +coverage==7.16.2