Repository navigation
Reproduction for sentry-python#7014 - #60
Closed
sentry-junior[bot] wants to merge 1 commit into
Closed
sentry-junior[bot] wants to merge 1 commit into
sentry-junior[bot] wants to merge 1 commit into
Conversation
Co-Authored-By: Johannes Daxböck <johannes.daxboeck@sentry.io>
sentrivana
pushed a commit
to getsentry/sentry-python
that referenced
this pull request
Oct 14, 2026
### Description This fixes the `AI (3.10)` litellm flake. The test that fails changes from run to run. Recent example: `test_binary_content_encoding_mixed_content[False-False]`, which failed **7 times on master** between Sep 30 and Oct 13 ([job](https://github.com/getsentry/sentry-python/actions/runs/38237427670/job/114771446502)). Other failures were in `test_no_integration[False-False]` and `test_binary_content_encoding_image_url[True-False]`. All 10 litellm failures I looked at on master have the same cause: ``` /usr/lib/python3.10/threading.py:1011: in _bootstrap_inner _sys.settrace(_trace_hook) E RuntimeError: Cannot install a trace function while another trace function is being installed ``` **Root cause** - On CPython < 3.12, `_PyEval_SetTrace` uses a `static int reentrant` flag shared by the whole process. The flag stays set while audit hooks run ([python/cpython#98257](python/cpython#98257), fixed in 3.12). - `filelock` 3.30.0 through 4.0.2 registers a Python audit hook (`sys.addaudithook(_audit_fork_safety)`) when it is imported. The py3.10 litellm envs pin `filelock==3.32.4`, and `huggingface_hub` imports it during the first test. - pytest-cov calls `threading.settrace`, so every new thread calls `sys.settrace()` while it starts up. If the GIL switches while another thread is inside filelock's hook, the new thread's `settrace()` raises and the thread dies before it runs. - The tests create a new single-worker executor (`reset_litellm_executor`), and LiteLLM finishes the `gen_ai.chat` span in a success callback submitted to it. A dead worker means the callback never runs, so the test gets `len(chat_spans) == 0`. **Fix** `filelock` 4.0.3 fixed this upstream: it no longer registers the hook on CPython < 3.12 (`_SETTRACE_SURVIVES_AUDIT_HOOKS`). This PR adds a `py3.10,py3.11` rule, `filelock>=4.0.3`, to the litellm config, then regenerates `tox.ini` with `scripts/generate-test-files.sh --only litellm`. - Only the four `py3.10-litellm-*` blocks change; they now pin `filelock==4.1.0`. py3.12 and later are unaffected and unchanged. - Re-resolving those four blocks also moved other transitive pins, for example `anthropic` 1.0.0 → 1.13.0. - `releases.jsonl` is unchanged. `--only` would have pruned it, so I restored it. `package_dependencies.jsonl` only gains the 3 new resolution entries. No retries, sleeps, skipped tests or weaker assertions. **Reproduction:** getsentry/repro#60 It is a standalone script with no litellm and no pytest. On Python 3.10 with filelock 3.32.4, it lost 14 of 32,000 callbacks in one run, and 3 of 5 shorter runs (16,000 callbacks each) reproduced it. Under the same conditions it lost 0 of 32,000 with filelock 4.0.3, and 0 on Python 3.12. Original suite, run locally (`py3.10-litellm-v1.83.14`, with coverage): - With default settings: 0 of 20 runs failed. - With GIL switching set to happen more often (`sys.setswitchinterval(1e-6)` via a `-p` plugin; no test code changed): 4 of 20 runs hit the `RuntimeError`, and 3 of 20 failed. This is a **draft** until validation of the fix against both the repro and the original suite is posted here. Possible follow-up, not changed here: `py3.11-anthropic-v0.16.0` also pins `filelock==3.32.4`. #### Issues * resolves: #7014 * Linear: PY-2660 <!-- junior-request-attribution:start --> via Junior system actor `event`. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:agent-dispatch%3Adispatch_6df7374375b237816e130534b6439833 --> -- [View Junior Session](https://junior-prod.sentry.dev/conversations/agent-dispatch%3Adispatch_6df7374375b237816e130534b6439833) [[Sentry]](https://sentry.sentry.io/explore/conversations/agent-dispatch%3Adispatch_6df7374375b237816e130534b6439833/?project=4510944073809921) <!-- junior-session-footer:end --> Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
|
Upstream issue getsentry/sentry-python#7014 has been closed. Closing this reproduction PR. Re-open if the issue resurfaces! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reproduction for getsentry/sentry-python#7014. This is the
RuntimeError: Cannot install a trace function while another trace function is being installedflake. It is currently the top flaky test on sentry-python master:tests/integrations/litellm/test_litellm.py::test_binary_content_encoding_mixed_content[False-False]onAI (3.10).What it demonstrates
Three things together cause the flake:
sys.settrace()with a single flag shared by the whole process (python/cpython#98257).threading.settrace.With all three, a new thread can die in
Thread._bootstrap. A single-workerThreadPoolExecutorthen never runs its callback, so the Sentry span that the callback should finish is lost. That is the same failure the litellm tests show.Run it
The README has more controls: py3.12,
--no-filelock,--no-tracer, and a realcoverage run.Local results (Python 3.10.22, 2000 iterations × 16 executors)
via johdax.
--
View Junior Session [Sentry]