Skip to content

fix(oauth): rebuild the OAuthRefreshTokenSource refresh lock per event loop - #3790

Open
pcbeingused333 wants to merge 1 commit into
deepset-ai:mainfrom
pcbeingused333:fix/oauth-refresh-lock-event-loop
Open

fix(oauth): rebuild the OAuthRefreshTokenSource refresh lock per event loop#3790
pcbeingused333 wants to merge 1 commit into
deepset-ai:mainfrom
pcbeingused333:fix/oauth-refresh-lock-event-loop

Conversation

@pcbeingused333

Copy link
Copy Markdown

Related Issues

Proposed Changes:

OAuthRefreshTokenSource.resolve_async built its asyncio.Lock lazily and then kept it for the lifetime of the source. An asyncio.Lock binds to the loop that first awaits it under contention and raises RuntimeError when awaited from another one, so a source reused across event loops — one asyncio.run per request is a common deployment — failed on the second loop's first contended refresh:

RuntimeError: <asyncio.locks.Lock object> is bound to a different event loop

Contention is not an edge case here: collapsing a burst of concurrent callers into a single network refresh is the only reason the lock exists.

_get_async_lock now returns the lock for the running loop and rebuilds it when the loop changes. A dedicated _async_lock_guard (a threading.Lock) covers the check and the assignment, never an await, so two threads driving separate loops cannot each install a lock and lose mutual exclusion between them.

_sync_lock was deliberately not reused for that guard: resolve holds it across a blocking network call, so an async caller waiting on it would stall its own event loop for the duration of a token refresh.

OAuthTokenExchangeSource needed no change — it holds a threading.Lock only around cache access and never across the network call, as its own comment says.

How did you test it?

Two tests added to TestOAuthRefreshTokenSource:

  • test_resolve_async_works_in_a_second_event_loop is the regression pin: it drives two concurrent callers under asyncio.run, then does it again on a fresh loop with the same source.
  • test_concurrent_resolve_async_refreshes_once pins the property the lock exists for — four concurrent callers, one network refresh. It passes before and after; it is there so a future change to the locking cannot quietly drop the coalescing.

I checked the regression test is not vacuous: with sources.py reverted to main it fails with the RuntimeError above, raised from asyncio/mixins.py:20 — the bug itself, not an incidental assertion mismatch. The standalone reproduction in #3789 was run both ways too (fails on main, passes on this branch).

Both halves of the test have to contend for the lock or it proves nothing: an uncontended Lock.acquire() returns before it ever calls _get_loop(), so the loop is never bound. That is exactly why the existing test_resolve_async_success_and_caches did not catch this, and the _slow_post helper exists to force the overlap.

hatch run test:unit tests/test_sources.py → 33 passed. hatch run test:types and hatch run fmt-check . clean.

Notes for the reviewer

Two judgement calls worth a look:

  1. Rebuilding on loop change rather than keeping a lock per loop. If two event loops in two threads use the same source simultaneously, each one rebuilds the lock and mutual exclusion between them is lost. A WeakKeyDictionary keyed by loop would hold in that case too. I did not do it because the class is documented as single-identity and "use a single instance in either sync or async mode, not both", so the realistic pattern is sequential loops, not simultaneous ones — but say the word and I will switch it.

  2. The extra threading.Lock. It is one more attribute on a class that already has one lock. Without it, the check-and-set is not atomic across threads. Happy to drop it if you consider cross-thread use out of scope.

Checklist

…t loop

`resolve_async` created its `asyncio.Lock` lazily and then kept it for the
lifetime of the source. An `asyncio.Lock` binds itself to the loop that first
awaits it under contention and raises `RuntimeError` when awaited from any
other one, so a source reused across event loops — one `asyncio.run` per
request is a common deployment — failed on the second loop's first contended
refresh with:

    RuntimeError: <asyncio.locks.Lock object> is bound to a different event loop

Contention is not an edge case here: collapsing a burst of concurrent callers
into a single network refresh is the only reason the lock exists. An
uncontended acquire never binds the loop, which is why the existing async test
did not catch this.

`_get_async_lock` now returns the lock for the running loop and rebuilds it
when the loop changes. A dedicated `_async_lock_guard` covers the check and the
assignment — never an await — so two threads driving separate loops cannot each
install a lock and lose mutual exclusion between them. `_sync_lock` was not
reused for this: `resolve` holds it across a blocking network call, and an
async caller waiting on it would stall its event loop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pcbeingused333
pcbeingused333 requested a review from a team as a code owner August 16, 2026 03:42
@pcbeingused333
pcbeingused333 requested review from bogdankostic and removed request for a team August 16, 2026 03:42
@github-actions github-actions Bot added integration:oauth type:documentation Improvements or additions to documentation labels Aug 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Coverage report (oauth)

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  integrations/oauth/src/haystack_integrations/utils/oauth
  sources.py
Project Total  

This report was generated by python-coverage-comment-action

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @pcbeingused333, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from bogdankostic August 16, 2026 04:59
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Aug 16, 2026
@HaystackBot
HaystackBot marked this pull request as draft August 16, 2026 04:59
@CLAassistant

CLAassistant commented Aug 17, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@HaystackBot
HaystackBot marked this pull request as ready for review August 17, 2026 22:55
@HaystackBot

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @pcbeingused333! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Aug 17, 2026
@sjrl sjrl assigned bogdankostic and unassigned bogdankostic Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

integration:oauth type:documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

oauth: OAuthRefreshTokenSource caches its asyncio.Lock across event loops, breaking reuse in a new loop

5 participants