Serialize LTI Advantage token refreshes with an advisory lock

[santicomp2014]

The problem

There is one cached LTIA token per (registration, scopes), enforced by a unique constraint, and JWTOAuth2TokenService.EXPIRATION_LEEWAY retires it 60s before it actually expires. Canvas tokens live 3600s, so once an hour that single row goes stale — and _get_access_token had no concurrency control, so every request in flight at that moment missed the cache and issued its own identical client_credentials POST.

Herd size is therefore equal to concurrency at the moment of expiry, which is why this has been harmless for years and then wasn't: a school with two concurrent grading reads sends two token requests, while one large institution at the start of term sends dozens, every hour, and the LMS starts throttling the developer key. Instructure throttles by queueing rather than returning 429, so the requests hang until they time out with no status code at all.

The fix

Take the same advisory lock oauth_http already takes for user-token refreshes, so exactly one request refreshes and the rest retry.

Two details worth knowing:

The lock is keyed on LTIRegistration.id, not on the jwt_oauth2_token row, because there is no row to key on the first time a registration needs a token. Registrations using several scope sets will serialize against each other, which is broader than strictly necessary and harmless.

The loser raises ConcurrentTokenRefreshError instead of waiting for the winner. It cannot usefully wait: the winner's token only becomes visible when its transaction commits, at the end of its request, not ours — so a waiter would re-read, find nothing, and refresh anyway. Raising surfaces as a 409, which the frontend already treats as retryable (utils/api.ts — status === 409, maxRetries = 10, 1s backoff), and the retry arrives as a fresh request with a fresh transaction that can see the committed token.

We re-read the token after acquiring the lock, since the winner may have committed between our first read and us getting the lock.

Changes

• lms/db/_locks.py — new LockType.LTIA_TOKEN_REFRESH = 2
• lms/services/jwt_oauth2_token.py — new try_lock_for_refresh()
• lms/services/ltia_http.py — _get_access_token now locks, re-reads, then refreshes

Testing

New tests added:

• test_try_lock_for_refresh — asserts the lock is keyed on the registration
• test_request_locks_before_refreshing_the_token
• test_request_raises_ConcurrentTokenRefreshError_when_locked — asserts the loser makes no token request
• test_request_uses_a_token_refreshed_while_it_waited_for_the_lock — covers the re-read

What to expect in Sentry after this deploys

Two things will look like regressions and aren't.

1. The token-timeout issue ID has moved. #7449 renamed the exception on this path to
LTIATokenRequestError, so LMS-1CY has flatlined and the same failure now lands under a new
issue. When verifying this PR, read the transaction rather than an issue ID:

transaction:lti_api.submissions.record error.type:[ExternalRequestError, LTIATokenRequestError]

2. concurrent_token_refresh 409s will appear on this path for the first time. That's the
lock working — the request that loses the refresh race raises ConcurrentTokenRefreshError,
which the existing exception view maps to 409, and the frontend already retries 409s. No
frontend change was needed.

How to read that rate:

• A few 409s = correct behaviour. One request refreshes, the rest retry and find the token.
• A flood of 409s means refreshes are taking long enough that losers exhaust maxRetries: 2.
  That points at the token endpoint still being slow, not at this change being wrong.

No threshold on that one yet — there's no baseline. Dashboard for a week, then set one.

Success criterion for this PR: token POSTs per expiry, per registration, should be 1
rather than equal to concurrency. ECU's submission-to-launch ratio should move from ~0.040 back
toward its healthy 0.241.

🤖 Generated with Claude Code