fix(flyteidl): auth interceptor CondWait deadlock on failed token refresh

[SVilgelm]

Why are the changes needed?

The auth interceptor's token-refresh path can permanently deadlock a caller. In NewAuthInterceptor, when a request fails with an authentication error:

if !tokenCache.TryLock() {
    tokenCache.CondWait()   // bare sync.Cond.Wait — no timeout
    return nil
}
defer tokenCache.Unlock()
...
newErr := MaterializeCredentials(...)
if newErr != nil {
    return errors.New(errString)  // returns WITHOUT CondBroadcast
}
tokenCache.CondBroadcast()        // broadcast only on success

Two defects combine:

1. The TryLock winner broadcasts only on success. If MaterializeCredentials (or the cache purge) fails — e.g. the auth server is briefly unreachable — it returns without waking waiters.
2. TokenCacheInMemoryProvider.CondWait is a bare sync.Cond.Wait over a no-op locker: no timeout, and no check-then-wait mutual exclusion, so a broadcast that fires just before the wait is lost (sync.Cond has no memory).

A waiter parked in CondWait then sleeps until the next successful refresh — which on a low-traffic client may never come. We hit this in production: a goroutine was blocked in CondWait for 9+ hours (confirmed by goroutine dump), hanging its caller indefinitely. The RPC deadline cannot help because the wait happens inside the interceptor, outside context control.

What changes were proposed in this pull request?

• auth_interceptor.go: broadcast is now deferred by the TryLock winner, so every exit path (including refresh failure) wakes waiters.
• cache/token_cache_inmemory.go: CondWait/CondBroadcast now use a close-and-replace notification channel (atomic.Pointer + Swap, so concurrent broadcasters never double-close) with a 30s bound on the wait. A missed or raced broadcast degrades to a short delay followed by the caller's normal retry instead of an unbounded hang.

No interface changes; NoopLocker is kept (still used by flytectl's keyring token cache).

How was this patch tested?

• New token_cache_inmemory_test.go: broadcast wakes all waiters; wait times out when no broadcast comes; wait after a missed (pre-wait) broadcast returns; concurrent broadcasts don't panic. Passes with -race.
• Extended the existing failed-refresh interceptor test to assert CondBroadcast is called on the error path.

Labels

fixed

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

unionai/flyte#1009 (same fix in Union's fork, where the production incident occurred)

Docs link

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.27%. Comparing base (e69f764) to head (1ad27f0).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7946      +/-   ##
==========================================
+ Coverage   57.25%   57.27%   +0.02%     
==========================================
  Files         931      931              
  Lines       58308    58311       +3     
==========================================
+ Hits        33383    33399      +16     
+ Misses      21866    21853      -13     
  Partials     3059     3059              

Flag	Coverage Δ
unittests-datacatalog	53.51% <ø> (ø)
unittests-flyteadmin	53.23% <ø> (ø)
unittests-flytecopilot	48.05% <ø> (ø)
unittests-flytectl	64.11% <ø> (ø)
unittests-flyteidl	76.63% <100.00%> (+0.92%)	⬆️
unittests-flyteplugins	60.45% <ø> (ø)
unittests-flytepropeller	53.81% <ø> (ø)
unittests-flytestdlib	64.39% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.