Skip to content

While GitHub is rate-limited, Azure DevOps (and everything else) stops updating for up to an hour: the shared update loop sleeps out GitHub's reset inside each request #384

Open

Description

@matt-edmondson

What's wrong

  • GitHub.MakeGitHubRequestAsync (BuildMonitor/Providers/GitHub.cs, around the "Use smart waiting" block) calls await Task.Delay(GetRateLimitWaitTime()) before every request.
  • When the provider is RateLimited with a known RateLimitResetTime, BuildProvider.GetRateLimitWaitTime() (BuildProvider.cs ~L335) returns min(reset − now + 5 s, 10 min).
  • BuildMonitor.UpdateAsync() (BuildMonitor.cs ~L1600–1661) runs one pass across all providers:
    • it awaits repository discovery for every owner (SyncGuard.RunAllAsync),
    • then Task.WhenAll over every provider's build and run syncs.
  • OnRender (~L439) starts a new UpdateAsync() only once the previous one has completed.

So a GitHub request that is sleeping out the rate-limit reset holds the whole pass open. Azure DevOps builds and runs are polled only from that same pass, so they stop updating too. So do GitHub owners whose own tokens still have budget.

Failure scenario

  1. GitHub returns 403 with X-RateLimit-Remaining: 0 and a reset up to 60 minutes away (primary limit).
  2. Every GitHub request in the pass now waits up to 10 min, fires, is refused again while the reset is still ahead, and the next pass waits again.
  3. For up to ~60–70 minutes, an in-progress or just-failed Azure DevOps run is never re-polled. The 10 s run interval and 30 s build interval have no effect, and the table shows stale ADO status with no indication why.

The review that found this verified it with a scratch test: with a GitHub provider rate-limited (reset 30 min ahead) and one owner, BuildMonitor.UpdateAsync() had not completed after 5 s (completed=False, status=RateLimited).

The same coupling, at a smaller scale, delays every provider whenever GitHub's adaptive pacing reaches its long-wait range.

Suggested fix

  • Don't sleep out a rate-limit reset inside the shared pass. When IsRateLimitedWithPendingReset() is true, skip that provider's syncs for this pass (leave them due) instead of awaiting the reset in MakeGitHubRequestAsync. Keep only short pacing delays in-line.
  • Alternatively, run an independent update loop per provider, so one provider's backoff can't hold up another's.

Acceptance criteria

  • With the GitHub provider rate-limited and its reset minutes away, Azure DevOps builds and runs keep being polled on their normal intervals. Add a test for this.
  • GitHub syncs resume on the first pass after the reset time.

Related, but not the same problem: #331 (one owner token's limit throttles the other GitHub owners, a single provider-wide budget), #341 (pacing runs per slot, so it goes too fast), #333 (budget never re-read after the reset), #305 (an ADO auth failure stalled the loop through a different cause).

Activity

  1. matt-edmondson commented on Oct 29, 2026

    @matt-edmondson
    ContributorAuthor

    Triage


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions