Skip to content

fix: Spread each bucket's hourly recheck by up to a tenth - #984

Merged
vvoytovych merged 1 commit into
mainfrom
fix/bucket-recheck-jitter
Nov 3, 2026
Merged

vvoytovych merged 1 commit into
mainfrom
fix/bucket-recheck-jitter

Conversation

@vvoytovych

Copy link
Copy Markdown
Contributor

Related to #978.

Every bucket comes back an hour after its last reconcile. After a restart the manager reconciles the buckets as it engages their projects (656 projects in 7 min on 2026-11-03), so their rechecks come back an hour later in the same wave, and every hour after unless an event moves one. Each recheck now waits wait.Jitter(time.Hour, 0.1): between 60 and 66 minutes, so the wave widens instead of repeating.

Corner cases

case what happens
A bucket with no events It comes back between 60 and 66 minutes after its last reconcile
A bucket with an event The event reconciles it at once, as before
The buckets a restart reconciles together Their rechecks spread over up to 6 more minutes each hour

Test plan

  • TestAllowanceBucketComesBackAfterTheRecheckInterval reconciles 20 times: each recheck lies in [1 h, 1.1 h), and they are not all equal. On main all 20 are 1h0m0s, so it fails; here it passes.
  • A planted factor of 0.2 turns it red. go test ./internal/quota/... passes with envtest 1.37.0.

Every bucket comes back an hour after its last reconcile. After a restart the manager reconciles the buckets as it
engages their projects, so their rechecks come back an hour later in the same wave, and every hour after unless an
event moves one. wait.Jitter spreads each recheck over up to 6 more minutes, so the wave widens instead of repeating.
@vvoytovych vvoytovych self-assigned this Nov 3, 2026
@vvoytovych
vvoytovych marked this pull request as ready for review November 3, 2026 20:13
@vvoytovych
vvoytovych requested a review from ecv November 3, 2026 20:13

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

wait.Jitter(bucketRecheckInterval, 0.1) returns between 60 and 66 minutes, so buckets reconciled together after a restart stop coming back in the same minute; this is the jitter follow-up from #978.

  • Jitter only ever lengthens the interval, so no bucket is rechecked sooner than before, and the stranded-reservation recovery still runs within 66 minutes.
  • The test bounds every result to [1h, 1.1h) and requires more than one distinct value across 20 reconciles, so it fails if the jitter is dropped. With continuous random durations a false failure is effectively impossible.

@vvoytovych
vvoytovych merged commit 7e6f18c into main Nov 3, 2026
9 checks passed
@vvoytovych
vvoytovych deleted the fix/bucket-recheck-jitter branch November 3, 2026 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants