Skip to content

fix: Decide a claim's requests of one type together - #978

Merged
vvoytovych merged 7 commits into
mainfrom
fix/bucket-requests-of-one-type
Nov 3, 2026
Merged

vvoytovych merged 7 commits into
mainfrom
fix/bucket-requests-of-one-type

Conversation

@vvoytovych

Copy link
Copy Markdown
Contributor

Related to #965.

These are ecv's three follow-ups from #965's review.

change why
A claim's requests of one type are decided as one request, their amounts summed. A claim holds one allocation per resource type, keyed by type. A reservation per request left one allocation and kept the other amount reserved. This is older than #949: the test fails on main too.
Every bucket comes back an hour after its last reconcile. A reservation that no claim holds shows as a cached total below Allocated once the cache is current, and that makes a live recompute. A quiet bucket gets no event to show it. Each recheck costs one bucket GET and one grant list, and a claim list only when the totals differ: one GET and one list an hour for each bucket.
An envtest case races 20 claims against a limit of 19 on a real API server. The fake client cannot make two writers meet a resourceVersion conflict.
One pending request per claim and type: the decision loop and its grant flag go. With the requests of one type summed, the loop ran at most once.

Test plan

  • TestAllowanceBucketDecidesAClaimsRequestsOfOneTypeTogether fails on main, where the claim holds 2 and the bucket reserves 3, and passes here. TestAllowanceBucketComesBackAfterTheRecheckInterval holds the recheck. The usual-denial test now also checks that a denied claim is not counted.
  • TestAllowanceBucketGrantsExactlyItsLimitAgainstARealAPIServer: 20 claims created while one controller decides them, and 20 pending claims when two controllers start at once. Both give exactly 19 grants and one denial, three runs each with -race, here and on fix: Stop listing all claims live on every bucket reconcile #965's head. A conflict retried with the stale allocation makes the second case fail, three runs of three.
  • Thirteen planted defects, one per mechanism of the bucket controller, each turn their own test red. go test -race ./internal/quota/... and go test -timeout 5m ./... pass with envtest 1.37.0. The core package now takes about 52 s.
  • After a staging rollout: the allowance-bucket reconcile rate rises by about one a bucket an hour. Cluster-scope ResourceClaim lists per bucket reconcile stay at or below fix: Stop listing all claims live on every bucket reconcile #965's on staging, by kind of minute: 0.04–0.07 in a grant burst, 0.31–0.33 in delete-heavy minutes.

A claim holds one allocation per resource type, but each of its requests of that type was reserved on its own; the
last grant overwrote the allocation and the bucket kept the other amount. The requests of one type now make one
request of their summed amount: one reservation, one allocation. Older than #949.
A reservation that no claim holds shows as a cached total below the live allocation once the cache is current, and
that makes a live recompute. A quiet bucket gets no event to show it, so a reconcile now comes back after an hour:
one bucket GET and one grant LIST of the consumer each time, a claim LIST only when the totals differ.
A claim's requests of one type are summed into one request, so the decision loop and its grant flag over at most one
request go; the claim is counted when that request is granted.
@vvoytovych vvoytovych self-assigned this Nov 3, 2026
@vvoytovych
vvoytovych marked this pull request as ready for review November 3, 2026 07:21
@vvoytovych
vvoytovych requested review from ecv and scotwells November 3, 2026 07:21

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

Allocations are keyed by resource type, so deciding a claim's same-type requests as one summed amount is the right unit: the old per-request loop reserved per request, and each claim write replaced the previous allocation of that type, leaking the earlier reservation. The all-or-nothing outcome on the sum is the correct answer when a claim is only partly satisfiable.

  • Partial fit: with limit 3, allocated 1 and a claim of 1+2, the old code granted the first, reserved 1, then denied the second and overwrote the allocation as Denied, so 1 stayed reserved with no claim holding it. When both fit, it reserved both but kept only the last allocation. Now the sum is one decision; a denial writes nothing to the bucket, and a grant reserves the sum that grantedAmount later sums back (TestAllowanceBucketDecidesAClaimsRequestsOfOneTypeTogether).
  • Concurrency: the reservation is still a bucket Status().Update on a live read's resourceVersion before the claim write, so with one active controller reservations serialize. The new envtest runs 20 claims against a limit of 19 with one and with two controllers and asserts 19 granted, 1 denied.
  • Stranded reservation: if the claim write fails after the bucket write, nothing reverts; the hourly requeue brings the bucket back, the cached total disagrees with Allocated, and it recomputes from live claims. TestAllowanceBucketComesBackAfterTheRecheckInterval pins only the requeue interval; the recompute is covered by the earlier tests.
  • ClaimCount counts grants only, as before; the updated test now pins that a denied claim is not counted.

Follow-ups:

  • Two active controllers (a leader handoff) can still over-grant, from reading the code: B reserves but has not written its claim; A reads the bucket, sees cached total below Allocated, recomputes Allocated from live claims without B's in-flight reservation, and A's update succeeds on that fresh resourceVersion. The two-controller envtest passing shows the race did not trigger, not that it is closed. Excluding in-flight reservations from the recompute, or not recomputing in the same pass that grants, would close it.
  • The envtest workflow runs on push only, so the race test does not gate pull requests.
  • The CRD schema does not reject two requests of one type (only the doc comment says once). A CEL rule would make the summing a safety net rather than a live path.
  • The hourly RequeueAfter has no jitter, so after a restart every bucket rechecks in the same minute. Add jitter if bucket count grows.

@vvoytovych

Copy link
Copy Markdown
Contributor Author

Your first follow-up holds, and it predates this PR.

I drove the handoff step by step against a real API server, with a limit of 1 and two pending claims:

  1. B reserves the last unit for handoff-b and has not written the claim yet.
  2. A reconciles in that gap. Its cached claims disagree with the bucket, so it recomputes Allocated from the live claims and drops B's reservation. It grants handoff-a and denies handoff-b.
  3. B applies handoff-b as granted. The apply carries no precondition, so it replaces A's denial.

Two claims end granted against a limit of 1. The test fails the same way on main (c457001) and on this PR's head (fde11cc), 3 runs of 3 on each. This PR does not change that path. So I would merge it as approved, and fix the handoff in its own PR next, with this test.

The other three follow-ups each get their own PR after this merges:

  • Run the envtest workflow on pull requests too, so the race tests gate a change.
  • Make the requests list +listType=map with +listMapKey=resourceType, so the API server rejects two requests of one type.
  • Add jitter to the hourly recheck.

@vvoytovych
vvoytovych merged commit 9594f1b into main Nov 3, 2026
9 checks passed
@vvoytovych
vvoytovych deleted the fix/bucket-requests-of-one-type branch November 3, 2026 15:34
@vvoytovych

Copy link
Copy Markdown
Contributor Author

One correction to my reply above. I wrote that the envtest workflow should run on pull requests so the race tests gate a change. They already do.

The required check on main is ci / Build and test. Its workflow, ci.yaml, runs on pull requests, installs envtest 1.37.0 and runs go test ./.... On this PR's run, the quota controller package took 51.7 s, so its envtests ran: https://github.com/milo-os/milo/actions/runs/41592340528

"IAM API testing" runs the same go test ./... on push only and is not required, so it duplicates that work. The follow-up becomes removing it, as datum-cloud/network-services-operator#849 did for NSO.

vvoytovych added a commit that referenced this pull request Nov 3, 2026
Related to #978.

The required check, `ci / Build and test`, runs `go test ./...` with
envtest 1.37.0 on every pull request and on `main`. "IAM API testing"
ran the same `go test ./...` on every push that changed Go files. It is
not required, so it gated nothing, and each pull request ran the suite
twice. It goes, as
[datum-cloud/network-services-operator#849](datum-cloud/network-services-operator#849)
did for NSO, and CI takes the concurrency group the other workflows
share.

## Where each step of IAM API testing runs now

| step | where |
|---|---|
| `go mod download` | CI: `go test` downloads what it needs |
| envtest 1.37.0 | CI: `setup-envtest use 1.37.0`, pinned by commit |
| `go test ./...` | CI, with a 10 min timeout instead of 5 |

## Corner cases

| case | what happens |
|---|---|
| A push to a branch with no pull request | No suite run until its pull
request opens. Milo has no release branches; it releases by tag from
`main`. |
| A newer push to a pull request | The older CI run is cancelled. |
| A push to `main` | Its run is never cancelled: `cancel-in-progress` is
false on the default branch. |

## Test plan

- [x] `ci / Build and test` ran the suite on #978's pull request and on
this one: the quota controller package's envtests took [51.7
s](https://github.com/milo-os/milo/actions/runs/41592340528) and [52.3
s](https://github.com/milo-os/milo/actions/runs/41706707445/job/126163821686).
- [x] `actionlint` passes on `ci.yaml`. The concurrency group is the one
in `fix-newlines.yaml`, `test-environment-validation.yaml` and
`test-prometheus-rules.yaml`, word for word.
- [x] Nothing in the repository names `testing.yaml` or "IAM API
testing".
- [ ] After the merge: the push to `main` runs CI's suite, and "IAM API
testing" no longer runs.
vvoytovych added a commit that referenced this pull request Nov 3, 2026
Related to #978.

A claim holds one allocation per resource type, but its schema let it
request one type twice; only the doc comment said once. The requests are
now a map list keyed by `resourceType`, as the claim's allocations
already are, so the API server refuses the second request. A claim
stored before keeps working, because the API server ratchets a list-map
rule that the stored object already broke.

## Corner cases

| case | what happens | proof |
|---|---|---|
| A new claim requests one type twice | Refused as invalid |
`TestResourceClaimRefusesTwoRequestsOfOneType` |
| A claim stored before, with one type twice | Its spec, status and
finalizer updates and its deletion are accepted |
`TestResourceClaimStoredWithTwoRequestsOfOneTypeStaysUpdatable`, and
apiextensions-apiserver's
[ratcheting](https://github.com/kubernetes/apiextensions-apiserver/blob/v0.37.1/pkg/registry/customresource/strategy.go#L299-L302)
|
| A ClaimCreationPolicy whose templates request one type twice | Its
claims are refused, so the resource it claims for is refused | None
does: the 28 policies across the services' repositories request one type
each |
| A server-side apply of a claim's requests | Merges by `resourceType`
instead of replacing the list | Milo writes a claim's spec only by
create (the admission plugin); its status and owner references by apply
|

## Test plan

- [x] Both tests fail on `main` (the claim is stored; the upgraded
schema never refuses) and pass here, 3 runs of 3, with envtest 1.37.0.
- [x] `task generate` changes only the claim CRD's two list lines. `go
test ./pkg/apis/quota/... ./internal/quota/...` passes.
- [x] Production holds no claim that requests one type twice: 0 of 6,166
claims in the 151 projects that hold claims, read 2026-11-03
16:47–16:50Z; the count finds a planted one. A code search of
datum-cloud and milo-os (102 files, all 47 YAML ones read) finds 28
ClaimCreationPolicies, in datum-cloud/datum,
datum-cloud/network-services-operator and this repository, and each
requests one type. A policy applied by hand in production cannot be
listed with a staff login.
vvoytovych added a commit that referenced this pull request Nov 3, 2026
Related to #978.

Every bucket comes back an hour after its last reconcile. After a
restart the manager reconciles the buckets as it engages their projects
(656 projects in 7 min on 2026-11-03), so their rechecks come back an
hour later in the same wave, and every hour after unless an event moves
one. Each recheck now waits `wait.Jitter(time.Hour, 0.1)`: between 60
and 66 minutes, so the wave widens instead of repeating.

## Corner cases

| case | what happens |
|---|---|
| A bucket with no events | It comes back between 60 and 66 minutes
after its last reconcile |
| A bucket with an event | The event reconciles it at once, as before |
| The buckets a restart reconciles together | Their rechecks spread over
up to 6 more minutes each hour |

## Test plan

- [x] `TestAllowanceBucketComesBackAfterTheRecheckInterval` reconciles
20 times: each recheck lies in [1 h, 1.1 h), and they are not all equal.
On `main` all 20 are 1h0m0s, so it fails; here it passes.
- [x] A planted factor of 0.2 turns it red. `go test
./internal/quota/...` passes with envtest 1.37.0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants