Repository navigation
fix: Decide a claim's requests of one type together - #978
Conversation
A claim holds one allocation per resource type, but each of its requests of that type was reserved on its own; the last grant overwrote the allocation and the bucket kept the other amount. The requests of one type now make one request of their summed amount: one reservation, one allocation. Older than #949.
A reservation that no claim holds shows as a cached total below the live allocation once the cache is current, and that makes a live recompute. A quiet bucket gets no event to show it, so a reconcile now comes back after an hour: one bucket GET and one grant LIST of the consumer each time, a claim LIST only when the totals differ.
… against a real API server
…t reservation shows as an over-grant
A claim's requests of one type are summed into one request, so the decision loop and its grant flag over at most one request go; the claim is counted when that request is granted.
ecv
left a comment
There was a problem hiding this comment.
Approve.
Allocations are keyed by resource type, so deciding a claim's same-type requests as one summed amount is the right unit: the old per-request loop reserved per request, and each claim write replaced the previous allocation of that type, leaking the earlier reservation. The all-or-nothing outcome on the sum is the correct answer when a claim is only partly satisfiable.
- Partial fit: with limit 3, allocated 1 and a claim of 1+2, the old code granted the first, reserved 1, then denied the second and overwrote the allocation as Denied, so 1 stayed reserved with no claim holding it. When both fit, it reserved both but kept only the last allocation. Now the sum is one decision; a denial writes nothing to the bucket, and a grant reserves the sum that
grantedAmountlater sums back (TestAllowanceBucketDecidesAClaimsRequestsOfOneTypeTogether). - Concurrency: the reservation is still a bucket
Status().Updateon a live read's resourceVersion before the claim write, so with one active controller reservations serialize. The new envtest runs 20 claims against a limit of 19 with one and with two controllers and asserts 19 granted, 1 denied. - Stranded reservation: if the claim write fails after the bucket write, nothing reverts; the hourly requeue brings the bucket back, the cached total disagrees with
Allocated, and it recomputes from live claims.TestAllowanceBucketComesBackAfterTheRecheckIntervalpins only the requeue interval; the recompute is covered by the earlier tests. ClaimCountcounts grants only, as before; the updated test now pins that a denied claim is not counted.
Follow-ups:
- Two active controllers (a leader handoff) can still over-grant, from reading the code: B reserves but has not written its claim; A reads the bucket, sees cached total below
Allocated, recomputesAllocatedfrom live claims without B's in-flight reservation, and A's update succeeds on that fresh resourceVersion. The two-controller envtest passing shows the race did not trigger, not that it is closed. Excluding in-flight reservations from the recompute, or not recomputing in the same pass that grants, would close it. - The envtest workflow runs on
pushonly, so the race test does not gate pull requests. - The CRD schema does not reject two requests of one type (only the doc comment says once). A CEL rule would make the summing a safety net rather than a live path.
- The hourly
RequeueAfterhas no jitter, so after a restart every bucket rechecks in the same minute. Add jitter if bucket count grows.
|
Your first follow-up holds, and it predates this PR. I drove the handoff step by step against a real API server, with a limit of 1 and two pending claims:
Two claims end granted against a limit of 1. The test fails the same way on The other three follow-ups each get their own PR after this merges:
|
|
One correction to my reply above. I wrote that the envtest workflow should run on pull requests so the race tests gate a change. They already do. The required check on "IAM API testing" runs the same |
Related to #978. The required check, `ci / Build and test`, runs `go test ./...` with envtest 1.37.0 on every pull request and on `main`. "IAM API testing" ran the same `go test ./...` on every push that changed Go files. It is not required, so it gated nothing, and each pull request ran the suite twice. It goes, as [datum-cloud/network-services-operator#849](datum-cloud/network-services-operator#849) did for NSO, and CI takes the concurrency group the other workflows share. ## Where each step of IAM API testing runs now | step | where | |---|---| | `go mod download` | CI: `go test` downloads what it needs | | envtest 1.37.0 | CI: `setup-envtest use 1.37.0`, pinned by commit | | `go test ./...` | CI, with a 10 min timeout instead of 5 | ## Corner cases | case | what happens | |---|---| | A push to a branch with no pull request | No suite run until its pull request opens. Milo has no release branches; it releases by tag from `main`. | | A newer push to a pull request | The older CI run is cancelled. | | A push to `main` | Its run is never cancelled: `cancel-in-progress` is false on the default branch. | ## Test plan - [x] `ci / Build and test` ran the suite on #978's pull request and on this one: the quota controller package's envtests took [51.7 s](https://github.com/milo-os/milo/actions/runs/41592340528) and [52.3 s](https://github.com/milo-os/milo/actions/runs/41706707445/job/126163821686). - [x] `actionlint` passes on `ci.yaml`. The concurrency group is the one in `fix-newlines.yaml`, `test-environment-validation.yaml` and `test-prometheus-rules.yaml`, word for word. - [x] Nothing in the repository names `testing.yaml` or "IAM API testing". - [ ] After the merge: the push to `main` runs CI's suite, and "IAM API testing" no longer runs.
Related to #978. A claim holds one allocation per resource type, but its schema let it request one type twice; only the doc comment said once. The requests are now a map list keyed by `resourceType`, as the claim's allocations already are, so the API server refuses the second request. A claim stored before keeps working, because the API server ratchets a list-map rule that the stored object already broke. ## Corner cases | case | what happens | proof | |---|---|---| | A new claim requests one type twice | Refused as invalid | `TestResourceClaimRefusesTwoRequestsOfOneType` | | A claim stored before, with one type twice | Its spec, status and finalizer updates and its deletion are accepted | `TestResourceClaimStoredWithTwoRequestsOfOneTypeStaysUpdatable`, and apiextensions-apiserver's [ratcheting](https://github.com/kubernetes/apiextensions-apiserver/blob/v0.37.1/pkg/registry/customresource/strategy.go#L299-L302) | | A ClaimCreationPolicy whose templates request one type twice | Its claims are refused, so the resource it claims for is refused | None does: the 28 policies across the services' repositories request one type each | | A server-side apply of a claim's requests | Merges by `resourceType` instead of replacing the list | Milo writes a claim's spec only by create (the admission plugin); its status and owner references by apply | ## Test plan - [x] Both tests fail on `main` (the claim is stored; the upgraded schema never refuses) and pass here, 3 runs of 3, with envtest 1.37.0. - [x] `task generate` changes only the claim CRD's two list lines. `go test ./pkg/apis/quota/... ./internal/quota/...` passes. - [x] Production holds no claim that requests one type twice: 0 of 6,166 claims in the 151 projects that hold claims, read 2026-11-03 16:47–16:50Z; the count finds a planted one. A code search of datum-cloud and milo-os (102 files, all 47 YAML ones read) finds 28 ClaimCreationPolicies, in datum-cloud/datum, datum-cloud/network-services-operator and this repository, and each requests one type. A policy applied by hand in production cannot be listed with a staff login.
Related to #978. Every bucket comes back an hour after its last reconcile. After a restart the manager reconciles the buckets as it engages their projects (656 projects in 7 min on 2026-11-03), so their rechecks come back an hour later in the same wave, and every hour after unless an event moves one. Each recheck now waits `wait.Jitter(time.Hour, 0.1)`: between 60 and 66 minutes, so the wave widens instead of repeating. ## Corner cases | case | what happens | |---|---| | A bucket with no events | It comes back between 60 and 66 minutes after its last reconcile | | A bucket with an event | The event reconciles it at once, as before | | The buckets a restart reconciles together | Their rechecks spread over up to 6 more minutes each hour | ## Test plan - [x] `TestAllowanceBucketComesBackAfterTheRecheckInterval` reconciles 20 times: each recheck lies in [1 h, 1.1 h), and they are not all equal. On `main` all 20 are 1h0m0s, so it fails; here it passes. - [x] A planted factor of 0.2 turns it red. `go test ./internal/quota/...` passes with envtest 1.37.0.
Related to #965.
These are ecv's three follow-ups from #965's review.
maintoo.Allocatedonce the cache is current, and that makes a live recompute. A quiet bucket gets no event to show it. Each recheck costs one bucket GET and one grant list, and a claim list only when the totals differ: one GET and one list an hour for each bucket.Test plan
TestAllowanceBucketDecidesAClaimsRequestsOfOneTypeTogetherfails onmain, where the claim holds 2 and the bucket reserves 3, and passes here.TestAllowanceBucketComesBackAfterTheRecheckIntervalholds the recheck. The usual-denial test now also checks that a denied claim is not counted.TestAllowanceBucketGrantsExactlyItsLimitAgainstARealAPIServer: 20 claims created while one controller decides them, and 20 pending claims when two controllers start at once. Both give exactly 19 grants and one denial, three runs each with-race, here and on fix: Stop listing all claims live on every bucket reconcile #965's head. A conflict retried with the stale allocation makes the second case fail, three runs of three.go test -race ./internal/quota/...andgo test -timeout 5m ./...pass with envtest 1.37.0. The core package now takes about 52 s.