feat(capacity): improve FlavorGroupCapacity CRD observability - #942
Merged
juliusclausnitzer merged 5 commits intoJul 17, 2026
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
🚧 Files skipped from review as they are similar to previous changes (6)
📝 WalkthroughWalkthroughThe capacity reconciler marks CRDs not ready after scheduler probe failures. Capacity calculation returns a dedicated not-ready error, the API responds with 503, and readiness is exported as a Prometheus gauge with a corresponding Cortex Nova alert. ChangesCapacity readiness handling
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Scheduler
participant CapacityReconciler
participant FlavorGroupCapacity
participant CapacityCalculator
participant ReportCapacityHandler
participant Prometheus
Scheduler-->>CapacityReconciler: return probe failure
CapacityReconciler->>FlavorGroupCapacity: patch Ready=False
CapacityCalculator-->>ReportCapacityHandler: return ErrCapacityNotReady
ReportCapacityHandler-->>ReportCapacityHandler: return HTTP 503
FlavorGroupCapacity-->>Prometheus: expose readiness gauge
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
juliusclausnitzer
marked this pull request as ready for review
July 16, 2026 09:41
juliusclausnitzer
requested review from
PhilippMatthes,
SoWieMarkus,
auhlig,
mblos and
umswmayj
as code owners
July 16, 2026 09:41
juliusclausnitzer
changed the title
mblos
requested changes
Jul 16, 2026
mblos
left a comment
mblos
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for the PR 👍
- we should update the capacity controller in this pr.. it never writes Ready=False on failure.
- once the controller sets ready=false, the capacity API should return 5xx
juliusclausnitzer
force-pushed
the
wm-469-flavor-group-capacity-observability
branch
3 times, most recently
from
July 17, 2026 08:37
19b7aa1 to
6f7ec22
Compare
mblos
previously approved these changes
Jul 17, 2026
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/scheduling/reservations/commitments/capacity.go`:
- Around line 93-96: Update the stale FlavorGroupCapacity log in the capacity
calculation flow to state that the calculation is being aborted or capacity is
unavailable, matching the error return of ErrCapacityNotReady; remove the
inaccurate “reporting capacity without usage” wording while preserving the
existing context fields.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: ba9315ae-96c3-4642-9687-00881e3fff23
📒 Files selected for processing (7)
helm/bundles/cortex-nova/templates/alerts.yamlinternal/scheduling/reservations/capacity/controller.gointernal/scheduling/reservations/capacity/controller_test.gointernal/scheduling/reservations/capacity/metrics.gointernal/scheduling/reservations/commitments/api/report_capacity.gointernal/scheduling/reservations/commitments/api/report_capacity_test.gointernal/scheduling/reservations/commitments/capacity.go
🚧 Files skipped from review as they are similar to previous changes (2)
- helm/bundles/cortex-nova/templates/alerts.yaml
- internal/scheduling/reservations/capacity/controller_test.go
Add cortex_committed_resource_capacity_ready{flavor_group, az} gauge that
is 1 when the FlavorGroupCapacity CRD's Ready condition is True and 0 when
False. Export it from the existing Monitor collector.
Add CortexNovaCommittedResourceCapacityNotReady PrometheusRule alert that
fires after 10 minutes of ready == 0, pointing to the committed-resource-
capacity playbook.
Add a playbook entry to docs/reservations/committed-resource-reservations.md
describing the alert and remediation steps.
Closes #469
Signed-off-by: Julius Clausnitzer <julius.clausnitzer@sap.com>
…ready The capacity controller previously skipped CRD writes silently on scheduler probe failure, leaving Ready=True indefinitely. The capacity API then served stale data without any visible signal to callers. - Controller: markCRDNotReady patches Ready=False on existing CRDs when probes fail, so the readiness gauge and alert fire correctly. - API: CalculateCapacity returns ErrCapacityNotReady when a CRD has Ready=False; the handler maps this to 503 Service Unavailable. - Metrics: use apimeta.IsStatusConditionTrue instead of a manual loop. Signed-off-by: Julius Clausnitzer <julius.clausnitzer@sap.com>
juliusclausnitzer
force-pushed
the
wm-469-flavor-group-capacity-observability
branch
2 times, most recently
from
July 17, 2026 12:22
5d0ad75 to
c9c289b
Compare
…pacity Signed-off-by: Julius Clausnitzer <julius.clausnitzer@sap.com>
Contributor
Test Coverage ReportTest Coverage 📊: 70.3% |
mblos
approved these changes
Jul 17, 2026
Merged
umswmayj
added a commit
that referenced
this pull request
Jul 23, 2026
## Release cortex v0.3.2 ### cortex v0.3.2 (sha-378ee2f5) Non-breaking changes: - Filter KVM hypervisors for non-KVM images — adds a new Nova scheduler filter that removes KVM hypervisors from scheduling candidates when the image declares a non-KVM hypervisor type ([#1067](#1067)) - Improve FlavorGroupCapacity CRD observability with better metrics and status reporting ([#942](#942)) - Protect pipeline maps with RWMutex to prevent concurrent map read/write data race ([#1066](#1066)) - Fix broken relative links in quickstart and develop docs ([#1063](#1063)) - Update `actions/setup-go` to v7 ([#1049](#1049)) - Update `actions/setup-python` to v7 ([#1062](#1062)) - Update `github.com/sapcc/go-bits` ([#1059](#1059), [#1071](#1071), [#1074](#1074)) - Update external dependencies: `actions/checkout` v7.0.1, `go-logr/logr` v1.4.4, `prometheus/client_golang` v1.24.0 ([#1072](#1072)) ### cortex-shim v0.1.8 (sha-378ee2f5) Includes updated image sha-378ee2f5. ### cortex-postgres v0.6.10 (sha-e06153f8) Includes updated image sha-e06153f8. ### cortex-nova v0.0.82 Includes updated charts cortex v0.3.2, cortex-postgres v0.6.10. - Add alert for missing hypervisor properties and KVM pipeline registration ([#1067](#1067)) - Update `kube-prometheus-stack` to v87.19.0 ([#1061](#1061), [#1075](#1075)) ### cortex-cinder v0.0.82 Includes updated charts cortex v0.3.2, cortex-postgres v0.6.10. ### cortex-manila v0.0.82 Includes updated charts cortex v0.3.2, cortex-postgres v0.6.10. ### cortex-crds v0.0.82 Includes updated chart cortex v0.3.2. ### cortex-ironcore v0.0.82 Includes updated chart cortex v0.3.2. ### cortex-pods v0.0.82 Includes updated chart cortex v0.3.2. ### cortex-placement-shim v0.1.8 Includes updated chart cortex-shim v0.1.8. ## Dependencies - Prep PR: #1076 (must be merged before this PR)
juliusclausnitzer
added a commit
that referenced
this pull request
Jul 28, 2026
Signed-off-by: Julius Clausnitzer <julius.clausnitzer@sap.com> Signed-off-by: juliusclausnitzer <julius.clausnitzer@sap.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.