Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
9f115a5
Add check-trigger-phrase job to claude workflow to reduce noisiness (…
PhilippMatthes Jul 6, 2026
9874e42
docs(apis): update FlavorGroupCapacity status fields after #983 (#1016)
cortex-ai-agents[bot] Jul 7, 2026
93e702a
docs: update changelog for release PR #1009 (#1018)
cortex-ai-agents[bot] Jul 8, 2026
e46710f
chore: add logger context for remote cluster communication (#1023)
SoWieMarkus Jul 8, 2026
70af93a
refactor: number of synced objects (#1024)
SoWieMarkus Jul 8, 2026
0900ede
bump app version [skip ci] (#1020)
github-actions[bot] Jul 8, 2026
b6f7beb
Renovate: Update External dependencies (#1026)
renovate[bot] Jul 8, 2026
628376e
Renovate: Update kube-prometheus-stack Docker tag to v87.12.1 (#1028)
renovate[bot] Jul 9, 2026
26105cb
Renovate: Update github.com/sapcc/go-bits digest to f8ef154 (#1030)
renovate[bot] Jul 10, 2026
10175b0
Renovate: Update External dependencies (#1029)
renovate[bot] Jul 10, 2026
be0cfea
Fix alert wording: 'LackingBehind' -> 'LaggingBehind' (#1031)
PhilippMatthes Jul 10, 2026
4ade756
feat(quota-enforcement): split accept_skipped decision into sub-bucke…
umswmayj Jul 10, 2026
f2d4c89
Renovate: Update External dependencies (#1032)
renovate[bot] Jul 11, 2026
d0691ba
fix: increase datasource lagging behind alert time frame to 30 min (#…
SoWieMarkus Jul 13, 2026
ee9cd48
feat(scheduling): migrate CR and failover calls to options-based pipe…
mblos Jul 13, 2026
1914e98
bump app version [skip ci] (#1027)
github-actions[bot] Jul 13, 2026
f710281
refactor(capacity): convert timer-based controller to controller-runt…
juliusclausnitzer Jul 13, 2026
2466110
Bump chart versions for release PR #1036 (#1038)
cortex-ai-agents[bot] Jul 13, 2026
4b1cf48
Add changelog entry for release PR #1036 (#1039)
cortex-ai-agents[bot] Jul 13, 2026
56f4ace
docs: add missing SkipCommittedResourceTracking to Options table and …
cortex-ai-agents[bot] Jul 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 23 additions & 2 deletions .github/workflows/claude-assistant.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,31 @@ jobs:
echo "allowed=false" >> $GITHUB_OUTPUT
fi

check-trigger-phrase:
runs-on: ubuntu-latest
needs: check-allowlist
outputs:
triggered: ${{ steps.check.outputs.triggered }}
steps:
- name: Check for trigger phrase in comment body
id: check
env:
COMMENT_BODY: ${{ github.event.comment.body }}
ISSUE_BODY: ${{ github.event.issue.body }}
ISSUE_TITLE: ${{ github.event.issue.title }}
run: |
if [[ "$COMMENT_BODY" == *"@claude"* ]] \
|| [[ "$ISSUE_BODY" == *"@claude"* ]] \
|| [[ "$ISSUE_TITLE" == *"@claude"* ]]; then
echo "triggered=true" >> $GITHUB_OUTPUT
else
echo "triggered=false" >> $GITHUB_OUTPUT
fi

claude:
environment: claude
needs: check-allowlist
if: needs.check-allowlist.outputs.allowed == 'true'
needs: [check-allowlist, check-trigger-phrase]
if: needs.check-allowlist.outputs.allowed == 'true' && needs.check-trigger-phrase.outputs.triggered == 'true'
runs-on: ubuntu-latest
permissions:
contents: write
Expand Down
73 changes: 67 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,79 @@
# Changelog

## 2026-07-03 — [#1009](https://github.com/cobaltcore-dev/cortex/pull/1009)
## 2026-07-13 — [#1036](https://github.com/cobaltcore-dev/cortex/pull/1036)

### cortex v0.2.2 (sha-f4d7df89)
### cortex v0.3.0 (sha-ee9cd485)

Breaking changes:
- `NumberOfObjects` field in `DatasourceStatus` changed from `int64` to `*int64` so `omitempty` no longer strips zero values ([#1024](https://github.com/cobaltcore-dev/cortex/pull/1024))

Non-breaking changes:
- Migrate CR and failover calls to options-based pipeline selection — eliminates dedicated CR and failover pipelines by encoding behavioral differences as call-time scheduling options ([#950](https://github.com/cobaltcore-dev/cortex/pull/950))
- Convert capacity controller from timer-based to controller-runtime reconciler with reactive watches and coalesced reconcile keys ([#1025](https://github.com/cobaltcore-dev/cortex/pull/1025))
- Split quota enforcement `accept_skipped` decision into sub-buckets for granular observability ([#1022](https://github.com/cobaltcore-dev/cortex/pull/1022))
- Fix alert wording: 'LackingBehind' → 'LaggingBehind' ([#1031](https://github.com/cobaltcore-dev/cortex/pull/1031))
- Increase datasource lagging behind alert time frame to 30 min ([#1035](https://github.com/cobaltcore-dev/cortex/pull/1035))
- Add logger context for remote cluster communication ([#1023](https://github.com/cobaltcore-dev/cortex/pull/1023))
- Update `golang.org/x/sync` v0.21.0→v0.22.0, `golang.org/x/term` v0.44.0→v0.45.0 ([#1026](https://github.com/cobaltcore-dev/cortex/pull/1026))
- Update `ironcore` v0.4.2→v0.4.3 ([#1029](https://github.com/cobaltcore-dev/cortex/pull/1029))
- Update `github.com/sapcc/go-bits` ([#1030](https://github.com/cobaltcore-dev/cortex/pull/1030))

### cortex-shim v0.1.6 (sha-ee9cd485)

Includes updated image sha-ee9cd485.

### cortex-nova v0.0.80

Includes updated charts cortex v0.3.0, cortex-postgres v0.6.8.

- Scope `SyncObjectsDroppedToZero` alert to only fire when previously non-zero ([#1024](https://github.com/cobaltcore-dev/cortex/pull/1024))
- Add `capacityMinReconcileInterval` config key for capacity controller ([#1025](https://github.com/cobaltcore-dev/cortex/pull/1025))

### cortex-cinder v0.0.80

Includes updated charts cortex v0.3.0, cortex-postgres v0.6.8.

- Scope `SyncObjectsDroppedToZero` alert to only fire when previously non-zero ([#1024](https://github.com/cobaltcore-dev/cortex/pull/1024))

### cortex-manila v0.0.80

Includes updated charts cortex v0.3.0, cortex-postgres v0.6.8.

- Scope `SyncObjectsDroppedToZero` alert to only fire when previously non-zero ([#1024](https://github.com/cobaltcore-dev/cortex/pull/1024))

### cortex-crds v0.0.80

Includes updated chart cortex v0.3.0.

### cortex-ironcore v0.0.80

Includes updated chart cortex v0.3.0.

### cortex-pods v0.0.80

Includes updated chart cortex v0.3.0.

### cortex-placement-shim v0.1.6

Includes updated chart cortex-shim v0.1.6.

## 2026-07-06 — [#1009](https://github.com/cobaltcore-dev/cortex/pull/1009)

### cortex v0.2.2 (sha-8d888397)

Breaking changes:
- Remove Nova image datasource — the `images` NovaDatasourceType is no longer available ([#1006](https://github.com/cobaltcore-dev/cortex/pull/1006))

Non-breaking changes:
- Fix stale RunningInstances/RunningResources fields that never zero out when all VMs in a flavor group are removed ([#1017](https://github.com/cobaltcore-dev/cortex/pull/1017))
- Include host information in error logs when multicluster connection fails ([#997](https://github.com/cobaltcore-dev/cortex/pull/997))
- Fix Perses config for local development ([#1007](https://github.com/cobaltcore-dev/cortex/pull/1007))
- Update `github.com/sapcc/go-bits` ([#1005](https://github.com/cobaltcore-dev/cortex/pull/1005))
- Update kube-prometheus-stack Docker tag to v87.4.0 ([#998](https://github.com/cobaltcore-dev/cortex/pull/998))
- Update `github.com/sapcc/go-bits` ([#1005](https://github.com/cobaltcore-dev/cortex/pull/1005), [#1013](https://github.com/cobaltcore-dev/cortex/pull/1013))
- Update kube-prometheus-stack Docker tag to v87.6.0 ([#998](https://github.com/cobaltcore-dev/cortex/pull/998), [#1014](https://github.com/cobaltcore-dev/cortex/pull/1014))

### cortex-shim v0.1.5 (sha-c98ddb05)
### cortex-shim v0.1.5 (sha-c8295850)

Includes updated image sha-c98ddb05.
Includes updated image sha-c8295850.

### cortex-nova v0.0.79

Expand Down
13 changes: 12 additions & 1 deletion api/external/nova/messages.go
Original file line number Diff line number Diff line change
Expand Up @@ -154,10 +154,15 @@ const (
EvacuateIntent v1alpha1.SchedulingIntent = "evacuate"
// CreateIntent indicates that the request is intended for creating a new VM.
CreateIntent v1alpha1.SchedulingIntent = "create"
// ReserveForFailoverIntent indicates that the request is for failover reservation scheduling.
// ReserveForFailoverIntent indicates that the request is for creating a new failover reservation slot.
ReserveForFailoverIntent v1alpha1.SchedulingIntent = "reserve_for_failover"
// ReuseFailoverReservationIntent indicates that the request is checking whether an existing
// failover reservation slot can be reused by a VM (compatibility check, not a new slot).
ReuseFailoverReservationIntent v1alpha1.SchedulingIntent = "reuse_failover_reservation"
// ReserveForCommittedResourceIntent indicates that the request is for CR reservation scheduling.
ReserveForCommittedResourceIntent v1alpha1.SchedulingIntent = "reserve_for_committed_resource"
// CapacityProbeIntent indicates that the request is a synthetic capacity probe (not a real VM placement).
CapacityProbeIntent v1alpha1.SchedulingIntent = "capacity_probe"

// HintKeyResourceGroup is the scheduler hint key used to pass the resource group
// (e.g., flavor group name) for failover reservation scheduling.
Expand Down Expand Up @@ -188,9 +193,15 @@ func (req ExternalSchedulerRequest) GetIntent() (v1alpha1.SchedulingIntent, erro
// Used by cortex failover reservation controller
case "reserve_for_failover":
return ReserveForFailoverIntent, nil
// Used by cortex failover reservation controller (reuse check)
case "reuse_failover_reservation":
return ReuseFailoverReservationIntent, nil
// Used by cortex committed resource reservation controller
case "reserve_for_committed_resource":
return ReserveForCommittedResourceIntent, nil
// Used by cortex capacity probe controller
case "capacity_probe":
return CapacityProbeIntent, nil
default:
return CreateIntent, nil
}
Expand Down
2 changes: 1 addition & 1 deletion api/v1alpha1/datasource_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ type DatasourceStatus struct {
// When the datasource was last successfully synced.
LastSynced metav1.Time `json:"lastSynced,omitempty"`
// The number of objects currently stored for this datasource.
NumberOfObjects int64 `json:"numberOfObjects,omitempty"`
NumberOfObjects *int64 `json:"numberOfObjects,omitempty"`
// Planned time for the next sync.
NextSyncTime metav1.Time `json:"nextSyncTime,omitempty"`

Expand Down
5 changes: 5 additions & 0 deletions api/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 8 additions & 5 deletions cmd/manager/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -765,22 +765,25 @@ func main() {
setupLog.Info("enabling controller", "controller", "capacity-controller")
capacityConfig := conf.GetConfigOrDie[capacity.Config]()
capacityConfig.ApplyDefaults()
if err := capacityConfig.Validate(); err != nil {
setupLog.Error(err, "invalid capacity-controller config")
os.Exit(1)
}

capacityMonitor := capacity.NewMonitor(multiclusterClient)
if err := metrics.Registry.Register(&capacityMonitor); err != nil {
setupLog.Error(err, "failed to register capacity monitor metrics, continuing without metrics")
}

capacityController := capacity.NewController(multiclusterClient, capacityConfig, commitmentsVMSource)
if err := mgr.Add(manager.RunnableFunc(func(ctx context.Context) error {
return capacityController.Start(ctx)
})); err != nil {
setupLog.Error(err, "unable to add capacity controller to manager")
if err := capacity.NewController(multiclusterClient, capacityConfig, commitmentsVMSource).
SetupWithManager(mgr, multiclusterClient); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "capacity")
os.Exit(1)
}
setupLog.Info("capacity-controller registered",
"schedulerURL", capacityConfig.SchedulerURL,
"reconcileInterval", capacityConfig.ReconcileInterval,
"minReconcileInterval", capacityConfig.MinReconcileInterval,
"totalPipeline", capacityConfig.TotalPipeline,
"placeablePipeline", capacityConfig.PlaceablePipeline)
}
Expand Down
5 changes: 3 additions & 2 deletions docs/apis.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,9 @@ The `scheduling.Options` struct configures a single pipeline invocation. All fie
| `MaxCandidates` | `int` | Maximum number of candidate hosts returned after weighing. 0 means no limit. |
| `SkipHistory` | `bool` | Skips recording the placement decision in placement history. |
| `SkipInflight` | `bool` | Skips creating pessimistic blocking reservations for returned candidates. |
| `SkipCommittedResourceTracking` | `bool` | Skips writing the placed VM UUID into the matching committed resource reservation slot. Required for ReadOnly runs. |

**Validation constraint:** A `ReadOnly` run must also set `SkipHistory=true` and `SkipInflight=true`. This is enforced by `Options.Validate()` — omitting either field causes validation to fail with an error before the pipeline executes.
**Validation constraint:** A `ReadOnly` run must also set `SkipHistory=true`, `SkipInflight=true`, and `SkipCommittedResourceTracking=true`. This is enforced by `Options.Validate()` — omitting any of these fields causes validation to fail with an error before the pipeline executes.

### Decisions

Expand Down Expand Up @@ -142,7 +143,7 @@ For more details on how committed resources interact with reservations, see [com
kubectl get flavorgroupcapacities
```

FlavorGroupCapacity caches pre-computed capacity data for one flavor group in one availability zone. One CRD exists per (flavor group × AZ) pair, maintained by the capacity controller on a fixed interval. The spec identifies the flavor group and AZ; the status holds per-flavor slot counts (`PlaceableVMs`, `PlaceableHosts`, `TotalCapacityVMSlots`, `TotalCapacityHosts`), aggregate fields (`CommittedCapacity`, `TotalCapacity`, `TotalInstances`), and a `LastReconcileAt` timestamp. The capacity API reads these CRDs instead of probing the scheduler on each request.
FlavorGroupCapacity caches pre-computed capacity data for one flavor group in one availability zone. One CRD exists per (flavor group × AZ) pair, maintained by the capacity controller on a fixed interval. The spec identifies the flavor group and AZ; the status holds per-flavor slot counts (`PlaceableVMs`, `PlaceableHosts`, `TotalCapacityVMSlots`, `TotalCapacityHosts`), aggregate fields (`CommittedCapacity`, `CommittedCapacityBytes`, `TotalCapacity`, `FreeCapacity`, `ExclusivelyFreeCapacity`, `ExclusivelyFreeSlots`, `RunningInstances`, `RunningResources`, `SmallestFlavorName`), and a `LastReconcileAt` timestamp. Where flavor groups share hosts, remaining capacity is fairly split across overlapping groups using a round-robin algorithm so that no group's exclusively-free capacity exceeds the actual installed resources. The capacity API reads these CRDs instead of probing the scheduler on each request.

### ProjectQuota

Expand Down
6 changes: 3 additions & 3 deletions docs/reservations/failover-reservations.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ We use three different scheduler pipelines for failover reservations, each servi

**Why:** When reusing a reservation, capacity is already reserved on the target host. We only need to verify that the VM is compatible with the host (traits, capabilities, AZ, etc.) without checking if there's enough free capacity.

Options: `ReadOnly: true, SkipHistory: true` — pure compatibility check, no state mutations.
Options: `ReadOnly: true, SkipHistory: true, SkipInflight: true, SkipCommittedResourceTracking: true` — pure compatibility check, no state mutations.

### `kvm-general-purpose-load-balancing` (new reservation)
**Used when:** Creating a new failover reservation.
Expand All @@ -175,14 +175,14 @@ Options: `ReadOnly: true, SkipHistory: true` — pure compatibility check, no st
1. Is compatible with the VM (traits, capabilities, AZ, etc.)
2. Has enough free capacity to accommodate the VM if it needs to evacuate

Options: `LockReservations: true, SkipHistory: true` — capacity check must see true remaining capacity with all reservation slots locked.
Options: `LockReservations: true, SkipHistory: true, SkipInflight: true, SkipCommittedResourceTracking: true` — capacity check must see true remaining capacity with all reservation slots locked.

### `kvm-acknowledge-failover-reservation`
**Used when:** Validating that an existing reservation is still valid (watch-based reconciliation).

**Why:** Periodically we need to verify that a VM could still evacuate to its reserved host. This sends an evacuation-style scheduling request with only the reservation's host as the eligible target. If the scheduler rejects it, the reservation is no longer valid and should be deleted so the periodic controller can create a new one on a valid host.

Options: `ReadOnly: true, SkipHistory: true` — validation only, no state mutations.
Options: `ReadOnly: true, LockReservations: true, SkipHistory: true, SkipInflight: true, SkipCommittedResourceTracking: true` — validation only, no state mutations.

## Data Model

Expand Down
16 changes: 8 additions & 8 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ require (
github.com/cobaltcore-dev/openstack-hypervisor-operator v1.2.3
github.com/go-gorp/gorp v2.2.0+incompatible
github.com/gophercloud/gophercloud/v2 v2.13.0
github.com/ironcore-dev/ironcore v0.4.2
github.com/ironcore-dev/ironcore v0.4.3
github.com/prometheus/client_golang v1.23.2
github.com/prometheus/client_model v0.6.2
github.com/sapcc/go-bits v0.0.0-20260703101005-27edea94477c
github.com/sapcc/go-bits v0.0.0-20260709170107-f8ef1547bb62
go.xyrillian.de/gg v1.11.1
k8s.io/api v0.36.2
k8s.io/apimachinery v0.36.2
Expand Down Expand Up @@ -111,15 +111,15 @@ require (
golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93 // indirect
golang.org/x/net v0.56.0 // indirect
golang.org/x/oauth2 v0.36.0 // indirect
golang.org/x/sync v0.21.0
golang.org/x/sys v0.46.0 // indirect
golang.org/x/term v0.44.0
golang.org/x/sync v0.22.0
golang.org/x/sys v0.47.0 // indirect
golang.org/x/term v0.45.0
golang.org/x/text v0.38.0 // indirect
golang.org/x/time v0.15.0 // indirect
gomodules.xyz/jsonpatch/v2 v2.5.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260226221140-a57be14db171 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260319201613-d00831a3d3e7 // indirect
google.golang.org/grpc v1.81.1 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260414002931-afd174a4e478 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260414002931-afd174a4e478 // indirect
google.golang.org/grpc v1.82.0 // indirect
google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af // indirect
gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
Expand Down
Loading
Loading