Release cortex v0.3.6

CHANGELOG.md

@@ -1,5 +1,61 @@
 # Changelog
 
+## 2026-08-10 — [#1129](https://github.com/cobaltcore-dev/cortex/pull/1129)
+
+### cortex v0.3.6 (sha-85d9d0cf)
+
+New features:
+- Implement in-flight reservations controller — adds a new controller that tracks reservations currently being fulfilled by monitoring VM creation state via Nova API, transitioning reservations through their lifecycle stages and cleaning up stale in-flight reservations ([#957](https://github.com/cobaltcore-dev/cortex/pull/957))
+- Add KPI that tracks CR count per configured cluster — introduces the `MulticlusterObjectCountKPI` plugin which counts custom resource objects (e.g. Hypervisors, Reservations) per cluster and exposes them as Prometheus metrics for deployment monitoring ([#1054](https://github.com/cobaltcore-dev/cortex/pull/1054))
+- Follow VM live migration in CR reservation reconciler — the committed resource reservation controller now detects when a VM has been live-migrated to a different host and updates the reservation's target host accordingly, preventing stale capacity accounting ([#1048](https://github.com/cobaltcore-dev/cortex/pull/1048))
+
+Bug fixes:
+- Subtract reservation CPU blocks when counting placeable slots — the capacity accounting now correctly deducts reserved CPU blocks from available capacity before calculating how many new instances can be placed, preventing over-commitment ([#1118](https://github.com/cobaltcore-dev/cortex/pull/1118))
+
+Non-breaking changes:
+- Skip non-candidate hypervisors in nova filters and weighers — filters and weighers now skip hypervisors that are not in the candidate set, improving performance and avoiding unnecessary processing ([#1117](https://github.com/cobaltcore-dev/cortex/pull/1117))
+- Add dynamic labels to pipeline step event metrics — scheduling pipeline step monitors can now emit events with dynamic label key-value pairs, enabling richer per-step observability ([#1108](https://github.com/cobaltcore-dev/cortex/pull/1108))
+- Update `go.xyrillian.de/gg` to v1.13.2 ([#1111](https://github.com/cobaltcore-dev/cortex/pull/1111), [#1122](https://github.com/cobaltcore-dev/cortex/pull/1122))
+- Update `github.com/sapcc/go-bits` ([#1124](https://github.com/cobaltcore-dev/cortex/pull/1124))
+- Update `kube-prometheus-stack` to v88.1.5 ([#1112](https://github.com/cobaltcore-dev/cortex/pull/1112), [#1119](https://github.com/cobaltcore-dev/cortex/pull/1119), [#1121](https://github.com/cobaltcore-dev/cortex/pull/1121))
+- Update `debian:trixie-slim` Docker digest ([#1120](https://github.com/cobaltcore-dev/cortex/pull/1120))
+
+### cortex-shim v0.1.12 (sha-85d9d0cf)
+
+Includes updated image sha-85d9d0cf with dependency updates.
+
+### cortex-postgres v0.6.12 (sha-85d9d0cf)
+
+Includes updated base image (debian:trixie-slim digest update).
+
+### cortex-nova v0.0.86
+
+Includes updated chart cortex v0.3.6.
+
+### cortex-cinder v0.0.86
+
+Includes updated chart cortex v0.3.6.
+
+### cortex-manila v0.0.86
+
+Includes updated chart cortex v0.3.6.
+
+### cortex-crds v0.0.86
+
+Includes updated chart cortex v0.3.6.
+
+### cortex-ironcore v0.0.86
+
+Includes updated chart cortex v0.3.6.
+
+### cortex-pods v0.0.86
+
+Includes updated chart cortex v0.3.6.
+
+### cortex-placement-shim v0.1.12
+
+Includes updated chart cortex-shim v0.1.12.
+
 ## 2026-08-03 — [#1114](https://github.com/cobaltcore-dev/cortex/pull/1114)
 
 ### cortex v0.3.5 (sha-684e0b07)

cmd/manager/main.go

@@ -63,6 +63,7 @@ import (
 	"github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/commitments"
 	commitmentsapi "github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/commitments/api"
 	"github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/failover"
+	"github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/inflight"
 	"github.com/cobaltcore-dev/cortex/internal/scheduling/reservations/quota"
 	"github.com/cobaltcore-dev/cortex/pkg/conf"
 	"github.com/cobaltcore-dev/cortex/pkg/monitoring"
@@ -499,6 +500,18 @@ func main() {
 			os.Exit(1)
 		}
 	}
+	if slices.Contains(mainConfig.EnabledControllers, "inflight-reservation-controller") {
+		setupLog.Info("enabling controller",
+			"controller", "inflight-reservation-controller")
+		config := conf.GetConfigOrDie[inflight.NovaVMClientConfig]()
+		vmClient := inflight.NewNovaVMClient(config)
+		controller := &inflight.Controller{Client: multiclusterClient, VMClient: vmClient}
+		if err := controller.SetupWithManager(ctx, mgr); err != nil {
+			setupLog.Error(err, "unable to create controller",
+				"controller", "inflight-reservation-controller")
+			os.Exit(1)
+		}
+	}
 	if slices.Contains(mainConfig.EnabledControllers, "nova-deschedulings-executor") {
 		setupLog.Info("enabling controller", "controller", "nova-deschedulings-executor")
 		executorConfig := conf.GetConfigOrDie[nova.DeschedulingsExecutorConfig]()

docs/apis.md

@@ -96,6 +96,10 @@ The `scheduling.Options` struct configures a single pipeline invocation. All fie
 
 **Validation constraint:** A `ReadOnly` run must also set `SkipHistory=true`, `SkipInflight=true`, and `SkipCommittedResourceTracking=true`. This is enforced by `Options.Validate()` — omitting any of these fields causes validation to fail with an error before the pipeline executes.
 
+#### Step Events
+
+Pipeline steps can report named events via the `Events []FilterWeigherPipelineStepEvent` field on their result. Each event carries a `Name` and an optional `Labels map[string]string` for dynamic Prometheus labels. The pipeline monitor exports these as the `cortex_filter_weigher_pipeline_step_events_total` counter with fixed labels `pipeline`, `step`, `event` plus any dynamic labels the step provides. Use events to surface noteworthy step conditions (skipped filtering, missing data) as observable metrics. See `filter_image_properties` for a reference implementation that emits an `image_properties_hv_type_undetermined` event with an `intent` label.
+
 ### Decisions
 
 ```bash

go.mod

@@ -9,8 +9,8 @@ require (
 	github.com/ironcore-dev/ironcore v0.5.0
 	github.com/prometheus/client_golang v1.24.1
 	github.com/prometheus/client_model v0.6.2
-	github.com/sapcc/go-bits v0.0.0-20260730170321-f6f727520601
-	go.xyrillian.de/gg v1.11.1
+	github.com/sapcc/go-bits v0.0.0-20260806170240-4bbc84d224db
+	go.xyrillian.de/gg v1.13.3
 	k8s.io/api v0.36.3
 	k8s.io/apimachinery v0.36.3
 	k8s.io/client-go v0.36.3
@@ -29,7 +29,7 @@ require (
 	github.com/go-openapi/swag/stringutils v0.25.1 // indirect
 	github.com/go-openapi/swag/typeutils v0.25.1 // indirect
 	github.com/go-openapi/swag/yamlutils v0.25.1 // indirect
-	github.com/gofrs/uuid/v5 v5.5.0 // indirect
+	github.com/gofrs/uuid/v5 v5.5.1 // indirect
 	github.com/gorilla/mux v1.8.1 // indirect
 	github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
 	k8s.io/streaming v0.36.3 // indirect

go.sum

@@ -102,8 +102,8 @@ github.com/go-sql-driver/mysql v1.5.0 h1:ozyZYNQW3x3HtqT1jira07DN2PArx2v7/mN66gG
 github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
 github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
 github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
-github.com/gofrs/uuid/v5 v5.5.0 h1:FkPv6jYQRbZtH3bD8yC7106u+CedTCLF8+t7CLHSZNo=
-github.com/gofrs/uuid/v5 v5.5.0/go.mod h1:bbAA98EoIlxyRHIVg6ektCSsZ5n8mSbwgEhvhMYlZgg=
+github.com/gofrs/uuid/v5 v5.5.1 h1:z1Ce19/JwNidXpy3tOQc3241lnJLKdKyq/xlNvlD4Ng=
+github.com/gofrs/uuid/v5 v5.5.1/go.mod h1:bbAA98EoIlxyRHIVg6ektCSsZ5n8mSbwgEhvhMYlZgg=
 github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
 github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
 github.com/golang-migrate/migrate/v4 v4.19.1 h1:OCyb44lFuQfYXYLx1SCxPZQGU7mcaZ7gH9yH4jSFbBA=
@@ -204,8 +204,8 @@ github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/sapcc/go-api-declarations v1.24.0 h1:sGBvOMVSM1olJlyvNoQSk7NX5uatXHKkztGDBPnTWMs=
 github.com/sapcc/go-api-declarations v1.24.0/go.mod h1:ZWRTijvgF8o8aHg5stgg7u4DF6jFrd0X97le/uGlZsA=
-github.com/sapcc/go-bits v0.0.0-20260730170321-f6f727520601 h1:pF4eF41dp0AxM0IfICn3ttpI5IwQoUmC0IkGuXLTmBA=
-github.com/sapcc/go-bits v0.0.0-20260730170321-f6f727520601/go.mod h1:4z4Vd1C7d6hDdbJ5q1CgzE9B8ESWuiMTbkZSJA5gSjA=
+github.com/sapcc/go-bits v0.0.0-20260806170240-4bbc84d224db h1:DfseqB6CZzdpbSdL4sGlqEBSYu3JpUpwYFiNzoYTkog=
+github.com/sapcc/go-bits v0.0.0-20260806170240-4bbc84d224db/go.mod h1:1wh2+fuMXNrJYttuaqiU7sNQgJhTmsZ/0S0SK1ttwQc=
 github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
 github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
 github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
@@ -250,8 +250,8 @@ go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
 go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
 go.uber.org/zap v1.28.0 h1:IZzaP1Fv73/T/pBMLk4VutPl36uNC+OSUh3JLG3FIjo=
 go.uber.org/zap v1.28.0/go.mod h1:rDLpOi171uODNm/mxFcuYWxDsqWSAVkFdX4XojSKg/Q=
-go.xyrillian.de/gg v1.11.1 h1:7P3kDFkTmR7jx2riYi0GwX5uhgrsL37QSrN16xH/n8E=
-go.xyrillian.de/gg v1.11.1/go.mod h1:DoO4fQSWIrBRlNlCjVyrYM0kAEBt/Jg2GkMH+cGRZ0k=
+go.xyrillian.de/gg v1.13.3 h1:Ulz3+eZnO2OUl7Bv+SWaA5ufDmjeLwwmICPP4dCurxA=
+go.xyrillian.de/gg v1.13.3/go.mod h1:DoO4fQSWIrBRlNlCjVyrYM0kAEBt/Jg2GkMH+cGRZ0k=
 go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
 go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
 go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=

helm/bundles/cortex-cinder/Chart.yaml

@@ -5,23 +5,23 @@ apiVersion: v2
 name: cortex-cinder
 description: A Helm chart deploying Cortex for Cinder.
 type: application
-version: 0.0.85
+version: 0.0.86
 appVersion: 0.1.0
 dependencies:
   # from: file://../../library/cortex-postgres
   - name: cortex-postgres
     repository: oci://ghcr.io/cobaltcore-dev/cortex/charts
-    version: 0.6.11
+    version: 0.6.12
 
   # from: file://../../library/cortex
   - name: cortex
     repository: oci://ghcr.io/cobaltcore-dev/cortex/charts
-    version: 0.3.5
+    version: 0.3.6
     alias: cortex-knowledge-controllers
   # from: file://../../library/cortex
   - name: cortex
     repository: oci://ghcr.io/cobaltcore-dev/cortex/charts
-    version: 0.3.5
+    version: 0.3.6
     alias: cortex-scheduling-controllers
 
   # Owner info adds a configmap to the kubernetes cluster with information on

helm/bundles/cortex-crds/Chart.yaml

@@ -5,13 +5,13 @@ apiVersion: v2
 name: cortex-crds
 description: A Helm chart deploying Cortex CRDs.
 type: application
-version: 0.0.85
+version: 0.0.86
 appVersion: 0.1.0
 dependencies:
   # from: file://../../library/cortex
   - name: cortex
     repository: oci://ghcr.io/cobaltcore-dev/cortex/charts
-    version: 0.3.5
+    version: 0.3.6
 
   # Owner info adds a configmap to the kubernetes cluster with information on
   # the service owner. This makes it easier to find out who to contact in case

helm/bundles/cortex-ironcore/Chart.yaml

@@ -5,13 +5,13 @@ apiVersion: v2
 name: cortex-ironcore
 description: A Helm chart deploying Cortex for IronCore.
 type: application
-version: 0.0.85
+version: 0.0.86
 appVersion: 0.1.0
 dependencies:
   # from: file://../../library/cortex
   - name: cortex
     repository: oci://ghcr.io/cobaltcore-dev/cortex/charts
-    version: 0.3.5
+    version: 0.3.6
 
   # Owner info adds a configmap to the kubernetes cluster with information on
   # the service owner. This makes it easier to find out who to contact in case

helm/bundles/cortex-manila/Chart.yaml

@@ -5,23 +5,23 @@ apiVersion: v2
 name: cortex-manila
 description: A Helm chart deploying Cortex for Manila.
 type: application
-version: 0.0.85
+version: 0.0.86
 appVersion: 0.1.0
 dependencies:
   # from: file://../../library/cortex-postgres
   - name: cortex-postgres
     repository: oci://ghcr.io/cobaltcore-dev/cortex/charts
-    version: 0.6.11
+    version: 0.6.12
 
   # from: file://../../library/cortex
   - name: cortex
     repository: oci://ghcr.io/cobaltcore-dev/cortex/charts
-    version: 0.3.5
+    version: 0.3.6
     alias: cortex-knowledge-controllers
   # from: file://../../library/cortex
   - name: cortex
     repository: oci://ghcr.io/cobaltcore-dev/cortex/charts
-    version: 0.3.5
+    version: 0.3.6
     alias: cortex-scheduling-controllers
 
   # Owner info adds a configmap to the kubernetes cluster with information on

helm/bundles/cortex-nova/Chart.yaml

@@ -5,23 +5,23 @@ apiVersion: v2
 name: cortex-nova
 description: A Helm chart deploying Cortex for Nova.
 type: application
-version: 0.0.85
+version: 0.0.86
 appVersion: 0.1.0
 dependencies:
   # from: file://../../library/cortex-postgres
   - name: cortex-postgres
     repository: oci://ghcr.io/cobaltcore-dev/cortex/charts
-    version: 0.6.11
+    version: 0.6.12
 
   # from: file://../../library/cortex
   - name: cortex
     repository: oci://ghcr.io/cobaltcore-dev/cortex/charts
-    version: 0.3.5
+    version: 0.3.6
     alias: cortex-knowledge-controllers
   # from: file://../../library/cortex
   - name: cortex
     repository: oci://ghcr.io/cobaltcore-dev/cortex/charts
-    version: 0.3.5
+    version: 0.3.6
     alias: cortex-scheduling-controllers
 
   # Owner info adds a configmap to the kubernetes cluster with information on

helm/bundles/cortex-nova/templates/alerts.yaml

@@ -303,9 +303,46 @@ spec:
           configuration. It is recommended to investigate the
           pipeline status and logs for more details.
 
+    - alert: CortexNovaInFlightReservationsBacklogGrowing
+      # Alert when the number of in-flight reservations stuck with
+      # reason=InstanceNotFound is trending upwards over a long window,
+      # rather than staying roughly the same or draining down. A healthy
+      # scheduler-hypervisor loop should quickly resolve in-flight
+      # reservations once the VM shows up on any hypervisor; a sustained
+      # positive derivative indicates the backlog is not being worked off.
+      #
+      # deriv() over 1h returns the per-second slope of the gauge; multiplied
+      # by 3600 it becomes "expected growth per hour". We require this to be
+      # meaningfully positive AND the current count to be non-trivial, so we
+      # don't page on a single lingering reservation.
+      expr: |
+        deriv(cortex_reservation_state{domain="nova",type="InFlightReservation",state="unknown",reason="InstanceNotFound"}[1h]) * 3600 > 5
+        and
+        cortex_reservation_state{domain="nova",type="InFlightReservation",state="unknown",reason="InstanceNotFound"} > 10
+      for: 2h
+      labels:
+        context: reservations
+        dashboard: cortex-status-dashboard/cortex-status-dashboard
+        service: cortex
+        severity: warning
+        support_group: workload-management
+        playbook: docs/support/playbook/cortex/alerts/unready
+      annotations:
+        summary: "In-flight reservation backlog with reason=InstanceNotFound keeps growing"
+        description: >
+          The number of in-flight reservations that have not yet observed
+          their VM on any hypervisor (reason=InstanceNotFound) has been
+          increasing steadily for at least two hours. Normally these
+          reservations resolve once the hypervisor operator reports the
+          instance; a sustained positive trend suggests either VMs are not
+          landing on hypervisors, or the hypervisor status feed is stalled,
+          or the in-flight reservation controller is not pruning cleanly.
+          Investigate the hypervisor operator, the nova build pipeline, and
+          the in-flight reservation controller logs.
+
     - alert: CortexNovaImagePropertiesHypervisorTypeUndetermined
       expr: |
-        sum by (pipeline, step) (rate(cortex_filter_weigher_pipeline_step_events_total{service="cortex-nova-metrics", event="image_properties_hv_type_undetermined"}[5m])) > 0.1
+        sum by (pipeline, step, intent) (rate(cortex_filter_weigher_pipeline_step_events_total{service="cortex-nova-metrics", event="image_properties_hv_type_undetermined"}[5m])) > 0.1
       for: 15m
       labels:
         context: scheduling
@@ -319,11 +356,32 @@ spec:
         description: >
           The `filter_image_properties` step in pipeline `{{ "{{" }} $labels.pipeline {{ "}}" }}`
           is frequently unable to determine the hypervisor type from the image properties
-          of incoming scheduling requests. In this case the filter is skipped and all hosts
-          are returned, so kvm-only images may be placed on non-kvm hypervisors (or vice
-          versa). This may indicate that images are missing the expected hypervisor
-          type property, or that the property format has changed. Investigate the
-          image metadata of the affected requests.
+          of incoming scheduling requests for intent `{{ "{{" }} $labels.intent {{ "}}" }}`.
+          In this case the filter is skipped and all hosts are returned, so kvm-only
+          images may be placed on non-kvm hypervisors (or vice versa). This may indicate
+          that images are missing the expected hypervisor type property, or that the
+          property format has changed. Investigate the image metadata of the affected
+          requests.
+
+    - alert: CortexNovaCapabilitiesUnknownHypervisorType
+      expr: |
+        sum by (pipeline, step, hypervisor_type) (rate(cortex_filter_weigher_pipeline_step_events_total{service="cortex-nova-metrics", event="filter_capabilities_unknown_hypervisor_type"}[5m])) > 0.1
+      for: 15m
+      labels:
+        context: scheduling
+        dashboard: cortex-status-dashboard/cortex-status-dashboard
+        service: cortex
+        severity: warning
+        support_group: workload-management
+        playbook: docs/support/playbook/cortex/alerts/scheduling
+      annotations:
+        summary: "Nova capabilities filter frequently encounters unknown hypervisor types"
+        description: >
+          The `filter_capabilities` step in pipeline `{{ "{{" }} $labels.pipeline {{ "}}" }}`
+          is frequently unable to determine the hypervisor type
+          (`{{ "{{" }} $labels.hypervisor_type {{ "}}" }}`) for candidate hosts.
+          This may indicate that hypervisors are reporting incomplete domain capabilities.
+          Investigate the affected hosts and hypervisor operator status.
 
     {{- if .Values.kvm.enabled }}
     - alert: CortexNovaDoesntFindValidKVMHosts

helm/bundles/cortex-nova/templates/kpis.yaml

@@ -146,6 +146,21 @@ spec:
 ---
 apiVersion: cortex.cloud/v1alpha1
 kind: KPI
+metadata:
+  name: cortex-nova-reservation-state
+spec:
+  schedulingDomain: nova
+  impl: reservation_state_kpi
+  opts:
+    reservationSchedulingDomain: nova
+  description: |
+    This KPI tracks the state of reservation resources managed by cortex,
+    labelled by reservation type (e.g. InFlightReservation) and the reason
+    on the Ready condition. It is used to alert on sustained increases of
+    unready in-flight reservations (e.g. reason=InstanceNotFound).
+---
+apiVersion: cortex.cloud/v1alpha1
+kind: KPI
 metadata:
   name: vmware-project-utilization
 spec: