Skip to content

fix(scheduler): keep PodScheduled in cached pods (cherry-pick for v0.18.6-meshy.2) - #4

Merged
liuxu623 merged 1 commit into
v0.18.6-pod-level-resourcesfrom
v0.18.6-keep-podscheduled-condition
Nov 3, 2026
Merged

liuxu623 merged 1 commit into
v0.18.6-pod-level-resourcesfrom
v0.18.6-keep-podscheduled-condition

Conversation

@liuxu623

@liuxu623 liuxu623 commented Nov 3, 2026 •

Copy link
Copy Markdown

Summary

Cherry-picks the upstream change from kai-scheduler#2545 (upstream issue kai-scheduler#2544) onto our v0.18.6 build branch, on top of the pod-level resources commit that v0.18.6-meshy.1 ships.

The scheduler's pod informer transform (compactConditions) kept only PodResizePending, so the status updater never found PodScheduled in the cached pod and patched every pending pod's condition on every cycle with a fresh lastTransitionTime. This change keeps PodScheduled in the cache, so the condition is patched only when its status, reason or message changes.

Why we need it now

On xinjiang the API server is reached over a 100 Mbps public link. With ~55 pending pods the scheduler produced ~1300 pod updates per minute (only lastTransitionTime changed), which fanned out to every cluster-wide pod watcher as ~11 MB/s of watch traffic and saturated the link. pod-grouper API calls then took ~5s each, its queue sat at ~95 pods, and new pods waited minutes for a PodGroup.

Plan

  • Merge, tag v0.18.6-meshy.2, build only the scheduler image (podgroupcontroller stays on meshy.1).
  • Bump the scheduler tag on xinjiang and fpt-k8s in taichi-dev/deployments.
  • Drop the commit once an upstream release carries it.

Testing

  • Clean cherry-pick (author preserved, -x trailer).
  • go test ./pkg/scheduler/cache/... passes, including status_updater and the new "keeps PodScheduled" spec.

🤖 Generated with Claude Code


Note

Low Risk
Narrow change to scheduler cache compaction with a targeted test; reduces API traffic without altering scheduling decisions.

Overview
Fixes excessive pod status patches for pending workloads by changing compactConditions in the scheduler informer transform: it now keeps both PodResizePending and PodScheduled in the cached pod copy, instead of retaining only resize-pending (and dropping scheduled state).

With PodScheduled present in cache, the status updater can compare the existing condition before patching, so unschedulable pending pods are not re-patched every scheduling cycle when only lastTransitionTime would change. That cuts API churn and watch traffic on clusters with many pending pods.

Adds a compactSchedulerPod test asserting both condition types survive compaction, plus an unreleased changelog entry for issue 2544.

Reviewed by Cursor Bugbot for commit 87311e8. Bugbot is set up for automated code reviews on this repo. Configure here.

…ons are not re-patched

The pod informer transform kept only PodResizePending, so the status updater, which compares the new
condition with the cached pod, never found PodScheduled and patched every pending pod's condition on
every cycle, with a fresh lastTransitionTime. Keep PodScheduled too: the condition is then patched
only when its status, reason or message changes.

Fixes kai-scheduler#2544

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Sam Huang <19882250+sam-huang1223@users.noreply.github.com>
(cherry picked from commit 4cf15ba)
@liuxu623
liuxu623 merged commit d96acbc into v0.18.6-pod-level-resources Nov 3, 2026
9 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants