Add support for broker demotion by kyguy · Pull Request #191 · strimzi/proposals

kyguy · 2026-01-13T03:15:15Z

Suggested solution

This feature would extend the KafkaRebalance resource to leverage Cruise Control’s /kafkacruisecontrol/demote_broker endpoint [1], allowing users to specify a list of brokers to be demoted. This would trigger the migration of partition leadership away from those brokers in preparation for decommissioning or maintenance.

Addresses issue discussed here [2]

[1] https://github.com/linkedin/cruise-control/wiki/REST-APIs#demote-a-list-of-brokers-from-the-kafka-cluster
[2] strimzi/strimzi-kafka-operator#11907

Signed-off-by: Kyle Liberti <kliberti.us@gmail.com>

see-quick

Thanks for the proposal, I think it looks good, just a few nits/questions...

Out of curiosity, I didn't find it in the proposal but what happens when demotion is in progress but target broker fails in the middle of operation?

Signed-off-by: Kyle Liberti <kliberti.us@gmail.com>

kyguy · 2026-01-14T18:06:03Z

Thanks for the review @see-quick

Out of curiosity, I didn't find it in the proposal but what happens when demotion is in progress but target broker fails in the middle of operation?

I just added this note to the Validation and constraints section of the proposal:

"If a target broker fails while leadership is being transferred to it, all demotion operations involving that broker are aborted, and the source brokers remain the leaders for the affected partitions.
In this case, the overall demotion request continues on a best-effort with the proposed operations, transferring the leadership on brokers that are available."

Signed-off-by: Kyle Liberti <kliberti.us@gmail.com>

fvaleri

Thanks @kyguy. I left some comments.

I think it is also important to document that broker demotion does NOT prevent new partition leaders to be scheduled on the selected broker.

tinaselenge

Thanks for the proposal. Overall, it looks good to me. I left a few comments to clarify.

tinaselenge · 2026-01-19T11:02:17Z

+| brokers                          | integer array    | List of ids of broker to be demoted in the cluster.                |
+| concurrentLeaderMovements        | integer | Upper bound of ongoing leadership swaps. Default is 1000.                   |
+| skipUrpDemotion                  | boolean | Whether to skip demoting leader replicas for under-replicated partitions.   |
+| excludeFollowerDemotion          | boolean | Whether to skip demoting follower replicas on the broker to be demoted.     |


what does demoting follower replicas mean?

It moves the ids of the demoted brokers to the end of the replica lists.

scholzj · 2026-01-19T19:19:08Z

+# Broker demotion support via KafkaRebalance resource
+
+This proposal extends the `KafkaRebalance` custom resource to support broker demotion by integrating with Cruise Control's `/demote_broker` endpoint. 
+This would allow users to demote brokers, removing them from partition leadership eligibility, in preparation for maintenance, decommissioning, or other operational needs.


How does this work? I get it that it removes them from partition leadership. But how does it ensure they are not eligible to become leaders again?

Cruise Control has the broker ids marked as ineligible in memory and moves the broker ids to the end of the replica lists, then triggers a leadership election. Since leadership election prioritizes choosing the first broker id of the replica list (the preferred leader) as the leader, the demoted brokers are less likely to be elected as leaders.

But how does it ensure they are not eligible to become leaders again?

If a rebalance or demotion request is made the broker ids marked as ineligible in memory are excluded from having partitions moved to them or being listed first in the replica lists.

What happens when CC pod is restarted (upgrade, migration to different node, etc).? User will have to trigger "rebalance" proposal and approval to refresh in-memory data?

At this time yes, to refresh the in-memory demotion data after Cruise Control pod restart, a user would have to trigger another KafkaRebalance demotion request.

scholzj · 2026-01-19T19:21:17Z

+**NOTE**: As part of this proposal, we will also add a `excludeRecentlyDemotedBrokers` field for the `full`, `add-brokers`, and `remove-brokers` KafkaRebalance modes to give users the ability to prevent to leader replicas to be moved to recently demoted brokers.
+When `excludeRecentlyDemotedBrokers` is set to `true`, a broker is considered demoted for the duration specified by the Cruise Control `demotion.history.retention.time.ms` server configuration.
+By default, this value is 1209600000 milleseconds (14 days) but is configurable in the `spec.cruiseControl.config` section of the `Kafka` custom resource.


How does this work? Where is the information about the demotion work?

Cruise Control stores the demotion information in memory (yes this is far from ideal). Cruise Control will make sure the ids of the demoted brokers are in the end of the replica lists and will exclude the brokers from being considered for partition movement when generating optimization proposals.

scholzj · 2026-01-19T19:29:07Z

+
+The implementation includes the following validation:
+
+* When `demote-brokers` mode is specified, the `brokers` field must be provided.


Will it be validated only in the broker code or also with CEL?

Was originally planning on validating the field in the operator code but this may change depending on the proposal surrounding the API changes discussed #191 (comment).

I'll link the separate proposal here when I have a draft ready and we can pick this up once that proposal and its implementation is sorted.

Signed-off-by: Kyle Liberti <kliberti.us@gmail.com>

kyguy · 2026-01-29T00:23:22Z

Thanks everyone for the reviews! I am putting this proposal on hold temporarily as we address some of the API concerns raised in the thread here: #191 (comment). We need devise a better path forward for the parameters that are used in conjunction with mode field of the KafkaRebalance resource, maybe by consolidating primitive fields into a generic config section to ensure the API can support future modes cleanly.

I am going to draft a separate proposal for those API changes and link it here. Once that proposal and its implementation is complete, we can continue this broker demotion proposal with a cleaner, more maintainable foundation.

Signed-off-by: Kyle Liberti <kliberti.us@gmail.com>

kyguy · 2026-04-08T22:32:47Z

I am going to draft a separate proposal for those API changes and link it here. Once that proposal and its implementation is complete, we can continue this broker demotion proposal with a cleaner, more maintainable foundation.

Opened a draft proposal for the KafkaRebalance API changes here #216. We can follow up on the conversation there and return to this proposal once the proposal is reviewed and implemented.

Signed-off-by: Kyle Liberti <kliberti.us@gmail.com>

kyguy · 2026-08-28T23:07:46Z

This proposal has been updated with the new KafkaRebalance API design as specified in the accepted proposal: #216), is unblocked, and ready for review!

kyguy mentioned this pull request Jan 13, 2026

[Enhancement]: Add support for broker demotion strimzi/strimzi-kafka-operator#11907

Open

kyguy force-pushed the kr-demote-brokers branch 2 times, most recently from bbb1dc1 to 7fc8acd Compare January 13, 2026 22:08

Add support for broker demotion

abb52f1

Signed-off-by: Kyle Liberti <kliberti.us@gmail.com>

kyguy force-pushed the kr-demote-brokers branch from 7fc8acd to abb52f1 Compare January 13, 2026 22:11

kyguy marked this pull request as ready for review January 13, 2026 22:12

see-quick reviewed Jan 14, 2026

View reviewed changes

Comment thread 128-broker-demotion-support.md Outdated

Comment thread 128-broker-demotion-support.md Outdated

Comment thread 128-broker-demotion-support.md Outdated

Comment thread 128-broker-demotion-support.md Outdated

Addressing feedback - mo

07cf465

Signed-off-by: Kyle Liberti <kliberti.us@gmail.com>

kyguy force-pushed the kr-demote-brokers branch from d308e3a to 07cf465 Compare January 14, 2026 18:06

More revisions

f2f77b9

Signed-off-by: Kyle Liberti <kliberti.us@gmail.com>

kyguy force-pushed the kr-demote-brokers branch from 7d66426 to f2f77b9 Compare January 14, 2026 21:33

kyguy requested review from Frawless, ShubhamRwt, fvaleri, im-konge, katheris, mimaison, ppatierno, scholzj, tinaselenge and tomncooper January 14, 2026 21:37

fvaleri reviewed Jan 15, 2026

View reviewed changes

tinaselenge reviewed Jan 19, 2026

View reviewed changes

scholzj reviewed Jan 19, 2026

View reviewed changes

ppatierno reviewed Jan 23, 2026

View reviewed changes

Comment thread 128-broker-demotion-support.md Outdated

kyguy added 2 commits January 27, 2026 19:50

Addressing some feedback - fv

2c01570

Signed-off-by: Kyle Liberti <kliberti.us@gmail.com>

Addressing feedback - fv, ts, js, pp

6a84e7f

Signed-off-by: Kyle Liberti <kliberti.us@gmail.com>

Addressing feedback - fv

a302d54

Signed-off-by: Kyle Liberti <kliberti.us@gmail.com>

ppatierno reviewed Feb 1, 2026

View reviewed changes

Comment thread 128-broker-demotion-support.md

mly-zju mentioned this pull request May 28, 2026

Add proposal for controller readiness probe based on KRaft raft state #225

Closed

kyguy force-pushed the kr-demote-brokers branch 2 times, most recently from b059881 to 146e85f Compare August 27, 2026 22:56

Update with new KafkaRebalance API design

cf20628

Signed-off-by: Kyle Liberti <kliberti.us@gmail.com>

kyguy force-pushed the kr-demote-brokers branch from 146e85f to cf20628 Compare August 28, 2026 23:04


		The implementation includes the following validation:

		* When `demote-brokers` mode is specified, the `brokers` field must be provided.

Uh oh!

Conversation

kyguy commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related problem

Suggested solution

Uh oh!

see-quick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kyguy commented Jan 14, 2026

Uh oh!

fvaleri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tinaselenge left a comment

Choose a reason for hiding this comment

Uh oh!

tinaselenge Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

kyguy Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

scholzj Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

kyguy Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Frawless Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

kyguy Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

scholzj Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

kyguy Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

scholzj Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

kyguy Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kyguy commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kyguy commented Apr 8, 2026

Uh oh!

kyguy commented Aug 28, 2026

Uh oh!

Reviewers

Assignees

kyguy commented Jan 13, 2026 •

edited

Loading

kyguy Jan 28, 2026 •

edited

Loading

kyguy Jan 28, 2026 •

edited

Loading

kyguy Jan 28, 2026 •

edited

Loading

kyguy commented Jan 29, 2026 •

edited

Loading