Skip to content

[Subscription] Prevent consumers from idling on dry regions - #18738

Merged
jt2594838 merged 2 commits into
apache:masterfrom
Caideyipi:fix/subscription-dry-region-polling-master
Sep 30, 2026
Merged

jt2594838 merged 2 commits into
apache:masterfrom
Caideyipi:fix/subscription-dry-region-polling-master

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Consensus subscription ownership can assign a consumer only regions that temporarily have no matching events. Even when another region has prefetched data ready, the consumer repeatedly receives empty polls and may hit its idle timeout, reducing the group and leaving a large WAL backlog.

This change keeps stable ownership as the preferred polling path, but when all assigned queues are empty it allows the consumer to borrow one already-prefetched event from an unassigned queue. Fallback queues are ordered by lag, are never waited on, and only one fallback event is returned per topic poll, preserving ownership affinity and response-size behavior.

Unit tests cover both fallback borrowing from a ready queue and retaining strict affinity when an assigned queue has data.


This PR has:

  • been self-reviewed.
  • added comments explaining the intent of the fallback polling path.
  • added unit tests for the new code paths.

Tests:

  • mvn spotless:apply -pl iotdb-core/datanode
  • mvn surefire:test -pl iotdb-core/datanode -Dtest=ConsensusSubscriptionBrokerPayloadLimitTest,ConsensusSubscriptionBrokerOwnershipTest -DforkCount=0 -Dsurefire.excludes=

Key changed/added classes
  • ConsensusSubscriptionBroker
  • ConsensusSubscriptionBrokerPayloadLimitTest

@jt2594838
jt2594838 merged commit acb801e into apache:master Sep 30, 2026
38 of 39 checks passed
@jt2594838
jt2594838 deleted the fix/subscription-dry-region-polling-master branch September 30, 2026 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants