Skip to content

[Feature] Support TAG value filtering for table subscriptions - #18639

Open
Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:feat/subscription-tag-filter
Open

Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:feat/subscription-tag-filter

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Description

Table subscriptions can select databases, tables, and output columns, but they cannot filter individual rows by TAG value. This PR adds the tag-filter topic attribute for table-model subscriptions. Its default value is true, so existing topics keep their current behavior.

Filter semantics

  • Parse a restricted SQL predicate over TAG columns. The supported forms are AND, OR, NOT, =, !=, IN, LIKE, REGEXP, and IS NULL, evaluated with SQL three-valued logic.
  • CREATE/ALTER validates the topic attributes and filter expression syntax. Each DataNode binds referenced identifiers against its current table schema when it refreshes the matcher. Missing schema, stale bindings, unknown fields, and non-TAG fields fail closed instead of leaking unfiltered rows.
  • Filter Tablet rows while preserving row order, data types, timestamps, and null bitmaps. TAG filtering runs before column-filter projection.

Delivery paths and configuration changes

  • Apply the filter to Record/Tablet and TsFile delivery, including normal subscription queues, consensus subscription queues, and Tree View projection.
  • Snapshot filter configuration per event batch so ALTER TOPIC takes effect on subsequent batches without changing the meaning of an in-flight batch.
  • Rebind matchers when table schema versions change, and clean up temporary TsFile rewrite resources on empty output and failure paths.
  • Append TagFilterStatus and TagFilterMessage to table-model SHOW TOPICS. The status reports MATCH_ALL, MATCH_NONE, ACTIVE, or ERROR for the DataNode executing the statement; the message contains the local matcher failure when the status is ERROR. Tree-model output remains unchanged.

Validation

  • Spotless and Checkstyle: passed for the changed modules.
  • Focused DataNode tests: 14 tests passed (ShowTopicsTaskTest, TabletTagFilterTest, and SubscriptionTopicAgentTest).
  • TopicConfigTest: 10 tests passed.
  • Chinese-locale DataNode test compilation: passed (mvn test-compile -pl iotdb-core/datanode -P with-zh-locale -DskipTests).
  • git diff --check: passed.

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added comments explaining the intent where it would not be obvious to an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths.
  • added integration tests.

Key changed/added classes (or packages if there are too many classes) in this PR
  • org.apache.iotdb.db.subscription.tagfilter
  • SubscriptionFilterSnapshot
  • SubscriptionTreeViewProjector
  • SubscriptionBrokerAgent
  • ConsensusLogToTabletConverter
  • PipeTabletEventTsFileBatch
  • ShowTopicsTask
  • TopicConfig and TopicMeta

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant