Skip to content

[Pipe] Add SHOW PIPE MEMORY for named memory blocks - #18575

Open
Caideyipi wants to merge 4 commits into
apache:masterfrom
Caideyipi:show-pipe-memory
Open

[Pipe] Add SHOW PIPE MEMORY for named memory blocks#18575
Caideyipi wants to merge 4 commits into
apache:masterfrom
Caideyipi:show-pipe-memory

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Description

Named Pipe memory blocks

  • Add an immutable name to every PipeMemoryBlock managed by PipeMemoryManager.
  • Propagate descriptive names through all Pipe memory allocation APIs and call sites.
  • Keep live zero-sized blocks visible until release, while using weak references so diagnostics do not retain otherwise unreachable zero-sized blocks.
  • Preserve one output row per block instance; blocks with the same name are not aggregated.

SHOW PIPE MEMORY and information_schema

  • Add SHOW PIPE MEMORY in the table SQL dialect.
  • Add information_schema.pipe_memory with:
    • name STRING TAG
    • memory_usage_in_bytes INT64 ATTRIBUTE
  • Expose floating memory as an independent FloatingMemory block.
  • Read the table from the local DataNode and require global system privilege.

Tests

  • Add parser and PipeMemoryManager unit coverage.
  • Update IoTDBDatabaseIT for table discovery, schema, permissions, SHOW/SELECT access, and system-table counts.
  • Verified:
    • mvn -o install -pl iotdb-core/node-commons,iotdb-core/relational-grammar -DskipTests
    • mvn -o test -pl iotdb-core/datanode -Dtest=ShowPipeMemoryTest,PipeMemoryManagerResizeTest,LoadTsFileMemoryManagerTest -Dcheckstyle.skip=true -Dspotless.check.skip=true
    • mvn checkstyle:check -pl iotdb-core/datanode
    • git diff --check

The integration test module could not be executed locally in offline mode because several required snapshot artifacts were unavailable.

This PR has:

  • been self-reviewed.
    • concurrent read and write
  • added comments explaining non-obvious lifecycle and accounting behavior.
  • added unit tests or modified existing tests to cover new code paths.
  • added integration tests.
Key changed/added classes
  • PipeMemoryBlock
  • PipeMemoryManager
  • InformationSchema
  • InformationSchemaContentSupplierFactory
  • RelationalSql.g4
  • AstBuilder
  • IoTDBDatabaseIT

jt2594838
jt2594838 reviewed Sep 8, 2026
View reviewed changes
Comment thread ...c/main/java/org/apache/iotdb/db/pipe/event/common/statement/PipeStatementInsertionEvent.java Outdated
Comment on lines +88 to +90
this.allocatedMemoryBlock =
PipeDataNodeResourceManager.memory().forceAllocateForTabletWithRetry(0);
PipeDataNodeResourceManager.memory()
.forceAllocateForTabletWithRetry(PipeStatementInsertionEvent.class.getSimpleName(), 0);

@jt2594838 jt2594838 Sep 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add some identifiers from the event?

@jt2594838 jt2594838 Sep 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same below.

@Caideyipi Caideyipi Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in 5c7cf74, with follow-up fixes in c0163e0. Event-owned blocks now record an assigner snapshot from the event. For EnrichedEvent, it uses coreReportMessage() and falls back to toString() when needed; the snapshot is bounded and does not retain the event object. The same handling is applied to the other event allocation sites.

@Caideyipi Caideyipi Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 5c7cf74, with follow-up fixes in c0163e0. This allocation site uses the same bounded event assigner snapshot, so the block retains the event identifier without retaining the event object.

Comment thread ...g/apache/iotdb/db/pipe/event/common/tsfile/parser/query/TsFileInsertionEventQueryParser.java
Comment on lines -224 to +226
memoryManager.forceAllocateForTabletWithRetry(currentModifications.ramBytesUsed());
memoryManager.forceAllocateForTabletWithRetry(
TsFileInsertionEventQueryParser.class.getSimpleName() + "#modifications",
currentModifications.ramBytesUsed());

@jt2594838 jt2594838 Sep 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include the file name?

@Caideyipi Caideyipi Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Included in 5c7cf74, with follow-up fixes in c0163e0. Parser child blocks inherit the source event diagnostic snapshot; PipeTsFileInsertionEvent.coreReportMessage() includes the tsFile identity, so the file name is visible in assigner while the snapshot remains bounded and detached from the event object.

Comment thread ...org/apache/iotdb/db/pipe/event/common/tsfile/parser/scan/TsFileInsertionEventScanParser.java Outdated
Comment on lines +184 to +189
this.allocatedMemoryBlockForBatchData =
memoryManager.forceAllocateForTabletWithRetry(
TsFileInsertionEventScanParser.class.getSimpleName() + "#batchData", 0);
this.allocatedMemoryBlockForChunk =
memoryManager.forceAllocateForTabletWithRetry(
TsFileInsertionEventScanParser.class.getSimpleName() + "#chunk", 0);

@jt2594838 jt2594838 Sep 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible that the PipeMemoryManager has some aggregated memory blocks, like "batchData", "chunk", "xxxEvent".
The memory blocks allocated come from the associated parent blocks, and can be shown as an aggregated result.

@jt2594838 jt2594838 Sep 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or, we can add a category for each memory block.

@Caideyipi Caideyipi Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in 5c7cf74, with follow-up fixes in c0163e0. Each event now has an aggregate EVENT block, and parser/tablet/metadata blocks register as children. Child allocations charge the global pool once through the root; parent rows expose aggregate usage and accounted_memory_usage_in_bytes identifies the bytes actually charged.

@Caideyipi Caideyipi Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in 5c7cf74, with follow-up fixes in c0163e0. In addition to the event-child hierarchy, every block has an explicit category such as EVENT, PARSER, TABLET, BATCH, CACHE, RECEIVER, SINK, and SUBSCRIPTION, exposed by both SHOW PIPE MEMORY and information_schema.pipe_memory.

@Caideyipi
[Pipe] Add hierarchical memory diagnostics to SHOW PIPE MEMORY
5c7cf74
@Caideyipi
[Pipe] Fix memory diagnostic allocation behavior
c0163e0
@Caideyipi
Merge branch 'master' of https://github.com/apache/iotdb into show-pi…
0062fee 
…pe-memory

# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/PipeTsFileInsertionEvent.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/memory/PipeMemoryManager.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFilePublicResource.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/payload/evolvable/batch/PipeTabletEventBatch.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jt2594838 jt2594838 jt2594838 left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Caideyipi @jt2594838