Skip to content

feat: experiment with RealUsagePool - #5666

Draft
comphead wants to merge 1 commit into
apache:mainfrom
comphead:oom_g
Draft

feat: experiment with RealUsagePool#5666
comphead wants to merge 1 commit into
apache:mainfrom
comphead:oom_g

Conversation

@comphead

@comphead comphead commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Experiment for #4576 .

Adopted from #4582

Rationale for this change

What changes are included in this PR?

How are these changes tested?

@comphead

comphead commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Tested on

/opt/homebrew/Cellar/apache-spark/3.5.5/bin/spark-submit  --master="local[*]"    --conf spark.driver.memory=8G     --conf spark.executor.instances=1     --conf spark.executor.cores=8     --conf spark.cores.max=8     --conf spark.executor.memory=16g     --conf spark.memory.offHeap.enabled=true     --conf spark.memory.offHeap.size=16g --conf spark.comet.explainFallback.enabled=true --conf spark.comet.logFallbackReasons.enabled=true --conf spark.eventLog.enabled=false     --jars $COMET_JAR     --driver-class-path $COMET_JAR     --conf spark.driver.extraClassPath=$COMET_JAR     --conf spark.executor.extraClassPath=$COMET_JAR     --conf spark.plugins=org.apache.spark.CometPlugin     --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager     --conf spark.comet.enabled=true     --conf spark.comet.exec.shuffle.enableFastEncoding=true      --conf spark.comet.cast.allowIncompatible=true  --conf spark.comet.exec.memoryGuard.enabled=true --conf spark.comet.exec.memoryPool=real_usage ../datafusion-benchmarks/runners/datafusion-comet/tpcbench.py     --benchmark tpcds     --data ../datafusion-benchmarks/tpcds/data_parquet/     --queries ../datafusion-benchmarks/tpcds/queries-spark  --query 67 --output . --iterations 1

Before PR the Q67 failed on

26/09/02 09:12:13 WARN TaskSetManager: Lost task 110.0 in stage 29.0 (TID 261) (192.168.4.142 executor driver): TaskKilled (Stage cancelled: Job aborted due to stage failure: Task 109 in stage 29.0 failed 1 times, most recent failure: Lost task 109.0 in stage 29.0 (TID 260) (192.168.4.142 executor driver): org.apache.comet.CometNativeException: Additional allocation failed for ExternalSorterMerge[0] with top memory consumers (across reservations) as:
  ExternalSorter[0]#1106(can spill: true) consumed 1024.0 MB, peak 1058.2 MB,
  ExternalSorterMerge[0]#1107(can spill: false) consumed 456.0 B, peak 10.0 MB,
  ShuffleRepartitioner[0]#1109(can spill: true) consumed 0.0 B, peak 0.0 B.
Error: Failed to acquire 228842 bytes, only got 11829 bytes. Reserved: 1073729995 bytes

After PR the query passed

@comphead
comphead marked this pull request as draft September 3, 2026 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant