[forsaken628]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

• Spilling is supported on the channel between the pipeline and the page manager, allowing the pipeline to continue executing while result set consumption is blocked.
• Introduced enable_result_set_spilling, which is turned off by default and is fully compatible with the previous behavior.
• Introduced result_set_spilling_to_disk_bytes_limit to support spilling result set data to local disk.

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

---

This change is 

[KKould]

please update the code synchronization #18709 This PR solves the CI test_private_tasks error

[forsaken628]

In this PR, the sender writes the spilled page into a file, which the receiver then restores.

However, this logic is not very reasonable because the size of the page depends on the transmission layer, which does not necessarily correspond to the spilling mode.

In the follow-up, a new spill unit called a "chunk" can be introduced. The sender writes the chunks, which then generates a page from it.
We also need to consider how to be compatible with the old PaginationConf.

[github-actions]

Docker Image for PR

• tag: pr-18679-0be4047-1757663905

note: this image tag is only available for internal use.

[bohutang]

Will do some test on prod this weekend.

[github-actions]

Docker Image for PR

• tag: pr-18679-ddaae69-1757775034

note: this image tag is only available for internal use.