Motivation
MSE hash aggregation spill currently uses mseAggregationSpillMaxGroups as a proxy for retained memory. Memory per group varies greatly across aggregation functions (e.g. SUM versus DISTINCTCOUNT/sketch functions) and the key representation. A single group-count value cannot provide a reliable memory bound or avoid unnecessary I/O across these queries.
Proposed direction
- Introduce an estimated retained-bytes signal for group keys and aggregation intermediate states. Define the estimate contract for
AggregationFunction (including variable-size functions and plugin implementations) without breaking existing plugins.
- Spill when the estimated memory budget or
mseAggregationSpillMaxGroups is exceeded; name the new query option by its units (for example, mseAggregationSpillMaxBytes).
- Evaluate the trigger during block processing when variable-size states can grow significantly within one input block.
- Preserve
numGroupsLimit as a per-hash-table hard ceiling and keep restore-side group limits distinct from the input spill trigger.
- Test memory estimation, mixed aggregate types, skewed keys, and the interaction of both triggers.
This is a follow-up to roadmap #19727: retained-memory-based triggering is outside the current four-PR, group-count-based spill scope and does not block it. Disk-byte quotas are separate from this memory budget.
Motivation
MSE hash aggregation spill currently uses
mseAggregationSpillMaxGroupsas a proxy for retained memory. Memory per group varies greatly across aggregation functions (e.g. SUM versus DISTINCTCOUNT/sketch functions) and the key representation. A single group-count value cannot provide a reliable memory bound or avoid unnecessary I/O across these queries.Proposed direction
AggregationFunction(including variable-size functions and plugin implementations) without breaking existing plugins.mseAggregationSpillMaxGroupsis exceeded; name the new query option by its units (for example,mseAggregationSpillMaxBytes).numGroupsLimitas a per-hash-table hard ceiling and keep restore-side group limits distinct from the input spill trigger.This is a follow-up to roadmap #19727: retained-memory-based triggering is outside the current four-PR, group-count-based spill scope and does not block it. Disk-byte quotas are separate from this memory budget.