Skip to content

Add a memory-based trigger for MSE hash aggregation spill #19666

Description

@wirybeaver

Motivation

MSE hash aggregation spill currently uses mseAggregationSpillMaxGroups as a proxy for retained memory. Memory per group varies greatly across aggregation functions (e.g. SUM versus DISTINCTCOUNT/sketch functions) and the key representation. A single group-count value cannot provide a reliable memory bound or avoid unnecessary I/O across these queries.

Proposed direction

  • Introduce an estimated retained-bytes signal for group keys and aggregation intermediate states. Define the estimate contract for AggregationFunction (including variable-size functions and plugin implementations) without breaking existing plugins.
  • Spill when the estimated memory budget or mseAggregationSpillMaxGroups is exceeded; name the new query option by its units (for example, mseAggregationSpillMaxBytes).
  • Evaluate the trigger during block processing when variable-size states can grow significantly within one input block.
  • Preserve numGroupsLimit as a per-hash-table hard ceiling and keep restore-side group limits distinct from the input spill trigger.
  • Test memory estimation, mixed aggregate types, skewed keys, and the interaction of both triggers.

This is a follow-up to roadmap #19727: retained-memory-based triggering is outside the current four-PR, group-count-based spill scope and does not block it. Disk-byte quotas are separate from this memory budget.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions