Skip to content

tp: fold tree values in a streaming pipeline - #7166

Open
LalitMaganti wants to merge 1 commit into
mainfrom
dev/lalitm/exec-tree-fold
Open

tp: fold tree values in a streaming pipeline#7166
LalitMaganti wants to merge 1 commit into
mainfrom
dev/lalitm/exec-tree-fold

Conversation

@LalitMaganti

@LalitMaganti LalitMaganti commented Aug 21, 2026

Copy link
Copy Markdown
Member

Once rows are in tree order, a fold only needs one running value per node.
TreeAccumulateUp consumes child-first rows and adds each completed child into
its parent. TreeAccumulateDown consumes parent-first rows and carries each
parent total into its children.

Both operators append the accumulated value and stream batches without keeping
another copy of the input.

The SqlScan test covers the complete path from SQLite variants through type
validation, node numbering, ordering, and accumulation.

@github-actions

github-actions Bot commented Aug 21, 2026

Copy link
Copy Markdown

🎨 Perfetto UI Builds

@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-dataframe-scan branch from 7e903d1 to 6354063 Compare August 21, 2026 13:33
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch from 0678132 to 147ef70 Compare August 21, 2026 13:33
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-dataframe-scan branch from 6354063 to ce88f4e Compare August 21, 2026 13:43
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch from 147ef70 to 873a588 Compare August 21, 2026 13:43
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-dataframe-scan branch from ce88f4e to dc9f5a5 Compare August 21, 2026 13:52
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch from 873a588 to 5876d07 Compare August 21, 2026 13:52
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-dataframe-scan branch from dc9f5a5 to cec7f13 Compare August 21, 2026 14:23
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch from 5876d07 to b3fdc90 Compare August 21, 2026 14:23
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-dataframe-scan branch from cec7f13 to 2c2ca49 Compare August 21, 2026 15:12
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch 5 times, most recently from 453bd8d to 9f295f9 Compare August 25, 2026 13:49
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-dataframe-scan branch 2 times, most recently from 300a805 to b34ba57 Compare August 25, 2026 13:54
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch from 9f295f9 to f011e48 Compare August 25, 2026 13:54
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-dataframe-scan branch from b34ba57 to 868c5e4 Compare August 25, 2026 14:21
@LalitMaganti
LalitMaganti changed the base branch from dev/lalitm/exec-dataframe-scan to dev/lalitm/exec-tree-order August 25, 2026 14:21
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch from f011e48 to f733cfc Compare August 25, 2026 14:22
@LalitMaganti LalitMaganti changed the title tp: fold a tree up and down over a stream of chunks tp: fold ordered streams over trees Aug 25, 2026
@LalitMaganti LalitMaganti changed the title tp: fold ordered streams over trees tp: fold tree values in a streaming pipeline Aug 25, 2026
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-order branch from 9fd6c36 to af2c641 Compare August 25, 2026 14:59
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch from f733cfc to d5c0236 Compare August 25, 2026 14:59
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-order branch from af2c641 to de35bf5 Compare August 25, 2026 16:50
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch from d5c0236 to a78c4b5 Compare August 25, 2026 16:50
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-order branch from de35bf5 to e416782 Compare August 26, 2026 16:44
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch from a78c4b5 to 59ba6af Compare August 26, 2026 16:44
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-order branch from e416782 to c443d73 Compare August 26, 2026 17:05
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch from 59ba6af to d2a2af9 Compare August 26, 2026 17:05
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-order branch from c443d73 to dd4e893 Compare August 26, 2026 17:13
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch from d2a2af9 to 9fd5851 Compare August 26, 2026 17:13
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-order branch from dd4e893 to 9546d09 Compare August 26, 2026 17:39
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch from 9fd5851 to 2f73d36 Compare August 26, 2026 17:39
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-order branch 11 times, most recently from d6666bc to 1bb978c Compare September 4, 2026 16:29
Base automatically changed from dev/lalitm/exec-tree-order to main September 4, 2026 16:59
Once rows are in tree order, a fold only needs one running value per node.
TreeAccumulateUp consumes child-first rows and adds each completed child into
its parent. TreeAccumulateDown consumes parent-first rows and carries each
parent total into its children.

Both operators append the accumulated value and stream batches without keeping
another copy of the input.

The SqlScan test covers the complete path from SQLite variants through type
validation, node numbering, ordering, and accumulation.
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/exec-tree-fold branch from 2f73d36 to b315baf Compare September 4, 2026 17:12
@LalitMaganti
LalitMaganti marked this pull request as ready for review September 4, 2026 17:13
@LalitMaganti
LalitMaganti requested a review from a team as a code owner September 4, 2026 17:13

@sashwinbalaji sashwinbalaji left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are CI failures as test written accoding to the old API

// The columns holding the tree structure and the values being summed. Node and
// parent columns must be flat, non-null Uint32 columns; values must be flat
// Int64. A null value contributes zero.
struct AccumulateSpec {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about TreeAccumulateSpec as it's in exec namespace


// What one execution carries between batches: a running total per node, and
// the totals computed for the current batch.
class AccumulateState : public OperatorState {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this to be in .h as we will anywya talk in terms of OperatorState right?

int64_t* totals = s.totals->data();
for (uint32_t row = 0; row < count; ++row) {
uint32_t node = nodes[row];
Grow(&s.by_node, node);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok so I was thinking why we need all the nodes (by_node) as I thought (maybe incorrectly) we wanted DFS post-order and also #7214 said,

Folding up a tree needs every child before its parent, and needs the order
to be a depth first post-order so the fold can carry a stack of the current
path instead of an array indexed by node.

But then I checked the pr again and see in tree_order some early exits which won't necessarily guarantee DFS post order right ?

if (s.child_first) {
  // Already in the requested order, so no reordering is needed.
  s.order.clear();
  return true;
}
if (s.parent_first) {
  for (uint32_t row = 0; row < rows; ++row) {
    s.order[row] = rows - 1 - row;
  }
  return true;
}

Just to make sure, I asked AI to find a possible scenario and it said:

for example, we can have this tree:

      0
     / \
    1   2
    |   |
    3   4

3, 4, 1, 2, 0 has every child before its parent, so we would pass it through, but it is not DFS post-order. A DFS post-order would be 3, 1, 4, 2, 0.

So I guess this is why we need by_node here. Wanted to just make sure this is intentional. And, yes I see the comment on TreeParentFirst saying below so I think we just need to add similar comment for child first also.

// The order is parent first and nothing more: not a pre-order, so a fold
// down keeps a value per node rather than a path. The input columns are node
// numbers, which TreeNumberNodes produces. No column is added.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants