tp: fold tree values in a streaming pipeline - #7166
Conversation
7e903d1 to
6354063
Compare
0678132 to
147ef70
Compare
6354063 to
ce88f4e
Compare
147ef70 to
873a588
Compare
ce88f4e to
dc9f5a5
Compare
873a588 to
5876d07
Compare
dc9f5a5 to
cec7f13
Compare
5876d07 to
b3fdc90
Compare
cec7f13 to
2c2ca49
Compare
453bd8d to
9f295f9
Compare
300a805 to
b34ba57
Compare
9f295f9 to
f011e48
Compare
b34ba57 to
868c5e4
Compare
f011e48 to
f733cfc
Compare
9fd6c36 to
af2c641
Compare
f733cfc to
d5c0236
Compare
af2c641 to
de35bf5
Compare
d5c0236 to
a78c4b5
Compare
de35bf5 to
e416782
Compare
a78c4b5 to
59ba6af
Compare
e416782 to
c443d73
Compare
59ba6af to
d2a2af9
Compare
c443d73 to
dd4e893
Compare
d2a2af9 to
9fd5851
Compare
dd4e893 to
9546d09
Compare
9fd5851 to
2f73d36
Compare
d6666bc to
1bb978c
Compare
Once rows are in tree order, a fold only needs one running value per node. TreeAccumulateUp consumes child-first rows and adds each completed child into its parent. TreeAccumulateDown consumes parent-first rows and carries each parent total into its children. Both operators append the accumulated value and stream batches without keeping another copy of the input. The SqlScan test covers the complete path from SQLite variants through type validation, node numbering, ordering, and accumulation.
2f73d36 to
b315baf
Compare
sashwinbalaji
left a comment
There was a problem hiding this comment.
There are CI failures as test written accoding to the old API
| // The columns holding the tree structure and the values being summed. Node and | ||
| // parent columns must be flat, non-null Uint32 columns; values must be flat | ||
| // Int64. A null value contributes zero. | ||
| struct AccumulateSpec { |
There was a problem hiding this comment.
how about TreeAccumulateSpec as it's in exec namespace
|
|
||
| // What one execution carries between batches: a running total per node, and | ||
| // the totals computed for the current batch. | ||
| class AccumulateState : public OperatorState { |
There was a problem hiding this comment.
Do we need this to be in .h as we will anywya talk in terms of OperatorState right?
| int64_t* totals = s.totals->data(); | ||
| for (uint32_t row = 0; row < count; ++row) { | ||
| uint32_t node = nodes[row]; | ||
| Grow(&s.by_node, node); |
There was a problem hiding this comment.
ok so I was thinking why we need all the nodes (by_node) as I thought (maybe incorrectly) we wanted DFS post-order and also #7214 said,
Folding up a tree needs every child before its parent, and needs the order
to be a depth first post-order so the fold can carry a stack of the current
path instead of an array indexed by node.
But then I checked the pr again and see in tree_order some early exits which won't necessarily guarantee DFS post order right ?
if (s.child_first) {
// Already in the requested order, so no reordering is needed.
s.order.clear();
return true;
}
if (s.parent_first) {
for (uint32_t row = 0; row < rows; ++row) {
s.order[row] = rows - 1 - row;
}
return true;
}
Just to make sure, I asked AI to find a possible scenario and it said:
for example, we can have this tree:
0
/ \
1 2
| |
3 4
3, 4, 1, 2, 0 has every child before its parent, so we would pass it through, but it is not DFS post-order. A DFS post-order would be 3, 1, 4, 2, 0.
So I guess this is why we need by_node here. Wanted to just make sure this is intentional. And, yes I see the comment on TreeParentFirst saying below so I think we just need to add similar comment for child first also.
// The order is parent first and nothing more: not a pre-order, so a fold
// down keeps a value per node rather than a path. The input columns are node
// numbers, which TreeNumberNodes produces. No column is added.
Once rows are in tree order, a fold only needs one running value per node.
TreeAccumulateUp consumes child-first rows and adds each completed child into
its parent. TreeAccumulateDown consumes parent-first rows and carries each
parent total into its children.
Both operators append the accumulated value and stream batches without keeping
another copy of the input.
The SqlScan test covers the complete path from SQLite variants through type
validation, node numbering, ordering, and accumulation.