compute: Collapse CollectionEdge to the columnar collection - #37797
Draft
antiguru wants to merge 5 commits into
Draft
compute: Collapse CollectionEdge to the columnar collection#37797antiguru wants to merge 5 commits into
antiguru wants to merge 5 commits into
Conversation
antiguru
force-pushed
the
columnar-te-collapse-enum
branch
from
July 21, 2026 19:53
d2580b4 to
291e330
Compare
antiguru
force-pushed
the
columnar-te-collapse-enum
branch
from
July 22, 2026 08:41
291e330 to
7934d1a
Compare
antiguru
force-pushed
the
columnar-te-collapse-enum
branch
from
July 22, 2026 16:25
7934d1a to
553989d
Compare
antiguru
force-pushed
the
columnar-te-collapse-enum
branch
from
July 22, 2026 17:50
553989d to
4c48faa
Compare
antiguru
force-pushed
the
columnar-te-collapse-enum
branch
from
August 19, 2026 12:01
4c48faa to
e5ae765
Compare
antiguru
force-pushed
the
columnar-te-collapse-enum
branch
from
August 19, 2026 13:17
e5ae765 to
a375205
Compare
antiguru
force-pushed
the
columnar-te-collapse-enum
branch
from
August 20, 2026 08:49
a375205 to
1f1da8a
Compare
antiguru
force-pushed
the
columnar-te-collapse-enum
branch
from
August 20, 2026 09:13
1f1da8a to
ef46557
Compare
Every producer emits the columnar edge and no producer constructs a `Vec` edge, so the `CollectionEdge` enum is now a single-variant wrapper. Replace it with a type alias for the columnar collection and delete the `Vec` variant. Deleting the variant is the compiler-enforced completeness check for the migration: nothing failed to compile, so no live dependency on the row-based edge remained. De-match the carrier operations, which no longer branch on a variant: `scope`, `enter_region`, and `leave_region` are the collection's own methods; `negate`, `concat_many`, `consolidate_named`, and `flat_map_datums` become single columnar free functions (the existing `columnar_negate` / `columnar_consolidate`, and new `concat_many` / `flat_map_datums`). The row-forming consumers (`arrange_collection`, `map_topk_key`, linear-join key preparation, FlatMap input) drop their `Vec` arms. The leaf conversions stay: `.into_vec()` method calls become `columnar_to_vec(...)` at the sanctioned leaves (sink, LetRec, temporal bucketing, TopK fallible-limit, linear-join initial closure, delta-join raw source), and `vec_to_columnar` remains the leaf encode. Pure type collapse with no runtime change; everything was already columnar. The cross-arm unit tests become single-path columnar correctness tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After the enum collapse the Vec arm is gone, so CollectionExt is no longer in scope and the intra-doc link cannot resolve. Reference it as a plain code span, matching the sibling comment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reword the vec_to_columnar/columnar_to_vec docs to describe them as the permanent leaf encode/decode for row-serializing leaves rather than a transitional seam. Split the relocated map_topk_key doc semicolon. Comment-only, no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment-only, no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Six leftovers from collapsing the edge enum, none of which a per-commit review could see, since each only becomes wrong once the enum is gone. Read the delta-join raw source and the TopK fallible limit from the borrowed column. Both consumers only read datums, so neither needs the owned rows that decoding to `Vec` built. The limit path is rarer, but it tees off the hot input, so the decode doubled that input's downstream work on those plans. Negate the diff column rather than re-encoding every record. Rows and times are handed over untouched for a typed input and copied in bulk for a serialized one, where before every row and time was pushed through a builder. Negation stays checked, so `-Diff::MIN` still reports overflow. Delete the FlatMap batch trait. Its `Vec` impl lost its caller here, and the remaining arm needs no abstraction. Its fuel test was passing a `Vec` stream, so the "fueling preserved" claim was being discharged against the dead arm; both tests now feed the columnar edge the operator is given in production. Give the accumulator's keying stage its own name. It shared "LinearJoinKeyPreparation" with the source edge's stage, so introspection could not tell the two apart, and its doc still claimed a caller it no longer has. Correct a doc that referred to `into_vec`, which no longer exists.
antiguru
force-pushed
the
columnar-te-collapse-enum
branch
from
September 6, 2026 18:23
ef46557 to
c4f0a5d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Collapse
CollectionEdgeto atype CollectionEdge = ColumnarCollectionalias: delete the Vec variant and all Vec arms, and de-match the carrier methods. A zero-warning build is the completeness signal that no live Vec dependency remains. The leaf encode/decode free functions (vec_to_columnar/columnar_to_vec) are retained for the sanctioned row-serializing leaves (sinks, LetRec, temporal-bucketing, join internals).Columnar dataflow-edge migration. Design doc:
doc/developer/design/20260720_columnar_dataflow_edges.md(#37744).Part of CPU-51.