variant feature branch - #22434
Closed
vuule wants to merge 109 commits into
Closed
Conversation
Implement get_variant_field, cast_variant, and extract_variant_field for Parquet VARIANT columns (struct<list<uint8> metadata, list<uint8> value>). Two-pass GPU kernels parse Variant binary encoding on-device: a sizing pass computes output lengths, followed by an offset scan and a copy pass. Includes Parquet reader support for the VARIANT logical type, unit tests (19 cases covering extraction, casting, nulls, edge cases, and Apache parquet-testing reference vectors), Parquet round-trip tests, and an NVBench divergence benchmark suite with 5 scenarios isolating dictionary scan, field scan, and found-vs-null divergence. Uses create_structs_hierarchy instead of make_structs_column to avoid expensive superimpose_and_sanitize_nulls when output rows contain nulls. Made-with: Cursor
…extraction-gpu
Recognize the Parquet VARIANT logical type (union field 16) along with UUID, FLOAT16, GEOMETRY, and GEOGRAPHY enumerators in the Thrift compact-protocol reader and writer, and route an unshredded VARIANT group's `metadata` / `value` BYTE_ARRAY children through the column buffer as `list<uint8>` via the new PARQUET_COLUMN_BUFFER_FLAG_VARIANT_BINARY flag. A VARIANT column read by libcudf surfaces as `struct<list<uint8> metadata, list<uint8> value>`. Adds pylibcudf roundtrip tests that build VARIANT parquet files in-memory (PyArrow + Thrift footer patching) and assert the resulting column shape. Made-with: Cursor
Replace the in-test PyArrow + Thrift-footer-patch construction with two committed unshredded VARIANT parquet fixtures under ``python/pylibcudf/tests/io/data/parquet/`` and load them in the roundtrip test. Drops the encoding helpers from the test module. Made-with: Cursor
The VARIANT extraction work only needs the reader to recognize the VARIANT logical type. Drop the unused UUID, FLOAT16, GEOMETRY, and GEOGRAPHY enumerators and the corresponding writer cases. The reader's ``function_builder`` already silently skips unknown union fields, so files annotated with those types continue to parse the same way. Made-with: Cursor
Adds duckdb_variant_sample.parquet (DuckDB-shredded VARIANT layout) and a test that reads it through pylibcudf and asserts: - the file reads without error - top-level id stays INT32 - the VARIANT struct's first two children remain list<uint8> (metadata/value pair was flagged correctly by select_columns) - the struct retains its additional shredded children (sanitize_schema did not drop the typed_value subtree) Also drops the redundant row-count assertion from the unshredded shape test now that the shredded test covers the metadata/value pair more broadly. Made-with: Cursor
…extraction-gpu
…m/vuule/cudf into pr/vuule/22416
…ion-gpu # Conflicts: # cpp/include/cudf/io/experimental/variant.hpp # cpp/src/io/parquet/experimental/variant_extract.cu
…ion-gpu Bring in the latest PR 22416 changes (read_uint64 rename, std::format error helper, refactored read_unquoted_name, doc comments, is_variant_castable traits, and relocation of variant tests to cpp/tests/io/experimental/). Conflict resolutions: - variant_path.cpp: keep the GPU branch's "[N]" array-index parsing while adopting pr2's 1-arg read_unquoted_name and std::format-based throw_parse_error. - variant_extract.cu: keep the GPU branch's read_dict_entry + binary-search find_key_in_metadata and locate_array_element, adapted to pr2's read_uint64 rename and non-inline __device__ style; keep pr2's is_variant_int/ is_variant_castable traits and updated doc comments. - cpp/tests/CMakeLists.txt: drop the stale io/variant_extract_test.cpp entry from the PARQUET_TEST bundle now that pr2 moved the test to its own VARIANT_EXTRACT_TEST target under io/experimental/.
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft PR for variant extraction GPU feature branch.
Made with Cursor