Repository navigation
Parquet filtering - #2693
Merged
Merged
Parquet filtering#2693
Conversation
- type-aware physical ordering for statistics (unsigned, decimal, float16, IEEE 754 total order) - column orders, statistics truncation and trusted min/max on read - UINT_64 read and written as numeric string - parquet.thrift and ThriftModel regenerated with nullable optional fields - arrow-ext writes ENUM logical type and matches PHP writer statistics - parquet-viewer shows statistics as logical values
- Filter API (parquet_column/all/any/not) and Query (parquet_query) with exact results - row groups pruned by chunk statistics, identical on PHP and arrow engines - reads take a ReadPlan instead of limit/offset, both extension ABIs bumped - ParquetFileReader exposes a lazy Footer - docs fail on a stale extension instead of writing wrong defaults
- FileScan carrier: FileExtractor::scan() replaces extract() limit/path filter - push every deterministic AND conjunct into file sources, shown in EXPLAIN - parquet: translate pushed predicates and skip row groups by statistics - string comparisons in byte order through ValueOrder - numeric_string comparable with string literals, ordered numerically
- PHP writer writes column and offset indexes (STATISTICS_LEVEL, COLUMN_INDEX_TRUNCATE_LENGTH) - page index pruning narrows read plans below row groups - both engines read only selected pages (arrow-ext on arrow-rs push decoder) - fix RLE_DICTIONARY chunks spanning multiple pages - page header statistics off by default (PAGE_HEADER_STATISTICS) - FLOW_ARROW_ABI 2
- split block bloom filters, arrow-rs compatible sizing and folding - COLUMNS_BLOOM_FILTERS option on the PHP and arrow writers - equals/in prune row groups before the page index - AND/OR truth combinators moved onto Truth - fixture generators moved out of the repository
- both engines decode filter columns first, others only at kept rows - arrow-ext filters with arrow-rs RowFilter, plan carries filter and limit - limit pushed to the reader when the whole filter is bound - PHP caches decoded filter pages and their dictionary for the output - arrow ABI bumped to 3
…d beside it - file extractors return only rows matching FileScan::$filter, conjunct by conjunct - pushed conjuncts leave their Filter node, pushed leaf first, stopping at a kept conjunct - a limit above a fully pushed filter reaches the scan - ReplaceLeaf renamed ReplaceNode
- simplify filter binding and evaluation - page reads through selected pages, malformed indexes ignored - arrow writer: dictionary encodings, page row limit, size validation - file sources filter through one shared evaluation - explain() wording and plan node replacement reworked - thrift generation behind bin/thrift.php parquet
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 1.x #2693 +/- ##
============================================
+ Coverage 89.39% 90.04% +0.64%
- Complexity 0 29380 +29380
============================================
Files 2117 2208 +91
Lines 82112 85514 +3402
============================================
+ Hits 73403 76998 +3595
+ Misses 8709 8516 -193 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves: #687
Change Log
Added
flow-php/parquet- filtered reads with row group, page index and bloom filter pruningflow-php/parquet- row filtering during decode, with a limit beside the filterflow-php/parquet- column index, offset index and bloom filter writingflow-php/etl- DataFrame filters pushed into file sourcesflow-php/etl-adapter-parquet- pushed filters translated into parquet readsflow-php/arrow-ext- page-range reads and row filtering through arrow-rsFixed
flow-php/parquet- corrupt multi-page RLE_DICTIONARY columns written by the PHP writerflow-php/parquet- UINT_64 values above PHP_INT_MAX read as numeric stringsflow-php/etl- string comparisons order by bytes instead of numericallyflow-php/arrow-ext- digit-only column option keys no longer ignoredChanged
flow-php/parquet- column statistics aligned with arrow-rsflow-php/parquet- ParquetFile reads take a Queryflow-php/parquet- PAGE_MAXIMUM_ROWS_COUNT and ROW_GROUP_SIZE_BYTES enforced by both writersflow-php/etl- file extractors read through FileScanflow-php/etl- explain() shows pushed filters and limits under Readflow-php/arrow-ext- built against arrow-rs 60Removed
flow-php/parquet- unused DICTIONARY_PAGE_MIN_CARDINALITY_RATION optionDeprecated
Security