HIVE-29834: Iceberg: Answer MIN, MAX and COUNT from the handler's column statistic - #6716
Open
deniskuzZ wants to merge 2 commits into
Open
HIVE-29834: Iceberg: Answer MIN, MAX and COUNT from the handler's column statistic#6716deniskuzZ wants to merge 2 commits into
deniskuzZ wants to merge 2 commits into
Conversation
deniskuzZ
force-pushed
the
col_stats_stats_opt
branch
from
August 23, 2026 06:26
27fc967 to
d0e8fea
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
deniskuzZ
changed the title
deniskuzZ
force-pushed
the
col_stats_stats_opt
branch
from
August 23, 2026 11:05
d0e8fea to
be100ec
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
deniskuzZ
force-pushed
the
col_stats_stats_opt
branch
from
August 24, 2026 13:09
be100ec to
efa65c1
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
deniskuzZ
force-pushed
the
col_stats_stats_opt
branch
from
August 24, 2026 20:55
efa65c1 to
8228b0b
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
deniskuzZ
force-pushed
the
col_stats_stats_opt
branch
from
August 25, 2026 08:22
8228b0b to
14823b9
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
deniskuzZ
force-pushed
the
col_stats_stats_opt
branch
from
August 25, 2026 22:13
14823b9 to
aa1cfb7
Compare
deniskuzZ
changed the title
deniskuzZ
force-pushed
the
col_stats_stats_opt
branch
from
August 28, 2026 13:10
4fadd60 to
876c853
Compare
asf-ci-hive
added
tests passed
tests pending
and removed
tests pending
tests passed
labels
deniskuzZ
force-pushed
the
col_stats_stats_opt
branch
from
August 28, 2026 23:55
876c853 to
a7fa8b2
Compare
asf-ci-hive
added
tests pending
tests passed
and removed
tests passed
tests pending
labels
deniskuzZ
force-pushed
the
col_stats_stats_opt
branch
from
August 29, 2026 04:39
a7fa8b2 to
4173058
Compare
asf-ci-hive
added
tests pending
tests passed
and removed
tests passed
tests pending
labels
deniskuzZ
force-pushed
the
col_stats_stats_opt
branch
from
August 29, 2026 11:03
4173058 to
27b2198
Compare
deniskuzZ
force-pushed
the
col_stats_stats_opt
branch
from
August 29, 2026 11:13
27b2198 to
fc3051a
Compare
asf-ci-hive
added
tests failed
tests pending
and removed
tests pending
tests failed
labels
deniskuzZ
force-pushed
the
col_stats_stats_opt
branch
from
August 29, 2026 14:48
fc3051a to
9fb4799
Compare
…ne them on read A partitioned table already kept its column statistics one entry per partition, but every gather held them all in memory, wrote each entry as a serialized Java object, and every read fetched one blob per partition and decoded all of it. None of that survives a table of any size. A gather is now pulled through the writer a batch at a time, so a table of two thousand partitions is never held at once, and an entry is written as the Thrift struct it is. A blob holds one partition, every column's entry behind the field it is for and its own length, so a scan decodes the columns it asked about and steps over the rest. The blobs a scan wants are read in as few requests as their places allow, merging what lies between them where crossing it costs less than the seek it saves - which the store says, not this. Every entry keeps the bit vector its distinct count is merged from. A count per partition is only worth having if the counts can be merged across the partitions a scan reads, and only a vector merges them: without one the fold falls back to the largest single partition's count, which for the partition column of an identity-partitioned table is one however many partitions are read. What still describes the data is decided from the snapshot a file was written for, so statistics separated from their rows are not served as though they stood. ANALYZE stores what it read, an overwrite what it replaced, and a major compaction of one current-spec partition refreshes it; a plain INSERT maintains none, and its partitions read as stale until recomputed. A branch's statistics stay on the branch, a file states every column it comes to hold so that whether a column has any is answerable from the table's metadata without opening it, and a file describing nothing is never committed in place of one that describes something. Where no stored statistics answer, the bounds the manifests already hold do, and they are folded a file at a time: a file states its metrics once for all of its columns, so what holds them is read where the file is in hand rather than again by each column of it. Where a column is one the spec partitions by identity, the partition is the value, so its bounds are exact and are stated even by a file that recorded no metrics of its own - which Iceberg stops inferring past its hundredth column.
…umn statistics MIN, MAX and COUNT over a column are facts the stored statistics already state, so a query asking only for them is answered from what a storage handler holds rather than by reading the rows. The statistics of every aggregate in a query are fetched at once, and a partitioned table is answered only from statistics that describe the partitions, the columns and the snapshot the scan asks about - a partition whose statistics do not cover every asked column, or which a live delete of no named partition may have changed, is not answered for. A table's size comes from its storage handler rather than from listing what its location holds, and whether a join can be a sort-merge is decided before its big table is elected, so an election made on the handler's numbers is not undone by one made on the listing's.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?
depends on #6707