perf(table): skip inactive parquet stats columns - #1918
Conversation
There was a problem hiding this comment.
LGTM. Nice, tight optimization — the reordering is output-identical: the set of columns that reach the colSizes/valueCounts accumulation is unchanged, splitOffsets still keys off ColumnChunk(0), and you preserved the resolveErr panic path (including fetching the chunk first so a doubly-broken column reports the ame error it did before). The none/skip/variant exclusion is already covered end-to-end by TestMetricsModeNone and TestColumnMetricsMode, both of which run through DataFileStatsFromMeta, so the perf win doesn't cost coverage.
The ColumnChunk(pos) fetch inside the resolveErr branch (parquet_files.go:1475) only exists to preserve which of two fatal panics fires. On a column that's both missing from the metrics plan and has undecodable chunk metadata, it surfaces the low-level parquet error instead of the more actionable "field id N not found in the metrics plan". Since both paths panic, you could just panic(column.resolveErr) directly — but it's behavior-preserving as written and this is an essentially unreachable path.
| colChunk, err = rowGroup.ColumnChunk(pos) | ||
| if err != nil { | ||
| panic(err) | ||
| } |
There was a problem hiding this comment.
I think this should be before invalidateCol check because otherwise there is a regression in metadata error reporting.
For example, an active field is using full metrics and StatsSet errors below for row group 1, then it will be added into invalidateCol in line 1503 and metadata failure will be skipped for row group 2 due to early continue in line 1487
There was a problem hiding this comment.
Thanks, I think you are right, I will push the fix.
What
ColumnChunkconstruction for variant-child, lineage, andnonemetrics columns.Why
DataFileStatsFromMetavisits every physical column in every row group. Inactive columns only added metadata wrapper work and allocations.Benchmark
Mostly inactive 32-row-group, 9-column case:
Checks
go test ./...go test -race ./table/internal ./tablegolangci-lint run --timeout=10m