Skip to content

perf(table): skip inactive parquet stats columns - #1918

Merged
laskoviymishka merged 2 commits into
apache:mainfrom
fallintoplace:perf/avoid-inactive-parquet-stats-columns
Aug 27, 2026
Merged

perf(table): skip inactive parquet stats columns#1918
laskoviymishka merged 2 commits into
apache:mainfrom
fallintoplace:perf/avoid-inactive-parquet-stats-columns

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

  • Skip ColumnChunk construction for variant-child, lineage, and none metrics columns.
  • Keep the existing stats behavior for active columns.

Why

DataFileStatsFromMeta visits every physical column in every row group. Inactive columns only added metadata wrapper work and allocations.

Benchmark

Mostly inactive 32-row-group, 9-column case:

  • 1,404 -> 636 allocs/op
  • 202 KB -> 114 KB/op

Checks

  • go test ./...
  • go test -race ./table/internal ./table
  • golangci-lint run --timeout=10m

@tanmayrauth tanmayrauth left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice, tight optimization — the reordering is output-identical: the set of columns that reach the colSizes/valueCounts accumulation is unchanged, splitOffsets still keys off ColumnChunk(0), and you preserved the resolveErr panic path (including fetching the chunk first so a doubly-broken column reports the ame error it did before). The none/skip/variant exclusion is already covered end-to-end by TestMetricsModeNone and TestColumnMetricsMode, both of which run through DataFileStatsFromMeta, so the perf win doesn't cost coverage.

The ColumnChunk(pos) fetch inside the resolveErr branch (parquet_files.go:1475) only exists to preserve which of two fatal panics fires. On a column that's both missing from the metrics plan and has undecodable chunk metadata, it surfaces the low-level parquet error instead of the more actionable "field id N not found in the metrics plan". Since both paths panic, you could just panic(column.resolveErr) directly — but it's behavior-preserving as written and this is an essentially unreachable path.

Comment on lines +1490 to +1493
colChunk, err = rowGroup.ColumnChunk(pos)
if err != nil {
panic(err)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be before invalidateCol check because otherwise there is a regression in metadata error reporting.

For example, an active field is using full metrics and StatsSet errors below for row group 1, then it will be added into invalidateCol in line 1503 and metadata failure will be skipped for row group 2 due to early continue in line 1487

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think you are right, I will push the fix.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@laskoviymishka
laskoviymishka merged commit 32bce4f into apache:main Aug 27, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants