Skip to content

perf(table): borrow remaining DataFile metadata in files tables - #1924

Merged
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/borrow-datafile-collections
Aug 27, 2026
Merged

perf(table): borrow remaining DataFile metadata in files tables#1924
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/borrow-datafile-collections

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

  • Borrow ColumnSizes, KeyMetadata, SplitOffsets, and EqualityFieldIDs for built-in manifest DataFile values.
  • Keep the public getter fallback for custom DataFile implementations.
  • Add focused tests for both paths.
  • Add a benchmark with empty, medium, and wide metadata.

Benchmark

Same benchmark fixture before and after. Apple M1 Pro, darwin/arm64, Go 1.26.3. -benchtime=3x.

  • files=4096/medium/borrowed: 13.16 ms -> 8.67 ms, 17.3 MB -> 11.3 MB, 45,773 -> 17,102 allocs/op.
  • files=16384/wide/borrowed: 181.0 ms -> 126.2 ms, 247.2 MB -> 153.5 MB, 197,465 -> 82,776 allocs/op.
  • The public wrapper stayed at about 247.2 MB and 197K allocs/op in the wide case.

Command:

go test ./table -run '^$' -bench 'BenchmarkInspectContentFileAppenderDataFileCollections' -benchmem -benchtime=3x -count=1

Checks

  • go test ./...
  • go test -race ./table
  • go test -tags=assert -run='^(TestInspectFilesTablesEarlyRelease|TestInspectDataFilesEmitsEmptyBatchWhenAllEntriesAreDeleted|TestInspectDataFilesEmptyTableEarlyRelease|TestInspectPositionDeletesEarlyRelease)$' ./table
  • go vet ./...
  • golangci-lint run --timeout=10m

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the borrowed metadata path preserves ownership and fallback semantics while materially reducing files-table allocations; tests and CI are green.


This review was drafted by an AI-assisted tool and confirmed by an Iceberg Go maintainer. The maintainer approving this PR has read the findings and signed off. If something feels off, please reply on the PR and a maintainer will follow up.

More on how Iceberg Go handles maintainer review:
CONTRIBUTING.md.

@zeroshade
zeroshade merged commit 4a36855 into apache:main Aug 27, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants