Skip to content

Give a small GeoParquet file at least ten row groups - #61

Merged
m-mohr merged 3 commits into
mainfrom
smaller-row-groups
Oct 6, 2026
Merged

m-mohr merged 3 commits into
mainfrom
smaller-row-groups

Conversation

@ivorbosloper

@ivorbosloper ivorbosloper commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

A file under 250,000 rows is now written in at least ten row groups instead of groups of 25,000 rows; larger files keep 25,000. With five or six groups, each one covers most of a small file even when its rows are Hilbert-sorted, so a reader can skip none of them, and Portolan's PTL-DAT-006 rejects the file.

  • Found on the canton of Zürich (100k–130k rows): sorted correctly, but its five or six groups each spanned most of the canton, because the canton straddles a quadrant of the CRS-wide Hilbert grid. At ten groups the same rows pass.
  • GeoParquet.row_group_size_for(num_rows) is the one rule; both write paths use it. It rounds down to whole 2,048-row groups, as DuckDB writes whole vectors, so a file under 20,480 rows gets fewer than ten groups.
  • The DuckDB path learns the row count only after writing, so it applies the size in its final rewrite. It rewrites a small file once even when the file is already sorted or cannot be sorted.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This comment was marked as resolved.

Both paths round to whole 2,048-row groups, and the DuckDB output is rebuilt in smaller groups even when it cannot be Hilbert-sorted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This comment was marked as resolved.

@m-mohr
m-mohr merged commit 05c48b2 into main Oct 6, 2026
7 checks passed
m-mohr added a commit that referenced this pull request Oct 6, 2026
Co-authored-by: Matthias Mohr <m.mohr@moregeo.it>
@m-mohr
m-mohr deleted the smaller-row-groups branch October 6, 2026 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants