Parquet: read-path redesign — serve bytes as they arrive, bounded coalescing, lifetime memory pools, read-ahead controller, filesystem-cache fixes

[UnamedRus]

Rework of the Parquet v3 read path for object storage, replacing the earlier read-sizing knobs in this PR with a measured redesign (spec: docs/superpowers/specs/2026-08-27-parquet-readpath-redesign.md on the branch).

Problem. On S3-backed Iceberg tables the reader was latency-bound: a coalesced read task blocked delivery until its last byte arrived; a 640-byte column could drag 3.5 MiB of unrelated bytes per row group through the coalescer (32 GB through the cache disk for 450 MB of pages on one bench query); the filesystem cache ignored the per-query boundary_alignment on the random-access path and always queued background fill of partially read segments (450 MB requested → 35 GB downloaded); memory was budgeted per pipeline stage so the high watermark was not a cap (delivered chunks were never charged); and reads were issued one row group at a time per file (~25–29 GETs in flight per node regardless of settings).

What changes.

• Prefetcher serves a coalesced read's bytes as they arrive (Task::bytes_ready, progress callback from readBigAt); decode starts on the first row group of a task while later bytes stream. New events ParquetPartialReadsServed, ParquetReadTasks, ParquetReadTaskBytes.

• Coalescing is bounded: input_format_parquet_max_read_amplification (default 4×) and input_format_parquet_coalesce_gap_bytes (default 2 MiB on remote storage; measured sweet spot vs the 4 MiB remote_read_min_bytes_for_seek).

• Filesystem cache: readBigAt honours the per-query filesystem_cache_boundary_alignment; FileSegmentsHolder honours filesystem_cache_allow_background_download.

• Memory is budgeted by lifetime — Metadata / Compressed (input_format_parquet_compressed_memory_fraction, default 0.35) / Decoded pools — and delivered chunks stay charged to Decoded until the pipeline drops them (ChunkMemoryInfo). input_format_parquet_prefetch_memory_fraction is kept for compatibility but no longer sizes memory.

• Reads are pre-issued for all row groups of a file (index reads at init, page reads once the offset index is known) from one queue, bounded by a bytes-in-flight target fitted online from measured first-byte time and bandwidth (Prefetcher::ReadStats; floor input_format_parquet_min_bytes_in_flight, default 64 MiB). Events ParquetPlannedReads, ParquetIssueQueueStalls, ParquetReadFirstByteMicroseconds, ParquetReadTransferMicroseconds.

• IO pool derived from the query (max(max_download_threads, min(max_parsing_threads, 16)), override input_format_parquet_max_io_threads); input_format_parquet_bytes_per_read_task.

• Read amplification is bounded by a ratio and a floor: input_format_parquet_max_read_amplification (default 8) only applies once a read wastes more than input_format_parquet_read_amplification_floor_bytes (default 256 KiB). A ratio alone cannot see that five columns of a 25 KiB file lie a few KB apart, and splitting such a read costs a round trip to save nothing.

• bytesInFlight counts bytes being read, not bytes queued: a task waiting for an IO thread no longer consumes the read-ahead target, which previously made input_format_parquet_min_bytes_in_flight a bound on queue length rather than on read depth.

• The coalescer reads through gaps the source already holds in its page cache, and the page cache itself reads through short islands of cached blocks; the budget for the latter is fitted by the cache from its own bandwidth and round-trip time and capped by a share of the request that the Parquet reader publishes from its IO concurrency. Event PageCacheReadThroughBytes.

Behaviour changes at defaults, also for local files: read amplification cap (8×, and only above 256 KiB of waste) applies everywhere (gap threshold for local files stays 8 KiB); IO pool 4 → min(max_parsing_threads, 16); up to input_format_parquet_min_bytes_in_flight (64 MiB) of index/page reads issued ahead per file. Pre-issue reads offset indexes and bloom-filter headers of row groups the column index later prunes and pages for PREWHERE-filtered rows — bounded extra bytes visible in the A/B. input_format_parquet_min_bytes_in_flight = 0 disables planning (previous behaviour). input_format_parquet_prefetch_memory_fraction is obsolete.

Known limitations to measure (not design flaws): the target is scaled by the query-wide IO pool per file; the Metadata pool share is a fixed 0.05 — on very wide files pre-issue may stall on it (ParquetIssueQueueStalls). Fitted read stats exclude page-cache-served reads. Found and fixed en route: AsynchronousBoundedReadBuffer::readBigAt consumed its pending prefetch future unsynchronised (segfault under concurrent positioned reads; from the ClickHouse#110263 backport) — to be reported upstream, together with FileSegmentsHolder::reset's non-advancing catch loop.

Measured. Single node, real S3 Iceberg, release binaries on both sides (base image vs the CI build_amd_release of this head), 23 IcebergBench queries, arm order alternated per query, results byte-identical on all 23.

The headline depends on which caches are warm, so all three states are given. The Iceberg metadata cache (use_iceberg_metadata_files_cache, on by default) is the one that decides:

cache state	base	this branch	delta
Iceberg metadata warm, data cold — the steady state after the first query on a table	10.7 s	8.5 s	-20%, faster on 23 of 23
fully warm	15.2 s	12.1 s	-20% (-26% on per-query minima)
fully cold, manifests included	37.6 s	37.0 s	-2%

The fully cold row is not a read-path result: 72% of a cold query on this dataset is the serial Iceberg manifest phase (27.7 s of the 37.6 s, 731 metadata files across the 23 queries, every pipeline thread blocked on it), which this PR does not touch. The part the read path owns — time the decoding threads spend blocked on file data — is 2.3 s of that 37.6 s, and the branch cuts it 35%, which is the -2%. Warm the metadata layer alone and the manifest phase disappears (37.6 s -> 10.7 s for base by itself), leaving the read path visible at -20%. The three queries that looked like cold regressions in a single sample are within noise at five cold reps (q04 1.56 -> 1.43 s, q18 1.48 -> 1.43 s, q02 1.46 -> 1.48 s).

Across all three states the branch reads 37-38% fewer bytes (1767 -> 1099 MiB metadata-warm; 1784 -> 1118 MiB cold) for ~12% more requests, with peak memory unchanged. Per-query wins metadata-warm: q20 -40%, q06 -32%, q21 -29%, q08 -28%, q04 -27%, q13 -22%, q05/q17 -20%. Read amplification (ReadBufferFromS3Bytes / ParquetReadTaskBytes) drops 41.5x -> 1.4x on q20, 6.2x -> 1.7x on q14, 4.3x -> 1.1x on q06, 3.1x -> 1.2x on q18. ParquetPartialReadsServed is 11-689 per query, so partial readiness serves decodes on the object-storage path; ParquetIssueQueueStalls is 0 everywhere, so the Metadata pool share never throttled pre-issue on this shape.

A 17,554-file Iceberg table (437 columns, ~70 GiB, mostly ~25 KiB files) is what produced input_format_parquet_read_amplification_floor_bytes. With the ratio bound alone the branch was 45% slower than base on a five-column month slice (20.1 s vs 13.9 s, 16,440 vs 9,968 requests, 502 vs 576 MiB): the bound split each small file's read three to four ways to avoid ~20 KB of waste. At the shipped defaults the branch matches base there (13.74 s, 9,968 requests, 576 MiB against base's 14.03 s, 9,968, 576), and setting the floor to 0 reproduces the old behaviour exactly (18.82 s, 16,440 requests, 502 MiB). input_format_parquet_max_io_threads makes no difference on this shape — it is bound by file count, not read depth.

A single large file (459 and 918 row groups on S3, plus a 200-file single-row-group control) is where the redesign is largest. The base reader reads such a file effectively serially — mean concurrent GET requests 0.98, ~90 MiB/s — while this branch reaches 19-70 concurrent requests and 1.1-1.2 GiB/s: 10-14x less wall time. Base does not improve when given the same IO pool (max_download_threads 4 -> 128 changes nothing), so the concurrency is what the redesign creates and the pool size only throttles it. On this shape the bytes-in-flight controller specifically is neutral — input_format_parquet_min_bytes_in_flight at 0, 64 MiB and 256 MiB are indistinguishable in wall time and concurrency, while the non-zero values cost ~2.3x peak memory — and the IO pool size is the lever that matters (3.6 s -> 1.2 s from 16 to 128 threads on the read-bound query).

A partially warm page cache is what the two cache-side commits address. Cache hits queued behind ~30 ms network reads, and cached blocks fragmenting the coalescing, used to make a half-cached file slower than an uncached one at a small IO pool (9.3 s vs 7.8 s). With both changes it is 5.8 s, requests drop 942 -> 201, and at a deep pool the read-through disables itself rather than spending bytes it cannot afford.

Related: #2266
Related: #2235
Related: ClickHouse#102282
Related: ClickHouse#103706
Related: ClickHouse#115816

Changelog category (leave one):

• Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Parquet reader on object storage: bytes of a coalesced read are decoded as they arrive; read coalescing is bounded by input_format_parquet_max_read_amplification and input_format_parquet_coalesce_gap_bytes; reads for all row groups of a file are issued ahead under a bytes-in-flight target (input_format_parquet_min_bytes_in_flight); reader memory is budgeted by lifetime (input_format_parquet_compressed_memory_fraction) and delivered chunks stay charged until consumed; the filesystem cache honours filesystem_cache_boundary_alignment and filesystem_cache_allow_background_download on the Parquet random-access path. New settings input_format_parquet_max_io_threads, input_format_parquet_bytes_per_read_task, input_format_parquet_read_amplification_floor_bytes (a read wasting little in absolute terms is exempt from the amplification bound); coalescing and the page cache now read through data that is already cached rather than splitting the read around it; new profile events ParquetPartialReadsServed, ParquetReadTasks, ParquetReadTaskBytes, ParquetPlannedReads, ParquetIssueQueueStalls, ParquetReadFirstByteMicroseconds, ParquetReadTransferMicroseconds, PageCacheReadThroughBytes.

Documentation entry for user-facing changes

All new settings are documented in their DECLARE doc strings; input_format_parquet_prefetch_memory_fraction's doc string notes it is superseded.

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)

[github-actions]

Workflow [PR], commit [6503da0]