datasource: opt-in open-ahead so FileStream overlaps the next file's open with scanning

[Vedin]

Summary

Adds datafusion.execution.file_stream_open_ahead (default false). When enabled, each FileStream partition claims its next file while the active reader is still producing batches and drives that file's planning until it either yields a ready morsel or blocks on its single outstanding I/O, which poll_scan resolves ahead of polling the reader. At most one file is in flight ahead of the reader, so the extra footprint is one file's metadata per partition.

Why

The morsel-driven FileStream (apache#21342, apache#21351) opens files strictly one after another per partition ("single I/O outstanding", next file only once the reader is drained). Before the rewrite the next file's open future was polled while the current file streamed. On many-file scans over object storage the per-file open latency is now exposed on every file. Measured on Embucket's SPCS M node, TPC-H SF100 Q1 (929 Iceberg/parquet lineitem files, 5 partitions): DataSourceExec time_elapsed_opening 2.2 s (DF 53) → 87.6 s (DF 55, summed over partitions) with metadata_load_time unchanged (~90 s); with this option on, the exposed opening time drops to 1.9 s and the same ~98 s of metadata loading overlaps with reads.

Measured (Embucket rustice, SPCS M, cold sweeps, same image, env toggle only)

	EMB server total, 22 TPC-H queries at SF100	Q1
control (option off)	551.8 s	30.1 s
option on	497.5 s (−9.8 %)	21.4 s (−29 %)

No query slower than control (per-query table in benchmark-results/bench-v6/scan-open-overlap/ of the rustice repo). Q6 on a warm metadata cache also improves (scanning-until-data 31.0 s → 23.4 s) because the next file's CPU planning (filter preparation, statistics pruning, stream construction) is overlapped too.

Notes

• Opening-timer semantics with the option on: time_elapsed_opening counts only the wait that is actually exposed (no reader left to drain); the non-exposed open time shows up in metadata_load_time as before.
• With shared work-stealing queues a partition claims one file early; the imbalance is bounded by one file per partition (same as the pre-morsel static assignment).
• Default off keeps the existing behaviour and every existing snapshot byte-identical; new snapshot test morsel_open_ahead_overlaps_next_file_io documents the interleaving.
• Docs + information_schema.slt updated; cargo test -p datafusion-datasource --lib file_stream 28/28; cargo clippy -p datafusion-datasource --all-targets -D warnings clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_012m5Yx6yEpZsacqAhZotgkT