Reduce CI time (Comet is consuming 50% of DataFusion CI, which is a lot)

[alamb]

What is the problem the feature request solves?

Apparently we (the DataFusion project) is consuming much of the overall ASF github runner allotment and this is causing queuing, among other things

See

• Reduce GitHub Actions usage (DataFusion is currently the top ASF consumer of GHA minutes) datafusion#25148

Describe the potential solution

No response

Additional context

No response

[blaginin]

I think some actions like Spark 4.1 — sql_hive-1 / Spark 4.1 — sql_core-1 / Spark 3.5 — sql_core-1 / Spark 3.5 — sql_hive-1 / TPC-H + TPC-DS have >99% pass rate yet run on every push. Maybe we can move them to main-only or to merge queue?

[comphead]

we should move some things to merge queue, we already discussed this last week on the call.

@andygrove @sunchao cc

[sunchao]

Thanks for pinging me! Agreed, we should reduce the CI time for Comet and maybe consider switching to a tiered CI architecture:

• On PR updates: Run linter, Rust/Comet tests, and Spark tests outside sql_core and sql_hive.
• Merge queue: Run required correctness coverage, including the Spark sql_core and sql_hive, and TPC-H/TPC-DS
• (optional) Nightly: Extended fuzzing, performance benchmarks, additional compatibility combinations

I had Codex to sample some recent CI runs, and found this information:

• One full CI run uses about 26 hours of machine time in total, and about 2 hours wall time excluding queue wait.
• The four Spark sql_core / sql_hive jobs mentioned above, plus TPC-H/TPC-DS, took 260 minutes, or approximately 17%.
• All Spark test shards together: approximately 33%.

[andygrove]

The merge queue idea seems like a good first step. We iterate very quickly with review/fix cycles now and the CI runs many times per PR. Running the Spark SQL and Iceberg tests in the merge queue once the PR is approved will save a lot of time.