[EPIC] Production-quality native Iceberg writes

[andygrove]

What is the problem the feature request solves?

Native Iceberg writes landed in three parts under #4322: the split writer/committer plan (#4658), the eligibility allowlist with fall-back reasons (#5298), and the iceberg-rust data-file writer with JVM-side manifest metrics (#5361). #4322 closed when #5361 merged, and no issue tracks what remains between "merged behind two off-by-default flags" and "production quality". The open follow-ups exist but are not linked to each other, and several gaps had no issue at all.

This epic is that plan. The goal is unchanged from #4322: an ETL job that scans, transforms, and writes Iceberg tables runs natively end to end, and falls back to iceberg-java wherever the native path cannot reproduce its outcome.

Current state. spark.comet.write.iceberg.splitOperator.enabled and spark.comet.iceberg.write.enabled both default to false. Nothing in CI runs with either flag on. Two correctness bugs are open on the native path. The user guide (iceberg-writes.md) still describes the writer as unimplemented in one section.

Describe the potential solution

Phased, in priority order. Each item is a linked issue; check it here when it closes.

Phase 1: correctness blockers

• Native Iceberg writes use an iceberg-rust revision missing partition-path escaping #5636 Bump the iceberg-rust pin past the partition-path escaping fix (a # in an S3 partition value commits a location iceberg-java resolves to a different key)
• Native Iceberg writes drop Hadoop GCS configuration for HadoopFileIO + gs:// #5637 Native writes drop Hadoop GCS configuration for HadoopFileIO + gs://; fail closed until the translation is complete
• Stale native Iceberg write wording in iceberg-writes.md #5641 Fix the stale "not yet wired in" wording and the divergence list in iceberg-writes.md

Phase 2: failure handling parity with iceberg-java

• Native Iceberg write: clean up task-attempt data files on task failure #5618 Clean up task-attempt data files on task failure
• Remove orphaned files in comet iceberg split writers #5277 Remove orphaned files when the commit fails
• Failure-injection tests for native Iceberg writes #5646 Failure-injection tests that assert both the visible outcome and the storage state

Phase 3: coverage so common workloads stay native

• Support Iceberg's system functions (bucket, truncate, years/months/days/hours) as native expressions #5635 Iceberg system functions (bucket, truncate, years/months/days/hours) as native expressions, so the default hash-distribution write is native end to end (PR feat: Support Iceberg system functions (bucket, truncate, years/months/days/hours) natively #5638)
• feat: add native support for MergeRowsExec (row-level MERGE INTO) #5318 Native MergeRowsExec
• Accelerate row-level MERGE / UPDATE / DELETE plans (MergeRowsExec, ReplaceDataExec, WriteDeltaExec) #5122 Row-level MERGE / UPDATE / DELETE plans
• Decide keep-or-lift for each remaining native Iceberg write eligibility restriction #5643 A keep-or-lift decision for every remaining eligibility restriction, so the fallback surface is intentional
• Reconcile the two config namespaces for native writes #5306 Reconcile the two config namespaces for native writes

Phase 4: enable by default

• Iceberg write split-operator plan: CI failures when enabled by default #5259 Fix the four CI failure buckets when the split plan is on by default
• Enable the split-operator plan and native Iceberg writes by default #5644 Graduation criteria and the two-step default flip (split plan first, native writer one release later), including a CI job that runs the Iceberg suites with both flags on

Phase 5: performance

• Add a native Iceberg write benchmark #5647 A native Iceberg write benchmark in the repository (unpartitioned, default hash distribution, fanout, copy-on-write delete)
• Account the native Iceberg writer's buffers in Comet's memory pool #5648 Account the writer's buffers in Comet's memory pool, especially the fanout writer's per-partition buffers
• Native Sort-Merge Writer for Iceberg ClusteredWriter Path #3595 Fused sort plus write for the clustered path

Ongoing

• Adopt a pin-bump policy for the iceberg-rust dependency #5645 A pin-bump policy for iceberg-rust, so upstream fixes to file bytes, manifests, or partition layout reach Comet on a schedule
• [EPIC] Iceberg Feature Matrix: Spark-Iceberg vs iceberg-rust vs datafusion-comet #3756 Update the Iceberg feature matrix, which still lists the write path as not implemented

Additional context

Done and merged: #4658 (split-operator plan), #5298 (eligibility detection), #5361 (native writer). Related umbrellas: #5121 (DataSource V2 writes in general), #1625 (native Parquet writes for the V1 path).

The accepted divergences between iceberg-rust and iceberg-java output are documented in iceberg-writes.md; changes to that list belong in the issues above rather than here.

[andygrove]

cc @jordepic @comphead @hsiang-c @mbutrovich. Let me know if I missed anything.