Delete data files written for a rejected commit

[Jprebys]

A transaction writes its data files to storage before it attempts the commit. When the catalog rejects that commit, nothing ever references those files: expiring snapshots cannot reclaim them, since no snapshot ever listed them, so they survive until a delete-orphan-files run, which deliberately waits for files to age before removing any. A copy-on-write delete makes that costly. Each attempt rewrites the surviving rows of every file its predicate only partly matched, so a writer retrying a contended commit strands a full copy of that data per attempt.

Track the data files a transaction writes itself, and delete them once the commit is definitively rejected. Only CommitFailedException qualifies: CommitStateUnknownException means the outcome is unknown and the files may be live table data, so those are left for orphan-file cleanup to age out. A with block abandoned by an exception cleans up for the same reason, having committed nothing.

The Java implementation draws the same line. SnapshotProducer.commit() rethrows CommitStateUnknownException without cleaning up, and calls cleanAll() for the failures marked CleanableFailure, of which CommitFailedException is one. It treats more failures as cleanable than this does — ValidationException, BadRequestException, ForbiddenException, NotAuthorizedException, ServiceUnavailableException and the NoSuchTable family also mean the request was never applied — so widening the set here is a reasonable follow-up. Starting with the one unambiguous case keeps the blast radius of a wrong call small.

Files handed to the transaction by a caller, through add_files or by appending a DataFile to a snapshot producer directly, are left alone. A caller may write its files once and re-append them across commit attempts rather than pay for the write again, and deleting them would break the attempt that follows.

[Jprebys]

@wallacms and @rj-imc Can you take a look at this?

[Jprebys]

After discussing, @wallacms pointed out that the upstream change apache#3320 addresses mostly the same issue. It doesn't cleanup orphaned data files after a failed transaction, but with the retries, that should be less of a problem. After we rebase to pick up that commit, if we still want the ability to cleanup data files, we can revisit.