GH-40502: [C++][Python] Expose NativeFile.abort()

[adrien-grl]

Rationale for this change

Closing an output stream always commits what was written, so a job that fails halfway through still leaves a partial object on S3. Arrow C++ already has Abort() for this, and the S3 stream implements it by aborting the multipart upload, but it wasn't reachable from Python.

What changes are included in this PR?

This adds NativeFile.abort(), which calls FileInterface::Abort() the same way close() calls Close(). The example from the issue now works:

with fs.open_output_stream("bucket/key") as f:
    try:
        do_something_that_may_explode(f)
    except Exception:
        f.abort()
        raise

It also changes the S3 output stream so that a failed abort still closes it. If the AbortMultipartUpload request failed, Abort() returned the error but left the stream open, so the close() from the with block (or the destructor) went on to complete the upload with the partial data. That's easy to run into, since s3:AbortMultipartUpload is a separate IAM permission: with a MinIO user that only had s3:PutObject, the partial object was committed anyway. It also didn't match the Abort() docs in interfaces.h, which say the stream is closed afterwards. The error is still returned, and the worst case is now an incomplete multipart upload, which a lifecycle rule can clean up.

Are these changes tested?

Yes. test_open_output_stream_abort runs on all the filesystem fixtures, with and without compression and buffering. It checks that nothing is written on S3, and that the mock filesystem sees an abort rather than a close. test_s3_output_stream_abort_after_part_upload aborts after a 10 MiB part has already been uploaded, which I don't think the C++ tests cover. test_s3_output_stream_failed_abort uses a MinIO user that isn't allowed to abort multipart uploads, and checks that the error is raised and that neither the with block nor the destructor completes the upload. There are also two small tests in test_io.py for aborting in-memory streams.

Backends other than S3 don't discard anything on abort today so the tests are pretty light for the other backends. I ran it against minio, azurite and the GCS testbench. Local, GCS and fsspec keep the written data, since their Abort() just closes. Azure leaves an empty blob, because the blob is created when the stream is opened. Those seem worth separate issues.

test_io.py and test_fs.py pass locally with S3, Azure and GCS enabled, and so does arrow-s3fs-test.

Are there any user-facing changes?

Yes, NativeFile.abort() is new.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

I used Claude Code to write the code, the tests and this description, and to run the tests locally.

PR code and description written by:

• Human
• AI

Reviewed before submission by:

• Human
• AI
• Not reviewed

• GitHub Issue: [Python] Provide a way to close a NativeFile without writing the contents #40502

[github-actions]

⚠️ GitHub issue #40502 has been automatically assigned in GitHub to PR creator.

[Copilot]

Copilot review overview

🟡 Changes recommended

A failed S3 abort can subsequently trigger close() and commit the upload.

Review effort: Balanced
Findings: 1

Open (1)

• Failed S3 abort can complete multipart upload during cleanup · New

What changed in this PR

Exposes C++ stream abort behavior through Python’s NativeFile.

Changes:

• Adds NativeFile.abort().
• Declares FileInterface::Abort() for Cython.
• Tests aborted memory and filesystem streams.

File	Description
python/​pyarrow/​io.pxi	Adds the Python abort API.
python/​pyarrow/​includes/​libarrow.pxd	Exposes the C++ abort method.
python/​pyarrow/​tests/​test_io.py	Tests in-memory abort behavior.
python/​pyarrow/​tests/​test_fs.py	Tests filesystem and S3 abort behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟢 Approval recommended

The implementation matches the stream contract and includes coverage for successful and failed abort paths.

Review effort: Balanced
Findings: 1

Open (1)

• Failed S3 abort can complete multipart upload during cleanup

[Copilot]

Copilot review overview

🟢 Approval recommended

The implementation matches the documented stream contract and includes focused coverage for success, failure, buffering, and compression paths.

Review effort: Balanced
Findings: 1

Open (1)

• Failed S3 abort can complete multipart upload during cleanup

[Copilot]

Copilot review overview

🔵 Needs a closer look

S3 abort can race with default background uploads and leave multipart-upload storage behind.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Failed S3 abort can complete multipart upload during cleanup

Previously missed (1)

In code that hasn't changed since last review

Wait for pending uploads before aborting multipart upload

cpp/​src/​arrow/​filesystem/​s3fs.cc:1754

Abort() can race with outstanding multipart uploads. With the default background_writes=True, a 10 MiB write queues UploadPart and returns, but this immediately sends AbortMultipartUpload; an in-flight part may finish after the abort and leave multipart-upload storage behind. The new test avoids this path by calling flush(). Wait for upload_state_->pending_uploads_completed (without committing current_part_) before sending the abort, and still issue the abort if an upload failed.

[adrien-grl]

Regarding Copilot's last comment: it seems to me that the background-upload race is Abort() pre-existing behaviour. Waiting for pending uploads would mean uploading everything already queued, which would be slow(er) and might defeat the point of aborting.
Skipping queued uploads when Abort() has been called could be a better fix, but I'm not sure this should be scoped in this PR?