GH-32438: [C++] Create target file when OpenAppendStream targets a nonexistent path

[mohammadnaqvi04]

Rationale for this change

FileSystem::OpenAppendStream is documented to create the target file if it doesn't already exist. On HDFS this instead raises a not-found error. HadoopFileSystemImpl::OpenWritable always opens in append mode when asked to append, but Hadoop's FileSystem.append() requires the file to already exist, unlike a local append-mode open. See #32438.

What changes are included in this PR?

• OpenWritable now only sets append mode when the target already exists; otherwise it falls back to a plain create, matching every other backend. Adds an existence check against the NameNode before an append-mode open, since that's the only way to know which mode to request.
• Removed TestHadoopFileSystemGeneric::allow_append_to_new_file's override to false, seemingly added when HDFS was first wired into the shared filesystem test suite. HDFS now runs the same append-to-new-file check as every other backend.

Are these changes tested?

Yes. With the override removed, TestHadoopFileSystemGeneric.OpenAppendStream fails on the unpatched code with the same not-found error from the issue, and passes with this change, run against the Docker-based HDFS test setup (ci/scripts/integration_hdfs.sh). The rest of the HDFS C++ suite and pyarrow.tests.test_fs pass with no regressions.

Unrelated to this change: ci/scripts/install_minio.sh currently fails for anyone building the CI images locally, since MinIO discontinued the binary distribution it fetches from. I patched around it locally to get a working build. If this is fixed properly upstream and CI behaves differently as a result, I wouldn't have had visibility into that from my local setup.

Are there any user-facing changes?

Yes: appending to a nonexistent file on HDFS now succeeds and creates the file, instead of raising an error.

This PR contains a "Critical Fix". It fixes a case where an operation documented to succeed instead raised an error.

• GitHub Issue: [C++] HadoopFileSystem open_append_stream throwing an error if file does not exists #32438

[github-actions]

⚠️ GitHub issue #32438 has been automatically assigned in GitHub to PR creator.

[mohammadnaqvi04]

@pitrou Bumping this for whenever you get a chance. Happy to address anything that needs changing!

[mohammadnaqvi04]

I've rebased onto current main to pick up the minio CI fix that was landing while this PR was open. Could someone please approve the workflow runs on the latest push?