Skip to content

Apache 5 causing S3 cross-region errors #7047

Closed

Description

@kschwarz1116

Describe the bug

With an S3 client with .crossRegionAccessEnabled(true), the recent update has started showing broken pipes on write requests.
Re-enabling expectContinue does not fix the behavior.

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

Cross region access to continue working out-of-the-box or the functionality to be removed if it is no longer supported.

Current Behavior

Here is a subset of the stack trace:

Caused by: java.net.SocketException: Broken pipe
	at java.base/sun.nio.ch.SocketDispatcher.write0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.write(Unknown Source)
	at java.base/sun.nio.ch.NioSocketImpl.tryWrite(Unknown Source)
	at java.base/sun.nio.ch.NioSocketImpl.implWrite(Unknown Source)
	at java.base/sun.nio.ch.NioSocketImpl.write(Unknown Source)
	at java.base/sun.nio.ch.NioSocketImpl$2.write(Unknown Source)
	at java.base/java.net.Socket$SocketOutputStream.implWrite(Unknown Source)
	at java.base/java.net.Socket$SocketOutputStream.write(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketOutputRecord.deliver(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
	at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection$1.write(DefaultBHttpClientConnection.java:259)
	at org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117)
	at org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:150)
	at org.apache.hc.core5.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:112)
	at org.apache.hc.core5.http.io.entity.InputStreamEntity.writeTo(InputStreamEntity.java:156)
	at org.apache.hc.core5.http.io.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:100)
	at software.amazon.awssdk.http.apache5.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:178)

Reproduction Steps

Attempt writing to S3 with a cross-region client across regions.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.46

JDK version used

25

Operating System and version

Amazon Linux 2023 (Lambda)

Activity

  1. added
    bugThis issue is a bug.
    needs-triageThis issue or PR still needs to be triaged.
    on Jun 17, 2026
  2. dagnir commented on Jun 17, 2026

    @dagnir
    Contributor

    hi @kschwarz1116 can you provide more details on your setup? I'm unable to repro with the following using 2.46.0

    edit: description said write, missed that! Writing also seems to work with 2.46.0 (locally, and in Lambda with JDK 25):

    public class Repro {
        private static final String BUCKET = "dongie-us-east-1";
        private static final String KEY = "test.txt";
    
        public static void main(String[] args) {
            S3Client s3 = S3Client.builder()
                    .region(Region.US_WEST_2)
                    .crossRegionAccessEnabled(true)
                    .build();
    
            s3.putObject(r -> r.bucket(BUCKET).key(KEY), RequestBody.fromString("test"));
    
            ResponseBytes<GetObjectResponse> object = s3.getObject(r -> r.bucket(BUCKET).key(KEY), ResponseTransformer.toBytes());
    
            System.out.println(object.asUtf8String());
    
        }
    }
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-0 >> "PUT /test.txt HTTP/1.1[\r][\n]"
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-0 >> "Host: dongie-us-east-1.s3.us-west-2.amazonaws.com[\r][\n]"
    ...
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-0 << "HTTP/1.1 301 Moved Permanently[\r][\n]"
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]"
    ...
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "PUT /test.txt HTTP/1.1[\r][\n]"
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "Host: dongie-us-east-1.s3.us-east-1.amazonaws.com[\r][\n]"
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "Content-Length: 45[\r][\n]"
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "Connection: keep-alive[\r][\n]"
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "[\r][\n]"
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "4[\r][\n]"
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "test[\r][\n]"
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "0[\r][\n]"
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "x-amz-checksum-crc32:2H9+DA==[\r][\n]"
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "[\r][\n]"
    ...
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "GET /test.txt HTTP/1.1[\r][\n]"
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "Host: dongie-us-east-1.s3.us-east-1.amazonaws.com[\r][\n]"
    ...
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 << "HTTP/1.1 200 OK[\r][\n]"
    2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 << "Content-Length: 20[\r][\n]"
    ...
    test
    
  3. added
    response-requestedWaiting on additional info and feedback. Will move to "closing-soon" in 10 days.
    and removed
    needs-triageThis issue or PR still needs to be triaged.
    on Jun 17, 2026
  4. dagnir commented on Jun 18, 2026

    @dagnir
    Contributor

    Okay, looks like a larger request body is needed to trigger this:

    public class Repro {
        private static final String BUCKET = "dongie-us-east-1";
        private static final String KEY = "test.dat";
        private static final Path FILE = Paths.get("128mb.dat");
    
        public static void main(String[] args) throws IOException {
            S3Client s3 = S3Client.builder()
                    .region(Region.US_WEST_2)
                    .crossRegionAccessEnabled(true)
                    .build();
    
            InputStream is = new FileInputStream(FILE.toFile());
            RequestBody requestBody = RequestBody.fromInputStream(is, Files.size(FILE));
    
            s3.putObject(r -> r.bucket(BUCKET).key(KEY), requestBody);
        }
    }

    Will continue to investigate this. As a workaround, you can switch back to the 4.x client:

            S3Client s3 = S3Client.builder()
                    .region(Region.US_WEST_2)
                    .httpClientBuilder(ApacheHttpClient.builder())
                    .crossRegionAccessEnabled(true)
                    .build();
  5. removed
    response-requestedWaiting on additional info and feedback. Will move to "closing-soon" in 10 days.
    on Jun 18, 2026
  6. kschwarz1116 commented on Jun 18, 2026

    @kschwarz1116
    Author

    Thanks, yep we're writing large payloads! Currently getting around it by just instantiating separate clients per-region, but going back to 4.x works too.

  7. dagnir commented on Jun 18, 2026

    @dagnir
    Contributor

    Looks like this is due to a difference in how Apache 5.x treats 3xx responses to requests with Expect: 100-continue, i.e. that they're not treated as final status codes. The client continues to send the entity even when a 3xx is received https://github.com/apache/httpcomponents-core/blob/d96a00fec9b2e19f8005e35681df5f6cd6e21a9e/httpcore5/src/main/java/org/apache/hc/core5/http/impl/io/HttpRequestExecutor.java#L186-L190

    There is an open issue on the Apache tracker for this, and I'm requesting for it to be fixed https://issues.apache.org/jira/browse/HTTPCORE-796

  8. dagnir commented on Jun 22, 2026

    @dagnir
    Contributor

    Update: this is fixed upstream in apache/httpcomponents-core#675 🎉 . Awaiting a Maven release to bump the SDK dependency.

  9. github-actions commented on Jun 25, 2026

    @github-actions

    This issue is now closed. Comments on closed issues are hard for our team to see.
    If you need more assistance, please open a new issue that references this one.

  10. kschwarz1116 commented on Jun 25, 2026

    @kschwarz1116
    Author

    Thank you very much for getting this addressed! I'll be on the lookout for 2.46.18!

  11. dagnir commented on Jun 25, 2026

    @dagnir
    Contributor

    @kschwarz1116 Next release should be Monday!

  12. added a commit that references this issue on Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugThis issue is a bug.p1This is a high priority issuepotential-regressionMarking this issue as a potential regression to be checked by team member

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions