Skip to content

Track upstream: botocore's own retries resend a stream body from position 0 #1416

Open

Description

@laughingman7743

What

botocore rewinds a stream body to position 0 before its own retries of a request, not to the position where the request started.

Upstream Defect PyAthena now
boto/botocore#3873, with a proposed fix in boto/botocore#3874 (a third-party PR, not reviewed yet) AWSPreparedRequest.reset_stream() calls self.body.seek(0) (botocore/awsrequest.py:543 on develop at ce33f9b1), and AwsChunkedWrapper.seek() rewinds its raw stream to 0 (botocore/httpchecksum.py:232) Not worked around. S3Core.call() restores the starting position before each attempt of PyAthena's retry policy (#1400), but not between botocore's attempts within one call

It affects a stream given as Body to the methods of S3Core, such as S3Core.put_object(path, Body=file), at a position other than 0, when botocore retries the request.
PyAthena's own uploads (S3File, to_parquet()) send bytes and are not affected.

Measured on botocore 1.44.2 against S3, with BytesIO(b"XXhello") at position 2 and the first attempt answered with 500:

  • With request_checksum_calculation="when_required", the retry sends all 7 bytes, and S3 stores XXhello.
  • With "when_supported" (the default), the retry's aws-chunked body is longer than its decoded length, S3 answers every retry with 500 InternalError, and the call fails. PyAthena's default policy retries InternalError (Retry the transient S3 errors with the default retry policy (GH-1375) #1400) and starts its next attempt from the starting position (not measured through PyAthena).

When a fixed release ships

  1. Find the botocore release whose changelog carries the fix, and check that it rewinds both the plain body and the aws-chunked wrapper to the starting position.
  2. Add an offline regression test next to TestS3Core.test_call_retries_stream_body (tests/pyathena/filesystem/test_s3_core.py): a stream at a position other than 0, botocore's retries on, and the first send answered with 500; the retry must not send the bytes before the starting position.
  3. Decide whether to raise the floor botocore>=1.43.31 to the fixed release, or to document that earlier releases resend the stream from position 0.
  4. Update "Retries" in docs/filesystem.md, which says only that each attempt of the policy sends the stream from the starting position.

How to notice

Nothing in the build fails when the fix lands.
Check boto/botocore#3873 and boto/botocore#3874, and botocore's changelog when the lock file's botocore is bumped.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugstatus:blocked-upstreamWaiting on an upstream project before repository work can complete

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions