You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Track upstream: botocore's own retries resend a stream body from position 0 #1416
AWSPreparedRequest.reset_stream() calls self.body.seek(0) (botocore/awsrequest.py:543 on develop at ce33f9b1), and AwsChunkedWrapper.seek() rewinds its raw stream to 0 (botocore/httpchecksum.py:232)
Not worked around. S3Core.call() restores the starting position before each attempt of PyAthena's retry policy (#1400), but not between botocore's attempts within one call
It affects a stream given as Body to the methods of S3Core, such as S3Core.put_object(path, Body=file), at a position other than 0, when botocore retries the request.
PyAthena's own uploads (S3File, to_parquet()) send bytes and are not affected.
Measured on botocore 1.44.2 against S3, with BytesIO(b"XXhello") at position 2 and the first attempt answered with 500:
With request_checksum_calculation="when_required", the retry sends all 7 bytes, and S3 stores XXhello.
With "when_supported" (the default), the retry's aws-chunked body is longer than its decoded length, S3 answers every retry with 500 InternalError, and the call fails. PyAthena's default policy retries InternalError (Retry the transient S3 errors with the default retry policy (GH-1375) #1400) and starts its next attempt from the starting position (not measured through PyAthena).
When a fixed release ships
Find the botocore release whose changelog carries the fix, and check that it rewinds both the plain body and the aws-chunked wrapper to the starting position.
Add an offline regression test next to TestS3Core.test_call_retries_stream_body (tests/pyathena/filesystem/test_s3_core.py): a stream at a position other than 0, botocore's retries on, and the first send answered with 500; the retry must not send the bytes before the starting position.
Decide whether to raise the floor botocore>=1.43.31 to the fixed release, or to document that earlier releases resend the stream from position 0.
Update "Retries" in docs/filesystem.md, which says only that each attempt of the policy sends the stream from the starting position.
How to notice
Nothing in the build fails when the fix lands.
Check boto/botocore#3873 and boto/botocore#3874, and botocore's changelog when the lock file's botocore is bumped.
Activity