Add CopyObject, UploadPartCopy and DeleteObjects operations - #1
Merged
Merged
Conversation
Implements the three S3 operations docker/distribution needs for
blob commits:
- CopyObject: PUT with x-amz-copy-source (URL-encoded, leading slash
optional), streamed server-side copy with MD5 ETag in the
CopyObjectResult response
- UploadPartCopy: PUT with partNumber/uploadId and x-amz-copy-source,
optional x-amz-copy-source-range (inclusive), returns CopyPartResult;
416 InvalidRange for out-of-bounds ranges
- DeleteObjects: POST /{bucket}?delete batch delete with quiet mode;
missing keys count as deleted per AWS semantics
Also fixes two issues surfaced during registry verification:
- All XML timestamps are now formatted in UTC; local-zone offsets
broke the AWS SDK's response parsing
- Verbose logging now includes the query string
Verified end-to-end against registry:2: docker push/pull of an image
with a 40MB layer (multipart copy path) and small layers (CopyObject
path) round-trips with no errors in the registry log.
The v4 lint action installs golangci-lint v1.x, which is built with Go 1.24 and refuses to target Go 1.25. Bump to golangci-lint-action@v8 (golangci-lint v2) and rewrite the config in the v2 schema, replacing the deprecated run.skip-dirs/skip-files and issues.exclude-rules keys.
The v2 config schema requires version to be a string; the action's config-verify step rejects a bare number.
The build step ran './s3dir --help', but s3dir has no flag handling - the binary started the real server and blocked the job indefinitely. (The step never ran before because the lint job it depends on was failing.) Replace it with a bounded smoke test that starts the server, checks the ListBuckets endpoint responds, and kills it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the three S3 operations docker/distribution's s3-aws storage driver needs for blob commits, which previously returned 405 (small blobs) or crashed the registry with a nil-pointer dereference (blobs over 32MB, where the driver switches to multipart copy and dereferences
CopyPartResultwithout a nil check).PUT /{bucket}/{key}withx-amz-copy-source(URL-encoded, leading slash optional). Streamed server-side copy using the existing 32KB-buffer temp-file-and-rename pattern, computing the content MD5 in flight. Returns<CopyObjectResult>with quoted MD5 ETag. 404NoSuchKeyfor a missing source.PUTwithx-amz-copy-sourceand optionalx-amz-copy-source-range: bytes=start-end(inclusive). Feeds a seek +io.LimitReaderof the source into the existingUploadPartpath, so part storage and ETag handling are unchanged. Returns<CopyPartResult>; 416InvalidRangefor out-of-bounds ranges.POST /{bucket}?deletebatch delete with quiet mode. Missing keys count as deleted, matching AWS semantics.All three respect read-only mode (403
AccessDenied).Also fixes two issues surfaced during verification:
+01:00) broke the AWS SDK's response parsing before any copy operation was even reachedTesting
pkg/s3/copy_test.go): encoded/leading-slash/bare copy sources, key with encoded spaces, MD5 ETag correctness, range boundaries (first/last byte, middle, full, out-of-bounds, inverted), multi-part ranged copy reassembled and content-verified, mixed existing/missing batch delete, quiet mode, malformed XML, read-only rejections for all three operationsgo test -race ./...green,golangci-lintcleanregistry:2backed by s3dir;docker push/docker pullof an image with a 40MB layer (exercises UploadPartCopy) plus small layers (CopyObject) round-trips with zero 405/500/panic in the registry log