Add ranges, listing pagination, MD5 ETags and metadata support - #2
Merged
Merged
Conversation
Broadens S3 API compatibility beyond the operations the Docker registry strictly needs: - Range GET: single byte ranges including open-ended and suffix forms, with 206/Content-Range responses and 416 InvalidRange when unsatisfiable; malformed or multi-range headers fall back to a full 200 per RFC 9110 - Conditional GET/HEAD: If-None-Match and If-Modified-Since return 304 - ListObjectsV2 (list-type=2) with KeyCount, continuation tokens and start-after; real pagination for v1 via marker/NextMarker. Listings are now in lexicographic key order with IsTruncated set correctly (previously hard-coded false, so buckets over max-keys silently looked complete) - MD5 ETags: PutObject, CopyObject and multipart complete persist the content MD5 (multipart: md5-of-md5s-N) in a metadata sidecar under .metadata/, so PUT, GET, HEAD and listings all agree; pre-existing objects without a sidecar keep the mtime-based fallback - Content-Type and x-amz-meta-* metadata round-trip through PUT, multipart and copy (honouring x-amz-metadata-directive) - GetBucketLocation and bucket subresource handling: versioning/acl stubs, AWS error codes for absent lifecycle/cors/policy/tagging/ encryption/object-lock config, and no-op PUT/DELETE so clients can no longer create or delete a bucket through a subresource URL, or corrupt object data via PUT ?acl/?tagging - ListBuckets no longer exposes the internal .multipart and .metadata directories as buckets Also fixes a listing bug where a non-matching file visited before the prefix could SkipDir the rest of its directory, dropping matching keys. Verified end-to-end against registry:2 (push/pull with a 40MB layer, zero errors) plus wire-format spot checks of location, V2 pagination, ranges and versioning against a live server.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Broadens S3 API compatibility beyond the copy/batch-delete operations in #1. Stacked on that branch — merge #1 first and this diff collapses to a single commit.
bytes=a-b,bytes=a-,bytes=-suffix) with 206 +Content-Range, 416InvalidRangewhen unsatisfiable, andAccept-Ranges: bytesadvertised. Malformed/multi-range headers fall back to a full 200 per RFC 9110.If-None-MatchandIf-Modified-Sinceon GET/HEAD return 304.list-type=2withKeyCount, opaque continuation tokens andstart-after; v1 gainsmarker/NextMarker. Listings are now in lexicographic key order andIsTruncatedis set correctly — previously it was hard-codedfalse, so any bucket over max-keys silently looked complete (this would have bittenregistry garbage-collecton a grown bucket). Also fixes a walk bug where a non-matching file couldSkipDirthe rest of its directory and drop matching keys.md5-of-md5s-N) in a JSON sidecar under.metadata/, so PUT responses, GET/HEAD and listings all agree. Objects created before this change keep the mtime-based fallback. Sync tools that compare ETag to MD5 (rclone, s3cmd) now behave.x-amz-meta-*— round-trip through PUT, multipart upload and server-side copy, honouringx-amz-metadata-directive: REPLACE.?locationand?versioning/?aclstubs; the AWS error codes clients expect for absent?lifecycle/?cors/?policy/?tagging/?encryption/?object-lockconfig. Subresource PUT/DELETE are accepted as no-ops — previouslyPUT /bucket?aclhit CreateBucket,DELETE /bucket?lifecycledeleted the bucket, andPUT /key?acloverwrote the object with the ACL body..multipart/.metadatadirectories as buckets.Out of scope (per discussion): aws-chunked streaming uploads, POST form uploads, presigned-URL semantics.
Testing
pkg/s3/compat_test.go: range boundaries (first/last byte, suffix, open-ended, clamped, unsatisfiable, malformed), conditional 304s, ETag agreement across PUT/GET/HEAD/List, metadata round-trips (plain, multipart, copy, REPLACE directive), v1 marker and v2 continuation-token pagination walks, delimiter rollup with pagination, subresource stubs and error codes, bucket-survives-DELETE ?lifecycle, object-survives-PUT ?aclgo test -race ./...green,golangci-lintclean?location, V2 pagination tokens, ranged reads and?versioning