Skip to content

Add ranges, listing pagination, MD5 ETags and metadata support - #2

Merged
stut merged 1 commit into
mainfrom
feat/s3-compat-improvements
Jul 20, 2026
Merged

stut merged 1 commit into
mainfrom
feat/s3-compat-improvements

Conversation

@stut

@stut stut commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Summary

Broadens S3 API compatibility beyond the copy/batch-delete operations in #1. Stacked on that branch — merge #1 first and this diff collapses to a single commit.

  • Range GET — single byte ranges (bytes=a-b, bytes=a-, bytes=-suffix) with 206 + Content-Range, 416 InvalidRange when unsatisfiable, and Accept-Ranges: bytes advertised. Malformed/multi-range headers fall back to a full 200 per RFC 9110.
  • Conditional requests — If-None-Match and If-Modified-Since on GET/HEAD return 304.
  • ListObjectsV2 + real pagination — list-type=2 with KeyCount, opaque continuation tokens and start-after; v1 gains marker/NextMarker. Listings are now in lexicographic key order and IsTruncated is set correctly — previously it was hard-coded false, so any bucket over max-keys silently looked complete (this would have bitten registry garbage-collect on a grown bucket). Also fixes a walk bug where a non-matching file could SkipDir the rest of its directory and drop matching keys.
  • MD5 ETags everywhere — PutObject, CopyObject and multipart completion persist the content MD5 (multipart: md5-of-md5s-N) in a JSON sidecar under .metadata/, so PUT responses, GET/HEAD and listings all agree. Objects created before this change keep the mtime-based fallback. Sync tools that compare ETag to MD5 (rclone, s3cmd) now behave.
  • Content-Type and x-amz-meta-* — round-trip through PUT, multipart upload and server-side copy, honouring x-amz-metadata-directive: REPLACE.
  • GetBucketLocation and subresource handling — ?location and ?versioning/?acl stubs; the AWS error codes clients expect for absent ?lifecycle/?cors/?policy/?tagging/?encryption/?object-lock config. Subresource PUT/DELETE are accepted as no-ops — previously PUT /bucket?acl hit CreateBucket, DELETE /bucket?lifecycle deleted the bucket, and PUT /key?acl overwrote the object with the ACL body.
  • ListBuckets no longer exposes the internal .multipart/.metadata directories as buckets.

Out of scope (per discussion): aws-chunked streaming uploads, POST form uploads, presigned-URL semantics.

Testing

  • New pkg/s3/compat_test.go: range boundaries (first/last byte, suffix, open-ended, clamped, unsatisfiable, malformed), conditional 304s, ETag agreement across PUT/GET/HEAD/List, metadata round-trips (plain, multipart, copy, REPLACE directive), v1 marker and v2 continuation-token pagination walks, delimiter rollup with pagination, subresource stubs and error codes, bucket-survives-DELETE ?lifecycle, object-survives-PUT ?acl
  • go test -race ./... green, golangci-lint clean
  • End-to-end: registry:2 push/pull round-trip (40MB multipart layer + small layers), zero 405/500/panic; live wire-format checks of ?location, V2 pagination tokens, ranged reads and ?versioning

Broadens S3 API compatibility beyond the operations the Docker
registry strictly needs:

- Range GET: single byte ranges including open-ended and suffix forms,
  with 206/Content-Range responses and 416 InvalidRange when
  unsatisfiable; malformed or multi-range headers fall back to a full
  200 per RFC 9110
- Conditional GET/HEAD: If-None-Match and If-Modified-Since return 304
- ListObjectsV2 (list-type=2) with KeyCount, continuation tokens and
  start-after; real pagination for v1 via marker/NextMarker. Listings
  are now in lexicographic key order with IsTruncated set correctly
  (previously hard-coded false, so buckets over max-keys silently
  looked complete)
- MD5 ETags: PutObject, CopyObject and multipart complete persist the
  content MD5 (multipart: md5-of-md5s-N) in a metadata sidecar under
  .metadata/, so PUT, GET, HEAD and listings all agree; pre-existing
  objects without a sidecar keep the mtime-based fallback
- Content-Type and x-amz-meta-* metadata round-trip through PUT,
  multipart and copy (honouring x-amz-metadata-directive)
- GetBucketLocation and bucket subresource handling: versioning/acl
  stubs, AWS error codes for absent lifecycle/cors/policy/tagging/
  encryption/object-lock config, and no-op PUT/DELETE so clients can
  no longer create or delete a bucket through a subresource URL, or
  corrupt object data via PUT ?acl/?tagging
- ListBuckets no longer exposes the internal .multipart and .metadata
  directories as buckets

Also fixes a listing bug where a non-matching file visited before the
prefix could SkipDir the rest of its directory, dropping matching keys.

Verified end-to-end against registry:2 (push/pull with a 40MB layer,
zero errors) plus wire-format spot checks of location, V2 pagination,
ranges and versioning against a live server.
@stut
stut merged commit 702f288 into main Jul 20, 2026
8 checks passed
@stut
stut deleted the feat/s3-compat-improvements branch July 20, 2026 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant