Skip to content

feat(core): copy_with can replace user metadata and content type - #8431

Open
dmxlol wants to merge 1 commit into
apache:mainfrom
dmxlol:feat-copy-user-metadata
Open

dmxlol wants to merge 1 commit into
apache:mainfrom
dmxlol:feat-copy-user-metadata

Conversation

@dmxlol

@dmxlol dmxlol commented Oct 25, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Closes #8430.

Rationale for this change

A metadata-only change, such as setting a file's mtime after upload, currently costs a full
read and rewrite. OpenDAL's copy cannot carry new metadata, and it refuses from == to.
S3 can make this change with one zero-body CopyObject that sends
x-amz-metadata-directive: REPLACE. The issue has the details.

What changes are included in this PR?

  • CopyOptions gets user_metadata and content_type, named as in WriteOptions and
    ComposeOptions. copy_with and copier_with get matching builder methods, and
    OpCopy gets matching getters.
  • New capability flags copy_with_user_metadata and copy_with_content_type.
    CorrectnessCheckLayer returns Unsupported when a service does not advertise them.
  • Operator::copy* accepts from == to when either option is set.
  • services-s3:
    • CopyObject sends x-amz-metadata-directive: REPLACE, Content-Type and
      x-amz-meta-*.
    • A multipart copy sets the same headers on CreateMultipartUpload.
    • Both capability flags are advertised.
  • Tests:
    • an OpCopy options round trip
    • a correctness-check test: Unsupported without the capability, and same-path copy
      allowed only with metadata
    • two S3 request-shape tests on the existing mock transport: single and multipart copy
    • a behavior test, test_copy_with_user_metadata_in_place, gated on
      copy_with_user_metadata

Tested against:

  • AWS S3 (eu-central-1, a fresh unversioned bucket): the copy behavior tests pass 19 of 21,
    including the new one. The two failures, test_copy_with_source_version_*, fail the same
    way on main against the same bucket ("must have version") because the bucket has no
    versioning. A same-key copy_with with user metadata, content type and a source length
    hint sent one request and kept the ETag; the metadata changed and the content did not.
  • Silo (the fixtures/s3 image): full s3 behavior suite, 128 of 128.
  • Garage v2.1.0: 115 of 128; the same 13 fail on main (conditional writes, response
    overrides, special characters), none of them copy.

Are there any user-facing changes?

Yes. This PR adds two options, two builder methods and two capability flags; nothing
existing changes. The behavior is documented on CopyOptions::user_metadata: when either
option is set, the destination does not inherit source metadata, and unset fields such as
Cache-Control take the service default.

Breaking changes

AI Usage Statement

  • Harness: Claude Code, a coordinator session that delegated to a subagent.
  • Model: Claude Opus 5.5 (claude-opus-5-5).
  • Effort: default.
  • Role: Drafted the change, the tests and this description under human direction. A human
    reviewed the change and submits it.
  • Assumptions and unknowns:
    • Testing used AWS S3, Silo and Garage, as listed above.
    • Not checked: S3 Express One Zone, R2, Neon and the other providers in
      compatible_services.md. All of them are advertised with the new flags.

🤖 Generated with Claude Code

Adds user_metadata and content_type to CopyOptions with matching capability flags. S3 sends x-amz-metadata-directive: REPLACE, so a same-key copy updates metadata in one zero-body CopyObject instead of a read and rewrite.
@dmxlol
dmxlol requested a review from Xuanwo as a code owner October 25, 2026 22:44
@github-actions github-actions Bot added core releases-note/feat The PR implements a new feature or has a title that begins with "feat" services/s3 size:L This PR changes 100-499 lines, ignoring generated files. labels Oct 25, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core releases-note/feat The PR implements a new feature or has a title that begins with "feat" services/s3 size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

new feature: copy_with should be able to replace user metadata and content type (S3 x-amz-metadata-directive REPLACE)

1 participant