Repository navigation
Conversation
ListObjectsV2 entries carried no ETag, although s3s-fs stores one per object at PUT, so a client that compares listed ETags (conditional writes, consistency checks) could not use the listing. Each listed object now carries the ETag from its stored internal info. Under encoding-type=url the listing echoed the encoding type but returned raw keys. Clients decode those keys and read `+` as a space, so the key `a+b` came back as `a b`. Listed keys and common prefixes are now percent-encoded when the request asks for url encoding. Two cases in tests/aws/list.rs pin both: the listed ETag equals the PUT response ETag, and under encoding-type=url `a b`, `a+b` and a prefix with a space list as `a%20b`, `a%2Bb` and `e%20f/`, while a listing without it returns the keys as stored. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Two listing defects have no test yet. An object imported under a long nested key, `segment/` 25 times then `object`, has short path components, but its sidecar name embeds the whole encoded key in one component and exceeds the 255-byte limit. The listing now reads that sidecar for the ETag, so the read fails with a name-too-long error and the whole listing fails. The new case imports such an object beside a stored one and expects both v1 and v2 listings to return it without an ETag, while the stored object keeps its ETag. Under encoding-type=url, S3 encodes the echoed Prefix, Delimiter and StartAfter in ListObjectsV2, and Prefix, Delimiter, Marker and NextMarker in ListObjects. s3s-fs returns them raw, and ListObjects does not echo Marker at all. The new cases list a literal `a%2Fb/` prefix with a `+` delimiter and expect the echoes encoded while filtering still uses the raw values. The v2 case also checks that the continuation token is returned and echoed unchanged and resumes the listing. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
A sidecar name holds the whole encoded bucket and key in one path component. An object imported under a long nested key has short path components of its own, but its sidecar name can exceed the 255-byte component limit. Reading that sidecar for the listed ETag failed with a name-too-long error, and the error failed the whole listing. No sidecar can exist under a name the filesystem cannot represent, so the listing now reads such a name as absent metadata and lists the object without an ETag, as it does when the sidecar is missing. The change is limited to the listing. HEAD, GET and writes keep their current behaviour. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Under encoding-type=url, S3 returns the echoed Prefix, Delimiter and StartAfter of ListObjectsV2 url-encoded, and the Prefix, Delimiter, Marker and NextMarker of ListObjects. s3s-fs encoded the keys and common prefixes but echoed these fields raw. A client decodes them, so a literal prefix `a%2Fb/` read back as `a/b/` and a `+` read back as a space. ListObjectsV2 now encodes the echoed prefix, delimiter and start-after after the walk has filtered with the raw values. The continuation tokens are opaque and stay as sent and as returned. ListObjects now echoes the marker, which it dropped before, and encodes NextMarker, which is a key in V1 and not an opaque token. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Member
Author
|
Validation done; pushed to s3s-project#1013. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Validation only for s3s-project#1013 review findings.