Repository navigation
Fix key encoding and virtual-prefix pagination in file listings - #30
Merged
Merged
Conversation
EncodingType=url means percent-encoding, but url_encode() replaced spaces with '+', so a file named "my file.txt" was listed as key "my+file.txt". Clients undo the encoding by unquoting (boto3) or with decodeURIComponent (Neuroglancer), and neither turns '+' back into a space, so a key taken from a listing named a file that does not exist and fetching it 404'd. Any name containing a space was affected: browsing a Fileglancer data link in Neuroglancer listed the entries but could not open them. quote(s, safe='/') also encodes a literal '+' as %2B, which tells the two apart. Added a listing test over a tree with spaced names that reads a reported key back through the app. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A listing whose prefix stopped inside the virtual prefix emitted its one synthetic CommonPrefixes entry unconditionally, so max-keys=0 answered with KeyCount=1, and a client that passed a continuation token was handed the same entry again. The filesystem branch of the same method honours both, so the two disagreed. That entry is the entire listing and is never truncated, so there is nothing to page: max-keys=0 now returns an empty listing, and a token can only be stale, which walk_path already treats as nothing to report. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
encoding-type=urlmeans percent-encoding, buturl_encode()replaced spaces with+, so a file namedmy file.txtwas listed as keymy+file.txt. Clients undo the encoding by unquoting (boto3) or withdecodeURIComponent(Neuroglancer), and neither turns+back into a space — so a key read out of a listing pointed at a file that doesn't exist, and fetching it 404'd.Any file or directory whose name contains a space was affected, which is why it turned up immediately on real data: browsing a Fileglancer data link in Neuroglancer showed the entries but couldn't open them.
quote(s, safe='/')now handles it, which also encodes a literal+as%2Band so tells the two apart./is still left alone so keys keep their structure.New test builds a target over a tree with spaced names, asserts the listing reports
my%20file.txtandsub%20dir/, and then reads the key back through the app to prove it resolves. Full suite passes (81).Also: pagination inside a virtual prefix
A listing whose prefix stopped inside the
virtual_prefixemitted its one syntheticCommonPrefixesentry unconditionally, somax-keys=0answered withKeyCount=1, and a client that passed a continuation token got the same entry again. The filesystem branch of the same method honours both, so the two disagreed:That entry is the whole listing and is never truncated, so there is nothing to page through:
max-keys=0now returns an empty listing, and a continuation token can only be stale, whichwalk_pathalready treats as nothing to report. Covered by a test over both cases.@StephanPreibisch