Skip to content

Fix key encoding and virtual-prefix pagination in file listings - #30

Merged
krokicki merged 2 commits into
mainfrom
fix-space-encoding
Aug 28, 2026
Merged

krokicki merged 2 commits into
mainfrom
fix-space-encoding

Conversation

@krokicki

@krokicki krokicki commented Aug 28, 2026 •

Copy link
Copy Markdown
Member

encoding-type=url means percent-encoding, but url_encode() replaced spaces with +, so a file named my file.txt was listed as key my+file.txt. Clients undo the encoding by unquoting (boto3) or with decodeURIComponent (Neuroglancer), and neither turns + back into a space — so a key read out of a listing pointed at a file that doesn't exist, and fetching it 404'd.

Any file or directory whose name contains a space was affected, which is why it turned up immediately on real data: browsing a Fileglancer data link in Neuroglancer showed the entries but couldn't open them.

quote(s, safe='/') now handles it, which also encodes a literal + as %2B and so tells the two apart. / is still left alone so keys keep their structure.

New test builds a target over a tree with spaced names, asserts the listing reports my%20file.txt and sub%20dir/, and then reads the key back through the app to prove it resolves. Full suite passes (81).

Also: pagination inside a virtual prefix

A listing whose prefix stopped inside the virtual_prefix emitted its one synthetic CommonPrefixes entry unconditionally, so max-keys=0 answered with KeyCount=1, and a client that passed a continuation token got the same entry again. The filesystem branch of the same method honours both, so the two disagreed:

prefix=abc&delimiter=/&max-keys=0     -> KeyCount=1 common=['abc123/']   # wrong
prefix=tests/&delimiter=/&max-keys=0  -> KeyCount=0 common=[]            # right

That entry is the whole listing and is never truncated, so there is nothing to page through: max-keys=0 now returns an empty listing, and a continuation token can only be stale, which walk_path already treats as nothing to report. Covered by a test over both cases.

@StephanPreibisch

EncodingType=url means percent-encoding, but url_encode() replaced spaces
with '+', so a file named "my file.txt" was listed as key "my+file.txt".
Clients undo the encoding by unquoting (boto3) or with decodeURIComponent
(Neuroglancer), and neither turns '+' back into a space, so a key taken
from a listing named a file that does not exist and fetching it 404'd.

Any name containing a space was affected: browsing a Fileglancer data
link in Neuroglancer listed the entries but could not open them.

quote(s, safe='/') also encodes a literal '+' as %2B, which tells the two
apart. Added a listing test over a tree with spaced names that reads a
reported key back through the app.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A listing whose prefix stopped inside the virtual prefix emitted its one
synthetic CommonPrefixes entry unconditionally, so max-keys=0 answered
with KeyCount=1, and a client that passed a continuation token was handed
the same entry again. The filesystem branch of the same method honours
both, so the two disagreed.

That entry is the entire listing and is never truncated, so there is
nothing to page: max-keys=0 now returns an empty listing, and a token can
only be stale, which walk_path already treats as nothing to report.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@krokicki krokicki changed the title Encode spaces as %20 in listings, not '+' Fix key encoding and virtual-prefix pagination in file listings Aug 28, 2026
@krokicki
krokicki merged commit 2786493 into main Aug 28, 2026
5 checks passed
@krokicki
krokicki deleted the fix-space-encoding branch August 28, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant