Repository navigation
Add virtual_prefix option to the file client - #28
Merged
Merged
Conversation
Mounts a file target at a key prefix that does not exist on disk: keys are reported with the prefix, and it is stripped from incoming keys. This is the inverse of the aioboto client's prefix option, which hides a prefix that does exist upstream. Fileglancer needs this to serve data links as a single S3 bucket, where the first segments of every key are a sharing key and a link name that have no counterpart in the filesystem. A prefix that stops inside the virtual prefix lists the next virtual segment as a common prefix, so those segments browse like real folders in clients such as Neuroglancer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mounts a file target at a key prefix that does not exist on disk: keys are reported with the prefix, and it is stripped from incoming keys. This is the inverse of the aioboto client's
prefixoption, which hides a prefix that does exist upstream. Default is empty, so nothing changes for existing targets./data/example/README.mdis then exposed as keyshared/my-data/README.md.Why
Fileglancer serves every data link out of a single S3 bucket, where the first segments of a key are a sharing key and a link name that have no counterpart in the filesystem. Without this the server would have to rewrite x2s3's XML after the fact, and it still couldn't answer correctly for a prefix that lands inside those virtual segments.
Behavior
virtual_prefixis walked on disk with the prefix stripped;Key,CommonPrefixes/PrefixandNextContinuationTokencome back with it prepended. Continuation tokens are de-prefixed on the way in, so paging round-trips.shared/,shared/my-, or empty) returns the next virtual segment as a common prefix, so those segments browse like real folders in clients such as Neuroglancer. Without a delimiter it lists the whole target instead, as S3 would.get_object/head_object: strip the prefix,NoSuchKeyif the key isn't under it.Tests
Six new tests in
tests/test_file.pyagainst amounted-filestarget: keys reported under the prefix, each prefix-stops-inside case, no-match, recursive listing without a delimiter, object GET/HEAD by virtual key plus 404 for the real path, and continuation-token paging. Full suite passes (79).@StephanPreibisch