Skip to content

Add virtual_prefix option to the file client - #28

Merged
krokicki merged 2 commits into
mainfrom
virtual-prefixes
Aug 28, 2026
Merged

krokicki merged 2 commits into
mainfrom
virtual-prefixes

Conversation

@krokicki

@krokicki krokicki commented Aug 28, 2026 •

Copy link
Copy Markdown
Member

Mounts a file target at a key prefix that does not exist on disk: keys are reported with the prefix, and it is stripped from incoming keys. This is the inverse of the aioboto client's prefix option, which hides a prefix that does exist upstream. Default is empty, so nothing changes for existing targets.

- name: mounted-files
  client: file
  options:
    path: /data/example
    virtual_prefix: shared/my-data

/data/example/README.md is then exposed as key shared/my-data/README.md.

Why

Fileglancer serves every data link out of a single S3 bucket, where the first segments of a key are a sharing key and a link name that have no counterpart in the filesystem. Without this the server would have to rewrite x2s3's XML after the fact, and it still couldn't answer correctly for a prefix that lands inside those virtual segments.

Behavior

  • Listing: a prefix under virtual_prefix is walked on disk with the prefix stripped; Key, CommonPrefixes/Prefix and NextContinuationToken come back with it prepended. Continuation tokens are de-prefixed on the way in, so paging round-trips.
  • Prefix stopping inside the virtual prefix (shared/, shared/my-, or empty) returns the next virtual segment as a common prefix, so those segments browse like real folders in clients such as Neuroglancer. Without a delimiter it lists the whole target instead, as S3 would.
  • Diverging prefix returns an empty listing.
  • get_object / head_object: strip the prefix, NoSuchKey if the key isn't under it.

Tests

Six new tests in tests/test_file.py against a mounted-files target: keys reported under the prefix, each prefix-stops-inside case, no-match, recursive listing without a delimiter, object GET/HEAD by virtual key plus 404 for the real path, and continuation-token paging. Full suite passes (79).

@StephanPreibisch

krokicki and others added 2 commits August 27, 2026 21:57
Mounts a file target at a key prefix that does not exist on disk: keys
are reported with the prefix, and it is stripped from incoming keys.
This is the inverse of the aioboto client's prefix option, which hides a
prefix that does exist upstream.

Fileglancer needs this to serve data links as a single S3 bucket, where
the first segments of every key are a sharing key and a link name that
have no counterpart in the filesystem. A prefix that stops inside the
virtual prefix lists the next virtual segment as a common prefix, so
those segments browse like real folders in clients such as Neuroglancer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@krokicki
krokicki merged commit 5eea793 into main Aug 28, 2026
5 checks passed
@krokicki
krokicki deleted the virtual-prefixes branch August 28, 2026 02:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant