Support secure RemoteArray, RemoteStore, and RemoteCTable references - #292
Open
FrancescAlted wants to merge 20 commits into
Open
FrancescAlted wants to merge 20 commits into
FrancescAlted wants to merge 20 commits into
Conversation
- Indirect remote-reference policy bypass through saved expressions. - HTTP 500 when moving SChunk .b2frame files. - Unnecessary whole-file reads during overwrite/delete. - RemoteStore missing/malformed member handling. - Caterva2’s embedded HDF5 frame reader, a test false positive, and stale documentation.
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved critical and moderate findings remain.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (3)
What changed in this PR
Adds secure RemoteArray, RemoteStore, and RemoteCTable support with remote policy enforcement, sparse caching, quota accounting, refresh APIs, metadata, client integration, documentation, tests, and benchmarks.
Changes:
- Adds secure remote-source validation and cache/quota lifecycle management.
- Adds RemoteStore and RemoteCTable serving, browsing, filtering, projections, and refresh.
- Extends REST, Python client, metadata, documentation, tests, and benchmarks.
Review findings:
- Critical (1 vote):
StorageBusygeneration conflicts can escape as 500 responses instead of documented 409 responses. - Moderate (1 vote):
cache_limit=0may still retain a disk generation. - Moderate (1 vote): The advertised POST fetch path currently receives 405.
- Moderate (2 votes): The storage-quota benchmark overstates bytes written.
- Moderate (1 vote): The remote proxy benchmark uses obsolete carrier markers.
- Moderate (1 vote): The contiguous benchmark does not select the contiguous backend.
- Nit (2 votes): Server documentation reports schema v2 while runtime uses v3.
- Nit (1 vote): The RemoteStore plan contains the same schema-version mismatch.
| File | Summary |
|---|---|
pyproject.toml |
Updates Blosc2 and server dependencies. |
plans/remote-store.md |
Documents RemoteStore design and acceptance criteria. |
plans/remote-ctable.md |
Documents RemoteCTable implementation and follow-up work. |
examples/benchmarks/remote_proxy_v7.py |
Provides remote-array route benchmarks. |
examples/benchmarks/remote_proxy_v7.md |
Documents benchmark methodology and results. |
examples/benchmarks/remote_proxy_v7_results/small-sparse.json |
Stores small sparse benchmark measurements. |
examples/benchmarks/remote_proxy_v7_results/small-contiguous.json |
Stores small contiguous benchmark measurements. |
examples/benchmarks/remote_proxy_v7_results/large-sparse.json |
Stores large sparse benchmark measurements. |
examples/benchmarks/remote_proxy_v7_results/large-contiguous.json |
Stores large contiguous benchmark measurements. |
examples/benchmark_storage_quota.py |
Benchmarks quota staging overhead. |
examples/benchmark_remote_store.py |
Benchmarks shared RemoteStore access. |
doc/utilities/cat2-server.md |
Documents server policy, quota, cache, and lifecycle behavior. |
doc/utilities/cat2-client.md |
Documents client attribute output. |
doc/reference/rest_api.rst |
Documents metadata attributes. |
caterva2/tests/test_treestore.py |
Tests TreeStore attributes. |
caterva2/tests/test_storage_quota.py |
Tests quota persistence and recovery. |
caterva2/tests/test_storage_quota_api.py |
Tests quota-controlled API behavior. |
caterva2/tests/test_sparse_cache.py |
Tests sparse-cache lifecycle and recovery. |
caterva2/tests/test_remote_proxy.py |
Tests remote-array policy and caching. |
caterva2/tests/test_providers.py |
Tests provider coupling and isolation. |
caterva2/tests/test_peers.py |
Tests peer metadata and table ranges. |
caterva2/tests/test_hdf5_tree.py |
Tests HDF5 attributes and metadata. |
caterva2/tests/test_ctable.py |
Tests table projections and refresh behavior. |
caterva2/tests/test_cli.py |
Tests CLI attribute rendering. |
caterva2/tests/test_chunk_writes.py |
Tests asynchronous publication verification. |
caterva2/tests/test_attrs.py |
Tests public attribute handling. |
caterva2/tests/test_api.py |
Tests remote API behavior and metadata. |
caterva2/services/templates/includes/info_metadata.html |
Renders public attributes. |
caterva2/services/storage_quota.py |
Implements coordinated storage admission and recovery. |
caterva2/services/srv_utils.py |
Provides metadata, dispatch, and table helpers. |
caterva2/services/remote_store.py |
Adds RemoteStore and RemoteCTable adapters. |
caterva2/services/remote_proxy.py |
Implements secure RemoteArray resolution and caching. |
caterva2/models.py |
Adds public attribute metadata fields. |
caterva2/hdf5.py |
Preserves HDF5 attributes and quota-aware writes. |
caterva2/clients/cli.py |
Displays dataset attributes. |
caterva2/client.py |
Adds attributes, projections, downloads, and refresh. |
caterva2/api_utils.py |
Supports cache-aware and cold download URLs. |
caterva2-server.sample.toml |
Documents remote policy and quota configuration. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+195
to
+198
| except ValueError as exc: | ||
| raise fastapi.HTTPException(status_code=400, detail=str(exc)) from exc | ||
| except (OSError, zipfile.BadZipFile) as exc: | ||
| raise fastapi.HTTPException(status_code=502, detail=str(exc)) from exc |
| samples.append(time.perf_counter() - tick) | ||
| np.testing.assert_array_equal(result, data[selection]) | ||
| if staged: | ||
| staged_bytes += path.stat().st_size |
Comment on lines
+228
to
+230
| Schema v2 is the initial Caterva2 runtime schema. The previous contiguous-carrier | ||
| implementation is retained only in the benchmark report and is not a supported | ||
| runtime backend. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch lets Caterva2 serve persisted python-blosc2 remote references while
keeping source access, cache growth, and storage accounting under server policy.
It updates the RemoteArray integration, adds RemoteStore and RemoteCTable
containers, and exposes them through the existing metadata, fetch, download,
browser, and Python client APIs.
Changes
public DNS addresses, and resource limits before outbound reads. Resolution is
disabled by default. Source authorization also covers linked stores, table
columns and batches, and cached-only operations.
discovery and an aggregate cache allowance across leaves, batches, indexes,
and linked references. Coordinate multi-process access, quota admission,
pruning, recovery, and portable warm or cold downloads with a SQLite ledger.
metadata, browser paging and sorting, row slices, filters, and one-column
projections. Local and remote table projections both return CTable cframes;
the Python client decodes nested table paths from metadata and preserves row
selections when a field is requested.
RemoteCTable carriers, plus Client.refresh(path). Refresh rediscovers the
source, publishes a cold reference atomically, and returns a fresh client
object. Missing targets, unsupported RemoteArray targets, denied calls,
source failures, and competing replacements receive explicit responses.
policy and API, and add benchmark scripts and regression coverage.
Compatibility and deployment
and integration commits 5f92a80d and d6554ad5. Version 4.14.0 has not been
released yet.
public, shared, and personal roots; a separate work budget covers staging.
Remote reference resolution remains opt-in through [server.remote_proxy].
Client.refresh() returns a newly loaded object for callers to use.