Writes fail against any custom-endpoint S3 (MinIO/Ceph/R2/moto): MemWAL writer drops storage_options

[beinan]

Summary

Writes are broken against any S3-compatible store that is not AWS proper — MinIO, Ceph RGW, Cloudflare R2, and moto. create() succeeds, then the first add() fails with Bucket '...' not found, even though the bucket demonstrably exists.

The root cause is upstream in lance 7.0.0: the MemWAL shard writer constructs its own object store from the URI alone and discards the dataset's configured storage_options, so aws_endpoint_url is dropped and the writer talks to real AWS.

This affects all three stores (ContextStore, RolloutStore, DatagenStore), since they all open shard writers the same way.

Reproduction

import subprocess, sys, socket, time, uuid, boto3
from botocore.config import Config
from lance_context import Context

# ... start `python -m moto.server` on `port`, then:
endpoint = f"http://127.0.0.1:{port}"
bucket = f"ctx{uuid.uuid4().hex[:8]}"
boto3.client("s3", endpoint_url=endpoint, ...).create_bucket(Bucket=bucket)

opts = {
    "aws_access_key_id": "test",
    "aws_secret_access_key": "test",
    "aws_region": "us-east-1",
    "aws_endpoint_url": endpoint,
    "aws_allow_http": "true",
}
ctx = Context.create(f"s3://{bucket}/ctx.lance", storage_options=opts)  # OK
ctx.add("user", "hello")                                                # FAILS

Observed:

create():  OK  (version=1)
add():     FAILED -> LanceError(IO): Generic S3 error: Bucket 'ctx2b9025f8' not found
boto3 confirms bucket exists: True

This is exactly what python/tests/test_persistence.py::test_s3_round_trip_with_storage_options and ::test_s3_deprecated_aws_kwargs_still_work do. Both are currently marked xfail(strict=True) pointing at this issue.

Diagnosis

Running moto with request logging is what makes it unambiguous: every request that reaches moto returns HTTP 200, and the failing request never arrives at all.

127.0.0.1 - - "PUT /ctxb8ce8c448 HTTP/1.1" 200 -
127.0.0.1 - - "GET /ctxb8ce8c448?list-type=2&prefix=ctx.lance/_versions/ HTTP/1.1" 200 -
127.0.0.1 - - "PUT /ctxb8ce8c448/ctx.lance/_transactions/0-....txn HTTP/1.1" 200 -
127.0.0.1 - - "PUT /ctxb8ce8c448/ctx.lance/_versions/1844....manifest HTTP/1.1" 200 -
127.0.0.1 - - "GET /ctxb8ce8c448?delimiter=/&list-type=2&prefix=ctx.lance/_mem_wal/ HTTP/1.1" 200 -

So the failure is not a rejected request — it is a request sent to a different endpoint. The writer is using a second, differently-configured object store.

Upstream, in lance/src/dataset/mem_wal/api.rs:607 (Dataset::mem_wal_writer):

// Get object store and base path
let base_uri = self.uri();
let (store, base_path) = ObjectStore::from_uri(base_uri).await?;   // <-- drops storage_options

ObjectStore::from_uri builds a store from the URI with ObjectStoreParams::default(), so every credential and endpoint override the caller configured is discarded. create() works because dataset creation goes through the options-aware path; add() is the first call that opens a shard writer.

Blast radius

All three stores open shard writers via Dataset::mem_wal_writer:

• crates/lance-context-core/src/store.rs:518
• crates/lance-context-core/src/rollout_store.rs:659
• crates/lance-context-core/src/datagen_store.rs:186

So this is not a test-only problem: anyone pointing lance-context at a non-AWS S3 endpoint gets a store they can create but cannot write to. Only AWS proper works today.

Reads are unaffected, which makes the failure mode more confusing: the dataset is created, listable, and appears healthy right up until the first write.

No local workaround

ShardWriterConfig has no field for storage options (grep -c storage_options lance-7.0.0/src/dataset/mem_wal/write.rs → 0), so this crate has no way to thread credentials through to the writer. The fix has to be upstream.

Suggested upstream fix

lance-io already exposes the options-aware constructor that from_uri itself delegates to:

// lance-io/src/object_store.rs:446
pub async fn from_uri_and_params(
    registry: Arc<ObjectStoreRegistry>,
    uri: &str,
    params: &ObjectStoreParams,
) -> Result<(Arc<Self>, Path)>

mem_wal_writer should pass the dataset's own object-store params (the same ones used to open the dataset) rather than defaults. Roughly a one-line change plus plumbing the params to the call site.

Current state in this repo

Both S3 tests are marked xfail(strict=True) — deliberately not skip. They keep running, and will fail loudly the moment a lance bump fixes this, which is the signal to remove the marker. See python/tests/test_persistence.py.

Note these two tests were passing silently in CI for ~100 PRs only because CI was collecting the wrong test directory (fixed in #195). Turning the real suite on is what surfaced this.

Environment

• lance 7.0.0 (lance-io 7.0.0)
• lance-context 0.6.4
• Reproduced on Linux x86_64 with moto 5.2.2; the same mechanism applies to MinIO / Ceph / R2 since it affects any aws_endpoint_url override.

Ask

Ideally this is filed upstream against lance and this issue tracks the version bump. Happy to open the upstream issue/PR if that is useful — the fix looks small and the reproduction is self-contained.