Blob v2: how to read payload bytes from C?

[MisterRaindrop]

I read Lance datasets from C through lance-c and am looking at blob columns.
Two questions, both about what is intended rather than what is broken.

What I see

lance-c v0.1.9 with lance at rev e934cc2c. Datasets written with pylance
11.0.0 at data_storage_version = "2.2"; the Blob v2 storage strategy is forced
through the field-metadata thresholds, so a small payload can stand in for a
large one.

A large_binary field tagged lance-encoding: blob (Blob v1): a projected scan
returns the payload bytes, and the stream schema keeps the metadata. Checked by
md5 on 1, 2 and 4 MiB values.

A Blob v2 field (Struct<data, uri> tagged
ARROW:extension:name = lance.blob.v2): a projected scan returns the
descriptor, for every strategy including inline. What arrives over the Arrow C
data interface is

format   = +s
metadata = (none)
children = [kind: C, position: L, size: L, blob_id: I, blob_uri: u]

kind follows the strategy (0 inline, 1 packed, 2 dedicated, matching the
threshold I set) and size is the true payload size.

1. Is a C entry point for blob payloads in scope?

lance.h has no blob functions, and the
liblance RFC does not
mention blobs in any of its four phases, so I cannot tell whether this is
deliberate or simply has not been asked for. The Rust crate lance-c depends on
has Dataset::take_blobs, read_blobs, read_blob_ranges and BlobFile at
the pinned rev.

From C I can read size and blob_uri out of the descriptor, which covers
inventory queries and, I assume, the external strategy. What I cannot do is
fetch the payload for the packed and dedicated strategies.

2. Is dropping ARROW:extension:name on the scan path intended?

lance_dataset_schema() carries ARROW:extension:name = lance.blob.v2 on the
field. The schema from lance_scanner_to_arrow_stream() does not: the
descriptor arrives as a bare five-field struct with no metadata at all. A
consumer that only sees the stream therefore cannot tell a blob descriptor from
an ordinary struct column, except by matching the child names.

---

If a C entry point for blob payloads is something you want, I am willing to
implement it. Tell me which shape you would accept — an opaque handle mirroring
BlobFile, or one planned batch read that returns Arrow for a row-id array —
and I will follow it, along with your error and lifetime conventions.

[jja725]

Thanks for the detailed report. I reproduced on main, which now pins lance v11.0.0 (ab6b5bbe, #77).

1. Reading blob payloads from C

Not deliberate. The RFC is from February and blob v2 landed in lance around June, so it was never scoped. Happy to take a PR.

With v11.0.0 on main, take_blobs, take_blobs_by_indices, read_blobs, read_blob_ranges, and BlobFile are all available to build on.

I'd follow the Java binding. It's the smallest surface that covers packed and dedicated reads, and it's what lance-spark uses. Two entry points, mirroring our existing lance_dataset_take / lance_dataset_take_rows split:

typedef struct LanceBlobFile LanceBlobFile;

/* Fills a caller-allocated array of n handles, in input order.
 * NULL entry for a null blob value. row_ids are _rowid values. */
int32_t lance_dataset_take_blobs(const LanceDataset*, const uint64_t* row_ids, size_t n,
                                 const char* column, LanceBlobFile** out);
int32_t lance_dataset_take_blobs_by_indices(const LanceDataset*, const uint64_t* indices, size_t n,
                                            const char* column, LanceBlobFile** out);

uint64_t lance_blob_file_size(const LanceBlobFile*);
int32_t  lance_blob_file_read(LanceBlobFile*, uint8_t* dst, size_t dst_len);        /* whole blob */
int32_t  lance_blob_file_read_up_to(LanceBlobFile*, uint8_t* dst, size_t len, size_t* bytes_read);
int32_t  lance_blob_file_read_range(const LanceBlobFile*, uint64_t offset, uint8_t* dst, size_t len);
int32_t  lance_blob_file_seek(LanceBlobFile*, uint64_t pos);
int32_t  lance_blob_file_tell(const LanceBlobFile*, uint64_t* pos);
void     lance_blob_file_close(LanceBlobFile*);

Reads copy into a caller buffer. Everything returns 0 or -1 with lance_last_error_* set, like the rest of the API. Validation lives in Rust with descriptive errors, ffi_try! at the boundary, declaration in lance.h, wrapper in lance.hpp, tests in tests/c_api_test.rs covering inline, packed, and dedicated values, null and empty blobs, and a multi-fragment dataset. At v11 take_blobs returns Option<BlobFile> per row, so a null blob value maps to a NULL handle as sketched above. Please cover that in the tests.

If you also want bytes inline in a scan, lance_scanner_set_blob_handling(scanner, LANCE_BLOB_HANDLING_ALL_BINARY) is a one-line pass-through to Scanner::blob_handling. Separate PR is fine.

2. Missing ARROW:extension:name on the scan path

That's upstream behavior. lance-c forwards the stream schema as is. In descriptor mode Lance swaps the field's type and metadata for its descriptor constant, and the descriptor struct isn't the same Arrow type as the lance.blob.v2 extension (storage type Struct<data, uri>), so tagging it with that name would be wrong. Lance detects descriptors by lance-encoding:blob = "true" plus the five child names. lance-spark does what you describe by hand: marker from lance_dataset_schema, shape from the scan.

On both revs I see lance-encoding:blob = "true" and lance-encoding:packed = "true" on the field, through the C stream and through lance_dataset_take. On the old pins I also tried every combination of projection, filter, with_row_id, and limit and couldn't get an empty metadata block. My dataset was written by the Rust crate rather than pylance 11, so if you still see none, can you dump the stream schema with pyarrow's RecordBatchReader._import_from_c and share the field metadata?

A dedicated extension name for the descriptor view would be a lance-format/lance change. We'd pass it through automatically.