MVR-Data

███╗   ███╗██╗   ██╗██████╗       ██████╗  █████╗ ████████╗ █████╗
████╗ ████║██║   ██║██╔══██╗      ██╔══██╗██╔══██╗╚══██╔══╝██╔══██╗
██╔████╔██║██║   ██║██████╔╝█████╗██║  ██║███████║   ██║   ███████║
██║╚██╔╝██║╚██╗ ██╔╝██╔══██╗╚════╝██║  ██║██╔══██║   ██║   ██╔══██║
██║ ╚═╝ ██║ ╚████╔╝ ██║  ██║      ██████╔╝██║  ██║   ██║   ██║  ██║
╚═╝     ╚═╝  ╚═══╝  ╚═╝  ╚═╝      ╚═════╝ ╚═╝  ╚═╝   ╚═╝   ╚═╝  ╚═╝

Important

Rewrite in progress. The repository contains the header-only C++ core and its native Python bindings. The command-line API is not implemented yet.

MVR-Data defines how multi-vector retrieval data is organized at runtime and packaged for interchange using Apache Arrow. It covers Raw multimodal content, Embedded multi-vector representations, and long-form relevance judgments.

Implementation

The replacement implementation will have one shared native core:

• a header-only C++ API under include/mvr_data/;
• a compiled pybind11 Python extension whose public class and method names match the C++ API, without invoking a subprocess;
• Apache Arrow C++ for schemas, arrays, record batches, IPC, and memory sharing;
• nlohmann/json for the small convention-based Manifest;
• LibRHash for whole-package SHA-256 integrity;
• a CMake INTERFACE target for C++ consumers;
• Doxygen comments in the public C++ headers and generated C++ API reference;
• Python modules that remain thin adapters rather than a second implementation.

MVR-Data will remain a reference implementation rather than a distributed data engine. Exact object-ID and ground-truth indexes may be held in memory, and the documentation will state those memory assumptions explicitly.

Although MVR-Data itself is header-only for C++ consumers, Apache Arrow and LibRHash are compiled dependencies. The Python package therefore builds a native extension.

CMake fetches pinned sources for all third-party libraries from GitHub and builds them locally. Apache Arrow also uses its bundled dependency mode; the build does not link against system-installed third-party package libraries.

Python binding

Create the reproducible Conda environment and install the package from this checkout:

conda env create -f environment.yml
conda activate mvr-data
python -m pip install . --no-build-isolation

The native binding exposes every public C++ API with the same name: PackageKind, TableRole, VectorConfig, WriterOptions, Manifest, RawObjectSchema, EmbeddedObjectSchema, GroundTruthSchema, VectorDtypes, DataTable, BatchReader, BatchAppender, DatasetPackage, and Checksum. C++ arrow::Schema, arrow::DataType, arrow::RecordBatchReader, and arrow::Table values cross the boundary through Arrow's C Data and C Stream PyCapsule protocols and appear as their native PyArrow counterparts.

import pyarrow as pa

import mvr_data

schema = mvr_data.EmbeddedObjectSchema.make(128, pa.float32())
dataset = mvr_data.DatasetPackage.open("/path/to/package")

# Stream batches without materializing the complete table.
batches = dataset.base_table.make_batched_reader()
for batch in batches:
    print(batch)

# Or materialize a native pyarrow.Table.
base = dataset.base_table.read_table()
mvr_data.Checksum.verify("/path/to/package")

Build a new package from streamed RecordBatches or complete Arrow Tables. The final directory is published only when finish() has closed every active table appender and generated package.sha256. The user-authored Manifest contains semantic metadata only and is copied byte-for-byte; shard paths are discovered from the fixed package layout.

dataset = mvr_data.DatasetPackage.create(
    "/path/to/new-package",
    "/path/to/manifest.json",
)

base_appender = dataset.base_table.make_batched_writer()
base_appender.append_batch(base_batch_1)
base_appender.append_batch(base_batch_2)
base_appender.close()

# A complete Table can also become one shard directly.
dataset.query_table.write_table(query_data)
dataset.finish()

For a local development build and the focused binding test suite:

cmake -S . -B build -G Ninja \
  -DMVR_DATA_BUILD_PYTHON=ON \
  -DBUILD_TESTING=ON
cmake --build build --parallel
ctest --test-dir build --output-on-failure
PYTHONPATH="$PWD/build/python" python -m pytest -q tests/test_python_bindings.py

DataTable.read_table() materializes the complete logical table in memory; prefer make_batched_reader() for large datasets. Each make_batched_writer() call creates the next shard and accepts multiple RecordBatches, while write_table() serializes one complete Arrow Table as one shard. DatasetPackage.finish() closes active table appenders, validates the fixed layout, writes package.sha256, and atomically publishes the staging directory. Each shard receives a generated role/part-NNNNN.arrow path. A CLI remains future work.

package.sha256 contains exactly one lowercase SHA-256 digest. MVR-Data walks all regular non-symlink files below the package root, excludes package.sha256 itself, converts every relative path to UTF-8 with / separators, and sorts those paths bytewise. The version-1 digest is then calculated as:

SHA256(
  "MVR-DATA-PACKAGE-SHA256-V1"
  || uint64_be(path_1.size) || path_1 || SHA256(file_1)
  || uint64_be(path_2.size) || path_2 || SHA256(file_2)
  || ...
)

Paths participate in the aggregate hash but are not written to the checksum file. Consequently, adding, deleting, renaming, or changing a package file changes the package digest.

Stable format concepts

An object is one retrievable content item, such as an illustrated article, an audio recording, a video, or another multimodal item.

• Raw data stores base and query objects as one or more ordered content components whose payloads live inside the package.
• Embedded data stores base and query object IDs with one or more ordered vectors. Dimension and numeric dtype are fixed package-wide.
• Ground-truth data stores judged query-object pairs with their relevance, split, judgment source, and annotation pool.

MVR-Data defines three physical Arrow Schemas: Raw objects, Embedded objects, and Ground Truth. The fixed base, query, and ground_truth directories are table roles, not three additional Schema kinds. In a Raw or Embedded package, the base and query roles always share the corresponding object Schema.

If Raw and Embedded packages describe the same collection, the same base or query object must use the same object_id in both packages. Row positions do not need to match; the ID is the link between content and vectors.

Raw base and query tables

Field	Arrow type	Meaning
object_id	string	Unique object identifier within the table.
components	list<struct>	One or more ordered content components of an object.
components[].component_id	string	Component identifier, unique within the object.
components[].modality	string	Extensible modality such as text, image, audio, or video.
components[].media_type	string	MIME type such as text/plain or image/png.
components[].payload_uri	string	Package-local content-addressed payload URI.

Embedded base and query tables

Field	Arrow type	Meaning
object_id	string	Unique object identifier within the table.
vectors	large_list<fixed_size_list<T, dimension>>	One or more ordered vectors using the package-wide dtype and dimension.

Ground-truth table

Field	Arrow type	Meaning
query_id	string	ID of an object in the query table.
object_id	string	ID of an object in the base table.
relevance	int16	Non-negative relevance level.
split_type	string	Data split such as test.
judgment_source	string	Label source such as human or adjudicated.
pool_id	string	Candidate annotation-pool identifier.

Every package also contains manifest.json, which describes the format version and data collection and records package-wide vector settings for Embedded data. Arrow shards use role/part-NNNNN.arrow; readers discover and numerically order the contiguous files in each fixed role directory.

Repository state

Path	Purpose during the rewrite
examples/	Example Raw and Embedded Manifests.
tests/	Behavioral requirements retained from the previous implementation.
include/	Header-only C++ schemas, Manifest, DataTable, DatasetPackage, and checksum APIs.
python/	pybind11 extension source, Python package, type stubs, and py.typed marker.
environment.yml	Reproducible Python 3.12 and C++ compiler environment.

Format and public API documentation is hosted at the MVR-Data documentation site.

The C++ format test and native binding tests are active. Some retained Python tests describe CLI behavior that has not been reintroduced yet.