Ready-to-use evaluators for 3D vision-language benchmarks, built on top of TorchMetrics.
Each metric downloads its ground truth automatically. You pass in your predictions keyed by sample ID and get the
benchmark's official numbers back. Because every evaluator is a TorchMetrics Metric, you also get:
- a standardized interface that makes results reproducible across projects
- automatic accumulation over batches (
update()many times,compute()once) - automatic synchronization across devices in distributed training
pip install torchmetrics-ext| Task | Benchmark | Metric class | Reported metrics | Ground truth |
|---|---|---|---|---|
| 3D Visual Grounding | ScanRefer | ScanReferMetric |
Acc@0.25IoU, Acc@0.5IoU | manual download* |
| Nr3D | Nr3DMetric |
Accuracy | automatic | |
| Multi3DRefer | Multi3DReferMetric |
F1@0.25IoU, F1@0.5IoU | automatic | |
| ViGiL3D | ViGiL3DMetric |
F1@0.25IoU, F1@0.5IoU | automatic | |
| 3D Visual Question Answering | ScanQA | ScanQAMetric |
BLEU-1, BLEU-4, METEOR, ROUGE-L, CIDEr | automatic |
| VSI-Bench | VSIBenchMetric |
Accuracy / MRA | automatic | |
| ReVSI | ReVSIMetric |
Accuracy / MRA | automatic | |
| 3D Object Detection | ScanNet | MeanAveragePrecisionMetric |
mAP@0.25IoU, mAP@0.5IoU | — (under development) |
* The ScanRefer license requires you to request the dataset yourself.
Every benchmark metric works the same way:
from torchmetrics_ext.metrics.visual_grounding import Multi3DReferMetric
metric = Multi3DReferMetric(split="validation") # downloads and caches the ground truth
ids = metric.get_all_data_ids() # all sample IDs of this split
# evaluate everything at once ...
results = metric(preds) # preds: {sample_id: prediction}
# ... or accumulate over batches (works across DDP processes)
for batch_preds in loader:
metric.update(batch_preds)
results = metric.compute()
metric.reset()All results are averaged over the predictions you submit, not over the whole split. To get official numbers, submit a
prediction for every ID in get_all_data_ids(). Grounding metrics are returned in [0, 1], VQA metrics in %.
Passing an ID that is not in the ground truth raises a KeyError.
Bounding boxes are axis-aligned and given as [[x_min, y_min, z_min], [x_max, y_max, z_max]]. ScanNet ground truth
boxes are axis-aligned with the transformation matrix from each scene's <scene_id>.txt. Your predictions must use
the same coordinate frame.
Predict one box per description, with shape (2, 3).
import torch
from torchmetrics_ext.metrics.visual_grounding import ScanReferMetric
metric = ScanReferMetric(dataset_file_path="./ScanRefer_filtered_val.json", split="validation")
# key: "{scene_id}_{object_id}_{ann_id}"
preds = {
"scene0011_00_0_0": torch.tensor([[0., 0., 0.], [0.5, 0.5, 0.5]]),
"scene0011_00_0_1": torch.tensor([[0., 0., 0.], [1., 1., 1.]]),
...
}
results = metric(preds)
# {"unique_0.25", "multiple_0.25", "all_0.25", "unique_0.5", "multiple_0.5", "all_0.5"}Predict the object ID of the target among the scene's candidate objects.
from torchmetrics_ext.metrics.visual_grounding import Nr3DMetric
metric = Nr3DMetric(split="test")
# key: stimulus_id, value: predicted object ID
preds = {
"scene0565_00-chair-4-25-0-1-24": 25,
"scene0653_00-desk-6-16-13-14-15-17-18": 16,
...
}
results = metric(preds)
# {"easy", "hard", "view_dep", "view_indep", "all"}Predict any number of boxes per description, with shape (N, 2, 3). Use an empty tensor to predict that nothing
matches. Predictions are matched to the ground truth with the Hungarian algorithm.
import torch
from torchmetrics_ext.metrics.visual_grounding import Multi3DReferMetric
metric = Multi3DReferMetric(split="validation")
# key: "{scene_id}_{ann_id}"
preds = {
"scene0011_00_0": torch.tensor([[[0., 0., 0.], [0.5, 0.5, 0.5]]]), # 1 box
"scene0011_00_1": torch.tensor([[[0., 0., 0.], [1., 1., 1.]], [[0., 0., 0.], [2., 2., 2.]]]), # 2 boxes
"scene0011_00_2": torch.tensor([]), # no box
...
}
results = metric(preds)
# zero-target: "zt_wo_d", "zt_w_d"
# single-target: "st_wo_d_{t}", "st_w_d_{t}"; multi-target: "mt_{t}"; overall: "all_{t}", for t in (0.25, 0.5)Set strict=True to require that each update() call covers exactly the full split.
Same input format and matching as Multi3DRefer. Scenes come from both ScanNet and ScanNet++.
import torch
from torchmetrics_ext.metrics.visual_grounding import ViGiL3DMetric
metric = ViGiL3DMetric(split="validation")
# key: "{scene_id}_{ann_id}"
preds = {
"scene0012_00_cf49717d-a751-417e-be93-32fa6a4aa1e4": torch.tensor([[[1.41, 1.13, 0.02], [1.54, 2.31, 1.60]]]),
...
}
results = metric(preds)
# {"zt", "st_0.25", "st_0.5", "mt_0.25", "mt_0.5", "all_0.25", "all_0.5"}Predictions are answer strings keyed by question ID.
Multiple-choice questions are scored by exact match. Numerical questions are scored by Mean Relative Accuracy (MRA),
following the official implementation. Only the first word of
each prediction is used, so "3 chairs." is read as "3".
from torchmetrics_ext.metrics.vqa import VSIBenchMetric, ReVSIMetric
metric = VSIBenchMetric(split="test")
# or: metric = ReVSIMetric(split="test", subset="all_frame")
# key: question "id", value: predicted answer
preds = {
0: "3",
1: "A",
...
}
results = metric(preds)
# {"<question_type>_acc", ..., "overall_acc"}Each answer is scored against all reference answers with the standard captioning metrics from pycocoevalcap.
METEOR and the PTB tokenizer need Java installed. Predictions are stored as strings, so this metric does not synchronize across devices. Gather predictions on one process before calling
compute().
from torchmetrics_ext.metrics.vqa import ScanQAMetric
metric = ScanQAMetric(split="validation")
# key: "question_id"
preds = {
"val-scene0441-17": "white rectangular",
...
}
results = metric(preds)
# {"BLEU_1", "BLEU_4", "METEOR", "ROUGE_L", "CIDEr"}pip install -e ".[dev]"
pytest testThe tests download the benchmark ground truth on their first run. Downloads are cached under the Hugging Face
datasets cache directory.