Describe the bug
Python test test_mg_ivf_flat.py::test_mg_ivf_flat_extend' reliably fails on multi-GPU systems with Blackwell GPUs (e.g. B200, GB200).
The failure mode (low recall) is always the same, but the exact values differ a lot across runs. Like this:
recall = calc_recall(neighbors, skl_idx)
# Multi-GPU implementation may have lower recall due to data distribution
# across GPUs
# This is acceptable as long as the functionality works correctly
> assert recall > 0.3, (
f"Recall too low: {recall:.3f} (n_lists={n_lists}, "
f"n_probes={search_params.get('n_probes', 'default')})"
)
E AssertionError: Recall too low: 0.120 (n_lists=30, n_probes=22)
E assert 0.12 > 0.3
tests/test_mg_ivf_flat.py:147: AssertionError
======================================================================= short test summary info =======================================================================
FAILED tests/test_mg_ivf_flat.py::test_mg_ivf_flat_extend[1-10] - AssertionError: Recall too low: 0.160 (n_lists=30, n_probes=22)
FAILED tests/test_mg_ivf_flat.py::test_mg_ivf_flat_extend[2-10] - AssertionError: Recall too low: 0.213 (n_lists=30, n_probes=22)
FAILED tests/test_mg_ivf_flat.py::test_mg_ivf_flat_extend[3-10] - AssertionError: Recall too low: 0.147 (n_lists=30, n_probes=22)
FAILED tests/test_mg_ivf_flat.py::test_mg_ivf_flat_extend[4-10] - AssertionError: Recall too low: 0.160 (n_lists=30, n_probes=22)
FAILED tests/test_mg_ivf_flat.py::test_mg_ivf_flat_extend[5-10] - AssertionError: Recall too low: 0.267 (n_lists=30, n_probes=22)
FAILED tests/test_mg_ivf_flat.py::test_mg_ivf_flat_extend[6-10] - AssertionError: Recall too low: 0.093 (n_lists=30, n_probes=22)
FAILED tests/test_mg_ivf_flat.py::test_mg_ivf_flat_extend[7-10] - AssertionError: Recall too low: 0.147 (n_lists=30, n_probes=22)
FAILED tests/test_mg_ivf_flat.py::test_mg_ivf_flat_extend[8-10] - AssertionError: Recall too low: 0.160 (n_lists=30, n_probes=22)
FAILED tests/test_mg_ivf_flat.py::test_mg_ivf_flat_extend[9-10] - AssertionError: Recall too low: 0.053 (n_lists=30, n_probes=22)
FAILED tests/test_mg_ivf_flat.py::test_mg_ivf_flat_extend[10-10] - AssertionError: Recall too low: 0.120 (n_lists=30, n_probes=22)
========================================================================= 10 failed in 17.36s =========================================================================
Steps/Code to reproduce bug
I was able to reproduce this today on latest release/26.06 (5b6fae9), on a B200x8 machine.
conda create \
--name cuvs-dev \
--yes \
--file ./conda/environments/all_cuda-132_arch-$(arch).yaml
conda install \
--yes \
--name cuvs-dev \
-c rapidsai-nightly \
-c conda-forge \
'cuvs=26.6.*' \
'pytest-repeat>=0.9.4'
source activate cuvs-dev
pushd ./python/cuvs/cuvs
pytest -ra --count=10 'tests/test_mg_ivf_flat.py::test_mg_ivf_flat_extend'Expected behavior
Expected this test to pass reliably, as I've observed on other multi-GPU setups like H100x2.
Environment details (please complete the following information):
output of 'nvidia-smi' (click me)
$ nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA B200 On | 00000000:1B:00.0 Off | 0 |
| N/A 34C P0 140W / 1000W | 0MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA B200 On | 00000000:43:00.0 Off | 0 |
| N/A 33C P0 141W / 1000W | 0MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA B200 On | 00000000:52:00.0 Off | 0 |
| N/A 36C P0 140W / 1000W | 0MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA B200 On | 00000000:61:00.0 Off | 0 |
| N/A 34C P0 143W / 1000W | 0MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 4 NVIDIA B200 On | 00000000:9D:00.0 Off | 0 |
| N/A 30C P0 141W / 1000W | 80MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 5 NVIDIA B200 On | 00000000:C3:00.0 Off | 0 |
| N/A 31C P0 140W / 1000W | 80MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 6 NVIDIA B200 On | 00000000:D1:00.0 Off | 0 |
| N/A 34C P0 142W / 1000W | 80MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 7 NVIDIA B200 On | 00000000:DF:00.0 Off | 0 |
| N/A 33C P0 141W / 1000W | 80MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
output of 'conda env export' (click me)
$ conda-env export
name: cuvs-dev
channels:
- conda-forge
- rapidsai-nightly
dependencies:
- _go_select=2.3.0=cgo
- _openmp_mutex=4.5=20_gnu
- _python_abi3_support=1.0=hd8ed1ab_2
- adwaita-icon-theme=49.0=unix_0
- alabaster=1.0.0=pyhd8ed1ab_1
- asttokens=3.0.1=pyhd8ed1ab_0
- at-spi2-atk=2.38.0=h0630a04_3
- at-spi2-core=2.40.3=h0630a04_0
- atk-1.0=2.38.0=h04ea711_2
- attrs=26.1.0=pyhcf101f3_0
- aws-c-auth=0.10.1=ha62d5e7_3
- aws-c-cal=0.9.13=h2c9d079_1
- aws-c-common=0.12.6=hb03c661_0
- aws-c-compression=0.3.2=h8b1a151_0
- aws-c-event-stream=0.7.0=h9b893ba_0
- aws-c-http=0.10.13=h4bacb7b_0
- aws-c-io=0.26.3=h692f434_1
- aws-c-mqtt=0.15.2=hc1936db_2
- aws-c-s3=0.12.2=he6ee468_1
- aws-c-sdkutils=0.2.4=h8b1a151_4
- aws-checksums=0.2.10=h8b1a151_0
- aws-crt-cpp=0.38.3=h745e52d_1
- aws-sdk-cpp=1.11.747=h41c0014_4
- azure-core-cpp=1.16.2=h206d751_0
- azure-identity-cpp=1.13.3=hed0cdb0_1
- azure-storage-blobs-cpp=12.16.0=hf824e48_2
- azure-storage-common-cpp=12.13.0=ha7a2c86_0
- azure-storage-files-datalake-cpp=12.14.0=h539c000_2
- babel=2.18.0=pyhcf101f3_1
- backports.zstd=1.5.0=py314h680f03e_0
- binutils=2.45.1=default_h4852527_102
- binutils_impl_linux-64=2.45.1=default_hfdba357_102
- binutils_linux-64=2.45.1=default_h4852527_102
- breathe=4.36.0=pyhd8ed1ab_0
- brotli-python=1.2.0=py314h3de4e8d_1
- bzip2=1.0.8=hda65f42_9
- c-ares=1.34.6=hb03c661_0
- c-compiler=1.11.0=h4d9bdce_0
- ca-certificates=2026.4.22=hbd8a1cb_0
- cachetools=7.1.1=pyhd8ed1ab_0
- cairo=1.18.4=he90730b_1
- certifi=2026.4.22=pyhd8ed1ab_0
- cffi=2.0.0=py314h4a8dc5f_1
- cfgv=3.5.0=pyhd8ed1ab_0
- charset-normalizer=3.4.7=pyhd8ed1ab_0
- clang=20.1.8=default_cfg_hcbb2b3e_15
- clang-20=20.1.8=default_h99862b1_15
- clang-format=20.1.8=default_h99862b1_15
- clang-format-20=20.1.8=default_h99862b1_15
- clang-tools=20.1.8=default_h57a47db_15
- clang_impl_linux-64=20.1.8=default_cfg_h027053c_15
- cmake=4.3.2=hc85cc9f_0
- colorama=0.4.6=pyhd8ed1ab_1
- commonmark=0.9.1=py_0
- compiler-rt=20.1.8=hb700be7_1
- compiler-rt_linux-64=20.1.8=hffcefe0_1
- coverage=7.14.0=py314h67df5f8_0
- cpython=3.14.4=py314hd8ed1ab_100
- cuda-bindings=13.2.0=py314ha0b5721_0
- cuda-cccl_linux-64=13.2.75=ha770c72_0
- cuda-core=0.7.0=cuda13_py314h025f531_0
- cuda-crt-dev_linux-64=13.2.78=ha770c72_0
- cuda-crt-tools=13.2.78=ha770c72_0
- cuda-cudart=13.2.75=hecca717_0
- cuda-cudart-dev=13.2.75=hecca717_0
- cuda-cudart-dev_linux-64=13.2.75=h376f20c_0
- cuda-cudart-static=13.2.75=hecca717_0
- cuda-cudart-static_linux-64=13.2.75=h376f20c_0
- cuda-cudart_linux-64=13.2.75=h376f20c_0
- cuda-driver-dev_linux-64=13.2.75=h376f20c_0
- cuda-nvcc=13.2.78=hcdd1206_0
- cuda-nvcc-dev_linux-64=13.2.78=he91c749_0
- cuda-nvcc-impl=13.2.78=h85509e4_0
- cuda-nvcc-tools=13.2.78=he02047a_0
- cuda-nvcc_linux-64=13.2.78=hb2fc203_0
- cuda-nvrtc=13.2.78=hecca717_0
- cuda-nvrtc-dev=13.2.78=hecca717_0
- cuda-nvtx=13.2.75=hecca717_0
- cuda-nvtx-dev=13.2.75=ha770c72_0
- cuda-nvvm-dev_linux-64=13.2.78=ha770c72_0
- cuda-nvvm-impl=13.2.78=h4bc722e_0
- cuda-nvvm-tools=13.2.78=h4bc722e_0
- cuda-pathfinder=1.5.4=pyhc364b38_0
- cuda-profiler-api=13.2.75=h7938cbb_0
- cuda-python=13.2.0=pyhc455866_0
- cuda-sanitizer-api=13.2.76=h10ca0ad_0
- cuda-version=13.2=he2cc418_3
- cudf=26.06.00a351=cuda13_cp311_abi3_260514_cb075d7e
- cuml=26.06.00a147=cuda13_cp311_abi3_260515_4ad59e41
- cupy=14.0.1=py314h31ce861_0
- cupy-core=14.0.1=py314hed3c566_0
- cuvs=26.06.00a112=cuda13_cp311_abi3_260514_5b6fae9b
- cxx-compiler=1.11.0=hfcd1e18_0
- cython=3.2.4=py314h1807b08_0
- dbus=1.16.2=h24cb091_1
- decorator=5.2.1=pyhd8ed1ab_0
- distlib=0.4.0=pyhd8ed1ab_0
- dlpack=0.8=h59595ed_3
- docutils=0.21.2=pyhd8ed1ab_1
- doxygen=1.13.2=h8e693c7_0
- epoxy=1.5.10=hb03c661_2
- exceptiongroup=1.3.1=pyhd8ed1ab_0
- executing=2.2.1=pyhd8ed1ab_0
- filelock=3.29.0=pyhd8ed1ab_0
- font-ttf-dejavu-sans-mono=2.37=hab24e00_0
- font-ttf-inconsolata=3.000=h77eed37_0
- font-ttf-source-code-pro=2.038=h77eed37_0
- font-ttf-ubuntu=0.83=h77eed37_3
- fontconfig=2.17.1=h27c8c51_0
- fonts-conda-ecosystem=1=0
- fonts-conda-forge=1=hc364b38_1
- fribidi=1.0.16=hb03c661_0
- fsspec=2026.4.0=pyhd8ed1ab_0
- future=1.0.0=pyhd8ed1ab_2
- gcc=14.3.0=h6f77f03_19
- gcc_impl_linux-64=14.3.0=h235f0fe_19
- gcc_linux-64=14.3.0=h50e9bb6_24
- gdk-pixbuf=2.44.6=h2b0a6b4_0
- gflags=2.2.2=h5888daf_1005
- glib-tools=2.88.1=hee1de02_2
- glog=0.7.1=hbabe93e_0
- go=1.26.3=h282a287_0
- graphite2=1.3.14=hecca717_2
- graphviz=14.1.2=h8b86629_0
- gtk3=3.24.52=ha5ea40c_0
- gts=0.7.6=h977cf35_4
- gxx=14.3.0=h76987e4_19
- gxx_impl_linux-64=14.3.0=h2185e75_19
- gxx_linux-64=14.3.0=h8a413ad_24
- h2=4.3.0=pyhcf101f3_0
- harfbuzz=14.2.0=h6083320_0
- hicolor-icon-theme=0.17=ha770c72_3
- hpack=4.1.0=pyhd8ed1ab_0
- hyperframe=6.1.0=pyhd8ed1ab_0
- icu=78.3=h33c6efd_0
- identify=2.6.19=pyhd8ed1ab_0
- idna=3.13=pyhcf101f3_0
- imagesize=2.0.0=pyhd8ed1ab_0
- importlib-metadata=8.8.0=pyhcf101f3_0
- importlib-resources=7.1.0=pyhd8ed1ab_0
- importlib_resources=7.1.0=pyhd8ed1ab_0
- iniconfig=2.3.0=pyhd8ed1ab_0
- ipython=9.13.0=pyh53cf698_0
- ipython_pygments_lexers=1.1.1=pyhd8ed1ab_0
- jedi=0.19.2=pyhd8ed1ab_1
- jinja2=3.1.6=pyhcf101f3_1
- joblib=1.5.3=pyhd8ed1ab_0
- jsonschema=4.26.0=pyhcf101f3_0
- jsonschema-specifications=2025.9.1=pyhcf101f3_0
- kernel-headers_linux-64=4.18.0=he073ed8_9
- keyutils=1.6.3=hb9d3cd8_0
- krb5=1.22.2=ha1258a1_0
- ld_impl_linux-64=2.45.1=default_hbd61a6d_102
- lerc=4.1.0=hdb68285_0
- libabseil=20260107.1=cxx17_h7b12aa8_0
- libarrow=24.0.0=h72d56d0_1_cuda
- libarrow-acero=24.0.0=h635bf11_1_cuda
- libarrow-compute=24.0.0=h53684a4_1_cuda
- libarrow-dataset=24.0.0=h635bf11_1_cuda
- libarrow-substrait=24.0.0=hb4dd7c2_1_cuda
- libblas=3.11.0=7_h4a7cf45_openblas
- libbrotlicommon=1.2.0=hb03c661_1
- libbrotlidec=1.2.0=hb03c661_1
- libbrotlienc=1.2.0=hb03c661_1
- libcap=2.77=hd0affe5_1
- libcblas=3.11.0=7_h0358290_openblas
- libclang=20.1.8=default_h99862b1_15
- libclang-cpp20.1=20.1.8=default_h99862b1_15
- libclang13=20.1.8=default_h746c552_15
- libcrc32c=1.1.2=h9c3ff4c_0
- libcublas=13.4.0.1=h676940d_0
- libcublas-dev=13.4.0.1=h676940d_0
- libcudf=26.06.00a351=cuda13_260514_cb075d7e
- libcufft=12.2.0.46=hecca717_0
- libcufile=1.17.1.22=h85c024f_0
- libcufile-dev=1.17.1.22=h676940d_0
- libcuml=26.06.00a147=cuda13_260515_4ad59e41
- libcups=2.3.3=h7a8fb5f_6
- libcurand=10.4.2.55=h676940d_0
- libcurand-dev=10.4.2.55=h676940d_0
- libcurl=8.20.0=hcf29cc6_0
- libcusolver=12.2.0.1=h676940d_0
- libcusolver-dev=12.2.0.1=h676940d_0
- libcusparse=12.7.10.1=hecca717_0
- libcusparse-dev=12.7.10.1=hecca717_0
- libcuvs=26.06.00a112=cuda13_260514_5b6fae9b
- libcuvs-headers=26.06.00a112=cuda13_260514_5b6fae9b
- libdeflate=1.25=h17f619e_0
- libdrm=2.4.125=hb03c661_1
- libedit=3.1.20250104=pl5321h7949ede_0
- libegl=1.7.0=ha4b6fd6_2
- libegl-devel=1.7.0=ha4b6fd6_2
- libev=4.33=hd590300_2
- libevent=2.1.12=hf998b51_1
- libexpat=2.8.0=hecca717_0
- libffi=3.5.2=h3435931_0
- libfreetype=2.14.3=ha770c72_0
- libfreetype6=2.14.3=h73754d4_0
- libgcc=15.2.0=he0feb66_19
- libgcc-devel_linux-64=14.3.0=hf649bbc_119
- libgcc-ng=15.2.0=h69a702a_19
- libgd=2.3.3=h5fbf134_12
- libgfortran=15.2.0=h69a702a_19
- libgfortran5=15.2.0=h68bc16d_19
- libgl=1.7.0=ha4b6fd6_2
- libgl-devel=1.7.0=ha4b6fd6_2
- libglib=2.88.1=h0d30a3d_2
- libglvnd=1.7.0=ha4b6fd6_2
- libglx=1.7.0=ha4b6fd6_2
- libglx-devel=1.7.0=ha4b6fd6_2
- libgomp=15.2.0=he0feb66_19
- libgoogle-cloud=3.3.0=h25dbb67_1
- libgoogle-cloud-storage=3.3.0=hdbdcf42_1
- libgrpc=1.78.1=h1d1128b_0
- libiconv=1.18=h3b78370_2
- libjpeg-turbo=3.1.4.1=hb03c661_0
- libkvikio=26.06.00a17=cuda13_260514_a07527ad
- liblapack=3.11.0=7_h47877c9_openblas
- libllvm20=20.1.8=hf7376ad_1
- liblzma=5.8.3=hb03c661_0
- libmpdec=4.0.0=hb03c661_1
- libnghttp2=1.68.1=h877daf1_0
- libnl=3.11.0=hb9d3cd8_0
- libnuma=2.0.18=hb03c661_3
- libnvcomp=5.2.0.10=h7bcfba5_0
- libnvcomp-dev=5.2.0.10=h7bcfba5_0
- libnvjitlink=13.2.78=hecca717_0
- libnvjitlink-dev=13.2.78=hecca717_0
- libnvptxcompiler-dev=13.2.78=ha770c72_0
- libnvptxcompiler-dev_linux-64=13.2.78=ha770c72_0
- libopenblas=0.3.33=pthreads_h94d23a6_0
- libopentelemetry-cpp=1.26.0=h9692893_0
- libopentelemetry-cpp-headers=1.26.0=ha770c72_0
- libparquet=24.0.0=h7376487_1_cuda
- libpciaccess=0.18=hb9d3cd8_0
- libpng=1.6.58=h421ea60_0
- libprotobuf=6.33.5=h2b00c02_0
- libraft=26.06.00a31=cuda13_260514_2f724f58
- libraft-headers=26.06.00a31=cuda13_260514_2f724f58
- libraft-headers-only=26.06.00a31=cuda13_260514_2f724f58
- libre2-11=2025.11.05=h0dc7533_1
- librmm=26.06.00a95=cuda13_260514_29a9ce33
- librsvg=2.62.1=h4c96295_0
- libsanitizer=14.3.0=h8f1669f_19
- libsqlite=3.53.1=h0c1763c_0
- libssh2=1.11.1=hcf80075_0
- libstdcxx=15.2.0=h934c35e_19
- libstdcxx-devel_linux-64=14.3.0=h9f08a49_119
- libstdcxx-ng=15.2.0=hdf11a46_19
- libsystemd0=260.1=h6569c3e_0
- libthrift=0.22.0=h7d032f7_2
- libtiff=4.7.1=h9d88235_1
- libucxx=0.50.00a40=cuda13_260514_b9ff05a4
- libudev1=260.1=h6569c3e_0
- libutf8proc=2.11.3=hfe17d71_0
- libuuid=2.42=h5347b49_0
- libuv=1.51.0=hb03c661_1
- libwebp-base=1.6.0=hd42ef1d_0
- libxcb=1.17.0=h8a09558_0
- libxkbcommon=1.13.1=hca5e8e5_0
- libxml2=2.15.3=h49c6c72_0
- libxml2-16=2.15.3=hca6bf5a_0
- libzlib=1.3.2=h25fd6f3_2
- llvm-openmp=22.1.5=h4922eb0_1
- llvmlite=0.46.0=py314h946fb2a_0
- lz4-c=1.10.0=h5888daf_1
- make=4.4.1=hb9d3cd8_2
- markdown=3.10.2=pyhcf101f3_0
- markdown-it-py=4.2.0=pyhd8ed1ab_0
- markupsafe=3.0.3=py314h67df5f8_1
- matplotlib-inline=0.2.2=pyhd8ed1ab_0
- mdurl=0.1.2=pyhd8ed1ab_1
- nccl=2.30.4.1=h6e9c2f8_0
- ncurses=6.6=hdb14827_0
- ninja=1.13.2=h171cf75_0
- nlohmann_json=3.12.0=h54a6638_1
- nodeenv=1.10.0=pyhd8ed1ab_0
- numba=0.64.0=py314h8169c2f_0
- numba-cuda=0.28.2=py314h118ebf0_0
- numpy=2.4.4=py314h2b28147_0
- numpydoc=1.10.0=pyhcf101f3_0
- nvidia-ml-py=13.595.45=pyhd8ed1ab_1
- nvtx=0.2.15=py314h5bd0f2a_0
- openblas=0.3.33=pthreads_h6ec200e_0
- openssl=3.6.2=h35e630c_0
- orc=2.3.0=h21090e2_0
- packaging=26.2=pyhc364b38_0
- pandas=2.3.3=py314ha0b5721_2
- pango=1.56.4=hda50119_1
- parso=0.8.7=pyhcf101f3_0
- pathspec=1.1.1=pyhd8ed1ab_0
- pcre2=10.47=haa7fec5_0
- pexpect=4.9.0=pyhd8ed1ab_1
- pip=26.1.1=pyh145f28c_0
- pixman=0.46.4=h54a6638_1
- platformdirs=4.9.6=pyhcf101f3_0
- pluggy=1.6.0=pyhf9edf01_1
- pre-commit=4.6.0=pyha770c72_0
- prometheus-cpp=1.3.0=ha5d0236_0
- prompt-toolkit=3.0.52=pyha770c72_0
- psutil=7.2.2=py314h0f05182_0
- pthread-stubs=0.4=hb9d3cd8_1002
- ptyprocess=0.7.0=pyhd8ed1ab_1
- pure_eval=0.2.3=pyhd8ed1ab_1
- pyarrow=24.0.0=py314hdafbbf9_0
- pyarrow-core=24.0.0=py314h57703d4_0_cuda
- pycparser=2.22=pyh29332c3_1
- pygments=2.20.0=pyhd8ed1ab_0
- pylibcudf=26.06.00a351=cuda13_cp311_abi3_260514_cb075d7e
- pylibraft=26.06.00a31=cuda13_cp311_abi3_260514_2f724f58
- pysocks=1.7.1=pyha55dd90_7
- pytest=9.0.3=pyhc364b38_1
- pytest-cov=7.1.0=pyhcf101f3_0
- pytest-repeat=0.9.4=pyhd8ed1ab_0
- python=3.14.4=habeac84_100_cp314
- python-dateutil=2.9.0.post0=pyhe01879c_2
- python-discovery=1.3.1=pyhcf101f3_0
- python-gil=3.14.4=h4df99d1_100
- python-tzdata=2026.2=pyhd8ed1ab_0
- python_abi=3.14=8_cp314
- pytz=2026.2=pyhcf101f3_0
- pyyaml=6.0.3=py314h67df5f8_1
- rapids-build-backend=0.4.1=py_0
- rapids-dependency-file-generator=1.20.3=py_0
- rapids-logger=0.2.3=h98325ef_0
- rdma-core=63.0=h192683f_1
- re2=2025.11.05=h5301d42_1
- readline=8.3=h853b02a_0
- recommonmark=0.7.1=pyhd8ed1ab_1
- referencing=0.37.0=pyhcf101f3_0
- requests=2.34.2=pyhcf101f3_0
- rhash=1.4.6=hb9d3cd8_1
- rich=15.0.0=pyhcf101f3_0
- rmm=26.06.00a95=cuda13_cp311_abi3_260514_29a9ce33
- roman-numerals=4.1.0=pyhd8ed1ab_0
- roman-numerals-py=4.1.0=pyhd8ed1ab_0
- rpds-py=0.30.0=py314h2e6c369_0
- rust=1.95.0=h53717f1_1
- rust-std-x86_64-unknown-linux-gnu=1.95.0=h2c6d0dc_1
- s2n=1.7.2=hc5a330e_1
- scikit-build-core=0.12.1=pyh04d0eab_0
- scikit-learn=1.8.0=np2py314hf09ca88_1
- scipy=1.16.3=py314hf07bd8e_2
- setuptools=82.0.1=pyh332efcf_0
- six=1.17.0=pyhe01879c_1
- snappy=1.2.2=h03e3b7b_1
- snowballstemmer=3.0.1=pyhd8ed1ab_0
- sphinx=8.2.3=pyhd8ed1ab_0
- sphinx-copybutton=0.5.2=pyhd8ed1ab_1
- sphinx-markdown-tables=0.0.17=pyhd8ed1ab_1
- sphinxcontrib-applehelp=2.0.0=pyhd8ed1ab_1
- sphinxcontrib-devhelp=2.0.0=pyhd8ed1ab_1
- sphinxcontrib-htmlhelp=2.1.0=pyhd8ed1ab_1
- sphinxcontrib-jsmath=1.0.1=pyhd8ed1ab_1
- sphinxcontrib-qthelp=2.0.0=pyhd8ed1ab_1
- sphinxcontrib-serializinghtml=1.1.10=pyhd8ed1ab_1
- stack_data=0.6.3=pyhd8ed1ab_1
- sysroot_linux-64=2.28=h4ee821c_9
- threadpoolctl=3.6.0=pyhecae5ae_0
- tk=8.6.13=noxft_h366c992_103
- tomli=2.4.1=pyhcf101f3_0
- tomlkit=0.15.0=pyha770c72_0
- traitlets=5.15.0=pyhcf101f3_0
- treelite=4.7.0=py314he2c2258_0
- typing_extensions=4.15.0=pyhcf101f3_0
- tzdata=2025c=hc9c84f9_1
- ucx=1.20.0=hf72d326_1
- ucxx=0.50.00a40=cuda13_cp311_abi3_260514_b9ff05a4
- ukkonen=1.1.0=py314h9891dd4_0
- urllib3=2.7.0=pyhd8ed1ab_0
- virtualenv=21.3.3=pyhcf101f3_0
- wayland=1.25.0=hd6090a7_0
- wcwidth=0.7.0=pyhd8ed1ab_0
- xkeyboard-config=2.47=hb03c661_0
- xorg-libice=1.1.2=hb9d3cd8_0
- xorg-libsm=1.2.6=he73a12e_0
- xorg-libx11=1.8.13=he1eb515_0
- xorg-libxau=1.0.12=hb03c661_1
- xorg-libxcomposite=0.4.7=hb03c661_0
- xorg-libxcursor=1.2.3=hb9d3cd8_0
- xorg-libxdamage=1.1.6=hb9d3cd8_0
- xorg-libxdmcp=1.1.5=hb03c661_1
- xorg-libxext=1.3.7=hb03c661_0
- xorg-libxfixes=6.0.2=hb03c661_0
- xorg-libxi=1.8.2=hb9d3cd8_0
- xorg-libxinerama=1.1.6=hecca717_0
- xorg-libxrandr=1.5.5=hb03c661_0
- xorg-libxrender=0.9.12=hb9d3cd8_0
- xorg-libxtst=1.2.5=hb9d3cd8_3
- xorg-libxxf86vm=1.1.7=hb03c661_0
- xorg-xorgproto=2025.1=hb03c661_0
- yaml=0.2.5=h280c20c_3
- zipp=3.23.1=pyhcf101f3_0
- zlib=1.3.2=h25fd6f3_2
- zstd=1.5.7=hb78ec9c_6
- pip:
- accessible-pygments==0.0.5
- beautifulsoup4==4.14.3
- nvidia-sphinx-theme==0.0.9.post1
- pydata-sphinx-theme==0.17.1
- soupsieve==2.8.3
prefix: /raid/jlamb/miniforge/envs/cuvs-devAdditional context
I observed this working on RAPIDS NVAIE builds. We've seen this exact type of failure in testing for those types of builds as far back as cuVS 25.10.
It may be worth revisiting whatever motivated this comment:
https://github.com/rapidsai/cuvs/blob/4c2407f459bb40abfd9e1d2959fb60180a8780e9/python/cuvs/cuvs/tests/test_mg_ivf_flat.py#L144-L146
Added in #1307 (cc @viclafargue)
Describe the bug
Python test
test_mg_ivf_flat.py::test_mg_ivf_flat_extend'reliably fails on multi-GPU systems with Blackwell GPUs (e.g. B200, GB200).The failure mode (low recall) is always the same, but the exact values differ a lot across runs. Like this:
Steps/Code to reproduce bug
I was able to reproduce this today on latest
release/26.06(5b6fae9), on a B200x8 machine.Expected behavior
Expected this test to pass reliably, as I've observed on other multi-GPU setups like H100x2.
Environment details (please complete the following information):
output of 'nvidia-smi' (click me)
output of 'conda env export' (click me)
Additional context
I observed this working on RAPIDS NVAIE builds. We've seen this exact type of failure in testing for those types of builds as far back as cuVS 25.10.
It may be worth revisiting whatever motivated this comment:
https://github.com/rapidsai/cuvs/blob/4c2407f459bb40abfd9e1d2959fb60180a8780e9/python/cuvs/cuvs/tests/test_mg_ivf_flat.py#L144-L146
Added in #1307 (cc @viclafargue)