fix(pathfinder): make find_nvidia_binary_utility deterministic, never search CWD

[aryanputta]

Summary

Fixes #2119.

find_nvidia_binary_utility previously delegated to shutil.which(name, path=trusted_dirs). On Windows, CPython's shutil.which prepends the process current working directory even when an explicit path= is supplied. That allowed a binary in an arbitrary CWD to shadow the CUDA / Conda / wheel binary that pathfinder is meant to discover through a bounded search.

This PR replaces that delegation with an explicit resolver over trusted directories only, then folds in the maintainer-requested CTK-root canary fallback so users who follow the CUDA installation guide continue to work when the toolkit is visible through the dynamic loader.

Fix

The binary lookup order is now:

1. NVIDIA wheel bin/ directories from site-packages.
2. CONDA_PREFIX.
3. CUDA_HOME / CUDA_PATH.
4. CTK-root canary fallback: resolve cudart through the OS dynamic loader, derive the CUDA Toolkit root, and search that root's bin layout.

The current working directory and ambient PATH are never consulted.

Notes:

• POSIX execute-bit semantics and Windows executable-extension semantics are preserved.
• Empty and duplicate trusted directories are skipped.
• The canary fallback only runs after the explicit trusted directories miss.
• The canary path reuses the dynamic-library CTK-root discovery helper, so library and binary discovery share the same root derivation.

Tests

• Reworked search-path tests to assert deterministic probe order.
• Added CWD-isolation coverage for the core find_nvidia_binary_utility() violates deterministic-search contract on Windows (CWD leakage via shutil.which) #2119 regression.
• Added first-match, empty/duplicate directory, and POSIX non-executable rejection coverage.
• Added CTK-root canary fallback coverage, including Windows bin/x64, bin/x86_64, and bin layout.
• Added coverage that the canary is not consulted when an earlier trusted directory hits.
• Added focused tests for shared resolve_ctk_root_via_canary(...).

Local checks: ruff check and ruff format --check pass on the touched files.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[rwgk]

Hi @aryanputta, this is great, but in isolation it'll probably break some users. This was pointed out by @kkraus14 here before:

• [codex] Avoid untrusted binary lookup paths #2111 (comment)

If someone follows this installation guide and sets LD_LIBRARY_PATH for libraries and PATH for executables ...

I ran the context and my thoughts by GPT around the same time, which lead to the conclusion at the end of this comment:

• [codex] Avoid untrusted binary lookup paths #2111 (comment)

Use a custom bounded finder, add CTK-root canary fallback, and do not implicitly search PATH.

I still think that's the right compromise:

• We will find the binary utility via LD_LIBRARY_PATH if set. It's still an attack vector, but significantly more difficult to exploit than hacking PATH.
• Users following the installation guide will not see a difference in most cases, and get a more self-consistent outcome in the rare case that the searches via PATH and LD_LIBRARY_PATH produce diverging results.

The behavior difference is significant enough though that I'd want to bump the pathfinder minor version, which we can do; I don't have concerns about that.

How do you feel about expanding this PR to fold in the CTK-root canary fallback? — I believe the code changes will still be quite modest.

[aryanputta]

Thanks @rwgk, that compromise makes sense and I've folded it in (pushed c3d39f0, no force-push). Agreed on the reasoning: the bounded finder shouldn't regress users who follow the Linux install guide and only set LD_LIBRARY_PATH + PATH, but reaching them via LD_LIBRARY_PATH instead of PATH is the safer trade.

What the new commit adds

Search order is unchanged for the explicit trusted dirs, with the canary as a strict fallback:

1. NVIDIA wheel bin/ (site-packages)
2. CONDA_PREFIX
3. CUDA_HOME / CUDA_PATH
4. CTK-root canary (new): resolve cudart through the OS dynamic loader, derive the CTK root from its absolute path, search that root's bin layout

Key points:

• The canary runs only when steps 1-3 miss, so the common wheel/conda/CUDA_HOME cases never spawn the resolver subprocess.
• It reuses the existing canary toolbox. I factored the cudart -> CTK-root resolution into resolve_ctk_root_via_canary(...) in load_nvidia_dynamic_lib.py and had the existing dynamic-library _try_ctk_root_canary call it too, so lib and binary discovery share one path. The resolver runs in the isolated subprocess that honors LD_LIBRARY_PATH on Linux and the native DLL search on Windows.
• PATH and the process CWD are still never consulted, so the LD_LIBRARY_PATH surface is the only added vector, exactly as you described.

Version

Added docs/source/release/1.6.0-notes.rst for the minor bump (version is tag-driven, so this is the doc side of it). Happy to renumber if you'd rather it land under a different version.

Tests

Added coverage for the fallback in tests/test_find_nvidia_binaries.py (found via canary, ordering vs CUDA_HOME, Windows bin/x64/bin/x86_64/bin layout, not consulted when an earlier dir hits, cached) and a focused resolve_ctk_root_via_canary unit test in tests/test_ctk_root_discovery.py. ruff check and ruff format --check are clean.

[aryanputta]

Quick check-in: is there anything else you would like changed here after the CTK-root canary fallback update, or is this mainly waiting on review / runner validation now? Happy to adjust if the version note or fallback behavior should be shaped differently.