Enable opt-in host-pageable CUDA GatherBlockQuantized

[Copilot]

Description

Add the enable_host_pageable_gather CUDA EP option and use it to keep constant FP8 com.microsoft::GatherBlockQuantized data in CPU/file-backed memory on devices that support pageable access through host page tables. Unsupported devices and CUDA Graph configurations warn and use a persistent device copy, including a thread-safe lazy copy when prepacking is disabled.

Also includes CUDA plugin plumbing, provider-option/policy/fallback/capability-gated tests, and documentation of memory-accounting, sharing, and performance limitations.

Motivation and Context

Large sparse FP8 embedding tables can exceed practical GPU memory capacity even though each inference accesses only a few rows. Direct host-pageable access preserves file-backed demand paging and avoids allocating or copying the complete table on supported unified-memory systems.

Validation: clang-format and secret scanning passed. Automated review reported no findings; its CodeQL database scan was skipped because the database was too large. A CPU provider-test build remains in progress, and the capability-gated direct CUDA test could not be run because this environment has no CUDA toolkit/hardware.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

The current implementation introduces an ABI hazard, default-path regressions, and unsafe lazy-copy state publication.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds opt-in host-pageable access for FP8 CUDA GatherBlockQuantized, with device-copy fallback and CUDA plugin support.

Changes:

• Adds provider-option plumbing and host/device data policies.
• Implements persistent and per-run CUDA fallback copies.
• Adds capability, CUDA Graph, option, and fallback tests plus documentation.

File summaries

File	Description
onnxruntime/test/util/default_providers.cc	Forwards the option to CUDA plugin tests.
onnxruntime/test/contrib_ops/gather_block_quantized_op_test.cc	Adds policy, fallback, direct-host, and graph tests.
onnxruntime/core/providers/cuda/plugin/cuda_kernel_adapter.h	Exposes plugin runtime configuration.
onnxruntime/core/providers/cuda/plugin/cuda_ep.h	Adds plugin configuration state.
onnxruntime/core/providers/cuda/plugin/cuda_ep.cc	Propagates plugin configuration to kernels.
onnxruntime/core/providers/cuda/plugin/cuda_ep_factory.cc	Parses the plugin option.
onnxruntime/core/providers/cuda/cuda_provider_factory.cc	Converts bundled provider options.
onnxruntime/core/providers/cuda/cuda_kernel.h	Exposes provider state to kernels.
onnxruntime/core/providers/cuda/cuda_execution_provider.h	Exposes the option from the CUDA EP.
onnxruntime/core/providers/cuda/cuda_execution_provider_info.h	Stores and hashes the option.
onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc	Parses and serializes the option.
onnxruntime/contrib_ops/cuda/quantization/gather_block_quantized.h	Defines policies and persistent storage.
onnxruntime/contrib_ops/cuda/quantization/gather_block_quantized.cuh	Identifies eligible FP8 types.
onnxruntime/contrib_ops/cuda/quantization/gather_block_quantized.cc	Implements direct-host and fallback paths.
include/onnxruntime/core/providers/cuda/cuda_provider_options.h	Adds the option to the V2 struct.
docs/cuda_host_pageable_gather.md	Documents requirements and limitations.

Review details

• Files reviewed: 16/16 changed files
• Comments generated: 4
• Review effort level: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.