Sync with Microsoft ONNX Runtime - 03092026 - #1282
Merged
Merged
Conversation
…rosoft#31668) ### Description The `Scan` operator's `num_scan_inputs` attribute determines how many of the node's variadic inputs are scan inputs (the remainder are loop state variables). In `scan::detail::Info::Info` (shared by the CPU `Scan-8`/`Scan-9` kernels, and reused as-is by the CUDA `Scan` kernel), this attribute was used in two unguarded subtractions: ```cpp num_loop_state_variables = num_variadic_inputs - num_scan_inputs; ... num_scan_outputs = num_outputs - num_loop_state_variables; ``` If `num_scan_inputs` is outside `[0, num_variadic_inputs]`, `num_loop_state_variables` becomes negative. That value is later used, unguarded, as a loop-start index into the node's inputs during `Compute()`, resulting in out-of-range indexing. This PR adds validation of `num_scan_inputs` (and the derived `num_loop_state_variables`) in the constructor, rejecting invalid values up front with a clear error message before any arithmetic derived from them is used for indexing. ### Testing Added regression tests for both opset 8 and opset 9+ with an out-of-range `num_scan_inputs` value: - Opset 8 hits the new kernel-construction-time check directly. - Opset 9+ is caught earlier during graph resolution's standard ONNX-level shape inference (before the kernel is even constructed), so its test asserts on the shape-inference error message instead. Both tests are guarded by `#if !defined(ORT_NO_EXCEPTIONS)` since they rely on exception-based failure reporting. Ran locally (CPU-only Debug build): - `onnxruntime_provider_test --gtest_filter='Scan*'` — all 34 tests pass, including the 2 new ones. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 77dcaf1b-748a-4379-94a7-478f7a924d73
## Summary Use a cooperative CUB block scan for low-lane int64 CumSum inputs. This changes decode-shaped scans from recomputing every prefix independently to linear work along the scan axis, while preserving the existing kernel for high-lane inputs and non-int64 types. ## Why Batch-1 position-id scans have one lane, so the existing one-thread-per-output implementation serializes an O(context) prefix scan and repeats earlier additions. The new path assigns one CUDA block per lane and carries the aggregate across 256-element tiles. ## Performance RTX A1000 (CC 8.6), CUDA-event timing of 100-node captured graphs, median of 30 replays: - width 640: 16.650 us -> 3.809 us (4.37x) - width 1000: 25.472 us -> 3.963 us (6.43x) - width 2600: 49.787 us -> 7.311 us (6.81x) - width 4096: 78.223 us -> 9.656 us (8.10x) - width 8192: 180.398 us -> 19.692 us (9.16x) The final width >= 4 gate avoids a measured regression at width 2. ## Correctness Integer addition preserves modulo-2^64 results under regrouping. CUDA validation was bit-exact for shapes 1x1000x1 and 2x513x3 across inclusive/exclusive and forward/reverse modes. The ORT test covers multi-tile scans and nontrivial outer/inner indexing with CPU fallback disabled. ## Validation - NVRTC compiled the exact CUB BlockScan kernel for compute capability 8.6 - CUDA functional matrix passed against a serial reference - clang-format and git diff --check passed - Two independent code reviews found no high-confidence issues Based on the mechanism validated in justinchuby/onnx-genai#1366. ### Description <!-- Describe your changes. --> ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…rosoft#29743) ### Description When building on ADO, clone DAWN via ADO mirrors & upstream github repos. ### Motivation and Context DAWN usually clones via `chromium.org` mirror, which isn't acceptable on ADO CI due to network isolation.
…soft#32268) ### Description Adds two `com.microsoft` contrib ops for DeepSeek Engram, each with CPU, CUDA, and WebGPU kernels: | Op | Purpose | | --- | --- | | `EngramGate` | Dual RMSNorm over key/query, sign-preserving sqrt gate, applied to `value` and broadcast across hyper-connections | | `NGramHashMapping` | Causal n-gram hash ids from compressed tokenizer ids, with optional cross-call `past_ids`/`present_ids` state | **Type coverage** | EP | `EngramGate` | `NGramHashMapping` | | --- | --- | --- | | CPU | `float`, `float16` | `int32`, `int64` | | CUDA | `float`, `float16`, `bfloat16` | `int32`, `int64` | | WebGPU | `WebGpuSupportedFloatTypes()` | `int32` | **Layout** - One file per op per EP under `contrib_ops/<ep>/bert/`, following the existing convention. - Shared math/launch helpers live in a per-EP `engram_helper` file rather than being duplicated per kernel: - `contrib_ops/cpu/bert/engram_helper.h` — `SigmoidFloat`, `EngramGateArg`, `PositiveMod`, `WrappedMultiply` - `contrib_ops/cuda/bert/engram_helper.cuh` — the same as device functions, plus `BlockSum3` and the shared `kThreads` / `GridSize` launch config - `contrib_ops/webgpu/bert/engram_helper.h` — WGSL snippets (`stable_sigmoid`, `engram_gate_arg`, `positive_mod`) - New entries are inserted in alphabetical order in each EP's `BuildKernelCreateInfo` table. ### Notes for review **No `ShortConv` op.** An earlier revision of this PR added a third op, `ShortConv` (per-hyper-connection RMSNorm + causal depthwise 1D conv with fused SiLU). It was dropped in review: the RMSNorm there is per-`(batch, token, hyper-connection)` over `hidden_size` only, so it never mixes positions and therefore commutes with the convolution's state threading. That makes `RMSNorm` → transpose → existing `CausalConvWithState(activation="silu")` → transpose exactly equivalent, including at chunk boundaries, so a second stateful causal depthwise convolution op in `com.microsoft` was not justified. See [this thread](microsoft#32268 (comment)). **Integer semantics are load-bearing, not stylistic.** - `WrappedMultiply` computes the hash mix through the unsigned counterpart of `T` so overflow wraps instead of being UB. - `PositiveMod` applies a Euclidean correction, because C++, CUDA, and WGSL all truncate `%` toward zero. `NGramHashMappingNegativeIds{Int32,Int64}` pins this with negative ids and a negative `pad_id`, driving 5 of 8 mixes negative so a regression to a plain `%` fails. **`EngramGate` zero-dot behavior.** `std::copysign`/`copysignf` cannot be used for `sign(dot) * sqrt(max(abs(dot), 1e-6))` — they map a zero dot product to `+sqrt(1e-6)`, giving a gate of ~0.50025 instead of exactly 0.5, and disagreeing with WGSL's `sign()`. `EngramGateZeroDotProduct` covers this. **`vocab_sizes` validation.** Entries must be strictly positive. CPU rejects a non-positive entry with an error naming the index; GPU EPs guard the modulo (to avoid a device-side divide by zero) and emit a hash id of `0`, since validating there would force a per-`Compute` device sync. Both the requirement and the GPU fallback are documented in the schema. **Docs.** `docs/ContribOperators.md` and `docs/OperatorKernels.md` were updated by hand to match `gen_doc` output; the Windows CI `--gen_doc validate` run is the authoritative check. **Tests.** `onnxruntime/test/contrib_ops/engram_ops_test.cc` (14 tests) covers both ops across the supported types, the vectorized WebGPU path, chunked-vs-full-sequence equivalence for the n-gram state, and the negative-id and invalid-`vocab_sizes` edge cases. ### Motivation and Context Engram currently decomposes into long chains of primitive ONNX ops (shifts, hashes, mod, dual RMSNorm, sigmoid gating), which is both slow and awkward to export. These two ops collapse those patterns into single kernels so Engram-based models can run efficiently across CPU, CUDA, and WebGPU. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
### Description Adds a safe way to use WebGPU graph capture from Python. Replay reuses the exact buffers recorded at capture, and Python had no way to allocate those buffers, update them, or release a captured graph so anything you tried either failed or quietly returned stale results. This adds session-owned WebGPU `OrtValue`s (plus support for ones from a shared allocator, which have no session), in-place updates, and `release_captured_graph(id)`, and the first capturing run now pins its `IOBinding` so rebinding is rejected instead of silently writing to the original buffers. There's also one C++ change: WebGPU's `CanCopy` accepted any GPU device pair without checking vendor ID, so in a session with WebGPU and CUDA registered a CUDA copy would get routed to WebGPU and handed to Dawn as a `WGPUBuffer`. WebGPU now refuses GPU buffers from other vendors ### Motivation and Context Graph capture already exists in the WebGPU EP, but there was no supported way to reach it from Python. You could turn the session option on, but without fixed device buffers the results were wrong without raising an error — replay just kept using whatever was bound the first time. On an NVIDIA TITAN V, capture takes YOLO26n from 5.885 ms to 5.475 ms p50, about 7%. Both numbers come from the script in this PR using its defaults, 5 fresh processes per arm with the arms alternating. Tested locally against a WebGPU build on the TITAN V: 80 passed, 2 skipped (no `onnx` installed), 3 failures that are pre-existing and unrelated (missing LoRA test data, custom-ops library not built). --------- Co-authored-by: Ananya Anand <t-anaanand@microsoft.com>
### Description `Im2ColMatMulProgram` refused any fused activation, so fused NHWC fp16 convs fell through to another kernel even where im2col was the better choice. This adds an activation epilogue to `im2col_matmul.wgsl.template` for the same six kinds the other Conv kernels support (Relu, Sigmoid, Clip, HardSigmoid, LeakyRelu, Tanh) and removes the `// TODO: Support fuse` guard. Depends on microsoft#32116 and should land after it. The template reads parameters from `uniforms.activation_param_0/1`, which only exist because of the uniforms refactor there. With values still baked into shader text, every distinct alpha or clip bound would have needed its own generated variant, which is why the TODO was there. **This path has not been executed anywhere.** `IsDeviceSupported()` requires vendor `intel` plus architecture `xe-2lpg`, `xe-2hpg`, `xe-3lpg` or `xe-3lpg-xs` (Lunar Lake, Battlemage, Panther Lake). No ORT CI agent has one and neither did any machine I had while writing this, so the parity tests skip everywhere and this code has never run. Please weigh it as unexecuted. Reachability is narrow, all five required: qualifying adapter, fp16 only, channels last, group 1, non 1x1 kernel. ### Motivation and Context Every other WebGPU Conv kernel already fuses activations. im2col was the only one that did not. Four parity tests cover Relu, LeakyRelu, HardSigmoid and Clip. The Clip one is new. Clip is the only two slot activation whose slots mean `{min, max}` instead of `{alpha, beta}`, so it is the one case where mis indexing `activation_param_0/1` would still pass the HardSigmoid test. It skips like the others but belongs in the file so it runs as soon as someone with the right hardware builds this. `static_assert`s pin each `ActivationKind` to the value the template checks, so reordering the enum breaks the build, and `IsActivationSupported` falls to `default: return false` for a new enumerator rather than emitting no epilogue. Goldens regenerated with `UPDATE_WGSL_GOLDEN=1 python wgsl_template/test/run_tests.py`. `generated/math/subgroup_matrix_*.h` are still missing because they belong to microsoft#32115, so the smoke test reports a file set difference for those three and no content mismatch.
ai-fw-intg
requested review from
Jaswanth51,
ankitm3k,
jatinwadhwa921 and
vthaniel
September 2, 2026 20:37
hdharpure9922
self-requested a review
September 3, 2026 05:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.