Repository navigation
fix(perf): avoid replaying terminal B1 decode - #2271
Merged
Merged
Conversation
Archive the pre-change profile, five-round acceptance records, and correctness evidence for issue #2227.
Add the bilingual technical report and record the verified performance, correctness, provenance, and Qwen prompt-cache limitation for PR #2271. Remove generated Nsight capture binaries from git while retaining their hashes, sizes, local provenance, and committed text exports under the repository's binary-assets policy. Validation: - `sha256sum -c SHA256SUMS` - `python3 scripts/ci/check_binary_assets.py` - `bash scripts/ci/check_binary_assets_test.sh` - archived acceptance analyzer: 4/4 cells passed Refs #2227
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For a fused-lookahead-eligible capped B=1 request, the scheduler submitted the next lookahead at the length boundary, discarded both speculative appends, and synchronously replayed a token it had already sampled. This changes the terminal B=1 path to commit that pending token through the existing finish/finalization logic, removing two complete decode forwards per affected request while leaving B>1 behavior unchanged. Closes #2227.
Profiling on GB10 showed row plumbing (0.0053%), metrics (0.0462%), provider channel send (0.0698%), and lookahead collect (0.0836%) below the issue's 0.1% threshold, so those candidates remain unchanged. A guard mutation makes the live-lookahead test fail with four forwards instead of two, confirming that the new branch accounts for the removed work.
Performance, five interleaved quiet-host rounds against pre-epic
4c44e317:All four medians pass the -1.0% threshold. Every candidate range overlaps its null range, so the measurements establish the threshold without claiming a speedup. The unexplained Llama 256 null outlier (-12.38% in round 3) remains in the raw data and analysis without exclusion or rerun.
Validation:
cargo fmt --all -- --checkcargo clippy --workspace --all-targets --features cuda -- -D warningsmlxcel runbefore/after generated text: byte-identicalThe full Qwen parity command still exits 1 on its prompt-cache miss/hit pair at greedy token 58 (
374versus594) and seeded token 61 (576versus362). Forced synchronous decode reproduces the same positions and token IDs, while the new guard is unreachable; the no-cache pairs affected by this change are identical. The benchmark report records this existing prompt-cache limitation, which is outside this issue's scope.Evidence: benchmark report and raw data.