Skip to content

fix(qwen): preserve compact logits indices across graph reuse - #426

Merged
0xShug0 merged 1 commit into
0xShug0:mainfrom
mirek190:fix/qwen-compact-logits-input-lifetime
Sep 3, 2026
Merged

fix(qwen): preserve compact logits indices across graph reuse#426
0xShug0 merged 1 commit into
0xShug0:mainfrom
mirek190:fix/qwen-compact-logits-input-lifetime

Conversation

@mirek190

@mirek190 mirek190 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Fix compact-logit row-index tensors in the shared Qwen causal decode runtime being overwritten when a cached prefill graph is reused.

The compact readback path creates an I32 tensor consumed by ggml_get_rows. It is populated by the host after graph allocation, but it was not marked as a graph input. The allocator could therefore alias its storage with an intermediate tensor. A first request could succeed while leaving corrupted indices for the next execution, which was observed as an illegal CUDA memory access on the second request.

Changes

  • mark compact-logit token IDs as a persistent ggml input
  • re-upload the IDs immediately before every cached prefill execution
  • cover both single and batched prefill paths
  • retain cached graph reuse rather than rebuilding compact-logit graphs

Validation

  • Windows CPU full-composite audiocpp_cli build: passed
  • Windows CPU engine_core rebuild: passed
  • Windows CUDA 12.4 engine_core build for SM 86: passed
  • git diff --check: passed

This is isolated from the MiraTTS model work and changes only the shared Qwen runtime lifecycle for configurations using logits_readback_token_ids.

@0xShug0
0xShug0 merged commit 4f3c9c2 into 0xShug0:main Sep 3, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants