Skip to content

Synchronize concurrent GPU chip proof stream#1331

Open
hero78119 wants to merge 1 commit intomasterfrom
bug/race_condition
Open

Synchronize concurrent GPU chip proof stream#1331
hero78119 wants to merge 1 commit intomasterfrom
bug/race_condition

Conversation

@hero78119
Copy link
Copy Markdown
Collaborator

@hero78119 hero78119 commented Apr 28, 2026

Problem

Concurrent GPU chip proving can return from a worker before its bound CUDA stream is fully drained. This can let the scheduler unbook/reuse resources while async work is still pending, causing intermittent proof corruption such as RootMismatch.

Change

Add a concurrent-mode-only stream synchronization at the end of create_chip_proof_gpu_impl before task resources are released.

Testing

cargo fmt --package ceno_zkvm
cargo check --features gpu --package ceno_zkvm --bin e2e

@hero78119 hero78119 added the regression-e2e-reth trigger regression test of https://github.com/scroll-tech/ceno-reth-benchmark label Apr 28, 2026
@hero78119 hero78119 changed the title add missing syncronization, avoid race condition Synchronize concurrent GPU chip proof stream Apr 28, 2026
@hero78119
Copy link
Copy Markdown
Collaborator Author

Benchmark result from scroll-tech/ceno-reth-benchmark.

@hero78119 hero78119 force-pushed the bug/race_condition branch from 8848c75 to cdbeb78 Compare April 28, 2026 00:18
@hero78119 hero78119 added regression-e2e-reth trigger regression test of https://github.com/scroll-tech/ceno-reth-benchmark and removed regression-e2e-reth trigger regression test of https://github.com/scroll-tech/ceno-reth-benchmark labels Apr 28, 2026
@hero78119
Copy link
Copy Markdown
Collaborator Author

Benchmark result from scroll-tech/ceno-reth-benchmark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

regression-e2e-reth trigger regression test of https://github.com/scroll-tech/ceno-reth-benchmark

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant