[maleadt]

Since #3404, every synchronization checks whether each context that kernels were compiled for is still valid, but the table of exception flags was never pruned. The core/exceptions test that launches a kernel in a temporary context leaves an entry behind, so every later synchronize() on that test worker calls cuCtxGetId on the destroyed context's handle. That usually returns ERROR_CONTEXT_IS_DESTROYED, but on the CI drivers (595.71, 610.43) it eventually returns ERROR_INVALID_CONTEXT:

CUDA error: invalid device context (code 201, ERROR_INVALID_CONTEXT)
Driver log:
  [CUDA][E] Green context used as CUcontext without cuCtxFromGreenCtx conversion
  [CUDA][E] Returning 201 (CUDA_ERROR_INVALID_CONTEXT) from cuCtxGetId
Stacktrace:
  [2] isvalid(ctx::CuContext)
  [3] check_exceptions()
  [4] synchronize(stream::CuStream; ...)

isvalid rethrew that, failing whichever test happened to run next on the worker (seen on main in build 8255, on #3439, and in GPUCompiler.jl's CUDA.jl job on its main branch and in JuliaGPU/GPUCompiler.jl#1052). This PR:

• treats ERROR_INVALID_CONTEXT like ERROR_CONTEXT_IS_DESTROYED in isvalid, as the pre-CUDA 12 code path already did. A CuContext is only constructed from a handle the driver accepted, so either error means it's gone. This also covers the other isvalid checks Don't touch destroyed contexts when releasing their resources #3404 added to the resource release paths;
• drops the exception flags of a context once it's found to be destroyed, so its handle isn't queried on every synchronization. Each of those queries also added an entry to the driver's error log, enough to push out the useful messages (the log buffer "rolled over" in build 8273);
• fixes a pre-existing race: check_exceptions iterated the Dict of flags without a lock while compiling a kernel for a new context inserted into it. The flags are now an immutable vector in an atomic field that writers replace under a lock, so the check on every synchronization stays lock-free.

I couldn't get a local driver (615.78) to report ERROR_INVALID_CONTEXT for a destroyed handle, so the regression test checks that the destroyed context is no longer tracked after synchronizing.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.69%. Comparing base (a9e28c0) to head (110873d).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3447      +/-   ##
==========================================
- Coverage   87.58%   85.69%   -1.89%     
==========================================
  Files         192      191       -1     
  Lines       19922    19762     -160     
==========================================
- Hits        17449    16936     -513     
- Misses       2473     2826     +353     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

CUDA.jl Benchmarks

Details

Benchmark suite	Current: 110873d	Previous: a9e28c0	Ratio
array/accumulate/Float32/1d	55196 ns	53368 ns	1.03
array/accumulate/Float32/dims=1	39911 ns	39146 ns	1.02
array/accumulate/Float32/dims=1L	82532 ns	82278 ns	1.00
array/accumulate/Float32/dims=2	50238 ns	49464 ns	1.02
array/accumulate/Float32/dims=2L	398519 ns	399782 ns	1.00
array/accumulate/Int64/1d	89448 ns	87737 ns	1.02
array/accumulate/Int64/dims=1	60717 ns	59754 ns	1.02
array/accumulate/Int64/dims=1L	140644 ns	139769 ns	1.01
array/accumulate/Int64/dims=2	69235 ns	68524 ns	1.01
array/accumulate/Int64/dims=2L	796980 ns	797529 ns	1.00
array/broadcast	10258 ns	10132 ns	1.01
array/broadcast launch	6716 ns	6591 ns	1.02
array/construct	892.8780487804878 ns	934.15 ns	0.96
array/copy	16984 ns	16973 ns	1.00
array/copyto!/cpu_to_gpu	208770 ns	206527 ns	1.01
array/copyto!/gpu_to_cpu	241075 ns	240008 ns	1.00
array/copyto!/gpu_to_gpu	8793 ns	8794 ns	1.00
array/iteration/findall/bool	50831 ns	47090 ns	1.08
array/iteration/findall/int	67687 ns	65086 ns	1.04
array/iteration/findfirst/bool	37067 ns	35389 ns	1.05
array/iteration/findfirst/int	36655 ns	34804 ns	1.05
array/iteration/findmin/1d	52048 ns	50387 ns	1.03
array/iteration/findmin/2d	54792 ns	51323 ns	1.07
array/iteration/logical	50516 ns	49399 ns	1.02
array/iteration/scalar	60432 ns	60128 ns	1.01
array/permutedims/2d	47446 ns	46699 ns	1.02
array/permutedims/3d	44633 ns	44729 ns	1.00
array/permutedims/4d	48457 ns	47477 ns	1.02
array/random/rand/Float32	10963 ns	10864 ns	1.01
array/random/rand/Int64	17712 ns	17540 ns	1.01
array/random/rand!/Float32	8365.666666666666 ns	8222.333333333334 ns	1.02
array/random/rand!/Int64	15574 ns	15438 ns	1.01
array/random/randn/Float32	34040 ns	33917 ns	1.00
array/random/randn!/Float32	24899 ns	24950 ns	1.00
array/reductions/mapreduce/Float32/1d	37053 ns	35344 ns	1.05
array/reductions/mapreduce/Float32/dims=1	20985 ns	19440 ns	1.08
array/reductions/mapreduce/Float32/dims=1L	58276 ns	57177 ns	1.02
array/reductions/mapreduce/Float32/dims=2	28542 ns	26865 ns	1.06
array/reductions/mapreduce/Float32/dims=2L	58713 ns	57426 ns	1.02
array/reductions/mapreduce/Int64/1d	37427 ns	35963 ns	1.04
array/reductions/mapreduce/Int64/dims=1	23227 ns	22076 ns	1.05
array/reductions/mapreduce/Int64/dims=1L	97810 ns	96669 ns	1.01
array/reductions/mapreduce/Int64/dims=2	30438 ns	28562 ns	1.07
array/reductions/mapreduce/Int64/dims=2L	84936 ns	82641 ns	1.03
array/reductions/reduce/Float32/1d	37251 ns	35477 ns	1.05
array/reductions/reduce/Float32/dims=1	21162 ns	19590 ns	1.08
array/reductions/reduce/Float32/dims=1L	58523 ns	56900 ns	1.03
array/reductions/reduce/Float32/dims=2	28898 ns	27235 ns	1.06
array/reductions/reduce/Float32/dims=2L	58549 ns	57560 ns	1.02
array/reductions/reduce/Int64/1d	37388 ns	36458 ns	1.03
array/reductions/reduce/Int64/dims=1	23174 ns	21785 ns	1.06
array/reductions/reduce/Int64/dims=1L	98450 ns	96838 ns	1.02
array/reductions/reduce/Int64/dims=2	30910 ns	28069 ns	1.10
array/reductions/reduce/Int64/dims=2L	84606 ns	82996 ns	1.02
array/reverse/1d	19835 ns	19163 ns	1.04
array/reverse/1dL	73845 ns	73140 ns	1.01
array/reverse/1dL_inplace	70859 ns	70001 ns	1.01
array/reverse/1d_inplace	12508 ns	11808 ns	1.06
array/reverse/2d	22716 ns	21957 ns	1.03
array/reverse/2dL	81276 ns	80616 ns	1.01
array/reverse/2dL_inplace	71151 ns	70530 ns	1.01
array/reverse/2d_inplace	15254 ns	14461 ns	1.05
array/sorting/1d	751451 ns	749551 ns	1.00
array/sorting/2d	195104 ns	193577 ns	1.01
array/sorting/by	929757 ns	925685 ns	1.00
cuda/graph/capture	35876 ns	35335 ns	1.02
cuda/graph/eager	33301 ns	33696 ns	0.99
cuda/graph/launch	14032 ns	13916 ns	1.01
cuda/synchronization/context/auto	2319.222222222222 ns	2327.1111111111113 ns	1.00
cuda/synchronization/context/blocking	880.3 ns	896.9761904761905 ns	0.98
cuda/synchronization/context/nonblocking	10436 ns	10394 ns	1.00
cuda/synchronization/stream/auto	891.75 ns	869.9464285714286 ns	1.03
cuda/synchronization/stream/blocking	766.6448598130842 ns	774.0733944954128 ns	0.99
cuda/synchronization/stream/nonblocking	8744.333333333334 ns	8489 ns	1.03
integration/byval/reference	147591 ns	147636 ns	1.00
integration/byval/slices=1	148697 ns	148857 ns	1.00
integration/byval/slices=2	290626 ns	290245 ns	1.00
integration/byval/slices=3	432331 ns	432133 ns	1.00
integration/cudadevrt	104983 ns	104872 ns	1.00
integration/volumerhs	9610794 ns	9610189 ns	1.00
kernel/indexing	11526 ns	11356 ns	1.01
kernel/indexing_checked	12125 ns	11875 ns	1.02
kernel/launch	2389.222222222222 ns	2348.3333333333335 ns	1.02
kernel/occupancy	703.7957746478874 ns	704.8057553956835 ns	1.00
kernel/rand	15048 ns	14971 ns	1.01
latency/import	4104982898 ns	4087629357 ns	1.00
latency/precompile	5030998895 ns	5041187065 ns	1.00
latency/ttfp	4973539429 ns	4978622722 ns	1.00

This comment was automatically generated by workflow using github-action-benchmark.