Skip to content

Cut allocations when launching CPU kernels - #768

Merged
vchuravy merged 1 commit into
mainfrom
mg/launch-allocations-p1
Sep 13, 2026
Merged

Cut allocations when launching CPU kernels#768
vchuravy merged 1 commit into
mainfrom
mg/launch-allocations-p1

Conversation

@giordano

@giordano giordano commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Redo of #766 in this repo, only first part

@giordano giordano mentioned this pull request Sep 11, 2026
Closed
@giordano
giordano added this pull request to stack #770 September 11, 2026 22:25
@giordano
giordano force-pushed the mg/launch-allocations-p1 branch from ca28982 to afd85a2 Compare September 11, 2026 22:25
@christiangnrd

This comment was marked as outdated.

Sign in to view
@github-actions

github-actions Bot commented Sep 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmark Results

Show table
main 1347c84... main / 1347c84...
launch/3D static workgroup, dynamic ndrange 22.4 ± 23 μs 21.9 ± 21 μs 1.02 ± 1.5
launch/3D static workgroup, static ndrange 23.3 ± 23 μs 21.7 ± 21 μs 1.08 ± 1.5
launch/dynamic workgroup, dynamic ndrange 21.2 ± 23 μs 22.7 ± 22 μs 0.934 ± 1.3
launch/dynamic workgroup, dynamic ndrange, workgroupsize given 22.4 ± 23 μs 21.1 ± 21 μs 1.06 ± 1.5
launch/static workgroup, dynamic ndrange 21.7 ± 23 μs 20.7 ± 19 μs 1.05 ± 1.5
launch/static workgroup, static ndrange 22.9 ± 23 μs 21.5 ± 19 μs 1.06 ± 1.4
partition/dynamic workgroup, dynamic ndrange 0.0338 ± 0.016 μs 0.0328 ± 0.016 μs 1.03 ± 0.7
partition/static workgroup, dynamic ndrange 0.0409 ± 0.0081 μs 0.0402 ± 0.0089 μs 1.02 ± 0.3
partition/static workgroup, static ndrange 1.12 ± 0.01 ns 1.54 ± 0.01 ns 0.727 ± 0.008
saxpy/default/Float16/1024 24.6 ± 23 μs 22.9 ± 23 μs 1.07 ± 1.5
saxpy/default/Float16/1048576 0.206 ± 0.013 ms 0.206 ± 0.015 ms 1 ± 0.098
saxpy/default/Float16/16384 31.6 ± 18 μs 30.5 ± 19 μs 1.04 ± 0.88
saxpy/default/Float16/2048 25.5 ± 22 μs 24 ± 22 μs 1.06 ± 1.3
saxpy/default/Float16/256 22.7 ± 23 μs 0.0321 ± 0.023 ms 0.708 ± 0.89
saxpy/default/Float16/262144 0.0727 ± 0.017 ms 0.0706 ± 0.017 ms 1.03 ± 0.34
saxpy/default/Float16/32768 0.0325 ± 0.018 ms 30.6 ± 18 μs 1.06 ± 0.87
saxpy/default/Float16/4096 27.5 ± 18 μs 24.7 ± 14 μs 1.11 ± 0.96
saxpy/default/Float16/512 24.2 ± 24 μs 25.6 ± 23 μs 0.945 ± 1.3
saxpy/default/Float16/64 22.1 ± 23 μs 29.3 ± 23 μs 0.753 ± 0.99
saxpy/default/Float16/65536 0.0372 ± 0.018 ms 0.0354 ± 0.019 ms 1.05 ± 0.75
saxpy/default/Float32/1024 23.9 ± 20 μs 21.9 ± 20 μs 1.09 ± 1.4
saxpy/default/Float32/1048576 0.208 ± 0.039 ms 0.202 ± 0.035 ms 1.03 ± 0.26
saxpy/default/Float32/16384 30.4 ± 18 μs 27.5 ± 16 μs 1.1 ± 0.92
saxpy/default/Float32/2048 25.2 ± 19 μs 23.2 ± 22 μs 1.08 ± 1.3
saxpy/default/Float32/256 22.7 ± 18 μs 22.1 ± 23 μs 1.03 ± 1.3
saxpy/default/Float32/262144 0.0717 ± 0.02 ms 0.0655 ± 0.02 ms 1.09 ± 0.46
saxpy/default/Float32/32768 31.6 ± 17 μs 30 ± 18 μs 1.05 ± 0.85
saxpy/default/Float32/4096 27.4 ± 18 μs 24.5 ± 17 μs 1.12 ± 1
saxpy/default/Float32/512 22.5 ± 18 μs 24 ± 23 μs 0.939 ± 1.2
saxpy/default/Float32/64 21.8 ± 18 μs 28.4 ± 23 μs 0.768 ± 0.88
saxpy/default/Float32/65536 0.0374 ± 0.018 ms 0.034 ± 0.019 ms 1.1 ± 0.81
saxpy/default/Float64/1024 24.2 ± 23 μs 22.7 ± 22 μs 1.07 ± 1.5
saxpy/default/Float64/1048576 0.271 ± 0.04 ms 0.265 ± 0.042 ms 1.02 ± 0.22
saxpy/default/Float64/16384 28.8 ± 18 μs 26.4 ± 17 μs 1.09 ± 0.97
saxpy/default/Float64/2048 24.5 ± 22 μs 22.5 ± 19 μs 1.09 ± 1.3
saxpy/default/Float64/256 21.5 ± 22 μs 27 ± 23 μs 0.795 ± 1.1
saxpy/default/Float64/262144 0.0832 ± 0.016 ms 0.0874 ± 0.019 ms 0.953 ± 0.27
saxpy/default/Float64/32768 0.0322 ± 0.018 ms 30.3 ± 18 μs 1.06 ± 0.86
saxpy/default/Float64/4096 25.2 ± 19 μs 23.3 ± 19 μs 1.08 ± 1.2
saxpy/default/Float64/512 22.4 ± 24 μs 23.1 ± 23 μs 0.967 ± 1.4
saxpy/default/Float64/64 22.3 ± 23 μs 22.6 ± 23 μs 0.988 ± 1.4
saxpy/default/Float64/65536 0.0377 ± 0.019 ms 0.0355 ± 0.019 ms 1.06 ± 0.78
saxpy/static workgroup=(1024,)/Float16/1024 23.9 ± 23 μs 22 ± 23 μs 1.08 ± 1.5
saxpy/static workgroup=(1024,)/Float16/1048576 0.206 ± 0.013 ms 0.202 ± 0.013 ms 1.02 ± 0.09
saxpy/static workgroup=(1024,)/Float16/16384 30.6 ± 18 μs 27.3 ± 18 μs 1.12 ± 0.99
saxpy/static workgroup=(1024,)/Float16/2048 27.5 ± 22 μs 22.2 ± 21 μs 1.24 ± 1.5
saxpy/static workgroup=(1024,)/Float16/256 24.8 ± 23 μs 22.4 ± 23 μs 1.11 ± 1.5
saxpy/static workgroup=(1024,)/Float16/262144 0.0718 ± 0.017 ms 0.0682 ± 0.016 ms 1.05 ± 0.35
saxpy/static workgroup=(1024,)/Float16/32768 0.0321 ± 0.018 ms 29.4 ± 18 μs 1.09 ± 0.91
saxpy/static workgroup=(1024,)/Float16/4096 27 ± 18 μs 23.6 ± 14 μs 1.14 ± 1
saxpy/static workgroup=(1024,)/Float16/512 24.3 ± 22 μs 22.1 ± 23 μs 1.1 ± 1.5
saxpy/static workgroup=(1024,)/Float16/64 24.3 ± 23 μs 21.7 ± 22 μs 1.12 ± 1.6
saxpy/static workgroup=(1024,)/Float16/65536 0.0371 ± 0.017 ms 0.0343 ± 0.018 ms 1.08 ± 0.76
saxpy/static workgroup=(1024,)/Float32/1024 22.3 ± 20 μs 22.5 ± 23 μs 0.992 ± 1.3
saxpy/static workgroup=(1024,)/Float32/1048576 0.196 ± 0.028 ms 0.189 ± 0.025 ms 1.03 ± 0.2
saxpy/static workgroup=(1024,)/Float32/16384 29.1 ± 17 μs 26.3 ± 16 μs 1.11 ± 0.93
saxpy/static workgroup=(1024,)/Float32/2048 26.2 ± 22 μs 22.5 ± 20 μs 1.17 ± 1.4
saxpy/static workgroup=(1024,)/Float32/256 24.5 ± 23 μs 22.1 ± 23 μs 1.11 ± 1.5
saxpy/static workgroup=(1024,)/Float32/262144 0.0694 ± 0.02 ms 0.0649 ± 0.021 ms 1.07 ± 0.45
saxpy/static workgroup=(1024,)/Float32/32768 31.5 ± 18 μs 28.2 ± 17 μs 1.12 ± 0.92
saxpy/static workgroup=(1024,)/Float32/4096 27.2 ± 18 μs 24.2 ± 17 μs 1.12 ± 1.1
saxpy/static workgroup=(1024,)/Float32/512 24.2 ± 22 μs 23.1 ± 22 μs 1.05 ± 1.4
saxpy/static workgroup=(1024,)/Float32/64 22.9 ± 20 μs 21.9 ± 23 μs 1.05 ± 1.4
saxpy/static workgroup=(1024,)/Float32/65536 0.0359 ± 0.018 ms 0.033 ± 0.019 ms 1.09 ± 0.83
saxpy/static workgroup=(1024,)/Float64/1024 23.3 ± 21 μs 20.9 ± 20 μs 1.11 ± 1.5
saxpy/static workgroup=(1024,)/Float64/1048576 0.271 ± 0.054 ms 0.219 ± 0.025 ms 1.24 ± 0.28
saxpy/static workgroup=(1024,)/Float64/16384 29.6 ± 18 μs 26.1 ± 16 μs 1.13 ± 1
saxpy/static workgroup=(1024,)/Float64/2048 24.3 ± 21 μs 21.8 ± 19 μs 1.12 ± 1.4
saxpy/static workgroup=(1024,)/Float64/256 28.8 ± 23 μs 21.9 ± 22 μs 1.32 ± 1.7
saxpy/static workgroup=(1024,)/Float64/262144 0.0819 ± 0.017 ms 0.0785 ± 0.015 ms 1.04 ± 0.29
saxpy/static workgroup=(1024,)/Float64/32768 0.0324 ± 0.018 ms 28.3 ± 18 μs 1.14 ± 0.95
saxpy/static workgroup=(1024,)/Float64/4096 25.8 ± 20 μs 23.4 ± 19 μs 1.1 ± 1.2
saxpy/static workgroup=(1024,)/Float64/512 25.9 ± 23 μs 25.5 ± 22 μs 1.01 ± 1.2
saxpy/static workgroup=(1024,)/Float64/64 22.4 ± 20 μs 21.4 ± 21 μs 1.05 ± 1.4
saxpy/static workgroup=(1024,)/Float64/65536 0.0387 ± 0.019 ms 0.034 ± 0.019 ms 1.14 ± 0.83
time_to_load 0.509 ± 0.006 s 0.545 ± 0.011 s 0.934 ± 0.022
main 1347c84... main / 1347c84...
launch/3D static workgroup, dynamic ndrange 0.047 k allocs: 3.06 kB 7 allocs: 0.297 kB 10.3
launch/3D static workgroup, static ndrange 0.038 k allocs: 2 kB 8 allocs: 0.328 kB 6.1
launch/dynamic workgroup, dynamic ndrange 0.054 k allocs: 2.38 kB 17 allocs: 0.469 kB 5.07
launch/dynamic workgroup, dynamic ndrange, workgroupsize given 0.042 k allocs: 2.12 kB 3 allocs: 0.172 kB 12.4
launch/static workgroup, dynamic ndrange 0.042 k allocs: 2 kB 3 allocs: 0.172 kB 11.6
launch/static workgroup, static ndrange 0.034 k allocs: 1.53 kB 4 allocs: 0.203 kB 7.54
partition/dynamic workgroup, dynamic ndrange 2 allocs: 0.0625 kB 2 allocs: 0.0625 kB 1
partition/static workgroup, dynamic ndrange 2 allocs: 32 B 2 allocs: 32 B 1
partition/static workgroup, static ndrange 0 allocs: 0 B 0 allocs: 0 B
saxpy/default/Float16/1024 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float16/1048576 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float16/16384 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float16/2048 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float16/256 0.064 k allocs: 3.22 kB 18 allocs: 0.484 kB 6.65
saxpy/default/Float16/262144 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float16/32768 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float16/4096 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float16/512 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float16/64 0.064 k allocs: 3.22 kB 18 allocs: 0.484 kB 6.65
saxpy/default/Float16/65536 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float32/1024 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float32/1048576 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float32/16384 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float32/2048 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float32/256 0.064 k allocs: 3.22 kB 18 allocs: 0.484 kB 6.65
saxpy/default/Float32/262144 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float32/32768 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float32/4096 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float32/512 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float32/64 0.064 k allocs: 3.22 kB 18 allocs: 0.484 kB 6.65
saxpy/default/Float32/65536 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float64/1024 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float64/1048576 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float64/16384 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float64/2048 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float64/256 0.064 k allocs: 3.22 kB 18 allocs: 0.484 kB 6.65
saxpy/default/Float64/262144 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float64/32768 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float64/4096 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float64/512 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/default/Float64/64 0.064 k allocs: 3.22 kB 18 allocs: 0.484 kB 6.65
saxpy/default/Float64/65536 0.074 k allocs: 3.38 kB 26 allocs: 0.609 kB 5.54
saxpy/static workgroup=(1024,)/Float16/1024 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float16/1048576 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float16/16384 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float16/2048 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float16/256 0.056 k allocs: 3.02 kB 3 allocs: 0.172 kB 17.5
saxpy/static workgroup=(1024,)/Float16/262144 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float16/32768 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float16/4096 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float16/512 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float16/64 0.056 k allocs: 3.02 kB 3 allocs: 0.172 kB 17.5
saxpy/static workgroup=(1024,)/Float16/65536 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float32/1024 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float32/1048576 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float32/16384 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float32/2048 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float32/256 0.056 k allocs: 3.02 kB 3 allocs: 0.172 kB 17.5
saxpy/static workgroup=(1024,)/Float32/262144 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float32/32768 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float32/4096 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float32/512 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float32/64 0.056 k allocs: 3.02 kB 3 allocs: 0.172 kB 17.5
saxpy/static workgroup=(1024,)/Float32/65536 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float64/1024 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float64/1048576 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float64/16384 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float64/2048 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float64/256 0.056 k allocs: 3.02 kB 3 allocs: 0.172 kB 17.5
saxpy/static workgroup=(1024,)/Float64/262144 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float64/32768 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float64/4096 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float64/512 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
saxpy/static workgroup=(1024,)/Float64/64 0.056 k allocs: 3.02 kB 3 allocs: 0.172 kB 17.5
saxpy/static workgroup=(1024,)/Float64/65536 0.059 k allocs: 3.06 kB 6 allocs: 0.219 kB 14
time_to_load 0.201 k allocs: 11.8 kB 0.201 k allocs: 11.8 kB 1

Benchmark Plots

A plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR.
Go to "Actions"->"Benchmark a pull request"->[the most recent run]->"Artifacts" (at the bottom).

@giordano

giordano commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

It'd probably be good to get #774 in first, so that we can actually track memory allocations (besides having more benchmarks)

@vchuravy

vchuravy commented Sep 12, 2026

Copy link
Copy Markdown
Member

Can you rebase to pick up the new benchmarks?

@giordano @claude
Cut allocations when launching CPU kernels
1347c84 
A warm launch of a two-argument kernel on the CPU backend allocated about
50 times (2.2 KiB). Most of it came from Julia not specializing on
`args...` that a method only splats onward, which boxed every kernel
argument on the way from `Kernel` to `clSetKernelArg`; the rest from
per-launch vectors for the work sizes and SVM pointers, a keyword splat
through the generated kernel call, and a device property query.

- Annotate the splatted arguments as `Vararg{Any, N}` in the `Kernel`
  call, the generated `AbstractKernel` call, `call` and `set_args!`, and
  unroll `set_args!` by recursion instead of `enumerate`.
- Pass each kernel argument to `clSetKernelArg` as a `Ref{T}` `ccall`
  argument, which copies it to the stack.
- Pass the global and local work sizes as three-element tuples through a
  `ccall` taking `Ref{NTuple{3, Csize_t}}` when there is no offset or
  device RNG state, skipping the `max_work_item_dims` query.
- The generated kernel call takes `global_size` and `local_size` as
  explicit keywords.
- `clconvert` and `call` take `nothing` for the SVM pointers instead of
  allocating an empty vector; no adaptor rule collects such pointers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VHciC8x39gm97sABrSvBkt
@giordano
giordano force-pushed the mg/launch-allocations-p1 branch from afd85a2 to 1347c84 Compare September 12, 2026 17:30
@codecov

codecov Bot commented Sep 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.30435% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.73%. Comparing base (7aabc96) to head (1347c84).

Files with missing lines Patch % Lines
src/pocl/nanoOpenCL.jl 88.88% 2 Missing ⚠️
Additional details and impacted files 
@@           Coverage Diff           @@
##             main     #768   +/-   ##
=======================================
  Coverage   63.72%   63.73%           
=======================================
  Files          23       23           
  Lines        1935     1941    +6     
=======================================
+ Hits         1233     1237    +4     
- Misses        702      704    +2

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@giordano

giordano commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator Author

Allocations reduction confirmed by the benchmarks 👀

@vchuravy
vchuravy merged commit 4135768 into main Sep 13, 2026
84 checks passed
@vchuravy
vchuravy deleted the mg/launch-allocations-p1 branch September 13, 2026 12:21
@giordano giordano mentioned this pull request Sep 14, 2026
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@giordano @christiangnrd @vchuravy