Resolve compiled CPU kernels through a fast path - #769
Merged
Merged
Conversation
giordano
added this pull request to stack #770
September 11, 2026 22:25
giordano
force-pushed
the
mg/launch-allocations
branch
from
September 11, 2026 22:25
b8c45de to
ef359b7
Compare
Contributor
Benchmark ResultsShow table
Benchmark PlotsA plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR. |
giordano
force-pushed
the
mg/launch-allocations
branch
from
September 12, 2026 17:30
ef359b7 to
15be7fd
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #769 +/- ##
==========================================
- Coverage 63.73% 61.28% -2.45%
==========================================
Files 23 23
Lines 1941 2162 +221
==========================================
+ Hits 1237 1325 +88
- Misses 704 837 +133 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Every launch went through `clfunction`, which built a compiler config,
looked up the method instance, allocated a `CompilerJob`, queried
GPUCompiler's cache and hashed its way into `_kernel_instances` before
returning the `HostKernel` it had returned last time. Keep the resolved
`HostKernel` per `HostKernel{F, tt}` type together with the world age and
context it was resolved in, and return it directly while no method has
been defined since and the context is unchanged. Reflection (a set
`compile_hook`) and explicit compiler keywords bypass the fast path.
Reading the `compile_hook` scoped value allocates, so it is only read
when a dynamic scope is active. The `Ref` used to find the kernel for the
current context and the `@something` in `compile_or_lookup` are replaced
with plain control flow, which also removes a boxed variable captured by
the `get!` closure.
A warm launch now allocates once, for the event handle.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VHciC8x39gm97sABrSvBkt
vchuravy
force-pushed
the
mg/launch-allocations
branch
from
September 13, 2026 12:21
15be7fd to
4cf9918
Compare
maleadt
requested changes
Sep 14, 2026
maleadt
left a comment
maleadt
left a comment
Member
There was a problem hiding this comment.
This looks questionable. AFAIU, it adds an ad-hoc caching layer over GPUCompiler, for the sake of saving 2 tiny (and thus fast) allocations? That doesn't seem worth it. It's also wrong: _kernel_fastpath is only caching on HostKernel{F,tt}, so distinct callable objects with the same type therefore return the first cached HostKernel, including its stored f.
Avoid duplicating GPUCompiler cache invalidation in a separate fast path. Instead, let compile_or_lookup specialize on the bounded back-end job type so CompilerJob remains allocation-free, and keep the method out of line to avoid duplicating it per kernel. This also preserves distinct HostKernel instances for same-type callables carrying different state.
Use a separate recursive helper so a leading Int kernel argument is not interpreted as the argument index.
Member
|
I don't think this needs a secondary cache, just some tweaks on how things are specialized. |
Member
|
Benchmarks look good. |
This was referenced Sep 14, 2026
Collaborator
Author
|
Is this good to go now? 👀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of the redo of #766 in this repo