Repository navigation
perf: improve TTFX by type-erasing data movement - #61
Merged
Merged
Conversation
Contributor
Benchmark Results (Julia v1)Time benchmarks
Memory benchmarks
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Beforerr
force-pushed
the
push-xokklmvxvyto
branch
3 times, most recently
from
August 31, 2026 03:53
8a94a21 to
d538e41
Compare
What changed
1. Type-erased data movement (src/loading/variable.jl). readblock! is now a thin wrapper; everything below it takes Ptr{UInt8} + element size + Vector{Int} dims/strides:
- _read_records! — record range → bytes, threaded decompression compiled once per file format instead of per shape.
- _copy_block! — one strided N-d gather kernel serving both the row→column majority swap and sub-block extraction.
- Big-endian byte swap keyed on word size, not T.
2. Array(var), collect(var), and show read via readblock! directly — DiskArrays' generic indexing costs ~60 ms compile per shape. Base.summary(::CDFVariable) is now cheap (Base's was ~190 ms/type).
3. Workload (src/precompile.jl): seeds one variable per common shape (floats ≤3-D, TT2000/Epoch 1-D, strings ≤2-D) with the ops people actually run — show, attribs, Array, var[1], record slice, DateTime.. Rare shapes skipped.
refactor: VVREntry records per-leaf whether it's a CVVR
The old code took the last leaf's type for the whole variable, which is wrong when a compressed variable stores a poorly-compressing block as a plain VVR.
Beforerr
force-pushed
the
push-xokklmvxvyto
branch
from
August 31, 2026 03:58
d538e41 to
e8dc10c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed