Decode bitmap containers into array containers with VPCOMPRESSB - #561
Merged
Conversation
fillArray and the fillArrayAND/ANDNOT/XOR helpers all carried a
"TODO: rewrite in assembly". Add the uint16 counterpart of the kernel
introduced for ToArray: one VPCOMPRESSB per word, widened 32 at a time with
VPMOVZXBW and written with a masked store, so exactly popcount(word) values
are written and the output needs no slack.
fillArraySkipVector additionally tests eight words at a time with VPTESTMQ
and skips empty groups, which wins only on very sparse containers;
fillArray picks between the two on cardinality, crossover just under 768.
The scalar loops keep their exact previous bodies, with only an early-out
added in front, so the non-AVX-512 path is textually unchanged.
Xeon Gold 6548N, ns per value:
fillArray card=64 10.21 -> 2.567 (3.98x)
fillArray card=1024 1.797 -> 1.118 (1.61x)
fillArray card=4096 0.911 -> 0.372 (2.45x)
fillArrayAND out=1044 1.678 -> 1.076 (1.56x)
fillArrayAND out=4076 0.903 -> 0.392 (2.30x)
The sparsest two-input case (out=249) is about 9% slower: at that density
the cost is scanning both containers, not emitting the values, so the
kernel has nothing to win. A size threshold made it worse rather than
better because inserting the guard shifts the scalar loop's alignment. It
does not show end to end -- andBitmap at that size is 1.01x -- and a
VPTESTMQ group-skip variant should fix it properly as a follow-up.
Claude-Session: https://claude.ai/code/session_0123ePBefqPjrCvhxdwWFakj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
bitmapContainer.fillArrayand thefillArrayAND/fillArrayANDNOT/fillArrayXORhelpers all carried a// TODO: rewrite in assembly. This does that, with the uint16 counterpart of the kernel in #557.Stacked on #557 - it reuses the
useVectorFillgate from that PR, so the base branch isvbmi2-toarray, notmaster.Type of Change
Changes Made
Five kernels in one new assembly file, same shape as
fillbits_vbmi2_amd64.s: oneVPCOMPRESSBper 64-bit word turns the word into 64 byte-sized bit positions, widened 32 at a time withVPMOVZXBWand written with a masked store, so exactlypopcount(word)uint16s are written and the output needs no slack. The AND/ANDNOT/XOR variants combine two containers word by word before emitting.fillArraySkipVectoradditionally tests eight words at a time withVPTESTMQand skips wholly empty groups. It wins only when the container is very sparse;bitmapContainer.fillArraypicks between the two on the container's cardinality, with a crossover measured just under 768.The scalar loops in
util.gokeep their exact previous bodies - only an early-out is added in front of each - so the non-AVX-512 path is textually unchanged.Performance Impact
Xeon Gold 6548N (Emerald Rapids), pinned with
taskset,-benchtime=2s, ns per value.bitmapContainer.fillArray:fillArrayAND, sized so the result stays at or belowarrayDefaultMaxSize(the only regime it is reached in):The sparsest two-input case is about 9% slower and I could not remove it. At that density the cost is scanning both 8 KB containers, not emitting the ~250 values that survive, so the vector kernel has nothing to win. Routing it back to the scalar loop on a size threshold made it worse (3.60), not better: inserting the guard shifts the scalar inner loop's alignment, and that loop is extremely alignment-sensitive here - in one build I measured byte-identical code 51% apart. Always taking the vector path was the best of the options I measured. A
VPTESTMQgroup-skip variant for the two-input kernels should fix it properly, since roughly 78% of the AND words are zero at that density; that is a follow-up.It does not show end to end. Whole container operations, ns/op:
andBitmapsrc=4096andBitmapsrc=8192andBitmapsrc=16384andNotBitmapsrc=4096Testing
go test ./...,roaring64, the smat corpus and smat-hit checks,go tool unconvert,gofmt,go vet, and cross-builds for 386/arm/arm64 and-tags appengine, on VBMI2 hardware so the kernels execute.TestFillArrayMatchesScalarchecks all four entry points at cardinalities 0, 1, 31, 32, 33, 63, 64, 65, 511, 512, 513, 4096, 30000, 65535 and 65536 - straddling the skip/compress crossover, the 32-values-per-store boundary and the 64-values-per-word boundary - against values derived independently from the source words rather than from the scalar implementation.Breaking Changes
None.
Additional Notes
A note on the benchmarks: at these sizes the scalar inner loop is dominated by branch prediction and is very sensitive to code alignment. Short
-benchtime=Nxruns moved identical code by up to 51%. All the numbers above use-benchtime=2swith the benchmark run alone in its own process, which was stable to under 1%.https://claude.ai/code/session_0123ePBefqPjrCvhxdwWFakj