roaring64: add portable 64-bit serialization API - #562
Merged
Conversation
Go equivalent of RoaringBitmap/RoaringBitmap#847, which added serializePortable / deserializePortable / portableSerializedSizeInBytes to the ART-backed Roaring64Bitmap. WriteTo/ReadFrom already speak the portable format, so WritePortableTo, ToPortableBytes and GetPortableSerializedSizeInBytes are named aliases rather than a second encoder. ReadPortableFrom adds the guarantees the Java PR introduced: the bucket count is bounded before any allocation, bucket keys must be strictly increasing, empty buckets are dropped, and the read is atomic -- a truncated or malformed stream leaves the receiver untouched. Nothing existing is modified. Claude-Session: https://claude.ai/code/session_014hucxKZQmUW5h6Homd2Q6m
The ppc64 CI job runs the compiled roaring64 test binary from the repository root, so a relative testdata path does not resolve. Locate the fixtures from the test source file instead. Claude-Session: https://claude.ai/code/session_014hucxKZQmUW5h6Homd2Q6m
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Go equivalent of RoaringBitmap/RoaringBitmap#847, which added
serializePortable/deserializePortable/portableSerializedSizeInBytesto the ART-backedRoaring64Bitmap.Unlike Java,
roaring64has only one 64-bit implementation andWriteTo/ReadFromalready speak the portable format. So this PR adds the explicitly named API surface, and with it the validation and atomicity guarantees the Java PR introduced. Nothing existing is modified — all new code lives inroaring64/serialization_portable.go.serializePortable(DataOutput)WritePortableTo(io.Writer) (int64, error)ToPortableBytes() ([]byte, error)deserializePortable(DataInput)ReadPortableFrom(io.Reader) (int64, error)portableSerializedSizeInBytes()GetPortableSerializedSizeInBytes() uint64WritePortableToandGetPortableSerializedSizeInBytesdelegate toWriteToandGetSerializedSizeInBytes: that path already emits the portable format, so these are named aliases rather than a second encoder.ReadPortableFromis where the behaviour differs fromReadFrom:uint32and must be strictly increasing, so a stream holds at most 2^32 buckets; anything larger is rejected.ReadFrompasses the declared count straight tomake([]uint32, size), so a corrupt header can request an enormous allocation. Preallocation here is additionally capped at 1024 entries, so a large-but-legal count costs nothing until the buckets actually arrive.Validatechecks.roaringArray64that is assigned only on success, so a truncated or malformed stream leaves the receiver untouched.ReadFromresizes the receiver up front and leaves it half-overwritten on failure.The copy-on-write setting is carried across the replacement, matching
ReadFrom.Note the bound is inclusive of 2^32, which differs by one from the Java PR:
#847rejectsbucketCount > MAX_UNSIGNED_INT(2^32-1), so it cannot represent a bitmap in which every one of the 2^32 high-32-bit prefixes is occupied. Unreachable in practice — that is an ~85-95 GB stream — but the inclusive bound is the one that matches the format.Container contents inside each 32-bit bucket are not validated — same as the Java PR, and consistent with the existing Go contract where the 32-bit
ReadFromdoesn't validate andValidate/MustReadFromare opt-in. This is documented on the method.Tests
roaring64/serialization_portable_test.go, plus the four CRoaring fixtures from the Java repo (64mapempty,64map32bitvals,64mapspreadvals,64maphighvals) inroaring64/testdata/. Coverage: byte-identical fixture round-trips with exact size prediction and cross-check againstReadFrom; rejection of an out-of-range bucket count and of duplicate/decreasing keys; empty-bucket dropping; atomicity at three truncation points; copy-on-write preservation; and a run-optimized round-trip.https://claude.ai/code/session_014hucxKZQmUW5h6Homd2Q6m