Skip to content

roaring64: add portable 64-bit serialization API - #562

Merged
lemire merged 2 commits into
masterfrom
roaring64-portable-serialization
Sep 2, 2026
Merged

roaring64: add portable 64-bit serialization API#562
lemire merged 2 commits into
masterfrom
roaring64-portable-serialization

Conversation

@lemire

@lemire lemire commented Sep 2, 2026

Copy link
Copy Markdown
Member

Go equivalent of RoaringBitmap/RoaringBitmap#847, which added serializePortable / deserializePortable / portableSerializedSizeInBytes to the ART-backed Roaring64Bitmap.

Unlike Java, roaring64 has only one 64-bit implementation and WriteTo/ReadFrom already speak the portable format. So this PR adds the explicitly named API surface, and with it the validation and atomicity guarantees the Java PR introduced. Nothing existing is modified — all new code lives in roaring64/serialization_portable.go.

Java (#847) Go
serializePortable(DataOutput) WritePortableTo(io.Writer) (int64, error)
ToPortableBytes() ([]byte, error)
deserializePortable(DataInput) ReadPortableFrom(io.Reader) (int64, error)
portableSerializedSizeInBytes() GetPortableSerializedSizeInBytes() uint64

WritePortableTo and GetPortableSerializedSizeInBytes delegate to WriteTo and GetSerializedSizeInBytes: that path already emits the portable format, so these are named aliases rather than a second encoder.

ReadPortableFrom is where the behaviour differs from ReadFrom:

  • Bucket count is bounded before any allocation. Bucket keys are uint32 and must be strictly increasing, so a stream holds at most 2^32 buckets; anything larger is rejected. ReadFrom passes the declared count straight to make([]uint32, size), so a corrupt header can request an enormous allocation. Preallocation here is additionally capped at 1024 entries, so a large-but-legal count costs nothing until the buckets actually arrive.
  • Bucket keys must be strictly increasing. The package assumes sorted keys throughout; this enforces the invariant at the load boundary instead of trusting the input.
  • Empty buckets are dropped. The format never writes one, and appending it would violate the "no empty container" rule that Validate checks.
  • Reads are atomic. Buckets accumulate in a fresh roaringArray64 that is assigned only on success, so a truncated or malformed stream leaves the receiver untouched. ReadFrom resizes the receiver up front and leaves it half-overwritten on failure.

The copy-on-write setting is carried across the replacement, matching ReadFrom.

Note the bound is inclusive of 2^32, which differs by one from the Java PR: #847 rejects bucketCount > MAX_UNSIGNED_INT (2^32-1), so it cannot represent a bitmap in which every one of the 2^32 high-32-bit prefixes is occupied. Unreachable in practice — that is an ~85-95 GB stream — but the inclusive bound is the one that matches the format.

Container contents inside each 32-bit bucket are not validated — same as the Java PR, and consistent with the existing Go contract where the 32-bit ReadFrom doesn't validate and Validate / MustReadFrom are opt-in. This is documented on the method.

Tests

roaring64/serialization_portable_test.go, plus the four CRoaring fixtures from the Java repo (64mapempty, 64map32bitvals, 64mapspreadvals, 64maphighvals) in roaring64/testdata/. Coverage: byte-identical fixture round-trips with exact size prediction and cross-check against ReadFrom; rejection of an out-of-range bucket count and of duplicate/decreasing keys; empty-bucket dropping; atomicity at three truncation points; copy-on-write preservation; and a run-optimized round-trip.

https://claude.ai/code/session_014hucxKZQmUW5h6Homd2Q6m

Go equivalent of RoaringBitmap/RoaringBitmap#847, which added
serializePortable / deserializePortable / portableSerializedSizeInBytes
to the ART-backed Roaring64Bitmap.

WriteTo/ReadFrom already speak the portable format, so WritePortableTo,
ToPortableBytes and GetPortableSerializedSizeInBytes are named aliases
rather than a second encoder. ReadPortableFrom adds the guarantees the
Java PR introduced: the bucket count is bounded before any allocation,
bucket keys must be strictly increasing, empty buckets are dropped, and
the read is atomic -- a truncated or malformed stream leaves the
receiver untouched.

Nothing existing is modified.

Claude-Session: https://claude.ai/code/session_014hucxKZQmUW5h6Homd2Q6m
The ppc64 CI job runs the compiled roaring64 test binary from the
repository root, so a relative testdata path does not resolve. Locate
the fixtures from the test source file instead.

Claude-Session: https://claude.ai/code/session_014hucxKZQmUW5h6Homd2Q6m
@lemire
lemire merged commit 65fb6f8 into master Sep 2, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant