[Experimental][WIP] Pfor delta encoding - #51150

Draft

prtkgaur wants to merge 84 commits into

apache:mainfrom

prtkgaur:pfor-delta-encoding

Commits on Jul 15, 2026

Add PFOR core algorithm and tests
sfc-gh-pgaur
committed
a7ac1ae
Integrate PFOR encoding into parquet encoder/decoder
sfc-gh-pgaur
committed
c8501f5
Add PFOR encoding benchmark
sfc-gh-pgaur
committed
8ac3f28
Use signed integer types consistently per Arrow style guide
sfc-gh-pgaur
committed
ff837d9
Use arrow::util::span for buffer parameters in Store/Load/Decode/Serialize
sfc-gh-pgaur
committed
f8c70e8
Return Result<T>/Status on decode paths instead of ARROW_DCHECK
sfc-gh-pgaur
committed
65b29f1
Add static_assert(ARROW_LITTLE_ENDIAN) and replace reinterpret_cast with SafeCopy
sfc-gh-pgaur
committed
7c5e6e4
Use single-call unpack() instead of manual batch loop + BitReader remainder
sfc-gh-pgaur
committed
0c5ef3e
Add pragma GCC unroll/ivdep to decode loops for better vectorization
sfc-gh-pgaur
committed
a99dee8
Add PforEncodedVectorView for zero-copy decode path
sfc-gh-pgaur
committed
c1a8eb5
Make vector_size configurable on encode path with default kPforVectorSize
sfc-gh-pgaur
committed
49ef1ce
Fix pfor_test.cc: unwrap Result<> from PforVectorInfo::Load()
sfc-gh-pgaur
committed
6f3a5bf
Convert PforWrapper::LoadHeader to return Result<PforHeader>
sfc-gh-pgaur
committed
cd2ed56
Use SafeLoadAs/SafeStore for header and offset array in PforWrapper
sfc-gh-pgaur
committed
b6aae63
Validate header fields in PforWrapper::LoadHeader
sfc-gh-pgaur
committed
374eae6
Use int64_t and global ::arrow:: prefix at PFOR encoder/decoder call sites
sfc-gh-pgaur
committed
87564b8
Use uint8_t* consistently in PforWrapper API instead of char*
sfc-gh-pgaur
committed
4f305b9
Convert PforVectorInfo to class with SafeLoadAs/SafeStore and bit_width validation
sfc-gh-pgaur
committed
9d5e82c
Convert PforEncodedVector and PforEncodedVectorView to classes
sfc-gh-pgaur
committed
5583424
Tighten PforVectorInfo encapsulation: validate num_exceptions in Load, use ctor in EncodeVector
sfc-gh-pgaur
committed
d801fe4
make
sfc-gh-pgaur
committed
9fabb7e
Use bit_util::IsPowerOf2/CeilDiv and add incremental encode/decode TODOs (PFOR)
sfc-gh-pgaur
committed
384e89f
Drop Snowflake attribution from PFOR header comments
sfc-gh-pgaur
committed
9d5470c
Fix IsPowerOf2(int32_t) ambiguity in pfor_wrapper.cc
sfc-gh-pgaur
committed
9bef47b
Add FastLanes auto-vectorized bit-packing library
sfc-gh-pgaur
committed
6ae3c9b
Add FastLanes-FOR to parquet-pfor-comparison-benchmark
sfc-gh-pgaur
committed
822f91b
FastLanes-FOR: add DecodeFlat (flat output) and benchmark it
sfc-gh-pgaur
committed
ef3e746
FastLanes: add toTransposed32; correct fromTransposed32 docstring
sfc-gh-pgaur
committed
4037bb2
PFOR: add PackingMode (BitPack | FastLanes) per-vector flag
sfc-gh-pgaur
committed
26ea3c4
PFOR: add OutputOrder { Flat, Transposed } decode option
sfc-gh-pgaur
committed
51f2568
Local: keep -O3 from CMake Release defaults (drop -O2 downgrade)
sfc-gh-pgaur
committed
d9ef31b

Commits on Jul 20, 2026

Commits on Jul 24, 2026

Commits on Jul 25, 2026

Add int64 (BIGINT) columns to the PFOR comparison benchmark
sfc-gh-pgaur
committed
8b5453c

Commits on Aug 21, 2026

Note the frame-of-reference bias fold as a TODO on the decode path
sfc-gh-pgaur
committed
db68831
Add a bias parameter to arrow::internal::unpack
sfc-gh-pgaur
committed
5f12e15
Keep unpack_full at memcpy speed when it carries a bias
sfc-gh-pgaur
committed
152e1d6
Reflow an unpack call site to clang-format 18
sfc-gh-pgaur
committed
8cae5e6
Store the PFOR bit width in 7 bits, not 6
sfc-gh-pgaur
committed
50fcf20
Fold the frame-of-reference add into the PFOR unpacker
sfc-gh-pgaur
committed
b0978e8

Commits on Aug 23, 2026

Spell the restrict qualifier portably in unpack_full
sfc-gh-pgaur
committed
ba66ee7

Commits on Aug 25, 2026

Fix two errors PFOR raises at the checkin warning level
sfc-gh-pgaur
committed
d995235
List PFOR in SupportedEncodings for INT32 and INT64
sfc-gh-pgaur
committed
1d36687
Use std::bit_width instead of __builtin_clz in PFOR
sfc-gh-pgaur
committed
6a285b4
Apply clang-format 18 to the PFOR sources
sfc-gh-pgaur
committed
aeffcb4
Reject a PFOR page header that disagrees with its buffer
sfc-gh-pgaur
committed
1a77108
Count PFOR exceptions in an unsigned field
sfc-gh-pgaur
committed
db4f45f
Put the output buffer last in PforWrapper::Decode
sfc-gh-pgaur
committed
f54ffdf
Return Status from PforWrapper::Encode
sfc-gh-pgaur
committed
9d302f6
Decode PFOR pages with null slots through the Arrow path
sfc-gh-pgaur
committed
f3e40a8
Note that incremental PFOR encode and decode come later
sfc-gh-pgaur
committed
1c304ed

Commits on Aug 27, 2026

Commits on Sep 3, 2026

Take the PFOR value count from the page's own header
sfc-gh-pgaur
committed
7dd4a9c
Validate the PFOR element count and offset chain before decoding
sfc-gh-pgaur
committed
16be70a
Encode an all-null PFOR page as a bare header
sfc-gh-pgaur
committed
f150979
Serialize PFOR little-endian instead of refusing to build
sfc-gh-pgaur
committed
239898f
Build the PFOR sources under meson too
sfc-gh-pgaur
committed
9d57072
Record what a batched PFOR read costs today
sfc-gh-pgaur
committed
3570bf1
Add end-to-end PFOR tests through the Arrow reader and writer
sfc-gh-pgaur
committed
87fdc1d
Name the PFOR headers _internal.h
sfc-gh-pgaur
committed
b5c850f
Give PFOR a delta mode and a frame it can search
sfc-gh-pgaur
committed
96c7a11
Cut the frame search's cost, and lower its frame onto a real value
sfc-gh-pgaur
committed
1d97580
Add the delta-mode column shapes to the codec comparison benchmark
sfc-gh-pgaur
committed
56e2003
Decline the PFOR delta mode from a sampled estimate
sfc-gh-pgaur
committed
10bd2d7
Gate PFOR and its delta mode behind writer properties
sfc-gh-pgaur
committed
06cd8cb
Harden the PFOR delta flag against a corrupt page
sfc-gh-pgaur
committed
55cb7ee
Note that the delta prefix sum could fold into the unpack kernel
sfc-gh-pgaur
committed
f2e3609