[WIP][POC] Pfor encoding#50088
Draft
prtkgaur wants to merge 31 commits into
Draft
Conversation
|
Thanks for opening a pull request! If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or See also: |
github-actions
Bot
added
Component: Parquet
Component: C++
awaiting review
Implements the PFOR (Patched Frame of Reference) integer compression algorithm as a standalone utility library in arrow/util/pfor/. Includes: - Cost model for optimal bit width selection (histogram-based) - Vector-level encode/decode with FOR + bit-packing + exceptions - Page-level wrapper with header, offset array, and multi-vector layout - Comprehensive unit tests covering edge cases and round-trips
Adds PFOR = 11 to the Encoding enum and wires it into the parquet read/write pipeline: - PforEncoder<DType> in encoder.cc (buffers values, calls PforWrapper::Encode) - PforDecoder<DType> in decoder.cc (decodes all values on first access) - PFOR case in column_reader.cc InitializeDataDecoder - Encoding string mapping in types.cc Supports INT32 and INT64 column types.
Benchmarks encode/decode throughput for int32/int64 across 10 data distributions inspired by Snowflake's NumericComprBenchmark: constant, sequential, small range, high-base-small-range (timestamps), with outliers (exception path), random, TPC-DS date/store/item/quantity keys. Each distribution runs at 1K/10K/100K/1M elements. Reports bytes/s, items/s, and compression ratio.
Load() now returns Result<PforVectorInfo> after the Status/Result refactoring. Use ASSERT_OK_AND_ASSIGN to properly unwrap the result in tests.
Make LoadHeader fallible: move the header-size check from Decode into LoadHeader, return Result<PforHeader>, and update Decode to use ARROW_ASSIGN_OR_RAISE. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
Replace std::memcpy / raw byte writes in PforWrapper::StoreHeader, LoadHeader, and the offset-array read/write paths with util::SafeLoadAs and util::SafeStore. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
Reject invalid packing_mode, value_byte_width mismatch, log_vector_size out of [kMin, kMax] range, and negative num_elements when loading the PFOR page header. Removes the redundant packing_mode and value_byte_width checks from Decode now that they live in LoadHeader. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
…sites Replace size_t with int64_t for max_size/comp_size to match the PforWrapper API signature, and qualify pfor::PforWrapper as ::arrow::util::pfor::PforWrapper to avoid ADL ambiguity.
Aligns with Arrow buffer conventions (Buffer::data() returns uint8_t*). Removes the reinterpret_cast<char*> at the parquet encoder/decoder call sites and switches std::vector<char> compressed buffers to std::vector<uint8_t> in the unit test and benchmark. Also fixes a pre-existing size_t / int64_t* mismatch in pfor_benchmark.cc that surfaced once the buffer pointer type was tightened. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
…th validation Per Google C++ style, replace the PforVectorInfo struct with a class that has private trailing-underscore members and getter/setter accessors. Replace std::memcpy calls in Store/Load and the exception patch loop in DecodeVector with util::SafeLoadAs / util::SafeStore. Add bit_width range validation inside Load() so callers don't have to repeat the check. Updates all access sites in pfor.cc and pfor_test.cc to go through the new accessors. Caches num_exceptions() in a local in DecodeVector so the #pragma GCC unroll can still see a constant loop bound. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
Per Google C++ style, both types become classes with private trailing-underscore members and const getters, mutable getters, and setters. Updates all access sites in pfor.cc (EncodeVector, LoadView, SerializedVectorSize, SerializeVector) and pfor_test.cc to go through the new accessors. Mirrors the corresponding ALP review fix on gh540-alp-pseudoDecimal-encoding.
…, use ctor in EncodeVector - Move the num_exceptions < 0 check from DecodeVector into PforVectorInfo::Load alongside the bit_width range check, so all loaded-data invariants are enforced at the same layer. - Use PforVectorInfo's parameterized constructor in EncodeVector instead of three separate setter calls on a default-constructed instance.
Commit 00b6318 introduced ARROW_DCHECK(bit_util::IsPowerOf2(vector_size)) in PforWrapper<T>::Encode, but vector_size is int32_t and bit_util has overloads only for int64_t and uint64_t -- the call is ambiguous and the file no longer compiles. Cast to int64_t to disambiguate. CeilDiv calls in the same file already promote to int64_t implicitly via its int64_t-only signature.
Portable C++ port of FastLanes (Afroozeh & Boncz, VLDB '23) for int32_t
columnar data. No SIMD intrinsics in the kernels — the inner lane loop
is structured (contiguous loads from packed[w*kLanes + lane], contiguous
stores to transposed[r*kLanes + lane]) so the compiler auto-vectorizes
to 4-wide NEON / 8-wide AVX2 / 16-wide AVX512 without source changes.
Layout: lane-interleaved 1024-bit format per the paper. 1024 values
pack as w u32 rows of 32 u32 lanes. FL_ORDER (8x16 -> 16x8 sub-block
transpose + 3-bit-reversal sub-block reorder) is applied OUTSIDE the
kernel: FastLanesForCodec::Encode gathers input[fromTransposed32(t)]
before packing; Decode produces output in transposed order (no scatter,
output[t] == input[fromTransposed32(t)] + min within each 1024-block).
FastLanesForCodec adds Frame-of-Reference on top:
- 2048-value chunks (2 FastLanes blocks per chunk)
- Per-chunk 5-byte header: [min(4B int32 LE)] [bit_width(1B)]
- Subtract min before packing; add back on decode
- bit_width=0 path stores no payload (constant chunk)
Files:
cpp/src/arrow/util/fastlanes/fastlanes_kernels.h
- PackBlock<W>(in, out) / UnpackBlock<W>(packed, out)
- W=32 fast path: std::memcpy
- fromTransposed32 helper
cpp/src/arrow/util/fastlanes/fastlanes_for.{h,cc}
- FastLanesForCodec::{Encode,Decode}
cpp/src/arrow/util/fastlanes/fastlanes_for_test.cc
- 5 round-trip tests (narrow range, single value, full int32 range,
multiple chunks, boundary values) — all passing
CMakeLists.txt wires the test as arrow-fastlanes-for-test.
Wires the new FastLanesForCodec into the existing pfor_comparison_benchmark harness alongside PFOR, DeltaBitPack, ZSTD, LZ4, RleBitPack, and Bss codecs. New BM_FastLanesEncode / BM_FastLanesDecode functions follow the same Gen32 + ::Apply(CustomArgs) shape; REGISTER_DATASET macro picks them up for every ClickBench dataset. Notes on the comparison: - FastLanes decoder produces output in TRANSPOSED order (output[chunk*2048 + block*1024 + t] == input[chunk*2048 + block*1024 + fromTransposed32(t)] + min). PFOR/DeltaBitPack produce flat output. The benchmark measures decoder throughput head-to-head; consumers of FastLanes output must be permutation-aware (which is the FastLanes paper's intended architecture). - num_values is rounded down to a multiple of 2048 (FastLanes chunk size) inside BM_FastLanesEncode / BM_FastLanesDecode for compatibility with the existing 102400-value test sizes. Also guards add_executable(parquet-pfor-comparison-benchmark) with if(ARROW_BUILD_BENCHMARKS) so non-benchmark configurations don't fail the cmake configure step. Bench numbers on aarch64 (102400 int32, 3-run median): EventDate decode: FastLanes 20us vs PFOR 36us vs Delta 122us EventTime decode: FastLanes 24us vs PFOR 56us vs Delta 140us GoodEvent decode: FastLanes 20us vs PFOR 34us vs Delta 119us Compression ratios match or slightly beat PFOR on every dataset tested.
FastLanesForCodec::DecodeFlat unpacks into a transposed scratch buffer
per chunk and then scatters via fromTransposed32 to produce output in
original input order — output[i] == input[i] for the encoded input.
This is the FL_ORDER inverse of the gather step in Encode.
Adds:
- DecodeFlat method + round-trip test (DecodeFlatIsIdentity) covering
4 chunks of random data. All 6 round-trip tests still pass.
- BM_FastLanesDecodeFlat in pfor_comparison_benchmark, registered in
the per-dataset macro for apples-to-apples vs PFOR / DeltaBitPack
(both of which produce flat output).
Bench (102400 int32, 3-run median, aarch64):
Dataset FL Decode FL DecodeFlat PFOR Decode Delta Decode
EventDate 20 us 108 us 38 us 123 us
EventTime 23 us 113 us 57 us 140 us
GoodEvent 20 us 107 us 35 us 119 us
The transposed-kernel decode beats every other codec by 1.5-7x. The
flat-output decode pays an ~85 us scatter cost per 100K values that
makes it slower than PFOR but still faster than DeltaBitPack. The gap
is exactly the FL_ORDER scatter — the reason FastLanes' intended
architecture keeps data in transposed order through the query.
The 8x16 -> 16x8 within-sub-block transpose is mutual-inverse with the 16x8 -> 8x16 transpose, NOT self-inverse. The previous docstring on fromTransposed32 said "Self-inverse: fromTransposed32 is also toTransposed32" — that was wrong. fromTransposed32(fromTransposed32(t)) does not equal t in general; e.g. fromTransposed32(1) = 16, fromTransposed32(16) = 2. Add the actual toTransposed32 (forward-direction mapping) and fix the docstring. Callers that need to invert a gather computed with fromTransposed32 (i.e. read out[i] = transposed["the t whose fromTransposed32(t) = i"]) must use toTransposed32(i).
Adds an additive packing-mode option to PFOR. Existing vectors round-trip
unchanged (default PackingMode::BitPack); new vectors can opt in to the
FastLanes lane-interleaved bit-packing layout via the per-vector flag.
On-disk format change (backwards-compatible):
- The 1-byte bit_width field of PforVectorInfo now packs two values:
bits 0..5 = the actual bit width (range 0..32 fits in 6 bits)
bit 7 = packing-mode flag (0 = BitPack, 1 = FastLanes)
bit 6 = reserved
- Legacy encoders only wrote the bit width, leaving high bits clear,
so they decode as PackingMode::BitPack via the new Load.
- PFOR header (page-level) is unchanged.
API:
- New enum class arrow::util::pfor::PackingMode { BitPack, FastLanes }.
- PforVectorInfo gains a packing_mode field and getter/setter.
- PforCompression<T>::EncodeVector takes an optional PackingMode (default
BitPack). FastLanes mode is only honored when num_elements equals the
FastLanes block size (1024) and T is 32-bit; otherwise it falls back
to BitPack per-vector (so tails and 64-bit values continue to work).
- PforCompression<T>::DecodeVector reads the per-vector flag and
dispatches between arrow::internal::unpack and the FastLanes kernel.
- PforWrapper<T>::Encode takes an optional PackingMode threaded down to
EncodeVector.
Decode-side perf (fused gather + FOR-add + SafeCopy):
The FL_ORDER inverse needs toTransposed32(i) — note: NOT
fromTransposed32(i), the two are mutual inverses, not self-inverse.
The scalar gather over can't be SIMD-vectorized, so
PFOR+FastLanes decode is ~2-3x slower than PFOR+BitPack end-to-end
despite the kernel itself being competitive. The win is only available
when the downstream consumer can work with data in FastLanes transposed
order (i.e. relax the flat-output contract).
Tests: 5 new tests in PforPackingModeTest cover round-trip identity for
both modes, the partial-tail fallback to BitPack, mixed-mode round-trip
through PforWrapper, and the bit_width=0 (constant vector) path. All 30
PFOR tests pass.
Benchmark: BM_PforFastLanesEncode / BM_PforFastLanesDecode added to
pfor_comparison_benchmark.cc, registered per dataset alongside the
existing 8 codec variants.
For FastLanes-encoded vectors the decoder previously always paid a
1024-element scalar FL_ORDER gather to produce flat output. That gather
is what made pfor+fastlanes 2-3x slower than pfor+bitpack overall, even
though the FastLanes unpack kernel itself is competitive.
The FastLanes paper's intended decode path is to NOT do that scatter at
all: keep the data in FastLanes stream order and let downstream
operators be permutation-aware (apply fromTransposed32 lazily, when
they need original index). This commit exposes that path.
API:
- New enum class arrow::util::pfor::OutputOrder { Flat, Transposed }.
- PforCompression<T>::DecodeVector and PforWrapper<T>::Decode take an
optional OutputOrder (default Flat, backwards-compatible).
- OutputOrder::Transposed only affects FastLanes-encoded vectors.
BitPack vectors have no permutation to skip, so they always produce
flat output regardless of the argument (mixed pages with a BitPack
tail end up flat in the tail, transposed in the full blocks).
Decoder paths in DecodeVector when packing_mode == FastLanes:
- Flat (existing): unpack -> scratch transposed[] -> fused
values[i] = SafeCopy(transposed[toTransposed32(i)] + FOR)
The toTransposed32 gather is scalar, breaks auto-vec.
- Transposed (new): unpack -> scratch transposed[] -> sequential
values[t] = SafeCopy(transposed[t] + FOR)
Pure sequential read/write, auto-vectorizes cleanly. Exceptions are
patched at toTransposed32(pos) so the stored-flat positions land in
the right transposed slots.
Tests: 4 new tests in PforOutputOrderTest cover (a) transposed output
satisfies the FL_ORDER relation, (b) manual inversion of the
permutation reconstructs the input, (c) BitPack vectors ignore the
Transposed request, (d) wrapper-level transposed decode across many
vectors. All 34 PFOR tests pass.
Benchmark: BM_PforFastLanesDecodeTransposed added, registered per
dataset. On 18 ClickBench-style datasets (102400 int32 each):
pfor+bitpack 33-57 us
pfor+fastlanes (flat) 98-108 us (0.34-0.53x — slower)
pfor+fastlanes (transp) 20-23 us (1.6-2.5x faster than bitpack)
The transposed path beats every other codec measured in the comparison
benchmark on every dataset.
Arrow's SetupCxxFlags.cmake explicitly appends '-O2' after CMake's default '-O3 -DNDEBUG' Release flags, downgrading optimization (the last -O flag wins in GCC). The comment 'Enable compiler optimizations' justifies this as a safety choice for the Arrow library at large, but for benchmarking the FastLanes path it's a measurable handicap: AutoVec relies on -O3's more aggressive inlining and loop unrolling to fully vectorize the row loop. Local override skips the append so Release builds get true -O3. Across the 18-dataset comparison benchmark, FL transposed decode picks up 10-17% (e.g. EventDate 20.3 us -> 17.8 us, TrafficSourceID 20.2 us -> 16.7 us). pfor+bitpack and the other codecs are unchanged within noise. Not intended for upstream — this is local-only for the bench. Revert with 'git checkout cpp/cmake_modules/SetupCxxFlags.cmake' to restore Arrow's default.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Doc : https://docs.google.com/document/d/1ZZOtxmq6K8pNU0npijfSglTJVkspXL5GLDKPSGj9HlA/edit?tab=t.0
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how to contribute here:
Please remove this line and the above text before creating your pull request.
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?
This PR includes breaking changes to public APIs. (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
This PR contains a "Critical Fix". (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)