[GLUTEN-12613][VL] Align VeloxBloomFilterAggregate buffer capacity with native bloom_filter_agg#12614
Open
brijrajk wants to merge 1 commit into
Open
Conversation
Contributor
Author
|
@zhztheplayer @philo-he this is the capacity-alignment fix split out per the sequencing discussed on #12151. Once this merges I'll update #12151 to add the injectFinal registration on top. |
Comment on lines
+62
to
+68
| private lazy val capacityFromNumBits: Int = { | ||
| val numBits = numBitsExpression.eval().asInstanceOf[Number].longValue | ||
| val maxNumBits = SQLConf.get | ||
| .getConfString("spark.sql.optimizer.runtime.bloomFilter.maxNumBits", "67108864") | ||
| .toLong | ||
| Math.max(1, Math.toIntExact(Math.min(numBits, maxNumBits) / 16)) | ||
| } |
Member
There was a problem hiding this comment.
Can we add a comment explaining how this code is aligned with Velox?
Contributor
Author
There was a problem hiding this comment.
Done -- added a comment citing the native code (velox/functions/sparksql/aggregates/BloomFilterAggAggregate.cpp, computeCapacity()) this formula mirrors, and explaining why the two engines must agree here.
…th native bloom_filter_agg VeloxBloomFilterAggregate's JVM-side buffer sizing (createAggregationBuffer) derived capacity purely from estimatedNumItems, ignoring numBits entirely. The native bloom_filter_agg Velox function derives capacity purely from numBits (min(numBits, maxNumBits) / 16), ignoring the raw item count once numBits is known. For the same input arguments, this produced deterministically different bit-array sizes (e.g. 16,777,216 bits on JVM vs 8,388,608 bits natively for Spark's plain defaults). Two-phase aggregation runs partial and final stages as separate physical operators that can independently land on the JVM or on native Velox. When they disagree on capacity, Velox's BloomFilter::merge silently corrupts the result instead of throwing, since its size-match guard is a VELOX_DCHECK compiled out in release builds -- values inserted relative to one array size are later queried relative to a different array size, producing bloom filter false negatives. Fix: size the JVM buffer from numBits using the same formula as the native aggregate (velox/functions/sparksql/aggregates/BloomFilterAggAggregate.cpp, computeCapacity()), so both engines agree on capacity for identical arguments regardless of which engine executes which stage. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
zhztheplayer
approved these changes
Jul 24, 2026
brijrajk
force-pushed
the
fix/12613-bloom-filter-capacity-mismatch
branch
from
July 24, 2026 17:46
fbcdd2a to
33660e1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Fixes #12613.
VeloxBloomFilterAggregate(the JVM-side expression used for Velox bloom filters) and thenative
bloom_filter_aggVelox function size their internal bit array differently, even whengiven the identical
estimatedNumItems/numBitsarguments:VeloxBloomFilterAggregate.createAggregationBuffer()calledVeloxBloomFilter.empty(estimatedNumItems), sizing the buffer purely from the raw itemcount and ignoring
numBitsentirely.BloomFilterAggAggregate.cppcomputescapacity_ = min(numBits, maxNumBits) / 16and ignores the raw item count once
numBitsis known.For example, given
estimatedNumItems=1000000, numBits=8388608(Spark's plain defaults),the JVM side allocated a 16,777,216-bit buffer while the native side allocated an
8,388,608-bit buffer -- exactly 2x different, deterministically, for any query using this
code path.
Why this matters
Two-phase aggregation runs the partial and final stages as separate physical operators,
which can independently land on the JVM or on native Velox (e.g. via Gluten's whole-stage
fallback policy, or via
spark.gluten.sql.columnar.hashagg.enabled=false). When the partialaggregate runs on one engine and the final aggregate (which merges partial buffers) runs on
the other, the merge combines two differently-sized bit arrays.
Velox's
BloomFilter::merge(velox/common/base/BloomFilter.h) guards the size match withVELOX_DCHECK_EQ, which is compiled out in release builds, so the mismatched merge proceedssilently instead of throwing. The result is silent data corruption: bits inserted relative
to one array size are later queried relative to a different array size, causing bloom
filter false negatives (values that were definitely inserted are reported as absent).
The fix
Make
VeloxBloomFilterAggregate's JVM-side buffer sizing use the same formula as the nativeside (derive capacity from
numBits, not from rawestimatedNumItems), so both enginesagree on capacity for the same input arguments regardless of which engine executes which
stage.
How was this patch tested?
Added a regression test to
GlutenBloomFilterAggregateQuerySuitethat runs the samebloom_filter_aggquery fully natively and fully on the JVM(
spark.gluten.sql.columnar.hashagg.enabled=false), and asserts the two produceidentically-sized serialized bytes.
Verified the test actually catches the bug: reverting only the fix (keeping the new test)
causes exactly this one test to fail, with 14 other tests in the same suite unaffected.
With the fix, all 15 tests in
GlutenBloomFilterAggregateQuerySuitepass.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5 (Anthropic)