GH-50149: [C++][Parquet] Handle OOM as soft failure in parquet encoding fuzzer#50154
Closed
omertt27 wants to merge 1 commit into
Closed
GH-50149: [C++][Parquet] Handle OOM as soft failure in parquet encoding fuzzer#50154omertt27 wants to merge 1 commit into
omertt27 wants to merge 1 commit into
Conversation
…encoding fuzzer When the CappedMemoryPool (2.2 GB limit) triggers OOM during encoding roundtrip, the resulting Status::OutOfMemory was passed to ARROW_CHECK_OK or .ValueOrDie(), both of which call ARROW_LOG(FATAL) -> std::abort(). This is not an exception, so BEGIN_PARQUET_CATCH_EXCEPTIONS cannot intercept it, and OSS-Fuzz sees a process crash instead of a resource-limit event. Add FuzzCheckOk(Status) in the anonymous namespace: returns the status if OutOfMemory (soft failure, propagates up to LogFuzzStatus which already handles it), calls ARROW_CHECK_OK for any other non-OK status (hard abort preserved so OSS-Fuzz still catches real decoder/encoder bugs). Six call sites fixed in TypedFuzzEncoding::Fuzz(); three invariant checks (decoder value count, BinaryBuilder state, byte count) intentionally left as hard aborts because they indicate actual implementation bugs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds fuzzing-specific status handling to treat OutOfMemory as a non-fatal outcome while ensuring all other non-OK statuses abort, improving OSS-Fuzz signal quality.
Changes:
- Introduce
FuzzCheckOkhelper to allow OOM to bubble up while aborting on other failures. - Replace several
ARROW_CHECK_OK/ValueOrDie()call sites withRETURN_NOT_OK(FuzzCheckOk(...))to avoid hard aborts on OOM. - Propagate OOM through decoding/validation/compare and chunked decoding paths.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Member
|
Closing this PR as it's superseded by #50150 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
When the
CappedMemoryPool(2.2 GB limit, added in GH-48105) triggers an OOM during encoding roundtrip verification, the resultingStatus::OutOfMemorypropagates back to call sites that usedARROW_CHECK_OK(...)or.ValueOrDie(). Both of these expand toARROW_LOG(FATAL)→std::abort(), which is not an exception —BEGIN_PARQUET_CATCH_EXCEPTIONScannot intercept it. OSS-Fuzz then sees a process crash instead of a resource-limit event.What changes are included in this PR?
Added
FuzzCheckOk(Status)in the anonymous namespace offuzz_encoding_internal.cc:Six call sites in
TypedFuzzEncoding::Fuzz()replaced withRETURN_NOT_OK(FuzzCheckOk(...)):reference_array_->ValidateFull()DecodeArrow(...).ValueOrDie()(replaced with status check +std::move(*result))array->ValidateFull()(on roundtrip result)CompareAgainstReference(array)RunOnDecodedChunks(...)× 2 (botharrow_supported()and non-Arrow branches)Three invariant checks intentionally left as hard aborts — they indicate actual decoder/encoder bugs, not resource limits:
ARROW_CHECK_LE(values_read, read_size)— decoder returning more values than requestedARROW_CHECK_EQ(acc.chunks.size(), 0)— BinaryBuilder invariantARROW_CHECK_EQ(offset, total_data_size)— byte count invariant inMakeArrowAre these changes tested?
The OOM path is exercised by
fuzzing_memory_pool()infuzz_internal.ccwhenever the cumulative allocation exceeds the 2.2 GB cap. The existingparquet-encoding-testsuite covers the non-OOM code paths.Closes #50149