forked from bitcoin/bitcoin
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
ci: add fuzz regression testing and continuous fuzzing infrastructure #7173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
thepastaclaw
wants to merge
9
commits into
dashpay:develop
Choose a base branch
from
thepastaclaw:ci/fuzz-regression
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
07b6f96
ci: add fuzz regression testing and continuous fuzzing infrastructure
thepastaclaw 093a849
test(fuzz): add Dash-specific deserialization and roundtrip fuzz targets
thepastaclaw 6178538
test(fuzz): add BLS cryptographic operations fuzz target
thepastaclaw e5395e6
test(fuzz): add CoinJoin protocol fuzz targets
thepastaclaw 3497a3c
test(fuzz): add governance proposal validation fuzz target
thepastaclaw 9531e45
test(fuzz): add special transaction and asset lock/unlock fuzz targets
thepastaclaw d38d7f9
test(fuzz): add LLMQ message and deterministic MN list fuzz targets
thepastaclaw 98860cf
test(fuzz): add Dash P2P message processing fuzz target
thepastaclaw 3ced565
build: register new Dash fuzz targets in Makefile.test.include
thepastaclaw File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,173 @@ | ||
| name: Fuzz regression | ||
|
|
||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| bundle-key: | ||
| description: "Key needed to access bundle of fuzz build artifacts" | ||
| required: true | ||
| type: string | ||
| build-target: | ||
| description: "Target name as defined by inputs.sh" | ||
| required: true | ||
| type: string | ||
| container-path: | ||
| description: "Path to built container at registry" | ||
| required: true | ||
| type: string | ||
| runs-on: | ||
| description: "Runner label to use" | ||
| required: false | ||
| default: ubuntu-24.04 | ||
| type: string | ||
|
|
||
| jobs: | ||
| fuzz-regression: | ||
| name: Fuzz regression | ||
| runs-on: ${{ inputs.runs-on }} | ||
| container: | ||
| image: ${{ inputs.container-path }} | ||
| options: --user root | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| ref: ${{ github.event.pull_request.head.sha }} | ||
| fetch-depth: 1 | ||
|
|
||
| - name: Download build artifacts | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| name: ${{ inputs.bundle-key }} | ||
|
|
||
| - name: Extract build artifacts | ||
| run: | | ||
| git config --global --add safe.directory "$PWD" | ||
| export BUILD_TARGET="${{ inputs.build-target }}" | ||
| export BUNDLE_KEY="${{ inputs.bundle-key }}" | ||
| ./ci/dash/bundle-artifacts.sh extract | ||
| shell: bash | ||
|
|
||
| - name: Download corpus | ||
| run: | | ||
| mkdir -p /tmp/fuzz_corpus | ||
|
|
||
| # Layer 1: bitcoin-core inherited corpus | ||
| if git clone --depth=1 https://github.com/bitcoin-core/qa-assets /tmp/qa-assets; then | ||
| if [ -d "/tmp/qa-assets/fuzz_corpora" ]; then | ||
| cp -r /tmp/qa-assets/fuzz_corpora/. /tmp/fuzz_corpus/ | ||
| echo "Loaded bitcoin-core corpus" | ||
| fi | ||
| else | ||
| echo "WARNING: Failed to clone bitcoin-core/qa-assets (non-fatal)" | ||
| fi | ||
|
|
||
| # Layer 2: Dash-specific corpus (overlays on top) | ||
| if git clone --depth=1 https://github.com/dashpay/qa-assets /tmp/dash-qa-assets; then | ||
| if [ -d "/tmp/dash-qa-assets/fuzz/corpora" ]; then | ||
| cp -r /tmp/dash-qa-assets/fuzz/corpora/. /tmp/fuzz_corpus/ | ||
| echo "Loaded Dash-specific corpus" | ||
| fi | ||
| else | ||
| echo "WARNING: Failed to clone dashpay/qa-assets (non-fatal)" | ||
| fi | ||
|
|
||
| # Layer 3: Generate synthetic seeds for Dash-specific targets | ||
| if [ -f "contrib/fuzz/seed_corpus_from_chain.py" ]; then | ||
| python3 contrib/fuzz/seed_corpus_from_chain.py --synthetic-only -o /tmp/fuzz_corpus | ||
| fi | ||
| shell: bash | ||
|
|
||
| - name: Run fuzz regression tests | ||
| id: fuzz-test | ||
| run: | | ||
| export BUILD_TARGET="${{ inputs.build-target }}" | ||
| source ./ci/dash/matrix.sh | ||
|
|
||
| BUILD_DIR="build-ci/dashcore-${BUILD_TARGET}" | ||
| FUZZ_BIN="${BUILD_DIR}/src/test/fuzz/fuzz" | ||
|
|
||
| if [ ! -x "$FUZZ_BIN" ]; then | ||
| echo "ERROR: Fuzz binary not found at $FUZZ_BIN" | ||
| exit 1 | ||
| fi | ||
|
|
||
| # detect_leaks=0 is intentional for fuzz regression due to libFuzzer/LSan noise. | ||
| export ASAN_OPTIONS="detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1:detect_leaks=0" | ||
| export LSAN_OPTIONS="suppressions=${BUILD_DIR}/test/sanitizer_suppressions/lsan" | ||
| export UBSAN_OPTIONS="suppressions=${BUILD_DIR}/test/sanitizer_suppressions/ubsan:print_stacktrace=1:halt_on_error=1:report_error_type=1" | ||
|
|
||
| # Get list of all targets | ||
| TARGETS=$(PRINT_ALL_FUZZ_TARGETS_AND_ABORT=1 "$FUZZ_BIN" 2>/tmp/fuzz_target_discovery.err || true) | ||
| TARGET_COUNT=$(echo "$TARGETS" | grep -c '[^[:space:]]' || true) | ||
| if [ "$TARGET_COUNT" -eq 0 ]; then | ||
| if [ -s /tmp/fuzz_target_discovery.err ]; then | ||
| cat /tmp/fuzz_target_discovery.err | ||
| fi | ||
| echo "::error::No fuzz targets found — binary may have failed to start" | ||
| exit 1 | ||
| fi | ||
| echo "Found $TARGET_COUNT fuzz targets" | ||
|
|
||
| FAILED=0 | ||
| PASSED=0 | ||
| FAILED_TARGETS="" | ||
|
|
||
| while IFS= read -r target; do | ||
| [ -z "$target" ] && continue | ||
| corpus_dir="/tmp/fuzz_corpus/${target}" | ||
|
|
||
| if [ ! -d "$corpus_dir" ] || [ -z "$(ls -A "$corpus_dir" 2>/dev/null)" ]; then | ||
| # No corpus for this target — run with empty input for 10s | ||
| # This catches basic initialization crashes | ||
| echo "::group::${target} (empty corpus, 10s run)" | ||
| mkdir -p "$corpus_dir" | ||
| # timeout(30) intentionally exceeds -max_total_time=10 to absorb startup/teardown jitter | ||
| # while still terminating genuinely hung processes. | ||
| if FUZZ="$target" timeout 30 "$FUZZ_BIN" \ | ||
| -rss_limit_mb=4000 \ | ||
| -max_total_time=10 \ | ||
| -reload=0 \ | ||
| "$corpus_dir" 2>&1; then | ||
| echo "PASS: $target (empty corpus)" | ||
| PASSED=$((PASSED + 1)) | ||
| else | ||
| EXIT_CODE=$? | ||
| echo "::error::FAIL: $target exited with code $EXIT_CODE" | ||
| FAILED=$((FAILED + 1)) | ||
| FAILED_TARGETS="${FAILED_TARGETS} - ${target} (exit code ${EXIT_CODE})\n" | ||
| fi | ||
| echo "::endgroup::" | ||
| continue | ||
| fi | ||
|
|
||
| # Run corpus regression (replay all inputs) | ||
| echo "::group::${target} ($(find "$corpus_dir" -maxdepth 1 -type f | wc -l) inputs)" | ||
| if FUZZ="$target" "$FUZZ_BIN" \ | ||
| -rss_limit_mb=4000 \ | ||
| -runs=0 \ | ||
| "$corpus_dir" 2>&1; then | ||
| echo "PASS: $target" | ||
| PASSED=$((PASSED + 1)) | ||
| else | ||
| echo "::error::FAIL: $target" | ||
| FAILED=$((FAILED + 1)) | ||
| FAILED_TARGETS="${FAILED_TARGETS} - ${target}\n" | ||
| fi | ||
| echo "::endgroup::" | ||
| done <<< "$TARGETS" | ||
|
|
||
| echo "" | ||
| echo "=== Fuzz Regression Summary ===" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It would be nice to have a list of failed targets here. |
||
| echo "Passed: $PASSED" | ||
| echo "Failed: $FAILED" | ||
| echo "Total: $TARGET_COUNT" | ||
|
|
||
| if [ $FAILED -gt 0 ]; then | ||
| echo "" | ||
| echo "=== Failed Targets ===" | ||
| printf '%b' "$FAILED_TARGETS" | ||
| echo "::error::$FAILED fuzz target(s) failed regression testing" | ||
| exit 1 | ||
| fi | ||
| shell: bash | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| # Dash Core Fuzz Testing Tools | ||
|
|
||
| This directory contains tools for continuous fuzz testing of Dash Core. | ||
|
|
||
| ## Overview | ||
|
|
||
| Dash Core inherits ~100 fuzz targets from Bitcoin Core and adds Dash-specific | ||
| targets for: | ||
| - Special transaction serialization (ProTx, CoinJoin, Asset Lock/Unlock, etc.) | ||
| - BLS operations and IES encryption | ||
| - LLMQ/DKG message handling | ||
| - Governance object validation | ||
| - Masternode list management | ||
|
|
||
| Some Dash-specific fuzz targets are planned/in-progress. Corpus tooling | ||
| pre-generates synthetic seeds for those target names so coverage is ready when | ||
| the targets are added. | ||
|
|
||
| ## Tools | ||
|
|
||
| ### `continuous_fuzz_daemon.sh` | ||
|
|
||
| A daemon script that continuously cycles through all fuzz targets with persistent | ||
| corpus storage and crash detection. | ||
|
|
||
| ```bash | ||
| # Run all targets, 10 minutes each, indefinitely | ||
| ./continuous_fuzz_daemon.sh --fuzz-bin /path/to/fuzz --time-per-target 600 | ||
|
|
||
| # Run specific targets only | ||
| ./continuous_fuzz_daemon.sh --targets bls_operations,bls_ies --time-per-target 3600 | ||
|
|
||
| # Single cycle (good for cron) | ||
| ./continuous_fuzz_daemon.sh --single-cycle --time-per-target 300 | ||
|
|
||
| # Dry run — list targets | ||
| ./continuous_fuzz_daemon.sh --dry-run | ||
| ``` | ||
|
|
||
| **Output directories:** | ||
| - `~/fuzz_corpus/<target>/` — persistent corpus per target | ||
| - `~/fuzz_crashes/<target>/` — crash artifacts (crash-*, timeout-*, oom-*) | ||
| - `~/fuzz_logs/` — per-target logs and daemon log | ||
|
|
||
| ### `seed_corpus_from_chain.py` | ||
|
|
||
| Extracts real-world data from a running Dash node into fuzzer-consumable corpus | ||
| files. Connects via `dash-cli` RPC. | ||
|
|
||
| ```bash | ||
| # Extract from a running node | ||
| ./seed_corpus_from_chain.py -o /path/to/corpus --blocks 500 | ||
|
|
||
| # Generate only synthetic seeds (no running node required) | ||
| ./seed_corpus_from_chain.py -o /path/to/corpus --synthetic-only | ||
| ``` | ||
|
|
||
| **What it extracts:** | ||
| - Serialized blocks and block headers | ||
| - Special transactions (ProRegTx, ProUpServTx, CoinJoin, Asset Lock, etc.) | ||
| - Governance objects and votes | ||
| - Masternode list entries | ||
| - Quorum commitment data | ||
|
|
||
| ## CI Integration | ||
|
|
||
| The `test-fuzz.yml` workflow runs fuzz regression tests on every PR: | ||
|
|
||
| 1. Builds fuzz targets with sanitizers (ASan + UBSan + libFuzzer) | ||
| 2. Downloads seed corpus from `bitcoin-core/qa-assets` + synthetic Dash seeds | ||
| 3. Replays all corpus inputs against every fuzz target | ||
| 4. Reports failures as CI errors | ||
|
|
||
| This catches regressions in seconds — any code change that causes a previously- | ||
| working input to crash will be caught. | ||
|
|
||
| ## Building Fuzz Targets | ||
|
|
||
| ```bash | ||
| # Configure with fuzzing + sanitizers | ||
| ./configure --enable-fuzz --with-sanitizers=fuzzer,address,undefined \ | ||
| CC='clang -ftrivial-auto-var-init=pattern' \ | ||
| CXX='clang++ -ftrivial-auto-var-init=pattern' | ||
|
|
||
| # Build | ||
| make -j$(nproc) | ||
|
|
||
| # The fuzz binary is at src/test/fuzz/fuzz | ||
| # Select target with FUZZ=<target_name> | ||
| FUZZ=bls_operations ./src/test/fuzz/fuzz corpus_dir/ | ||
| ``` | ||
|
|
||
| ## Contributing Corpus Inputs | ||
|
|
||
| Found an interesting input? Add it to the appropriate corpus directory: | ||
|
|
||
| ```bash | ||
| # The filename should be the sha256 of the content (for dedup) | ||
| sha256sum input_file | ||
| cp input_file fuzz_corpus/<target_name>/<sha256_prefix> | ||
| ``` | ||
|
|
||
| Crash-reproducing inputs are especially valuable — they become regression tests. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing
-runs=0causes the corpus replay to run indefinitely, hanging CI.By default, the fuzzing process will continue indefinitely after replaying the existing corpus. Without
-runs=0, libFuzzer will replay all corpus inputs and then switch to mutation-based fuzzing with no time bound. If you want to re-execute a directory of test cases without actually fuzzing, you can run with-runs=0. The corpus replay step has neither-runs=0nor a-max_total_timeguard, so any target with a non-trivial corpus will stall the CI job indefinitely.🐛 Proposed fix
if FUZZ="$target" "$FUZZ_BIN" \ -rss_limit_mb=4000 \ + -runs=0 \ "$corpus_dir" 2>&1; then🤖 Prompt for AI Agents
✅ Confirmed as addressed by @UdjinM6
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
proof: https://github.com/UdjinM6/dash/actions/runs/22323052205/job/64588995007
With this patch applied it can finally finish: https://github.com/UdjinM6/dash/actions/runs/22329270486/job/64608898233
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@UdjinM6, excellent catch and thank you for the confirmation! The evidence clearly shows the issue—the CI job was hanging indefinitely without-runs=0, and your test run proves the fix resolves it completely. The corpus replay now properly terminates after replaying all inputs instead of entering mutation mode.🧠 Learnings used
If you found this review helpful, would you consider giving us a shout-out on X?
Thank you for using CodeRabbit!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure why @coderabbitai marked this one as resolved, it's still not fixed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@UdjinM6, you're absolutely right—I apologize for the confusion. I incorrectly marked the issue as resolved when you provided proof that the fix works in your test branch. That was only validation of the solution, not confirmation that you'd applied it to this PR.Looking at the current code at lines 134-136, the
-runs=0flag is indeed still missing:The original issue remains: without
-runs=0, libFuzzer will replay the corpus and then enter indefinite mutation mode, hanging the CI. Your test run at https://github.com/UdjinM6/dash/actions/runs/22329270486/job/64608898233 proved that adding-runs=0resolves this completely.To fix this in the PR, please add the flag:
Sorry for the premature closure!
🧠 Learnings used