Skip to content

Reduce debug logging noise#3036

Open
liquidsec wants to merge 4 commits into3.0from
logging-cleanup
Open

Reduce debug logging noise#3036
liquidsec wants to merge 4 commits into3.0from
logging-cleanup

Conversation

@liquidsec
Copy link
Copy Markdown
Contributor

Summary

Debug output (-d) was producing ~154k lines for a moderate subdomain-enum scan, making it nearly impossible to follow. These changes cut that roughly in half by removing redundant/noisy messages while keeping genuinely useful debug info.

Changes

Event pipeline (biggest win):

  • Remove "Not intercepting", "Forwarding", "Not queueing" from intercept modules and output modules — these fire for every event through every module and say "nothing happened"
  • Keep "Intercepting" (something actually happened) and "Not forwarding" (event actively rejected)
  • Collapse per-event lifecycle from 4 lines to 1: remove "Got", "passed post-check", "Finished handling" — keep only "Handling {event} from {module}"

Paramminer:

  • Log binary_search entry only on top-level call, not every recursion (~14k lines eliminated)
  • Remove per-word "Adding to wordlist" debug

HttpCompare (diff.py):

  • Remove per-comparison "status code different", "headers different", "body different", "ignored header removed", "switching to text parsing" — these fire on every single HTTP comparison

Serial submodule:

  • Remove 4 routine per-payload messages (baseline match, probe result dump, "potential finding", "500 or body match")
  • Keep useful ones: HttpCompareError, header-only diffs, unexpected status codes, transient baseline, error string detection

Telerik:

  • Remove pointless self.debug(root_tool_path) line

@liquidsec
Copy link
Copy Markdown
Contributor Author

improving issue raised in #2446

@github-actions
Copy link
Copy Markdown
Contributor

📊 Performance Benchmark Report

Comparing 3.0 (baseline) vs logging-cleanup (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name 📏 Base 📏 Current 📈 Change 🎯 Status
Bloom Filter Dns Mutation Tracking Performance 4.25ms 5.22ms +22.8% 🔴🔴🔴 ⚠️
Bloom Filter Large Scale Dns Brute Force 18.62ms 17.75ms -4.7%
Large Closest Match Lookup 358.77ms 352.55ms -1.7%
Realistic Closest Match Workload 191.40ms 189.82ms -0.8%
Event Memory Medium Scan 1783 B/event 1781 B/event -0.1%
Event Memory Large Scan 1768 B/event 1768 B/event +0.0%
Event Validation Full Scan Startup Small Batch 409.32ms 408.79ms -0.1%
Event Validation Full Scan Startup Large Batch 589.91ms 585.72ms -0.7%
Make Event Autodetection Small 31.29ms 31.53ms +0.8%
Make Event Autodetection Large 317.74ms 319.01ms +0.4%
Make Event Explicit Types 14.02ms 14.19ms +1.2%
Excavate Single Thread Small 4.153s 4.046s -2.6%
Excavate Single Thread Large 9.896s 9.872s -0.3%
Excavate Parallel Tasks Small 4.383s 4.185s -4.5%
Excavate Parallel Tasks Large 7.332s 7.270s -0.8%
Is Ip Performance 3.21ms 3.21ms -0.0%
Make Ip Type Performance 11.59ms 11.69ms +0.8%
Mixed Ip Operations 4.58ms 4.82ms +5.2%
Memory Use Web Crawl 44.7 MB 43.1 MB -3.6%
Memory Use Subdomain Enum 19.3 MB 19.4 MB +0.1%
Scan Throughput 100 7.964s 7.454s -6.4%
Scan Throughput 1000 43.074s 34.383s -20.2% 🟢🟢🟢 🚀
Typical Queue Shuffle 66.21µs 62.08µs -6.2%
Priority Queue Shuffle 730.03µs 700.35µs -4.1%

🎯 Performance Summary

+ 1 improvement 🚀
! 1 regression ⚠️
  22 unchanged ✅

🔍 Significant Changes (>10%)

  • Bloom Filter Dns Mutation Tracking Performance: 22.8% 🐌 slower
  • Scan Throughput 1000: 20.2% 🚀 faster

🐍 Python Version 3.11.15

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 13, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91%. Comparing base (39b9d31) to head (c1579ca).
⚠️ Report is 2 commits behind head on 3.0.

Additional details and impacted files
@@          Coverage Diff          @@
##             3.0   #3036   +/-   ##
=====================================
+ Coverage     91%     91%   +1%     
=====================================
  Files        440     440           
  Lines      37411   37388   -23     
=====================================
- Hits       33890   33873   -17     
+ Misses      3521    3515    -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant