Skip to content

[Snyk] Security upgrade urllib3 from 1.24.3 to 1.26.19#45

Open
matholiveira91 wants to merge 1 commit intomasterfrom
snyk-fix-9f8dde973e5b5ceba5d4fd69b8d1ff87
Open

[Snyk] Security upgrade urllib3 from 1.24.3 to 1.26.19#45
matholiveira91 wants to merge 1 commit intomasterfrom
snyk-fix-9f8dde973e5b5ceba5d4fd69b8d1ff87

Conversation

@matholiveira91
Copy link
Collaborator

This PR was automatically created by Snyk using the credentials of a real user.


![snyk-top-banner](https://github.com/andygongea/OWASP-Benchmark/assets/818805/c518c423-16fe-447e-b67f-ad5a49b5d123)

Snyk has created this PR to fix 1 vulnerabilities in the pip dependencies of this project.

Snyk changed the following file(s):

  • requirements.txt
⚠️ Warning ``` requests 2.20.1 has requirement urllib3<1.25,>=1.21.1, but you have urllib3 2.0.7.
</details>





---

> [!IMPORTANT]
>
> - Check the changes in this PR to ensure they won't cause issues with your project.
> - Max score is 1000. Note that the real score may have changed since the PR was raised.
> - This PR was automatically created by Snyk using the credentials of a real user.
> - Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

---

**Note:** _You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs._

For more information: <img src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiI0NTQ1MDU2OC03ZTcyLTRjMjctYjMwOS0xZWEwOThkZGRmOGIiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6IjQ1NDUwNTY4LTdlNzItNGMyNy1iMzA5LTFlYTA5OGRkZGY4YiJ9fQ==" width="0" height="0"/>
🧐 [View latest project report](https://app.snyk.io/org/matholiveira91/project/a1962abd-b101-4be6-935a-ae9a181c2ab6?utm_source&#x3D;github&amp;utm_medium&#x3D;referral&amp;page&#x3D;fix-pr)
📜 [Customise PR templates](https://docs.snyk.io/scan-using-snyk/pull-requests/snyk-fix-pull-or-merge-requests/customize-pr-templates)
🛠 [Adjust project settings](https://app.snyk.io/org/matholiveira91/project/a1962abd-b101-4be6-935a-ae9a181c2ab6?utm_source&#x3D;github&amp;utm_medium&#x3D;referral&amp;page&#x3D;fix-pr/settings)
📚 [Read about Snyk's upgrade logic](https://support.snyk.io/hc/en-us/articles/360003891078-Snyk-patches-to-fix-vulnerabilities)

---

**Learn how to fix vulnerabilities with free interactive lessons:**

🦉 [Learn about vulnerability in an interactive lesson of Snyk Learn.](https://learn.snyk.io/?loc&#x3D;fix-pr)

[//]: # 'snyk:metadata:{"customTemplate":{"variablesUsed":[],"fieldsUsed":[]},"dependencies":[{"name":"urllib3","from":"1.24.3","to":"1.26.19"}],"env":"prod","issuesToFix":[{"exploit_maturity":"No Known Exploit","id":"SNYK-PYTHON-URLLIB3-7267250","priority_score":586,"priority_score_factors":[{"type":"freshness","label":true,"score":71},{"type":"fixability","label":true,"score":214},{"type":"cvssScore","label":"6","score":300},{"type":"scoreVersion","label":"v1","score":1}],"severity":"medium","title":"Improper Removal of Sensitive Information Before Storage or Transfer"},{"exploit_maturity":"No Known Exploit","id":"SNYK-PYTHON-URLLIB3-7267250","priority_score":586,"priority_score_factors":[{"type":"freshness","label":true,"score":71},{"type":"fixability","label":true,"score":214},{"type":"cvssScore","label":"6","score":300},{"type":"scoreVersion","label":"v1","score":1}],"severity":"medium","title":"Improper Removal of Sensitive Information Before Storage or Transfer"},{"exploit_maturity":"No Known Exploit","id":"SNYK-PYTHON-URLLIB3-7267250","priority_score":586,"priority_score_factors":[{"type":"freshness","label":true,"score":71},{"type":"fixability","label":true,"score":214},{"type":"cvssScore","label":"6","score":300},{"type":"scoreVersion","label":"v1","score":1}],"severity":"medium","title":"Improper Removal of Sensitive Information Before Storage or Transfer"},{"exploit_maturity":"No Known Exploit","id":"SNYK-PYTHON-URLLIB3-7267250","priority_score":586,"priority_score_factors":[{"type":"freshness","label":true,"score":71},{"type":"fixability","label":true,"score":214},{"type":"cvssScore","label":"6","score":300},{"type":"scoreVersion","label":"v1","score":1}],"severity":"medium","title":"Improper Removal of Sensitive Information Before Storage or Transfer"}],"prId":"45450568-7e72-4c27-b309-1ea098dddf8b","prPublicId":"45450568-7e72-4c27-b309-1ea098dddf8b","packageManager":"pip","priorityScoreList":[586],"projectPublicId":"a1962abd-b101-4be6-935a-ae9a181c2ab6","projectUrl":"https://app.snyk.io/org/matholiveira91/project/a1962abd-b101-4be6-935a-ae9a181c2ab6?utm_source=github&utm_medium=referral&page=fix-pr","prType":"fix","templateFieldSources":{"branchName":"default","commitMessage":"default","description":"default","title":"default"},"templateVariants":["updated-fix-title","pr-warning-shown","priorityScore"],"type":"auto","upgrade":[],"vulns":["SNYK-PYTHON-URLLIB3-7267250"],"patch":[],"isBreakingChange":false,"remediationStrategy":"vuln"}'

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-URLLIB3-7267250
matholiveira91 pushed a commit that referenced this pull request Mar 4, 2026
…rovements

Rewrites the three main Python modules to address performance bottlenecks
identified in the scraping, text processing and headless browser pipelines.
GoMutation is preserved unchanged pending evaluation.

modules/scraper.py
- Replace sequential requests with asyncio + aiohttp parallel pipeline
- Add configurable concurrency semaphore (default: 10 simultaneous requests)
- Switch HTML parser from html.parser to lxml (up to 10x faster parsing)
- Add SHA-256 disk cache per URL to skip redundant fetches on re-runs
- Add automatic retry with exponential backoff (3 attempts per URL)
- Use set() for immediate deduplication during word collection
- Expose synchronous scrape() entry point for backward compatibility

modules/aggressive.py
- Replace per-URL browser instantiation with a single reusable Playwright instance
- Add async tab pool with configurable concurrency (default: 4 simultaneous tabs)
- Add JS-detection heuristic to delegate non-JS pages to the faster aiohttp path
- Retain geckodriver support as --use-gecko fallback for legacy environments
- geckodriver path also improved: single driver instance reused across all URLs

modules/wordlist.py
- Replace manual frequency dict with collections.Counter (C-level implementation)
- Switch file reading to line-by-line streaming to avoid full file loading in RAM
- Add Unicode normalization (NFKD) for correct handling of accented characters
- Deduplicate early with set(); Counter.most_common() replaces manual sort
- Fix bug #17: static text list not saved from interactive mode — add explicit
  flush + fsync to guarantee writes before process exit
- GoMutation invoked via stdin pipe instead of temp file, reducing disk I/O
- GoMutation binary preserved and unchanged

main
- Dispatch to correct Python module based on CLI flag (-w, -t, -b)
- Conditional GoMutation compilation preserved (go build only when binary absent)
- Interactive menu retained for no-argument invocations

requirements.txt
- Add aiohttp>=3.9.3 and playwright>=1.43.0 for async/headless improvements
- Bump urllib3 to >=1.26.19 to address open security PR #45 (Snyk CVE fix)
- Pin lxml>=5.1.0 and beautifulsoup4>=4.12.3

tests/test_improvements.py (new)
- Unit tests for normalize(), tokenize(), Counter pipeline, top_words()
- Streaming file reader test with 10k-line corpus
- save_wordlist() test asserting bug #17 regression does not reoccur
- extract_words_from_html() tests covering script stripping and deduplication
- Cache path determinism and collision-resistance tests

.github/workflows/ci.yml (new)
- Test matrix across Python 3.10, 3.11 and 3.12
- Bandit static security analysis on modules/
- pip-audit dependency vulnerability scan on each PR
- ShellCheck linting for main, functions.sh and load.sh

Expected performance gains:
- Standard mode (-w / -t): 5–20x faster on multi-URL targets
- HTML parsing:            up to 10x faster with lxml
- Aggressive mode (-a):    3–10x faster with browser tab pool
- Repeated runs:           near-instant via disk cache
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants