-
Notifications
You must be signed in to change notification settings - Fork 1
Add performance regression test suite to CI #391
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| name: Performance Tests | ||
|
|
||
| on: | ||
| pull_request: | ||
| paths: | ||
| - 'main.py' | ||
| - 'tests/test_performance_*.py' | ||
| push: | ||
| branches: [main] | ||
|
|
||
| jobs: | ||
| benchmark: | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: read | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Install uv | ||
| uses: astral-sh/setup-uv@v4 | ||
| with: | ||
| enable-cache: true | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.13' | ||
|
|
||
| - name: Install dependencies | ||
| run: uv sync --all-extras | ||
|
|
||
| - name: Run performance tests | ||
| run: | | ||
| uv run pytest tests/test_performance_regression.py \ | ||
| --benchmark-only \ | ||
| --benchmark-autosave \ | ||
| -v | ||
|
|
||
| - name: Store benchmark results | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: benchmark-results | ||
| path: .benchmarks/ |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,4 +14,5 @@ dev = [ | |
| "pytest>=7.0.0", | ||
| "pytest-mock>=3.10.0", | ||
| "pytest-xdist>=3.0.0", | ||
| "pytest-benchmark>=4.0.0", | ||
| ] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| """ | ||
| Performance regression tests with baseline thresholds. | ||
|
|
||
| These benchmarks guard against unintentional slowdowns on hot paths. | ||
| Run with: uv run pytest tests/test_performance_regression.py --benchmark-only | ||
| """ | ||
|
|
||
| import sys | ||
| import os | ||
|
|
||
| import pytest | ||
Check warningCode scanning / Prospector (reported by Codacy) Unused import pytest (unused-import) Warning test
Unused import pytest (unused-import)
Check noticeCode scanning / Pylint (reported by Codacy) Unused import pytest Note test
Unused import pytest
Check noticeCode scanning / Pylintpython3 (reported by Codacy) Unused import pytest Note test
Unused import pytest
|
||
| import httpx | ||
Check warningCode scanning / Prospector (reported by Codacy) Unable to import 'httpx' (import-error) Warning test
Unable to import 'httpx' (import-error)
|
||
|
|
||
| sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) | ||
| import main | ||
Check warningCode scanning / Pylint (reported by Codacy) Import "import main" should be placed at the top of the module Warning test
Import "import main" should be placed at the top of the module
Check warningCode scanning / Prospector (reported by Codacy) Cannot import 'main' due to syntax error 'invalid syntax (, line 1286)' (syntax-error) Warning test
Cannot import 'main' due to syntax error 'invalid syntax (, line 1286)' (syntax-error)
Check warningCode scanning / Pylintpython3 (reported by Codacy) Import "import main" should be placed at the top of the module Warning test
Import "import main" should be placed at the top of the module
|
||
|
|
||
| # Maximum acceptable mean execution time (seconds) for hot-path operations. | ||
| _MAX_MEAN_S = 0.001 # 1 ms | ||
|
|
||
|
|
||
| class TestPerformanceRegression: | ||
| """Performance regression tests with baseline thresholds.""" | ||
|
|
||
| def test_validate_hostname_performance(self, benchmark): | ||
Check warningCode scanning / Pylint (reported by Codacy) Method could be a function Warning test
Method could be a function
Check warningCode scanning / Pylintpython3 (reported by Codacy) Method could be a function Warning test
Method could be a function
|
||
| """Hostname validation should complete in <1ms on average.""" | ||
| # Use a private IP so no real DNS lookup occurs, making the test fast and deterministic | ||
| main.validate_hostname.cache_clear() | ||
| result = benchmark(main.validate_hostname, "192.168.1.1") | ||
| assert result is False # Private IP is rejected | ||
Check noticeCode scanning / Bandit Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
Check noticeCode scanning / Bandit (reported by Codacy) Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
|
||
| assert benchmark.stats["mean"] < _MAX_MEAN_S | ||
Check noticeCode scanning / Bandit Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
Check noticeCode scanning / Bandit (reported by Codacy) Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
|
||
|
|
||
| def test_validate_hostname_cached_performance(self, benchmark): | ||
Check warningCode scanning / Pylint (reported by Codacy) Method could be a function Warning test
Method could be a function
Check warningCode scanning / Pylintpython3 (reported by Codacy) Method could be a function Warning test
Method could be a function
|
||
| """Cached hostname validation should be significantly faster than <1ms.""" | ||
| # Prime the cache with a known private IP | ||
| main.validate_hostname.cache_clear() | ||
| main.validate_hostname("10.0.0.1") | ||
| # Now benchmark the cached call | ||
| result = benchmark(main.validate_hostname, "10.0.0.1") | ||
| assert result is False | ||
Check noticeCode scanning / Bandit Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
Check noticeCode scanning / Bandit (reported by Codacy) Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
|
||
| assert benchmark.stats["mean"] < _MAX_MEAN_S | ||
Check noticeCode scanning / Bandit Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
Check noticeCode scanning / Bandit (reported by Codacy) Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
|
||
|
|
||
| def test_rate_limit_parsing_performance(self, benchmark): | ||
Check warningCode scanning / Pylint (reported by Codacy) Method could be a function Warning test
Method could be a function
Check warningCode scanning / Pylintpython3 (reported by Codacy) Method could be a function Warning test
Method could be a function
|
||
| """Rate limit header parsing should complete in <1ms on average.""" | ||
| headers = { | ||
| "X-RateLimit-Limit": "5000", | ||
| "X-RateLimit-Remaining": "4999", | ||
| "X-RateLimit-Reset": "1640000000", | ||
| } | ||
| mock_response = httpx.Response(200, headers=headers) | ||
|
|
||
| result = benchmark(main._parse_rate_limit_headers, mock_response) | ||
Check warningCode scanning / Prospector (reported by Codacy) Access to a protected member _parse_rate_limit_headers of a client class (protected-access) Warning test
Access to a protected member _parse_rate_limit_headers of a client class (protected-access)
Check noticeCode scanning / Pylint (reported by Codacy) Access to a protected member _parse_rate_limit_headers of a client class Note test
Access to a protected member _parse_rate_limit_headers of a client class
Check noticeCode scanning / Pylintpython3 (reported by Codacy) Access to a protected member _parse_rate_limit_headers of a client class Note test
Access to a protected member _parse_rate_limit_headers of a client class
|
||
| assert result is None # Function returns None | ||
Check noticeCode scanning / Bandit Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
Check noticeCode scanning / Bandit (reported by Codacy) Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
|
||
| assert benchmark.stats["mean"] < _MAX_MEAN_S | ||
Check noticeCode scanning / Bandit Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
Check noticeCode scanning / Bandit (reported by Codacy) Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
|
||
|
|
||
| def test_rate_limit_parsing_empty_headers_performance(self, benchmark): | ||
Check warningCode scanning / Pylint (reported by Codacy) Method could be a function Warning test
Method could be a function
Check warningCode scanning / Pylintpython3 (reported by Codacy) Method could be a function Warning test
Method could be a function
|
||
| """Rate limit parsing with no rate-limit headers should complete in <1ms.""" | ||
| mock_response = httpx.Response(200, headers={}) | ||
| result = benchmark(main._parse_rate_limit_headers, mock_response) | ||
Check warningCode scanning / Prospector (reported by Codacy) Access to a protected member _parse_rate_limit_headers of a client class (protected-access) Warning test
Access to a protected member _parse_rate_limit_headers of a client class (protected-access)
Check noticeCode scanning / Pylint (reported by Codacy) Access to a protected member _parse_rate_limit_headers of a client class Note test
Access to a protected member _parse_rate_limit_headers of a client class
Check noticeCode scanning / Pylintpython3 (reported by Codacy) Access to a protected member _parse_rate_limit_headers of a client class Note test
Access to a protected member _parse_rate_limit_headers of a client class
|
||
| assert result is None | ||
Check noticeCode scanning / Bandit Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
Check noticeCode scanning / Bandit (reported by Codacy) Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
|
||
| assert benchmark.stats["mean"] < _MAX_MEAN_S | ||
Check noticeCode scanning / Bandit Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
Check noticeCode scanning / Bandit (reported by Codacy) Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. Note test
Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
|
||
Check warning
Code scanning / Prospector (reported by Codacy)
Unable to import 'pytest' (import-error) Warning test