Skip to content

Add -k filtering to pytest-benchmark compare#303

Merged
ionelmc merged 1 commit intoionelmc:masterfrom
akx:compare-filter
Mar 26, 2026
Merged

Add -k filtering to pytest-benchmark compare#303
ionelmc merged 1 commit intoionelmc:masterfrom
akx:compare-filter

Conversation

@akx
Copy link
Copy Markdown
Contributor

@akx akx commented Mar 18, 2026

... just like pytest -k, and using the same machinery.

$ pytest-benchmark compare .benchmarks/Darwin-CPython-3.14-64bit/*.json -kmultiple --columns=ops

---------------------- benchmark: 4 tests ---------------------
Name (time in us)                                 OPS
---------------------------------------------------------------
test_aggregate_multiple (0001_main-ad)     3,787.8262 (0.98)
test_aggregate_multiple (0002_sql-mic)     3,874.5312 (1.0)
test_order_by_multiple (0001_main-ad)        684.4064 (0.18)
test_order_by_multiple (0002_sql-mic)        694.5633 (0.18)
---------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

@akx akx changed the title Add -k to pytest-benchmark compare Add -k filtering to pytest-benchmark compare Mar 18, 2026
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 18, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.79%. Comparing base (cf97f26) to head (1abd48f).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #303      +/-   ##
==========================================
+ Coverage   90.72%   90.79%   +0.07%     
==========================================
  Files          28       28              
  Lines        3052     3076      +24     
  Branches      336      337       +1     
==========================================
+ Hits         2769     2793      +24     
  Misses        208      208              
  Partials       75       75              
Flag Coverage Δ
py310-pytest84-nodist-cover (macos/arm64) 86.50% <100.00%> (+0.10%) ⬆️
py310-pytest84-nodist-cover (ubuntu/x64) 86.99% <100.00%> (+0.10%) ⬆️
py310-pytest84-nodist-cover (windows/x64) 86.08% <100.00%> (-0.09%) ⬇️
py310-pytest84-xdist-cover (macos/arm64) 86.83% <100.00%> (+0.10%) ⬆️
py310-pytest84-xdist-cover (ubuntu/x64) 87.12% <100.00%> (-0.10%) ⬇️
py310-pytest84-xdist-cover (windows/x64) 86.41% <100.00%> (-0.09%) ⬇️
py310-pytest90-nodist-cover (macos/arm64) 86.47% <100.00%> (+0.10%) ⬆️
py310-pytest90-nodist-cover (ubuntu/x64) 86.76% <100.00%> (+0.10%) ⬆️
py310-pytest90-nodist-cover (windows/x64) 86.05% <100.00%> (+0.10%) ⬆️
py310-pytest90-xdist-cover (macos/arm64) 86.60% <100.00%> (?)
py310-pytest90-xdist-cover (ubuntu/x64) 87.09% <100.00%> (+0.10%) ⬆️
py310-pytest90-xdist-cover (windows/x64) 86.57% <100.00%> (+0.10%) ⬆️
py311-pytest84-nodist-cover (macos/arm64) 86.50% <100.00%> (+0.10%) ⬆️
py311-pytest84-nodist-cover (ubuntu/x64) 86.83% <100.00%> (+0.13%) ⬆️
py311-pytest84-nodist-cover (windows/x64) 86.28% <100.00%> (+0.10%) ⬆️
py311-pytest84-xdist-cover (macos/arm64) 86.83% <100.00%> (+0.10%) ⬆️
py311-pytest84-xdist-cover (ubuntu/x64) 87.15% <100.00%> (+0.10%) ⬆️
py311-pytest84-xdist-cover (windows/x64) 86.41% <100.00%> (+0.10%) ⬆️
py311-pytest90-nodist-cover (macos/arm64) 86.47% <100.00%> (+0.10%) ⬆️
py311-pytest90-nodist-cover (ubuntu/x64) 86.76% <100.00%> (+0.10%) ⬆️
py311-pytest90-nodist-cover (windows/x64) 86.05% <100.00%> (-0.09%) ⬇️
py311-pytest90-xdist-cover (macos/arm64) 86.80% <100.00%> (+0.10%) ⬆️
py311-pytest90-xdist-cover (ubuntu/x64) 87.09% <100.00%> (-0.13%) ⬇️
py311-pytest90-xdist-cover (windows/x64) 86.57% <100.00%> (+0.30%) ⬆️
py312-pytest84-nodist-cover (macos/arm64) 86.50% <100.00%> (+0.10%) ⬆️
py312-pytest84-nodist-cover (ubuntu/x64) 86.80% <100.00%> (-0.10%) ⬇️
py312-pytest84-nodist-cover (windows/x64) 86.28% <100.00%> (+0.10%) ⬆️
py312-pytest84-xdist-cover (macos/arm64) 86.83% <100.00%> (+0.10%) ⬆️
py312-pytest84-xdist-cover (ubuntu/x64) 87.15% <100.00%> (+0.10%) ⬆️
py312-pytest84-xdist-cover (windows/x64) 86.41% <100.00%> (+0.10%) ⬆️
py312-pytest90-nodist-cover (macos/arm64) 86.28% <100.00%> (-0.09%) ⬇️
py312-pytest90-nodist-cover (ubuntu/x64) 86.96% <100.00%> (+0.26%) ⬆️
py312-pytest90-nodist-cover (windows/x64) 86.24% <100.00%> (+0.30%) ⬆️
py312-pytest90-xdist-cover (macos/arm64) 86.80% <100.00%> (+0.10%) ⬆️
py312-pytest90-xdist-cover (ubuntu/x64) 87.28% <100.00%> (+0.26%) ⬆️
py312-pytest90-xdist-cover (windows/x64) 86.57% <100.00%> (+0.10%) ⬆️
py313-pytest84-nodist-cover (macos/arm64) 86.50% <100.00%> (+0.10%) ⬆️
py313-pytest84-nodist-cover (ubuntu/x64) 86.80% <100.00%> (+0.10%) ⬆️
py313-pytest84-nodist-cover (windows/x64) 86.08% <100.00%> (-0.09%) ⬇️
py313-pytest84-xdist-cover (macos/arm64) 86.83% <100.00%> (+0.10%) ⬆️
py313-pytest84-xdist-cover (ubuntu/x64) 87.32% <100.00%> (+0.09%) ⬆️
py313-pytest84-xdist-cover (windows/x64) 86.41% <100.00%> (-0.09%) ⬇️
py313-pytest90-nodist-cover (macos/arm64) 86.47% <100.00%> (+0.10%) ⬆️
py313-pytest90-nodist-cover (ubuntu/x64) 86.80% <100.00%> (-0.10%) ⬇️
py313-pytest90-nodist-cover (windows/x64) 86.24% <100.00%> (+0.30%) ⬆️
py313-pytest90-xdist-cover (macos/arm64) 86.80% <100.00%> (+0.10%) ⬆️
py313-pytest90-xdist-cover (ubuntu/x64) 87.28% <100.00%> (+0.26%) ⬆️
py313-pytest90-xdist-cover (windows/x64) 86.57% <100.00%> (+0.30%) ⬆️
py314-pytest84-nodist-cover (macos/arm64) 89.69% <100.00%> (?)
py314-pytest84-nodist-cover (ubuntu/x64) 89.98% <100.00%> (-0.12%) ⬇️
py314-pytest84-nodist-cover (windows/x64) 89.43% <100.00%> (+0.08%) ⬆️
py314-pytest84-xdist-cover (macos/arm64) 89.82% <100.00%> (?)
py314-pytest84-xdist-cover (ubuntu/x64) 90.27% <100.00%> (-0.13%) ⬇️
py314-pytest84-xdist-cover (windows/x64) 89.56% <100.00%> (-0.12%) ⬇️
py314-pytest90-nodist-cover (macos/arm64) 89.69% <100.00%> (+0.08%) ⬆️
py314-pytest90-nodist-cover (ubuntu/x64) 90.14% <100.00%> (+0.27%) ⬆️
py314-pytest90-nodist-cover (windows/x64) 89.23% <100.00%> (+0.08%) ⬆️
py314-pytest90-xdist-cover (macos/arm64) 89.82% <100.00%> (-0.12%) ⬇️
py314-pytest90-xdist-cover (ubuntu/x64) 90.27% <100.00%> (-0.16%) ⬇️
py314-pytest90-xdist-cover (windows/x64) 89.75% <100.00%> (+0.27%) ⬆️
pypy311-pytest84-nodist-cover (macos/arm64) 85.69% <100.00%> (+0.11%) ⬆️
pypy311-pytest84-nodist-cover (ubuntu/x64) 85.98% <100.00%> (+0.11%) ⬆️
pypy311-pytest84-nodist-cover (windows/x64) 85.46% <100.00%> (+0.31%) ⬆️
pypy311-pytest84-xdist-cover (macos/arm64) 85.82% <100.00%> (-0.09%) ⬇️
pypy311-pytest84-xdist-cover (ubuntu/x64) 86.34% <100.00%> (+0.10%) ⬆️
pypy311-pytest84-xdist-cover (windows/x64) 85.59% <100.00%> (+0.11%) ⬆️
pypy311-pytest90-nodist-cover (macos/arm64) 85.69% <100.00%> (?)
pypy311-pytest90-nodist-cover (ubuntu/x64) 85.98% <100.00%> (+0.07%) ⬆️
pypy311-pytest90-nodist-cover (windows/x64) 85.27% <100.00%> (-0.09%) ⬇️
pypy311-pytest90-xdist-cover (macos/arm64) 86.02% <100.00%> (+0.10%) ⬆️
pypy311-pytest90-xdist-cover (ubuntu/x64) 86.31% <100.00%> (+0.07%) ⬆️
pypy311-pytest90-xdist-cover (windows/x64) 85.59% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@akx akx force-pushed the compare-filter branch from 57532b1 to 88edeff Compare March 24, 2026 21:21
@ionelmc
Copy link
Copy Markdown
Owner

ionelmc commented Mar 25, 2026

Looks like this needs a rebase.

@akx akx force-pushed the compare-filter branch from 88edeff to 1abd48f Compare March 25, 2026 19:27
@akx
Copy link
Copy Markdown
Contributor Author

akx commented Mar 26, 2026

@ionelmc Rebased, build is green.

@ionelmc ionelmc merged commit db50ab1 into ionelmc:master Mar 26, 2026
222 of 223 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants