Skip to content

ASV demo#487

Open
Micky774 wants to merge 3 commits intodevfrom
zain/asv-demo
Open

ASV demo#487
Micky774 wants to merge 3 commits intodevfrom
zain/asv-demo

Conversation

@Micky774
Copy link
Contributor

@Micky774 Micky774 commented Mar 16, 2026

Description

This PR is a port of #478 to ASV. The main benefits are:

  1. Since ASV is a popular and mature OSS project, and is broadly adopted by other large communities (e.g. pandas, scikit-learn, numpy, etc.), there is a significantly reduced maintenance and development burden. It's already feature-rich and tested through the community, allowing us to make use of it and focus on the actual benchmark design rather than implementation detials.
  2. ASV accumulates results per commit on dev in Artifactory, so each CI run only benchmarks the
    current commit on an update to dev.
  3. ASV's time_* methods automatically adapt the number of iterations used, wherein fast operations (e.g. casting) get more iterations for statistical significance, slow operations (e.g. large attention) get fewer to avoid wasting time. This can be customized (e.g. setting a max total time).
  4. Benchmark classes follow a well-documented convention (setup/time_*), making it easy for anyone familiar with ASV to add new benchmarks without understanding custom infrastructure. Similarly, people unfamiliar with ASV can quickly onboard with existing documentation.
  5. ASV's params/param_names class attributes handle the cross-product automatically, and failures are reported per parameter combination without crashing the suite.
  6. ASV stores results per commit as JSON in Artifactory, enabling asv publish to generate an HTML dashboard showing performance trends across the full commit history. (Note that this can be done statically via github pages in a separate repo).
  7. The ASV CI steps are ~75 lines total (restore/run/upload), with no custom run management.
  8. ASV handles all storage (JSON) and reporting internally.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Adds ASV benchmarks
  • Updates CI to read/generate/write ASV results to artifactory
  • Adds README.md for documentation

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@Micky774 Micky774 marked this pull request as ready for review March 17, 2026 13:58
@Micky774
Copy link
Contributor Author

Note the CI failure is unrelated

@Micky774
Copy link
Contributor Author

I've added a helper script like @alextmagro had suggested, as well as corresponding documentation to the README.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant