Skip to content

test(serialize): improved benchmarking#142

Open
zsilbi wants to merge 36 commits intomainfrom
perf/benchmarks
Open

test(serialize): improved benchmarking#142
zsilbi wants to merge 36 commits intomainfrom
perf/benchmarks

Conversation

@zsilbi
Copy link
Copy Markdown
Collaborator

@zsilbi zsilbi commented Mar 4, 2025

I created a small util to automatically download specific versions from GitHub to compare with each other and the current version.

import { bench } from "vitest";
import { getVersions } from "./fixtures/utils/versions";

// Options for v1
const hashOptions = { unorderedArrays: true, unorderedSets: true };
const versions = await getVersions(["v1.1.5", "v2.0.4", "v2.0.11"]);
for (const version of versions) { 
  bench(version.name, () => {
    version.serialize("hello world", hashOptions); 
  });
}

Added custom benchmarks.
Added the same benchmarks for bun using mitata.
Added bench:bun command to run the bun benchmark.

@pi0
Copy link
Copy Markdown
Member

pi0 commented Mar 5, 2025

Tip: we can use "ohash-v1": "npm:ohash@^1" in devDependencies to have same effect.

@zsilbi
Copy link
Copy Markdown
Collaborator Author

zsilbi commented Mar 5, 2025

Tip: we can use "ohash-v1": "npm:ohash@^1" in devDependencies to have same effect.

This allows me to use any branch name, commit hash or version tag to benchmark with the current dev version (and against each other).
If I only wanted to bench dev with v1 that would have been an easier solution of course 👍🏻

For example:

export const benchConfig: BenchConfig = {
  versions: [
    "v2.0.11",
    "7a52c4ac82c33396c2e2ea90e2896ccf3a03256b",
    "main",
  ],
  ...
}
benchmark                   avg (min … max) p75 / p99    (min … top 1%)
------------------------------------------- -------------------------------
• count:1, size:small
------------------------------------------- -------------------------------
ohash @ v2.0.11              497.56 ns/iter 477.08 ns ▆█
                    (457.46 ns … 905.65 ns) 841.28 ns ██
                    (  0.00  b … 726.00  b)  12.63  b ██▃▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▁▂▁

ohash @ 7a52c4a              453.85 ns/iter 431.23 ns █▅
                    (415.98 ns … 825.99 ns) 787.21 ns ██
                    (  0.00  b …   1.03 kb)  31.17  b ██▂▂▁▁▁▁▁▁▁▁▂▁▂▁▁▁▂▁▁

ohash @ main                 475.14 ns/iter 448.85 ns 
                      (431.98 ns … 1.01 µs) 851.39 ns ██
                    (  0.00  b …   1.29 kb)  36.44  b ██▁▁▁▂▁▁▁▁▁▁▂▂▁▁▁▁▁▁▁

ohash @ dev                  406.66 ns/iter 386.36 ns █▃
                    (370.59 ns … 962.38 ns) 769.31 ns ██
                    (  0.00  b …   1.10 kb)  29.73  b ██▂▁▁▁▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁

summary
  ohash @ dev
   1.12x faster than ohash @ 7a52c4a
   1.17x faster than ohash @ main
   1.22x faster than ohash @ v2.0.11

@zsilbi
Copy link
Copy Markdown
Collaborator Author

zsilbi commented Mar 9, 2025

I decided to remove vitest benchmarks for serialize and use only mitata for both Bun and node for easier maintainabilty.
Added jiti as dev. dep. to use mitata with node.
Added the option to use arguments to turn on specific benchmarks (--presets (default) / --combined / --custom / --full).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants