When developing a plugin for software created by others and then benchmarking it, it's often desirable to compare performance with and without your plugin.
describe("Bench for test case #1", () => {
bench("without my plugin", () => {
// bench
});
bench("with my plugin", () => {
// bench
});
});
The current CodSpeed can only detect whether each benchmark has performance degraded against the same benchmark on the main branch. It should be possible to use other benchmarks within the same branch and commit as metrics.