(WIP) compare and chart implementations from crates.io

[nkconnor]

I'm not sure the preferred way to start integrating this. What came to mind is creating a benchmarks directory with a small CLI utility for specifying options, e.g.

cargo run --impls contrie,flurry --mix read_heavy --seeds 5

Python is used for the charting, but it'd probably be valuable to have CSV as an option so the user can create their own charts. With the existing output, there's only a handful of ways to slice the pie, so we could also add some additional preset charts with ease.

Also note, this module requires nightly since I pulled in nkconnor/sharded. I'm not sure if that is desirable.

Looking forward to any feedback!

[jonhoo]

Yeah, this will end up never releasing garbage. You'd have to explicitly repin it to release garbage, which there isn't really a good way to do in flurry today.

[jonhoo]

This looks great, and is a good idea. I'd love to have @xacrimon's input on this too.

I don't think having the benchmarker be nightly-only is a problem.

[xacrimon]

Well this looks just fantastic! I like it.

[xacrimon]

One thing I think needs change though is the default hasher. In a performance critical and concurrent environment the default hash_map::RandomState isn't a good default. It's quite slow and has a habit of obscuring benchmarks. fxhash, fnv-1a, twox-hash and wyhash are good benchmarking hashers.

[nkconnor]

@xacrimon one thing I can do is add a feature flag, or probably runtime flag to select a hasher, with some sensible default.

Also, (and maybe this is overkill) if we target a config instead then it becomes somewhat natural to parameterize hashing as part of the benchmark.

seeds = 5

[[trials]]
label = "dashmap<S=ahash>"
impl = "dashmap"
hasher = "ahash"

[[trials]]
label = "dashmap<S=fxhash>"
impl = "dashmap"
hasher = "fxhash"

[nkconnor]

I haven't put more effort into this yet but starting to look at it again. It looks like xacrimon et. al have advanced their bench suite quite a bit since this PR: https://github.com/xacrimon/conc-map-bench