Dealing with many, noisy, secondary measurements

[Kha]

We finally have some first measurements live on our instance: http://speedcenter.informatik.kit.edu/velcom/home
I've changed the benchmark configuration a bit from the codespeed instance, we now record many additional quantities such as branch/cache misses using perf stat. The primary purpose of these additions is not detecting regressions, but giving possible explanations for why a benchmark's run time might have regressed. Thus we could call them "secondary" measurements.

As you can see, some of these measurements completely dominate the Home view currently since they turn out to be quite noisy. Respecting the stddev in the significance predicate would help, but would not hide all of these events. They also make the table of benchmark results for a single commit quite long. So it would be useful to be able to mark measurements as secondary and then special-case them in the UI, e.g.

• never mark them as significant changes
• hide them in the benchmark results table by default

[Garmelon]

At the moment, the unit and interpretation are retrieved from the benchmark script output and stored separately for each individual run and individual measurement. When the unit or interpretation need to be displayed or used, the latest run's units and interpretation are used, but old units and interpretations are not lost. Thus, if the newest run is deleted, the second newest run's units and interpretations will be used.

We could use the same system for marking certain measurements as significant/insignificant by adding a corresponding flag in the benchmark script output format. That way, we could use the existing system.

We could also change to a system where unit, measurement and significance of metrics are stored globally, not per-run. New runs would then update the global config values. This change would probably require more effort to initially implement but the behaviour of velcom might become more intuitive. Also, giving the unit, interpretation or significance in the benchmark output could become entirely optional.

[Kha]

We could also change to a system where unit, measurement and significance of metrics are stored globally, not per-run.

That does sound like a good idea in the long term. In the short term, I could turn of the secondary metrics completely for now, but I don't really want to discard all that data. We could change the output format as you said, though we would have to change temci as well then. A more hackish, but perhaps the simplest solution would be to configure a single repo-wide regex that matches secondary metrics. I just want a mostly usable solution for now until we can fix it properly at some later point.

[Kha]

Alternatively, I can move the regex into a jq script post-processing the temci output after you've extended the output format. I would be ok with that for now.

[Kha]

I've mostly mitigated the issue for now by removing the cache metrics - they are probably too noisy to be of any help at all. http://speedcenter.informatik.kit.edu/velcom

[Garmelon]

Partially fixed by #193 and manually editing the dimension table. There is no way to edit the significance of dimensions via the UI yet.