[Need Help] Relative benchmarks (baseline)?

[ig-sinicyn]

Hi!
I am trying to write perf tests for new implementation of some code and I want to compare its performance with the old version.

Ideally I want to mark one benchmark as a baseline and specify for others that there will be (lets say) no more than 0.9x by time and no more than 1.2x allocations compared to the baseline benchmark.
Also, if possible, I want to ignore peaks and assert for 90-95th percentile of samples.

Is something alike supported / possible with NBench? Thanks!

[Aaronontheweb]

This is not possible to do with NBench currently because it's stateful - you need to have something that retains the state of one benchmark and can use it as a comparison. Our current design doesn't support this at the moment but it's something I would like to see too as it's inherently useful.

[ig-sinicyn]

Ok, thanks!

[antondelsink]

@Aaronontheweb is the implementation of BenchmarkRunReport, MetricRunReport and BenchmarkFinalResults to be considered stable? I'm thinking of writing them to an Azure SQL DB using a new (additional) implementation of IBenchmarkOutput. That way I can point PowerBI at the data.

But for this we would need an async approach. Any preferences?

[nkast]

Is it possible to change the output format into something more accessible, like XML, cvs, tab delimited, ...?
I could them combine and compare the values from the two files (baseline+current).