Compile time benchmarks in CI?

[Kmeakin]

I've submitted some PRs in the past that remove some inefficiencies in the compiler (#427, #426). In those cases, they were obvious performance wins, but in the future we might want to include compile time benchmarks in CI, so we can measure performance changes and catch regressions.

Information on benchmarking rust code in CI is scarce. Criterion recommends iai. We should also investigate what other compilers (eg rustc, cranelift) use.

[brendanzab]

Yeah, this would be nice! I need to be better at profiling as well to be honest. Stuff like memory usage and performance of binary parsing would be nice as well perhaps?

I think I’ve also seen some places that it’s worth taking CI-based profiling with a grain of salt due to it being run on shared infrastructure. Curious to know about best practices but it still might be a useful thing to do at any rate.