2025 benchmark update

[Kobzol]

In 2025, it will be time to refresh and update our benchmarks again, based on our benchmark update policy.

The update policy, and a guide of how to actually update the benchmarks is described here. In short:

1. Download the new benchmark

$ cargo run --bin collector download -c Primary -a <library/binary> crate <name> <version>

2. Add Cargo.lock in the downloaded crate to git, even if it is ignored!
3. Commit
4. Copy perf-config.json and .patch files from the old version
5. Add [workspace] to Cargo.toml if it's not already there
6. Check that

$ cargo build -p collector; cargo run --bin collector bench_local `rustup +stable which rustc` --include <benchmark-name>

works

• If it fails because "patch does not apply", re-create the old patch files based on the updated benchmark source code. For each patch file, perform the same change as was in the original patch file, and then run git diff --patch --relative from the directory of the benchmark and store the output into the patch filename.

7. Modify collector/compile-benchmarks/README.md
8. Modify collector/compile-benchmarks/REUSE.toml
9. Commit
10. Run timings in old and new version and post the results to the PR description

time -p CARGO_INCREMENTAL=0 cargo check ; cargo clean
time -p CARGO_INCREMENTAL=0 cargo build ; cargo clean
time -p CARGO_INCREMENTAL=0 cargo build --release ; cargo clean

Progress

• bitmaps (@Kobzol)
  • Updated to 3.2.1 (Update bitmaps to 3.2.1 #2040, 14. 4.)
  • Removed old version (Remove bitmaps-3.1.0 benchmark #2075)
• cargo (@Kobzol)
  • Updated to 0.87.1 (Update cargo to 0.87.1 #2071, 16. 4.)
  • Removed old version
• clap
  • Updated to 4.5.36 (Update clap to 4.5.36 #2073)
  • Removed old version
• cranelift-codegen
  • Updated to 0.119.0 (Benchmark 2025 update cranelift codegen to 0.119.0 #2077, 23. 4.)
  • Removed old version
• diesel
  • Updated to X.Y.Z
  • Removed old version
• exa
  • Updated to X.Y.Z
  • Removed old version
• html5ever (@Kobzol)
  • Updated to 0.31.0 (Update html5ever to 0.31.0 #2069, 15. 4.)
  • Removed old version (Remove html5ever-0.26.0 benchmark #2076)
• hyper
  • Updated to X.Y.Z
  • Removed old version
• image
  • Updated to X.Y.Z
  • Removed old version
• libc
  • Updated to X.Y.Z
  • Removed old version
• nalgebra
  • Updated to X.Y.Z
  • Removed old version
• regex
  • Updated to X.Y.Z
  • Removed old version
• ripgrep
  • Updated to X.Y.Z
  • Removed old version
• ripgrep-tiny
  • Updated to X.Y.Z
  • Removed old version
• serde
  • Updated to X.Y.Z
  • Removed old version
• serde_derive
  • Updated to X.Y.Z
  • Removed old version
• stm32f4
  • Updated to X.Y.Z
  • Removed old version
• syn
  • Updated to X.Y.Z
  • Removed old version
• typenum
  • Updated to X.Y.Z
  • Removed old version
• unicode-normalization
  • Updated to X.Y.Z
  • Removed old version

[nnethercote]

As per the discussion on Zulip, we're going to wait a little bit and get some ducks lined up in a row before starting on this.

[Kobzol]

We have started the benchmark update process this week.