Add post-mono MIR optimizations

[saethlin]

Before this PR, all MIR passes had to operate on polymorphic MIR. Thus any MIR transform maybe unable to determine the type of an argument or local (because it's still generic) or it may be unable to determine which function a Call terminator is calling (because it's still generic).

MIR transforms are a highly maintainable solution to a number of compiler problems, but this polymorphic limitation means that they are cannot solve some of our problems that we'd like them to; the most recent examples that come to mind are #134082 which has extra limitations because of the polymorphic inliner, and #139088 which is explicitly waiting for post-mono MIR passes to happen.

In addition, the lack of post-mono MIR optimizations means that MIR optimizations just miss out on profitable optimizations, which are so valuable that we've added kludges like #121421 (a MIR traversal that you better only run at mono-time).

In addition, rustc_codegen_ssa is riddled with on-the-fly monomorphization and optimization; the logic for these trick that we do in codegen in my experience are hard to maintain, and I would much rather have those implemented in a MIR transform.

So this PR adds a new query codegen_mir (the MIR for codegen, not that I like the name). I've then replaced some of the kludges in rustc_codegen_ssa with PostMono variants of existing MIR transforms.

I've also un-querified check_mono_item and put it at the end of the post-mono pass list. Those checks should be post-mono passes too, but I've tried to keep this PR to a reviewable size. It's easy to imagine lots of other places to use post-mono MIR opts and I want the usefulness of this to be clear while the diff is also manageable.

---

This PR has a perf regression. I've hammered on the perf in a number of ways to get it down to what it is. incr-full builds suffer the most because they need to clone, intern, and cache a monomorphized copy of every MIR body. Things are mixed for every other build scenario. In almost all cases, binary sizes improve.

[saethlin]

@bors try @rust-timer queue

[bors]

⌛ Trying commit a211812 with merge b141564...

[bors]

☀️ Try build successful - checks-actions
Build commit: b141564 (b1415647cdfcdd1b8dc5ed5f9a5aba87ade0b225)

[rust-timer]

Finished benchmarking commit (b141564): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

	mean	range	count
Regressions ❌ (primary)	12.2%	[0.2%, 93.7%]	163
Regressions ❌ (secondary)	6.9%	[0.2%, 266.3%]	119
Improvements ✅ (primary)	-0.7%	[-3.0%, -0.2%]	6
Improvements ✅ (secondary)	-11.1%	[-33.8%, -0.2%]	12
All ❌✅ (primary)	11.7%	[-3.0%, 93.7%]	169

Max RSS (memory usage)

Results (primary 14.5%, secondary 1.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	14.5%	[0.7%, 56.9%]	108
Regressions ❌ (secondary)	4.5%	[0.6%, 12.8%]	34
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-22.2%	[-24.2%, -19.2%]	4
All ❌✅ (primary)	14.5%	[0.7%, 56.9%]	108

Cycles

Results (primary 22.8%, secondary 13.8%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	23.0%	[0.8%, 108.5%]	111
Regressions ❌ (secondary)	19.4%	[1.0%, 223.4%]	42
Improvements ✅ (primary)	-3.0%	[-3.0%, -3.0%]	1
Improvements ✅ (secondary)	-33.2%	[-42.8%, -1.3%]	5
All ❌✅ (primary)	22.8%	[-3.0%, 108.5%]	112

Binary size

Results (primary -0.3%, secondary -2.3%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.8%	[0.0%, 2.3%]	7
Regressions ❌ (secondary)	0.1%	[0.1%, 0.1%]	1
Improvements ✅ (primary)	-0.4%	[-1.7%, -0.0%]	76
Improvements ✅ (secondary)	-2.4%	[-25.8%, -0.0%]	65
All ❌✅ (primary)	-0.3%	[-1.7%, 2.3%]	83

Bootstrap: 781.427s -> 807.023s (3.28%)
Artifact size: 331.96 MiB -> 332.21 MiB (0.08%)

[saethlin]

@bors try @rust-timer queue

[bors]

⌛ Trying commit 6f6737a with merge 9233d9f...

[bors]

☀️ Try build successful - checks-actions
Build commit: 9233d9f (9233d9f83ca672be3b2cfa697806fdb7c8970490)

[rust-timer]

Finished benchmarking commit (9233d9f): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

	mean	range	count
Regressions ❌ (primary)	7.6%	[0.1%, 59.9%]	151
Regressions ❌ (secondary)	2.9%	[0.2%, 18.7%]	107
Improvements ✅ (primary)	-3.0%	[-3.0%, -3.0%]	1
Improvements ✅ (secondary)	-6.6%	[-64.0%, -0.3%]	11
All ❌✅ (primary)	7.5%	[-3.0%, 59.9%]	152

Max RSS (memory usage)

Results (primary 11.3%, secondary 2.4%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	12.9%	[1.3%, 52.1%]	93
Regressions ❌ (secondary)	3.6%	[2.2%, 5.9%]	10
Improvements ✅ (primary)	-2.7%	[-4.3%, -0.8%]	10
Improvements ✅ (secondary)	-3.4%	[-3.5%, -3.4%]	2
All ❌✅ (primary)	11.3%	[-4.3%, 52.1%]	103

Cycles

Results (primary 10.6%, secondary 3.2%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	10.7%	[1.0%, 50.1%]	94
Regressions ❌ (secondary)	5.4%	[1.7%, 18.4%]	37
Improvements ✅ (primary)	-3.1%	[-3.1%, -3.1%]	1
Improvements ✅ (secondary)	-17.2%	[-62.3%, -1.6%]	4
All ❌✅ (primary)	10.6%	[-3.1%, 50.1%]	95

Binary size

Results (primary -0.1%, secondary -0.3%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.7%	[0.0%, 2.4%]	9
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.2%	[-0.8%, -0.0%]	69
Improvements ✅ (secondary)	-0.3%	[-0.8%, -0.0%]	51
All ❌✅ (primary)	-0.1%	[-0.8%, 2.4%]	78

Bootstrap: 782.104s -> 806.252s (3.09%)
Artifact size: 332.57 MiB -> 332.81 MiB (0.07%)

[saethlin]

@bors try @rust-timer queue

[bors]

⌛ Trying commit 4ae3542 with merge 174810c...

[saethlin]

Not sure who's good to review PRs about query system and codegen. @oli-obk ?

r? oli-obk

[rustbot]

Some changes occurred in compiler/rustc_codegen_ssa

cc @WaffleLapkin

Some changes occurred in tests/codegen/sanitizer

cc @rcvalle

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

[oli-obk]

@bors try @rust-timer queue

[bors]

⌛ Trying commit 6a7b022 with merge 10fa210...

[bors]

☀️ Try build successful - checks-actions
Build commit: 10fa210 (10fa210a552831aeacbe37bdaf65c8f8b932a56d)

[rust-timer]

Finished benchmarking commit (10fa210): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

	mean	range	count
Regressions ❌ (primary)	0.8%	[0.2%, 3.9%]	135
Regressions ❌ (secondary)	1.1%	[0.1%, 19.6%]	78
Improvements ✅ (primary)	-0.9%	[-1.6%, -0.3%]	4
Improvements ✅ (secondary)	-1.0%	[-2.8%, -0.3%]	5
All ❌✅ (primary)	0.8%	[-1.6%, 3.9%]	139

Max RSS (memory usage)

Results (primary 8.4%, secondary 3.1%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	8.7%	[0.5%, 31.2%]	58
Regressions ❌ (secondary)	3.1%	[2.8%, 3.6%]	3
Improvements ✅ (primary)	-0.8%	[-1.1%, -0.5%]	2
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	8.4%	[-1.1%, 31.2%]	60

Cycles

Results (primary 2.2%, secondary 3.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.2%	[0.4%, 4.7%]	43
Regressions ❌ (secondary)	4.3%	[1.1%, 18.4%]	9
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.7%	[-1.7%, -1.7%]	1
All ❌✅ (primary)	2.2%	[0.4%, 4.7%]	43

Binary size

Results (primary -0.4%, secondary -0.3%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.4%	[0.1%, 1.2%]	4
Regressions ❌ (secondary)	0.1%	[0.1%, 0.1%]	1
Improvements ✅ (primary)	-0.4%	[-1.3%, -0.0%]	91
Improvements ✅ (secondary)	-0.3%	[-0.8%, -0.0%]	57
All ❌✅ (primary)	-0.4%	[-1.3%, 1.2%]	95

Bootstrap: 776.211s -> 784.617s (1.08%)
Artifact size: 365.21 MiB -> 364.79 MiB (-0.12%)

[oli-obk]

Uh I guess remove this function in a follow-up? 😆

[saethlin]

Yeah. Leaving this stub in is part of me trying to keep the diff reviewable.

[oli-obk]

one thing you can try to improve compile times here is no_hash. Since we are not storing it in the incremental caches anyway, we need to recompute it always as the only callers are from codegen, which either gets loaded from cache, so we don't need to call this one, or it doesn't get loaded from cache, then we can just reevaluate it. I think eval_always would make perf worse again, as we'd rerun it even if it's not needed?

[oli-obk]

Some of the perf regressions show about twice as many build_codegen_mir invocations as optimized_mir invocations. Could it make sense to run everything from build_codegen_mir in optimized_mir for already monomorphic bodies? And then have build_codegen_mir just forward that body (without even re-arena-allocating the body)?

[saethlin]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

[bors]

🔒 Merge conflict

This pull request and the master branch diverged in a way that cannot be automatically merged. Please rebase on top of the latest master branch, and let the reviewer approve again.

How do I rebase?

Assuming self is your fork and upstream is this repository, you can resolve the conflict following these steps:

1. git checkout post-mono-mir-opts (switch to your branch)
2. git fetch upstream master (retrieve the latest master)
3. git rebase upstream/master -p (rebase on top of it)
4. Follow the on-screen instruction to resolve conflicts (check git status if you got lost).
5. git push self post-mono-mir-opts --force-with-lease (update this PR)

You may also read Git Rebasing to Resolve Conflicts by Drew Blessing for a short tutorial.

Please avoid the "Resolve conflicts" button on GitHub. It uses git merge instead of git rebase which makes the PR commit history more difficult to read.

Sometimes step 4 will complete without asking for resolution. This is usually due to difference between how Cargo.lock conflict is handled during merge and rebase. This is normal, and you should still perform step 5 to update this PR.

Error message

Removing tests/ui/mir/validate/critical-edge.rs
Removing tests/codegen/skip-mono-inside-if-false.rs
Auto-merging compiler/rustc_monomorphize/src/collector.rs
Auto-merging compiler/rustc_mir_transform/src/validate.rs
Auto-merging compiler/rustc_mir_transform/src/lib.rs
CONFLICT (content): Merge conflict in compiler/rustc_mir_transform/src/lib.rs
Auto-merging compiler/rustc_middle/src/ty/mod.rs
Auto-merging compiler/rustc_middle/src/query/mod.rs
Auto-merging compiler/rustc_middle/src/mir/mod.rs
Auto-merging compiler/rustc_codegen_ssa/src/mir/block.rs
Auto-merging compiler/rustc_codegen_cranelift/src/base.rs
Auto-merging Cargo.lock
Automatic merge failed; fix conflicts and then commit the result.

[rust-log-analyzer]

The job x86_64-gnu-llvm-19 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

#18 exporting to docker image format
#18 sending tarball 20.6s done
#18 DONE 27.1s
##[endgroup]
Setting extra environment values for docker:  --env ENABLE_GCC_CODEGEN=1 --env GCC_EXEC_PREFIX=/usr/lib/gcc/
[CI_JOB_NAME=x86_64-gnu-llvm-19]
[CI_JOB_NAME=x86_64-gnu-llvm-19]
debug: `DISABLE_CI_RUSTC_IF_INCOMPATIBLE` configured.
---
sccache: Listening on address 127.0.0.1:4226
##[group]Configure the build
configure: processing command line
configure: 
configure: build.configure-args := ['--build=x86_64-unknown-linux-gnu', '--llvm-root=/usr/lib/llvm-19', '--enable-llvm-link-shared', '--set', 'rust.randomize-layout=true', '--set', 'rust.thin-lto-import-instr-limit=10', '--set', 'build.print-step-timings', '--enable-verbose-tests', '--set', 'build.metrics', '--enable-verbose-configure', '--enable-sccache', '--disable-manage-submodules', '--enable-locked-deps', '--enable-cargo-native-static', '--set', 'rust.codegen-units-std=1', '--set', 'dist.compression-profile=balanced', '--dist-compression-formats=xz', '--set', 'rust.lld=false', '--disable-dist-src', '--release-channel=nightly', '--enable-debug-assertions', '--enable-overflow-checks', '--enable-llvm-assertions', '--set', 'rust.verify-llvm-ir', '--set', 'rust.codegen-backends=llvm,cranelift,gcc', '--set', 'llvm.static-libstdcpp', '--set', 'gcc.download-ci-gcc=true', '--enable-new-symbol-mangling']
configure: build.build          := x86_64-unknown-linux-gnu
configure: target.x86_64-unknown-linux-gnu.llvm-config := /usr/lib/llvm-19/bin/llvm-config
configure: llvm.link-shared     := True
configure: rust.randomize-layout := True
configure: rust.thin-lto-import-instr-limit := 10
---
[RUSTC-TIMING] rustc_ty_utils test:false 22.111
error[E0061]: this function takes 2 arguments but 1 argument was supplied
   --> compiler/rustc_mir_transform/src/lib.rs:797:39
    |
797 |         transform_to_codegen_mir(tcx, Instance::mono(did), &mut body);
    |                                       ^^^^^^^^^^^^^^-----
    |                                                     ||
    |                                                     |expected `TyCtxt<'_>`, found `LocalDefId`
    |                                                     argument #2 of type `rustc_span::def_id::DefId` is missing
    |
note: associated function defined here
   --> /checkout/compiler/rustc_middle/src/ty/instance.rs:502:12
    |
502 |     pub fn mono(tcx: TyCtxt<'tcx>, def_id: DefId) -> Instance<'tcx> {
    |            ^^^^
help: provide the argument
    |
797 -         transform_to_codegen_mir(tcx, Instance::mono(did), &mut body);
797 +         transform_to_codegen_mir(tcx, Instance::mono(/* rustc_middle::ty::TyCtxt<'_> */, /* rustc_span::def_id::DefId */), &mut body);
    |

error[E0308]: mismatched types
   --> compiler/rustc_mir_transform/src/lib.rs:815:9
    |
803 | pub fn build_codegen_mir<'tcx>(tcx: TyCtxt<'tcx>, instance: Instance<'tcx>) -> Body<'tcx> {
    |                                                                                ---------- expected `rustc_middle::mir::Body<'tcx>` because of return type
...
815 |         body
    |         ^^^^ expected `Body<'_>`, found `&Body<'_>`

Some errors have detailed explanations: E0061, E0308.
For more information about an error, try `rustc --explain E0061`.
[RUSTC-TIMING] rustc_mir_transform test:false 5.261
error: could not compile `rustc_mir_transform` (lib) due to 2 previous errors

[saethlin]

Could it make sense to run everything from build_codegen_mir in optimized_mir for already monomorphic bodies? And then have build_codegen_mir just forward that body (without even re-arena-allocating the body)?

I got most of the way through implementing this before I realized that it's not this easy.

One of the special powers that post-mono GVN and InstSimplify get is to always evaluate the UbChecks intrinsic and do simplifications based on that. If we run those post-mono passes as part of optimized_mir, we'd strip the UB check out of hint::unreachable_unchecked.

I'm still curious to benchmark this because maybe there's some kind of angle where we check is_polymorphic || contains_ub_check