Auto merge of #142134 - workingjubilee:reject-unsupported-abi, r=<try>

Reject `extern "{abi}"` when the target does not support it

## What

Promote [`unsupported_fn_ptr_calling_conventions`] from a warning to a hard error, making sure edge-cases will not escape. We now emit hard errors for every case we would return `Invalid` from `AbiMap::canonize_abi` during AST to HIR lowering. In particular, these architecture-specific ABIs now only compile on their architectures[^1]:
  - amdgpu: "gpu-kernel"
  - arm: "aapcs", "C-cmse-nonsecure-entry"
  - avr: "avr-interrupt", "avr-non-blocking-interrupt"
  - msp430: "msp430-interrupt"
  - nvptx64: "gpu-kernel", "ptx-kernel"
  - riscv32 and riscv64: "riscv-interrupt-machine", "riscv-interrupt-supervisor"
  - x86: "thiscall"
  - x86 and x86_64: "x86-interrupt"
  - x86_64: "sysv64", "win64"

The panoply of ABIs that are logically x86-specific but actually permitted on all Windows targets remain supported on Windows, as they were before. For non-Windows targets they error if the architecture does not match.

Moving the check into AST lowering **is itself a breaking change in rare cases**, above and beyond the cases rustc currently warns about. See "Why or Why Not" for details.

## How

We modify rustc_ast_lowering to prevent unsupported ABIs from leaking through the HIR without being checked for target support. Previously ad-hoc checking on various HIR items required making sure we check every HIR item which could contain an `extern "{abi}"` string. This is a losing proposition compared to gating the lowering itself.

As a consequence, unsupported ABI strings will now hard-error instead of triggering the FCW `unsupported_fn_ptr_calling_conventions`.

However, per #86232 this does cause errors for rare usages of `extern "{abi}"` that were theoretically possible to write in Rust source, without previous warning or error. For instance, trait declarations without impls were never checked. These are the exact kinds of leakages that this new approach prevents.

This differs from the following PRs:
- #141435 is orthogonal, as it adds a new lint for ABIs we have not warned on and are not touched by this PR
- #141877 is subsumed by this, in that this simply cuts out bad functionality instead of adding epicycles for stable code

## Why or Why Not

We already made the decision to issue the `unsupported_fn_ptr_calling_conventions` future compatibility warning. It has warned in dependencies since #135767, which reached stable with Rust 1.87. That was released on 2025 May 17, and it is now June. As we already had erred on these ABI strings in most other positions, and warn on stable for function pointer types, this breakage has had reasonable foreshadowing.

Upgrading the warning to an error addresses a real problem. In some cases the Rust compiler can attempt to actually compute the ABI for calling a function. We could accept this case and compute unsupported ABIs according to some other ABI, silently[^0]. However, this obviously exposes Rust to errors in codegen. We cannot lower directly to the "obvious" ABI and then trust code generators like LLVM to reliably error on these cases, either.

Refactoring the compiler so we could defer more ABI computations would be possible, but seems weakly motivated. Even if we succeeded, we would at minimum risk:
- exposing the "whack-a-mole" problem but "approaching linking" instead of "leaving AST"
- making it harder to reason about functions we *can* lower further
- complicating the compiler for no clear benefit

A deprecation cycle for the edge-cases could be implemented first, but it is not very useful for such marginal cases, like this trait declaration without a definition:
```rust
pub trait UsedToSneakBy {
    pub extern "gpu-kernel" fn sneaky();
}
```

Upon any impl, even for provided fn within trait declarations, e.g. `pub extern "gpu-kernel" fn sneaky() {}`, different HIR types were used which would, in fact, get checked. Likewise with anything with function pointers. Thus we would be discussing deprecation cycles for code that is impotent or forewarned[^2].

Implementing a deprecation cycle _is_ possible, but it would likely require emitting multiple of a functionally identical warning or error on code that would not have multiple warnings or errors before. It is also not clear to me we would not find **another**, even more marginal edge-case that slipped through, as "things slip through" is the motivation for checking earlier. Additional effort spent on additional warnings should require committing to a hard limit first.

r? lang

Fixes #86232
Fixes #132430
Fixes #138738
Fixes #142107

[`unsupported_fn_ptr_calling_conventions`]: #130260
[^1]: Some already will not compile, due to reaching ICEs or LLVM errors.
[^0]:  We already do this for all `AbiStr` we cannot parse, pretending they are `ExternAbi::Rust`, but we also emit an error to prevent reaching too far into codegen.
[^2]: It actually did appear in two cases in rustc's test suite because we are a collection of Rust edge-cases by the simple fact that we don't care if the code actually runs. These cases were excised in c1db989.

Author: rust-bors[bot]

compiler/rustc_ast_lowering/src/item.rs

@@ -2,7 +2,7 @@ use rustc_abi::ExternAbi;
 use rustc_ast::ptr::P;
 use rustc_ast::visit::AssocCtxt;
 use rustc_ast::*;
-use rustc_errors::ErrorGuaranteed;
+use rustc_errors::{E0570, ErrorGuaranteed, struct_span_code_err};
 use rustc_hir::def::{DefKind, PerNS, Res};
 use rustc_hir::def_id::{CRATE_DEF_ID, LocalDefId};
 use rustc_hir::{self as hir, HirId, LifetimeSource, PredicateOrigin};
@@ -1617,9 +1617,20 @@ impl<'hir> LoweringContext<'_, 'hir> {
             self.error_on_invalid_abi(abi_str);
             ExternAbi::Rust
         });
-        let sess = self.tcx.sess;
-        let features = self.tcx.features();
-        gate_unstable_abi(sess, features, span, extern_abi);
+        let tcx = self.tcx;
+
+        // we can't do codegen for unsupported ABIs, so error now so we won't get farther
+        if !tcx.sess.target.is_abi_supported(extern_abi) {
+            struct_span_code_err!(
+                tcx.dcx(),
+                span,
+                E0570,
+                "`{extern_abi}` is not a supported ABI for the current target",
+            )
+            .emit();
+        }
+        // stability gate even things that are already errored on
+        gate_unstable_abi(tcx.sess, tcx.features(), span, extern_abi);
         extern_abi
     }
 

compiler/rustc_hir_analysis/src/check/check.rs

@@ -10,9 +10,7 @@ use rustc_hir::def::{CtorKind, DefKind};
 use rustc_hir::{LangItem, Node, intravisit};
 use rustc_infer::infer::{RegionVariableOrigin, TyCtxtInferExt};
 use rustc_infer::traits::{Obligation, ObligationCauseCode};
-use rustc_lint_defs::builtin::{
-    REPR_TRANSPARENT_EXTERNAL_PRIVATE_FIELDS, UNSUPPORTED_FN_PTR_CALLING_CONVENTIONS,
-};
+use rustc_lint_defs::builtin::REPR_TRANSPARENT_EXTERNAL_PRIVATE_FIELDS;
 use rustc_middle::hir::nested_filter;
 use rustc_middle::middle::resolve_bound_vars::ResolvedArg;
 use rustc_middle::middle::stability::EvalResult;
@@ -35,28 +33,6 @@ use {rustc_attr_data_structures as attrs, rustc_hir as hir};
 use super::compare_impl_item::check_type_bounds;
 use super::*;
 
-pub fn check_abi(tcx: TyCtxt<'_>, span: Span, abi: ExternAbi) {
-    if !tcx.sess.target.is_abi_supported(abi) {
-        struct_span_code_err!(
-            tcx.dcx(),
-            span,
-            E0570,
-            "`{abi}` is not a supported ABI for the current target",
-        )
-        .emit();
-    }
-}
-
-pub fn check_abi_fn_ptr(tcx: TyCtxt<'_>, hir_id: hir::HirId, span: Span, abi: ExternAbi) {
-    if !tcx.sess.target.is_abi_supported(abi) {
-        tcx.node_span_lint(UNSUPPORTED_FN_PTR_CALLING_CONVENTIONS, hir_id, span, |lint| {
-            lint.primary_message(format!(
-                "the calling convention {abi} is not supported on this target"
-            ));
-        });
-    }
-}
-
 fn check_struct(tcx: TyCtxt<'_>, def_id: LocalDefId) {
     let def = tcx.adt_def(def_id);
     let span = tcx.def_span(def_id);
@@ -779,7 +755,6 @@ pub(crate) fn check_item_type(tcx: TyCtxt<'_>, def_id: LocalDefId) {
             let hir::ItemKind::ForeignMod { abi, items } = it.kind else {
                 return;
             };
-            check_abi(tcx, it.span, abi);
 
             for item in items {
                 let def_id = item.id.owner_id.def_id;

compiler/rustc_hir_analysis/src/check/mod.rs

@@ -72,8 +72,7 @@ pub mod wfcheck;
 
 use std::num::NonZero;
 
-pub use check::{check_abi, check_abi_fn_ptr};
-use rustc_abi::{ExternAbi, VariantIdx};
+use rustc_abi::VariantIdx;
 use rustc_data_structures::fx::{FxHashSet, FxIndexMap};
 use rustc_errors::{Diag, ErrorGuaranteed, pluralize, struct_span_code_err};
 use rustc_hir::def_id::{DefId, LocalDefId};

compiler/rustc_hir_analysis/src/hir_ty_lowering/mod.rs

@@ -50,7 +50,6 @@ use rustc_trait_selection::traits::wf::object_region_bounds;
 use rustc_trait_selection::traits::{self, ObligationCtxt};
 use tracing::{debug, instrument};
 
-use crate::check::check_abi_fn_ptr;
 use crate::errors::{AmbiguousLifetimeBound, BadReturnTypeNotation};
 use crate::hir_ty_lowering::errors::{GenericsArgsErrExtend, prohibit_assoc_item_constraint};
 use crate::hir_ty_lowering::generics::{check_generic_arg_count, lower_generic_args};
@@ -2736,12 +2735,6 @@ impl<'tcx> dyn HirTyLowerer<'tcx> + '_ {
         let fn_ty = tcx.mk_fn_sig(input_tys, output_ty, decl.c_variadic, safety, abi);
         let bare_fn_ty = ty::Binder::bind_with_vars(fn_ty, bound_vars);
 
-        if let hir::Node::Ty(hir::Ty { kind: hir::TyKind::BareFn(bare_fn_ty), span, .. }) =
-            tcx.hir_node(hir_id)
-        {
-            check_abi_fn_ptr(tcx, hir_id, *span, bare_fn_ty.abi);
-        }
-
         // reject function types that violate cmse ABI requirements
         cmse::validate_cmse_abi(self.tcx(), self.dcx(), hir_id, abi, bare_fn_ty);
 

compiler/rustc_hir_typeck/src/lib.rs

@@ -49,7 +49,6 @@ use rustc_errors::{Applicability, ErrorGuaranteed, pluralize, struct_span_code_e
 use rustc_hir as hir;
 use rustc_hir::def::{DefKind, Res};
 use rustc_hir::{HirId, HirIdMap, Node};
-use rustc_hir_analysis::check::check_abi;
 use rustc_hir_analysis::hir_ty_lowering::HirTyLowerer;
 use rustc_infer::traits::{ObligationCauseCode, ObligationInspector, WellFormedLoc};
 use rustc_middle::query::Providers;
@@ -150,8 +149,6 @@ fn typeck_with_inspect<'tcx>(
             tcx.fn_sig(def_id).instantiate_identity()
         };
 
-        check_abi(tcx, span, fn_sig.abi());
-
         // Compute the function signature from point of view of inside the fn.
         let mut fn_sig = tcx.liberate_late_bound_regions(def_id.to_def_id(), fn_sig);
 

compiler/rustc_lint/src/lib.rs

@@ -610,6 +610,7 @@ fn register_builtins(store: &mut LintStore) {
         "converted into hard error, see issue #127323 \
          <https://github.com/rust-lang/rust/issues/127323> for more information",
     );
+    store.register_removed("unsupported_fn_ptr_calling_conventions", "converted into hard error");
     store.register_removed(
         "undefined_naked_function_abi",
         "converted into hard error, see PR #139001 \

compiler/rustc_lint_defs/src/builtin.rs

@@ -123,7 +123,6 @@ declare_lint_pass! {
         UNSAFE_OP_IN_UNSAFE_FN,
         UNSTABLE_NAME_COLLISIONS,
         UNSTABLE_SYNTAX_PRE_EXPANSION,
-        UNSUPPORTED_FN_PTR_CALLING_CONVENTIONS,
         UNUSED_ASSIGNMENTS,
         UNUSED_ASSOCIATED_TYPE_BOUNDS,
         UNUSED_ATTRIBUTES,

tests/crashes/132430.rs

@@ -19,7 +19,7 @@
 // gdb-check:type = type_names::GenericStruct<type_names::mod1::Struct2, type_names::mod1::mod2::Struct3>
 
 // gdb-command:whatis generic_struct2
-// gdb-check:type = type_names::GenericStruct<type_names::Struct1, extern "fastcall" fn(isize) -> usize>
+// gdb-check:type = type_names::GenericStruct<type_names::Struct1, extern "system" fn(isize) -> usize>
 
 // gdb-command:whatis mod_struct
 // gdb-check:type = type_names::mod1::Struct2
@@ -379,7 +379,7 @@ fn main() {
     let simple_struct = Struct1;
     let generic_struct1: GenericStruct<mod1::Struct2, mod1::mod2::Struct3> =
         GenericStruct(PhantomData);
-    let generic_struct2: GenericStruct<Struct1, extern "fastcall" fn(isize) -> usize> =
+    let generic_struct2: GenericStruct<Struct1, extern "system" fn(isize) -> usize> =
         GenericStruct(PhantomData);
     let mod_struct = mod1::Struct2;
 

tests/crashes/138738.rs

@@ -440,7 +440,7 @@ trait TraitAddExternModifier {
 
 // Change extern "C" to extern "stdcall"
 #[cfg(any(cfail1,cfail4))]
-trait TraitChangeExternCToRustIntrinsic {
+trait TraitChangeExternCToExternSystem {
     // --------------------------------------------------------------
     // -------------------------
     // --------------------------------------------------------------
@@ -458,7 +458,7 @@ trait TraitChangeExternCToRustIntrinsic {
     #[rustc_clean(cfg="cfail3")]
     #[rustc_clean(except="opt_hir_owner_nodes,fn_sig", cfg="cfail5")]
     #[rustc_clean(cfg="cfail6")]
-    extern "stdcall" fn method();
+    extern "system" fn method();
 }
 
 

tests/debuginfo/type-names.rs

@@ -1,4 +1,4 @@
-#![feature(abi_vectorcall)]
+#![feature(rust_cold_cc)]
 
 //@ is "$.index[?(@.name=='AbiRust')].inner.type_alias.type.function_pointer.header.abi" \"Rust\"
 pub type AbiRust = fn();
@@ -15,8 +15,5 @@ pub type AbiCUnwind = extern "C-unwind" fn();
 //@ is "$.index[?(@.name=='AbiSystemUnwind')].inner.type_alias.type.function_pointer.header.abi" '{"System": {"unwind": true}}'
 pub type AbiSystemUnwind = extern "system-unwind" fn();
 
-//@ is "$.index[?(@.name=='AbiVecorcall')].inner.type_alias.type.function_pointer.header.abi.Other" '"\"vectorcall\""'
-pub type AbiVecorcall = extern "vectorcall" fn();
-
-//@ is "$.index[?(@.name=='AbiVecorcallUnwind')].inner.type_alias.type.function_pointer.header.abi.Other" '"\"vectorcall-unwind\""'
-pub type AbiVecorcallUnwind = extern "vectorcall-unwind" fn();
+//@ is "$.index[?(@.name=='AbiRustCold')].inner.type_alias.type.function_pointer.header.abi.Other" '"\"rust-cold\""'
+pub type AbiRustCold = extern "rust-cold" fn();

tests/incremental/hashes/trait_defs.rs

@@ -1,4 +1,4 @@
-#![feature(abi_vectorcall)]
+#![feature(rust_cold_cc)]
 
 //@ is "$.index[?(@.name=='abi_rust')].inner.function.header.abi" \"Rust\"
 pub fn abi_rust() {}
@@ -15,8 +15,5 @@ pub extern "C-unwind" fn abi_c_unwind() {}
 //@ is "$.index[?(@.name=='abi_system_unwind')].inner.function.header.abi" '{"System": {"unwind": true}}'
 pub extern "system-unwind" fn abi_system_unwind() {}
 
-//@ is "$.index[?(@.name=='abi_vectorcall')].inner.function.header.abi.Other" '"\"vectorcall\""'
-pub extern "vectorcall" fn abi_vectorcall() {}
-
-//@ is "$.index[?(@.name=='abi_vectorcall_unwind')].inner.function.header.abi.Other" '"\"vectorcall-unwind\""'
-pub extern "vectorcall-unwind" fn abi_vectorcall_unwind() {}
+//@ is "$.index[?(@.name=='abi_rust_cold')].inner.function.header.abi.Other" '"\"rust-cold\""'
+pub extern "rust-cold" fn abi_rust_cold() {}

tests/rustdoc-json/fn_pointer/abi.rs

@@ -1,5 +1,4 @@
-#![feature(abi_vectorcall)]
-
+#![feature(rust_cold_cc)]
 //@ has "$.index[?(@.name=='Foo')]"
 pub struct Foo;
 
@@ -19,11 +18,8 @@ impl Foo {
     //@ is "$.index[?(@.name=='abi_system_unwind')].inner.function.header.abi" '{"System": {"unwind": true}}'
     pub extern "system-unwind" fn abi_system_unwind() {}
 
-    //@ is "$.index[?(@.name=='abi_vectorcall')].inner.function.header.abi.Other" '"\"vectorcall\""'
-    pub extern "vectorcall" fn abi_vectorcall() {}
-
-    //@ is "$.index[?(@.name=='abi_vectorcall_unwind')].inner.function.header.abi.Other" '"\"vectorcall-unwind\""'
-    pub extern "vectorcall-unwind" fn abi_vectorcall_unwind() {}
+    //@ is "$.index[?(@.name=='abi_rust_cold')].inner.function.header.abi.Other" '"\"rust-cold\""'
+    pub extern "rust-cold" fn abi_rust_cold() {}
 }
 
 pub trait Bar {
@@ -42,9 +38,6 @@ pub trait Bar {
     //@ is "$.index[?(@.name=='trait_abi_system_unwind')].inner.function.header.abi" '{"System": {"unwind": true}}'
     extern "system-unwind" fn trait_abi_system_unwind() {}
 
-    //@ is "$.index[?(@.name=='trait_abi_vectorcall')].inner.function.header.abi.Other" '"\"vectorcall\""'
-    extern "vectorcall" fn trait_abi_vectorcall() {}
-
-    //@ is "$.index[?(@.name=='trait_abi_vectorcall_unwind')].inner.function.header.abi.Other" '"\"vectorcall-unwind\""'
-    extern "vectorcall-unwind" fn trait_abi_vectorcall_unwind() {}
+    //@ is "$.index[?(@.name=='trait_abi_rust_cold')].inner.function.header.abi.Other" '"\"rust-cold\""'
+    extern "rust-cold" fn trait_abi_rust_cold() {}
 }

tests/rustdoc-json/fns/abi.rs

@@ -0,0 +1,27 @@
+#![feature(abi_vectorcall)]
+//@ only-x86_64
+
+//@ is "$.index[?(@.name=='AbiVectorcall')].inner.type_alias.type.function_pointer.header.abi.Other" '"\"vectorcall\""'
+pub type AbiVectorcall = extern "vectorcall" fn();
+
+//@ is "$.index[?(@.name=='AbiVectorcallUnwind')].inner.type_alias.type.function_pointer.header.abi.Other" '"\"vectorcall-unwind\""'
+pub type AbiVectorcallUnwind = extern "vectorcall-unwind" fn();
+
+//@ has "$.index[?(@.name=='Foo')]"
+pub struct Foo;
+
+impl Foo {
+    //@ is "$.index[?(@.name=='abi_vectorcall')].inner.function.header.abi.Other" '"\"vectorcall\""'
+    pub extern "vectorcall" fn abi_vectorcall() {}
+
+    //@ is "$.index[?(@.name=='abi_vectorcall_unwind')].inner.function.header.abi.Other" '"\"vectorcall-unwind\""'
+    pub extern "vectorcall-unwind" fn abi_vectorcall_unwind() {}
+}
+
+pub trait Bar {
+    //@ is "$.index[?(@.name=='trait_abi_vectorcall')].inner.function.header.abi.Other" '"\"vectorcall\""'
+    extern "vectorcall" fn trait_abi_vectorcall() {}
+
+    //@ is "$.index[?(@.name=='trait_abi_vectorcall_unwind')].inner.function.header.abi.Other" '"\"vectorcall-unwind\""'
+    extern "vectorcall-unwind" fn trait_abi_vectorcall_unwind() {}
+}

tests/rustdoc-json/methods/abi.rs

@@ -0,0 +1,7 @@
+#![feature(abi_gpu_kernel)]
+// Check we error before unsupported ABIs reach codegen stages.
+
+fn main() {
+    let a = unsafe { core::mem::transmute::<usize, extern "gpu-kernel" fn(i32)>(4) }(2);
+    //~^ ERROR E0570
+}

tests/rustdoc-json/vectorcall.rs

@@ -0,0 +1,9 @@
+error[E0570]: `"gpu-kernel"` is not a supported ABI for the current target
+  --> $DIR/unsupported-abi-transmute.rs:5:59
+   |
+LL |     let a = unsafe { core::mem::transmute::<usize, extern "gpu-kernel" fn(i32)>(4) }(2);
+   |                                                           ^^^^^^^^^^^^
+
+error: aborting due to 1 previous error
+
+For more information about this error, try `rustc --explain E0570`.

tests/ui/abi/unsupported-abi-transmute.rs

@@ -0,0 +1,9 @@
+//@ compile-flags: --crate-type=lib
+//@ edition: 2018
+#![feature(abi_gpu_kernel)]
+struct Test;
+
+impl Test {
+    pub extern "gpu-kernel" fn test(val: &str) {}
+    //~^ ERROR [E0570]
+}

tests/ui/abi/unsupported-abi-transmute.stderr

@@ -0,0 +1,9 @@
+error[E0570]: `"gpu-kernel"` is not a supported ABI for the current target
+  --> $DIR/unsupported-extern-abi-in-type-impl.rs:7:16
+   |
+LL |     pub extern "gpu-kernel" fn test(val: &str) {}
+   |                ^^^^^^^^^^^^
+
+error: aborting due to 1 previous error
+
+For more information about this error, try `rustc --explain E0570`.

tests/ui/abi/unsupported-extern-abi-in-type-impl.rs

@@ -0,0 +1,11 @@
+//@ compile-flags: --crate-type=lib
+
+#![feature(abi_gpu_kernel)]
+
+trait T {
+    extern "gpu-kernel" fn mu();
+    //~^ ERROR[E0570]
+}
+
+type TAU = extern "gpu-kernel" fn();
+//~^ ERROR[E0570]

tests/ui/abi/unsupported-extern-abi-in-type-impl.stderr

@@ -0,0 +1,15 @@
+error[E0570]: `"gpu-kernel"` is not a supported ABI for the current target
+  --> $DIR/unsupported-trait-decl.rs:6:12
+   |
+LL |     extern "gpu-kernel" fn mu();
+   |            ^^^^^^^^^^^^
+
+error[E0570]: `"gpu-kernel"` is not a supported ABI for the current target
+  --> $DIR/unsupported-trait-decl.rs:10:19
+   |
+LL | type TAU = extern "gpu-kernel" fn();
+   |                   ^^^^^^^^^^^^
+
+error: aborting due to 2 previous errors
+
+For more information about this error, try `rustc --explain E0570`.

tests/ui/abi/unsupported-trait-decl.rs

@@ -0,0 +1,18 @@
+// FIXME(workingjubilee): add revisions and generalize to other platform-specific varargs ABIs,
+// preferably after the only-arch directive is enhanced with an "or pattern" syntax
+//@ only-x86_64
+
+// We have to use this flag to force ABI computation of an invalid ABI
+//@ compile-flags: -Clink-dead-code
+
+#![feature(extended_varargs_abi_support)]
+
+// sometimes fn ptrs with varargs make layout and ABI computation ICE
+// as found in https://github.com/rust-lang/rust/issues/142107
+
+fn aapcs(f: extern "aapcs" fn(usize, ...)) {
+//~^ ERROR [E0570]
+// Note we DO NOT have to actually make a call to trigger the ICE!
+}
+
+fn main() {}

tests/ui/abi/unsupported-trait-decl.stderr

@@ -0,0 +1,9 @@
+error[E0570]: `"aapcs"` is not a supported ABI for the current target
+  --> $DIR/unsupported-varargs-fnptr.rs:13:20
+   |
+LL | fn aapcs(f: extern "aapcs" fn(usize, ...)) {
+   |                    ^^^^^^^
+
+error: aborting due to 1 previous error
+
+For more information about this error, try `rustc --explain E0570`.