Refinement of Providers into Providers and ExternProviders

src/query.md

@@ -71,22 +71,24 @@ are cheaply cloneable; insert an `Rc` if necessary).
 
 If, however, the query is *not* in the cache, then the compiler will
 call the corresponding **provider** function. A provider is a function
-implemented in a specific module and **manually registered** into the
-[`Providers`][providers_struct] struct during compiler initialization.
-The macro system generates the [`Providers`][providers_struct] struct,
-which acts as a function table for all query implementations, where each
+implemented in a specific module and **manually registered** into either
+the [`Providers`][providers_struct] struct (for local crate queries) or
+the [`ExternProviders`][extern_providers_struct] struct (for external crate queries)
+during compiler initialization. The macro system generates both structs,
+which act as function tables for all query implementations, where each
 field is a function pointer to the actual provider.
 
-**Note:** The `Providers` struct is generated by macros and acts as a function table for all query implementations.
-It is **not** a Rust trait, but a plain struct with function pointer fields.
+**Note:** Both the `Providers` and `ExternProviders` structs are generated by macros and act as function tables for all query implementations.
+They are **not** Rust traits, but plain structs with function pointer fields.
 
 **Providers are defined per-crate.** The compiler maintains,
 internally, a table of providers for every crate, at least
-conceptually. Right now, there are really two sets: the providers for
-queries about the **local crate** (that is, the one being compiled)
-and providers for queries about **external crates** (that is,
-dependencies of the local crate). Note that what determines the crate
-that a query is targeting is not the *kind* of query, but the *key*.
+conceptually. There are two sets of providers:
+- The `Providers` struct for queries about the **local crate** (that is, the one being compiled)
+- The `ExternProviders` struct for queries about **external crates** (that is,
+dependencies of the local crate)
+
+Note that what determines the crate that a query is targeting is not the *kind* of query, but the *key*.
 For example, when you invoke `tcx.type_of(def_id)`, that could be a
 local query or an external query, depending on what crate the `def_id`
 is referring to (see the [`self::keys::Key`][Key] trait for more
@@ -119,59 +121,101 @@ they define both a `provide` and a `provide_extern` function, through
 
 ### How providers are set up
 
-When the tcx is created, it is given the providers by its creator using
-the [`Providers`][providers_struct] struct. This struct is generated by
-the macros here, but it is basically a big list of function pointers:
-
-[providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.Providers.html
+When the tcx is created, it is given both the local and external providers by its creator using
+the `Providers` struct from `rustc_middle::util`. This struct contains both the local and external providers:
 
 ```rust,ignore
-struct Providers {
-    type_of: for<'tcx> fn(TyCtxt<'tcx>, DefId) -> Ty<'tcx>,
-    // ... one field for each query
+pub struct Providers {
+    pub queries: crate::query::Providers,        // Local crate providers
+    pub extern_queries: crate::query::ExternProviders,  // External crate providers
+    pub hooks: crate::hooks::Providers,
 }
 ```
 
+Each of these provider structs is generated by the macros and contains function pointers for their respective queries.
+
 #### How are providers registered?
 
-The `Providers` struct is filled in during compiler initialization, mainly by the `rustc_driver` crate.  
+The provider structs are filled in during compiler initialization, mainly by the `rustc_driver` crate.  
 But the actual provider functions are implemented in various `rustc_*` crates (like `rustc_middle`, `rustc_hir_analysis`, etc).
 
 To register providers, each crate exposes a [`provide`][provide_fn] function that looks like this:
 
 [provide_fn]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/hir/fn.provide.html
 
 ```rust,ignore
-pub fn provide(providers: &mut Providers) {
-    *providers = Providers {
-        type_of,
-        // ... add more providers here
-        ..*providers
-    };
+pub fn provide(providers: &mut rustc_middle::util::Providers) {
+    providers.queries.type_of = type_of;
+    // ... add more local providers here
+
+    providers.extern_queries.type_of = extern_type_of;
+    // ... add more external providers here
+
+    providers.hooks.some_hook = some_hook;
+    // ... add more hooks here
 }
 ```
 
 - This function takes a mutable reference to the `Providers` struct and sets the fields to point to the correct provider functions.
-- You can also assign fields individually, e.g. `providers.type_of = type_of;`.
+- You can assign fields individually for each provider type (local, external, and hooks).
 
 #### Adding a new provider
 
 Suppose you want to add a new query called `fubar`. You would:
 
 1. Implement the provider function:
     ```rust,ignore
-    fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: DefId) -> Fubar<'tcx> { ... }
+    fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: LocalDefId) -> Fubar<'tcx> { ... }
     ```
 2. Register it in the `provide` function:
     ```rust,ignore
-    pub fn provide(providers: &mut Providers) {
-        *providers = Providers {
-            fubar,
-            ..*providers
+    pub fn provide(providers: &mut rustc_middle::util::Providers) {
+        providers.queries.fubar = fubar;
+    }
+    ```
+
+### How queries interact with external crate metadata
+
+When a query is made for an external crate (i.e., a dependency), the query system needs to load the information from that crate's metadata. 
+This is handled by the [`rustc_metadata` crate][rustc_metadata], which is responsible for decoding and providing the information stored in the `.rmeta` files.
+
+The process works like this:
+
+1. When a query is made, the query system first checks if the `DefId` refers to a local or external crate by checking if `def_id.krate == LOCAL_CRATE`.
+   This determines whether to use the local provider from [`Providers`][providers_struct] or the external provider from [`ExternProviders`][extern_providers_struct].
+
+2. For external crates, the query system will look for a provider in the [`ExternProviders`][extern_providers_struct] struct.
+   The `rustc_metadata` crate registers these external providers through the `provide_extern` function in `rustc_metadata/src/rmeta/decoder/cstore_impl.rs`. Just like:
+    ```rust
+    pub fn provide_extern(providers: &mut ExternProviders) {
+        providers.foo = |tcx, def_id| {
+            // Load and decode metadata for external crate
+            let cdata = CStore::from_tcx(tcx).get_crate_data(def_id.krate);
+            cdata.foo(def_id.index)
         };
+        // Register other external providers...
     }
     ```
 
+3. The metadata is stored in a binary format in `.rmeta` files that contains pre-computed information about the external crate, such as types, function signatures, trait implementations, and other information needed by the compiler. When an external query is made, the `rustc_metadata` crate:
+   - Loads the `.rmeta` file for the external crate
+   - Decodes the metadata using the `Decodable` trait
+   - Returns the decoded information to the query system
+
+This approach avoids recompiling external crates, allows for faster compilation of dependent crates, and enables incremental compilation to work across crate boundaries.
+Here is a simplified example, when you call `tcx.type_of(def_id)` for a type defined in an external crate, the query system will:
+1. Detect that the `def_id` refers to an external crate by checking `def_id.krate != LOCAL_CRATE`
+2. Call the appropriate provider from `ExternProviders` which was registered by `rustc_metadata`
+3. The provider will load and decode the type information from the external crate's metadata
+4. Return the decoded type to the caller
+
+This is why most `rustc_*` crates only need to provide local providers - the external providers are handled by the metadata system. 
+The only exception is when a crate needs to provide special handling for external queries, in which case it would implement both local and external providers.
+
+[rustc_metadata]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/index.html
+[providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.Providers.html
+[extern_providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.ExternProviders.html
+
 ---
 
 ## Adding a new query