Refinement of Providers into Providers and ExternProviders #2469
+76
−32
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rustbot
added
the
S-waiting-on-review
Signed-off-by: xizheyin <[email protected]>
Signed-off-by: xizheyin <[email protected]>
|
first commit for |
Noratrieb
reviewed
Jun 16, 2025
src/query.md
Outdated
| pub fn provide(providers: &mut rustc_middle::util::Providers) { | ||
| providers.queries.fubar = fubar; | ||
| // If you need an external provider: | ||
| providers.extern_queries.fubar = extern_fubar; |
There was a problem hiding this comment.
this is not really helpful, I'd prefer to remove this and instead have a section on how the query system interacts with metadata for extern providers (you don't have to write that now, you can just remove the example)
|
|
||
| #### Adding a new provider | ||
|
|
||
| Suppose you want to add a new query called `fubar`. You would: | ||
|
|
||
| 1. Implement the provider function: | ||
| ```rust,ignore | ||
| fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: DefId) -> Fubar<'tcx> { ... } | ||
| fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: LocalDefId) -> Fubar<'tcx> { ... } |
There was a problem hiding this comment.
@tshepang suggest to make it LocalDefId. #2465 (comment)
I just went through the source code and realized that the key for local providers is pretty much LocalDefId. Which one are we going to choose?
Signed-off-by: xizheyin <[email protected]>
xizheyin
commented
Jun 16, 2025
Comment on lines
+177
to
+217
| ### How queries interact with external crate metadata | ||
|
|
||
| When a query is made for an external crate (i.e., a dependency), the query system needs to load the information from that crate's metadata. | ||
| This is handled by the [`rustc_metadata` crate][rustc_metadata], which is responsible for decoding and providing the information stored in the `.rmeta` files. | ||
|
|
||
| The process works like this: | ||
|
|
||
| 1. When a query is made, the query system first checks if the `DefId` refers to a local or external crate by checking if `def_id.krate == LOCAL_CRATE`. | ||
| This determines whether to use the local provider from [`Providers`][providers_struct] or the external provider from [`ExternProviders`][extern_providers_struct]. | ||
|
|
||
| 2. For external crates, the query system will look for a provider in the [`ExternProviders`][extern_providers_struct] struct. | ||
| The `rustc_metadata` crate registers these external providers through the `provide_extern` function in `rustc_metadata/src/rmeta/decoder/cstore_impl.rs`. Just like: | ||
| ```rust | ||
| pub fn provide_extern(providers: &mut ExternProviders) { | ||
| providers.foo = |tcx, def_id| { | ||
| // Load and decode metadata for external crate | ||
| let cdata = CStore::from_tcx(tcx).get_crate_data(def_id.krate); | ||
| cdata.foo(def_id.index) | ||
| }; | ||
| // Register other external providers... | ||
| } | ||
| ``` | ||
|
|
||
| 3. The metadata is stored in a binary format in `.rmeta` files that contains pre-computed information about the external crate, such as types, function signatures, trait implementations, and other information needed by the compiler. When an external query is made, the `rustc_metadata` crate: | ||
| - Loads the `.rmeta` file for the external crate | ||
| - Decodes the metadata using the `Decodable` trait | ||
| - Returns the decoded information to the query system | ||
|
|
||
| This approach avoids recompiling external crates, allows for faster compilation of dependent crates, and enables incremental compilation to work across crate boundaries. | ||
| Here is a simplified example, when you call `tcx.type_of(def_id)` for a type defined in an external crate, the query system will: | ||
| 1. Detect that the `def_id` refers to an external crate by checking `def_id.krate != LOCAL_CRATE` | ||
| 2. Call the appropriate provider from `ExternProviders` which was registered by `rustc_metadata` | ||
| 3. The provider will load and decode the type information from the external crate's metadata | ||
| 4. Return the decoded type to the caller | ||
|
|
||
| This is why most `rustc_*` crates only need to provide local providers - the external providers are handled by the metadata system. | ||
| The only exception is when a crate needs to provide special handling for external queries, in which case it would implement both local and external providers. | ||
|
|
||
| [rustc_metadata]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/index.html | ||
| [providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.Providers.html | ||
| [extern_providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.ExternProviders.html |
There was a problem hiding this comment.
I read the code in rustc_metadata related to the extern providers and added it to the new commit, but I'm not sure there are parts I wrote correctly, if you have time to look at it. Thanks @Noratrieb
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refined some vague concepts and pseudo-code in Query System
In the past, we only used vague concepts like local and extern to differentiate between different queries. however the code has more detailed
ProvidersandExternProvidersto differentiate between them, so we should mention both of them in the documentation as well.It is also important to indicate that
rustc_middle::util::Providersis made up ofProvidersandExternProvidersin the query mod.This PR also updates the example of registering a new Provider to consider
ExternProviders.r? @tshepang