From ab17a36d7d23d9f582cb026441616f940673d881 Mon Sep 17 00:00:00 2001 From: mark Date: Wed, 24 Jun 2020 11:53:38 -0500 Subject: [PATCH 01/10] rewrite/update compiler source code chapter --- src/compiler-src.md | 282 +++++++++++++++++++++++--------------------- 1 file changed, 147 insertions(+), 135 deletions(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index 25b3baa3e..efb9ae896 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -1,137 +1,149 @@ # High-level overview of the compiler source -## Crate structure - -The main Rust repository consists of a `src` directory, under which -there live many crates. These crates contain the sources for the -standard library and the compiler. This document, of course, focuses -on the latter. - -Rustc consists of a number of crates, including `rustc_ast`, -`rustc`, `rustc_target`, `rustc_codegen`, `rustc_driver`, and -many more. The source for each crate can be found in a directory -like `src/libXXX`, where `XXX` is the crate name. - -(N.B. The names and divisions of these crates are not set in -stone and may change over time. For the time being, we tend towards a -finer-grained division to help with compilation time, though as incremental -compilation improves, that may change.) - -The dependency structure of these crates is roughly a diamond: - -```text - rustc_driver - / | \ - / | \ - / | \ - / v \ -rustc_codegen rustc_borrowck ... rustc_metadata - \ | / - \ | / - \ | / - \ v / - rustc_middle - | - v - rustc_ast - / \ - / \ - rustc_span rustc_builtin_macros -``` - -The `rustc_driver` crate, at the top of this lattice, is effectively -the "main" function for the rust compiler. It doesn't have much "real -code", but instead ties together all of the code defined in the other -crates and defines the overall flow of execution. (As we transition -more and more to the [query model], however, the -"flow" of compilation is becoming less centrally defined.) - -At the other extreme, the `rustc_middle` crate defines the common and -pervasive data structures that all the rest of the compiler uses -(e.g. how to represent types, traits, and the program itself). It -also contains some amount of the compiler itself, although that is -relatively limited. - -Finally, all the crates in the bulge in the middle define the bulk of -the compiler – they all depend on `rustc_middle`, so that they can make use -of the various types defined there, and they export public routines -that `rustc_driver` will invoke as needed (more and more, what these -crates export are "query definitions", but those are covered later -on). - -Below `rustc_middle` lie various crates that make up the parser and error -reporting mechanism. They are also an internal part -of the compiler and not intended to be stable (though they do wind up -getting used by some crates in the wild; a practice we hope to -gradually phase out). - -## The main stages of compilation - -The Rust compiler is in a bit of transition right now. It used to be a -purely "pass-based" compiler, where we ran a number of passes over the -entire program, and each did a particular check of transformation. We -are gradually replacing this pass-based code with an alternative setup -based on on-demand **queries**. In the query-model, we work backwards, -executing a *query* that expresses our ultimate goal (e.g. "compile -this crate"). This query in turn may make other queries (e.g. "get me -a list of all modules in the crate"). Those queries make other queries -that ultimately bottom out in the base operations, like parsing the -input, running the type-checker, and so forth. This on-demand model -permits us to do exciting things like only do the minimal amount of -work needed to type-check a single function. It also helps with -incremental compilation. (For details on defining queries, check out -the [query model].) - -Regardless of the general setup, the basic operations that the -compiler must perform are the same. The only thing that changes is -whether these operations are invoked front-to-back, or on demand. In -order to compile a Rust crate, these are the general steps that we -take: - -1. **Parsing input** - - this processes the `.rs` files and produces the AST - ("abstract syntax tree") - - the AST is defined in `src/librustc_ast/ast.rs`. It is intended to match the lexical - syntax of the Rust language quite closely. -2. **Name resolution, macro expansion, and configuration** - - once parsing is complete, we process the AST recursively, resolving - paths and expanding macros. This same process also processes `#[cfg]` - nodes, and hence may strip things out of the AST as well. -3. **Lowering to HIR** - - Once name resolution completes, we convert the AST into the HIR, - or "[high-level intermediate representation]". The HIR is defined in - `src/librustc_middle/hir/`; that module also includes the [lowering] code. - - The HIR is a lightly desugared variant of the AST. It is more processed - than the AST and more suitable for the analyses that follow. - It is **not** required to match the syntax of the Rust language. - - As a simple example, in the **AST**, we preserve the parentheses - that the user wrote, so `((1 + 2) + 3)` and `1 + 2 + 3` parse - into distinct trees, even though they are equivalent. In the - HIR, however, parentheses nodes are removed, and those two - expressions are represented in the same way. -3. **Type-checking and subsequent analyses** - - An important step in processing the HIR is to perform type - checking. This process assigns types to every HIR expression, - for example, and also is responsible for resolving some - "type-dependent" paths, such as field accesses (`x.f` – we - can't know what field `f` is being accessed until we know the - type of `x`) and associated type references (`T::Item` – we - can't know what type `Item` is until we know what `T` is). - - Type checking creates "side-tables" (`TypeckTables`) that include - the types of expressions, the way to resolve methods, and so forth. - - After type-checking, we can do other analyses, such as privacy checking. -4. **Lowering to MIR and post-processing** - - Once type-checking is done, we can lower the HIR into MIR ("middle IR"), - which is a **very** desugared version of Rust, well suited to borrowck - but also to certain high-level optimizations. -5. **Translation to LLVM and LLVM optimizations** - - From MIR, we can produce LLVM IR. - - LLVM then runs its various optimizations, which produces a number of - `.o` files (one for each "codegen unit"). -6. **Linking** - - Finally, those `.o` files are linked together. - - -[query model]: query.html -[high-level intermediate representation]: hir.html -[lowering]: lowering.html +> **NOTE**: The structure of the repository is going through a lot of +> transitions. In particular, we want to get to a point eventually where the +> top-level directory has separate directories for the compiler, build-system, +> std libs, etc, rather than one huge `src/` directory. + +## Workspace structure + +The `rust-lang/rust` repository consists of a single large cargo workspace +containing the compiler, the standard library (core, alloc, std, etc), and +`rustdoc`, along with the build system and bunch of tools and submodules for +building a full Rust distribution. + +As of this writing, this structure is gradually undergoing some transformation +to make it a bit less monolithic and more approachable, especially to +newcommers. + +> Eventually, the hope is for the standard library to live in a `stdlib/` +> directory, while the compiler lives in `compiler/`. However, as of this +> writing, both live in `src/`. + +The repository consists of a `src` directory, under which there live many +crates, which are the source for the compiler, standard library, etc, as +mentioned above. + +## Standard library + +The standard library crates are obviously named `libstd`, `libcore`, +`liballoc`, etc. There is also `libproc_macro`, `libtest`, and other runtime +libraries. + +This code is fairly similar to most other Rust crates except that it must be +built in a special way because it can use unstable features. + +## Compiler + +The compiler crates all have names starting with `librustc_*`. These are a large +collection of interdependent crates. There is also the `rustc` crate which is +the actual binary. It doesn't actually do anything besides calling the compiler +main function elsewhere. + +The dependency structure of these crates is complex, but roughly it is +something like this: + +- `rustc` (the binary) calls [`rustc_driver::main`][main]. + - [`rustc_driver`] depends on a lot of other crates, but the main one is + [`rustc_interface`]. + - [`rustc_interface`] depends on most of the other compiler crates. It + is a fairly generic interface for driving the whole compilation. + - The most of the other `rustc_*` crates depend on [`rustc_middle`], + which defines a lot of central data structures in the compiler. + - [`rustc_middle`] and most of the other crates depend on a + handful of crates representing the early parts of the + compiler (e.g. the parser), fundamental data structures (e.g. + [`Span`]), or error reporting: [`rustc_data_strucutres`], + [`rustc_span`], [`rustc_errors`], etc. + +[main]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/fn.main.html +[`rustc_driver`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/index.html +[`rustc_interface`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/index.html +[`rustc_middle`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/index.html +[`rustc_data_strucutres`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_strucutres/index.html +[`rustc_span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/index.html +[`Span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/struct.Span.html +[`rustc_errors`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html + +You can see the exact dependencies by reading the `Cargo.toml` for the various +crates, just like a normal Rust crate. + +You may ask why the compiler is broken into so many crates. There are two major reasons: + +1. Organization. The compiler is a _huge_ codebase; it would be an impossibly large crate. +2. Compile time. By breaking the compiler into multiple crates, we can take + better advantage of incremental/parallel compilation using cargo. In + particular, we try to have as few dependencies between crates as possible so + that we dont' have to rebuild as many crates if you change one. + +Most of this book is about the compiler, so we won't have any further +explanation of these crates here. + +One final thing: [`src/llvm-project`] is a submodule for our fork of LLVM. + +[`src/llvm-project`]: https://github.com/rust-lang/rust/tree/master/src + +## rustdoc + +The bulk of `rustdoc` is in [`librustdoc`]. However, the `rustdoc` binary +itself is [`src/tools/rustdoc`], which does nothing except call [`rustdoc::main`]. + +There is also javascript and CSS for the rustdocs in [`src/tools/rustdoc-js`] +and [`src/tools/rustdoc-themes`]. + +You can read more about rustdoc in [this chapter][rustdocch]. + +[`librustdoc`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/index.html +[`rustdoc::main`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/fn.main.html +[`src/tools/rustdoc`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc +[`src/tools/rustdoc-js`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-js +[`src/tools/rustdoc-themes`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-themes + +[rustdocch]: ./rustdoc-internals.md + +## Tests + +The test suite for all of the above is in [`src/test/`]. You can read more +about the test suite [in this chapter][testsch]. + +The test harness itself is in [`src/tools/compiletest`]. + +[testsch]: ./tests/intro.md + +[`src/test/`]: https://github.com/rust-lang/rust/tree/master/src/test +[`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest + +## Build System + +There are a number of tools in the repository just for building the compiler, +standard library, rustdoc, etc, along with testing, building a full Rust +distribution, etc. + +One of the primary tools is [`src/bootstrap`]. You can read more about +bootstrapping [in this chapter][bootstch]. The process may also use other tools +from `src/tools/`, such as [`tidy`] or [`compiletest`]. + +[`src/bootstrap`]: https://github.com/rust-lang/rust/tree/master/src/bootstrap +[`tidy`]: https://github.com/rust-lang/rust/tree/master/src/tools/tidy +[`compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest + +[bootstch]: ./building/bootstrapping.md + +## Other + +There are a lot of other things in the `rust-lang/rust` repo that are related +to building a full rust distribution. Most of the time you don't need to worry +about them. + +These include: +- [`src/ci`]: The CI configuration. This actually quite extensive because we + run a lot of tests on a lot of platforms. +- [`src/doc`]: Various documentation, including submodules for a few books. +- [`src/etc`]: Miscellaneous utilities. +- [`src/tools/rustc-workspace-hack`], and others: Various workarounds to make cargo work with bootstrapping. +- And more... + +[`src/ci`]: https://github.com/rust-lang/rust/tree/master/src/ci +[`src/doc`]: https://github.com/rust-lang/rust/tree/master/src/doc +[`src/etc`]: https://github.com/rust-lang/rust/tree/master/src/etc +[`src/tools/rustc-workspace-hack`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustc-workspace-hack From f1f34b3cfe75b94fce1c46d77a5e5b2991c4db21 Mon Sep 17 00:00:00 2001 From: mark Date: Wed, 24 Jun 2020 12:38:44 -0500 Subject: [PATCH 02/10] line length --- src/compiler-src.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index efb9ae896..e0789e2a2 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -140,7 +140,8 @@ These include: run a lot of tests on a lot of platforms. - [`src/doc`]: Various documentation, including submodules for a few books. - [`src/etc`]: Miscellaneous utilities. -- [`src/tools/rustc-workspace-hack`], and others: Various workarounds to make cargo work with bootstrapping. +- [`src/tools/rustc-workspace-hack`], and others: Various workarounds to make + cargo work with bootstrapping. - And more... [`src/ci`]: https://github.com/rust-lang/rust/tree/master/src/ci From 6eed5b47b02e6fd278f828db390ab70c9417e096 Mon Sep 17 00:00:00 2001 From: mark Date: Wed, 24 Jun 2020 13:24:40 -0500 Subject: [PATCH 03/10] fix typo --- src/compiler-src.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index e0789e2a2..27d656ead 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -53,14 +53,14 @@ something like this: - [`rustc_middle`] and most of the other crates depend on a handful of crates representing the early parts of the compiler (e.g. the parser), fundamental data structures (e.g. - [`Span`]), or error reporting: [`rustc_data_strucutres`], + [`Span`]), or error reporting: [`rustc_data_structures`], [`rustc_span`], [`rustc_errors`], etc. [main]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/fn.main.html [`rustc_driver`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/index.html [`rustc_interface`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/index.html [`rustc_middle`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/index.html -[`rustc_data_strucutres`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_strucutres/index.html +[`rustc_data_structures`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_structures/index.html [`rustc_span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/index.html [`Span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/struct.Span.html [`rustc_errors`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html From 89018e58430679c7a3d4cad84ce439e923a5f7a4 Mon Sep 17 00:00:00 2001 From: mark Date: Fri, 26 Jun 2020 19:59:46 -0500 Subject: [PATCH 04/10] add a bit more discussion of big picture --- src/compiler-src.md | 55 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 47 insertions(+), 8 deletions(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index 27d656ead..5d8c3ca04 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -68,20 +68,59 @@ something like this: You can see the exact dependencies by reading the `Cargo.toml` for the various crates, just like a normal Rust crate. -You may ask why the compiler is broken into so many crates. There are two major reasons: +One final thing: [`src/llvm-project`] is a submodule for our fork of LLVM. + +Most of this book is about the compiler, so we won't have any further +explanation of these crates here. + +[`src/llvm-project`]: https://github.com/rust-lang/rust/tree/master/src + +### Big picture -1. Organization. The compiler is a _huge_ codebase; it would be an impossibly large crate. +The dependency structure is influenced strongly by two main factors: + +1. Organization. The compiler is a _huge_ codebase; it would be an impossibly + large crate. In part, the dependency structure reflects the code structure + of the compiler. 2. Compile time. By breaking the compiler into multiple crates, we can take better advantage of incremental/parallel compilation using cargo. In particular, we try to have as few dependencies between crates as possible so that we dont' have to rebuild as many crates if you change one. -Most of this book is about the compiler, so we won't have any further -explanation of these crates here. - -One final thing: [`src/llvm-project`] is a submodule for our fork of LLVM. - -[`src/llvm-project`]: https://github.com/rust-lang/rust/tree/master/src +At the very bottom of the dependency tree are a handful of crates that are used +by the whole compiler (e.g. [`rustc_span`]). The very early parts of the +compilation process (e.g. parsing and the AST) depend on only these. + +Pretty soon after the AST is constructed, the compiler's [query system][query] +gets set up. The query system is set up in a clever way using function +pointers. This allows us to break dependencies between crates, allowing more +parallel compilation. + +However, since the query system is defined in [`rustc_middle`], nearly all +subsequent parts of the compiler depend on this crate. It is a really large +crate, leading to long compile times. Some efforts have been made to move stuff +out of it with limited success. Another unfortunate sideffect is that sometimes +related functionality gets scattered across different crates. For example, +linting functionality is scattered across earlier parts of the crate, +[`rustc_lint`], [`rustc_middle`], and other places. + +More generally, in an ideal world, it seems like there would be fewer, more +cohesive crates, with incremental and parallel compilation making sure compile +times stay reasonable. However, our incremental and parallel compilation haven't +gotten good enough for that yet, so breaking things into separate crates has +been our solution so far. + +At the top of the dependency tree are the [`rustc_interface`] and +[`rustc_driver`] crates. [`rustc_interface`] is an unstable wrapper around the +query system that helps to drive the various stages of compilation. Other +consumers of the compiler may use this interface in different ways (e.g. +rustdoc or maybe eventually rust-analyzer). The [`rustc_driver`] crate first +parses command line arguments and then uses [`rustc_interface`] to drive the +compilation to completion. + +[query]: ./query.md + +[orgch]: ./overview.md ## rustdoc From ff15aaf3672b6c1c8de2837a29e038649d78db3d Mon Sep 17 00:00:00 2001 From: mark Date: Sat, 27 Jun 2020 11:07:25 -0500 Subject: [PATCH 05/10] missing link --- src/compiler-src.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/compiler-src.md b/src/compiler-src.md index 5d8c3ca04..fc7118734 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -104,6 +104,8 @@ related functionality gets scattered across different crates. For example, linting functionality is scattered across earlier parts of the crate, [`rustc_lint`], [`rustc_middle`], and other places. +[`rustc_lint`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lint/index.html + More generally, in an ideal world, it seems like there would be fewer, more cohesive crates, with incremental and parallel compilation making sure compile times stay reasonable. However, our incremental and parallel compilation haven't From 6f99ad6d4ee92e9132ea76711087368a0c744d53 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Mon, 20 Jul 2020 19:34:11 -0500 Subject: [PATCH 06/10] Link to overview ch instead of internals Co-authored-by: Joshua Nelson --- src/compiler-src.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index fc7118734..67940d367 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -140,7 +140,7 @@ You can read more about rustdoc in [this chapter][rustdocch]. [`src/tools/rustdoc-js`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-js [`src/tools/rustdoc-themes`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-themes -[rustdocch]: ./rustdoc-internals.md +[rustdocch]: ./rustdoc.md ## Tests From dcf8c635ae8dffe5f2b7f214f0c024a07d1400f7 Mon Sep 17 00:00:00 2001 From: mark Date: Mon, 20 Jul 2020 19:42:01 -0500 Subject: [PATCH 07/10] mention the overview chapter --- src/compiler-src.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/compiler-src.md b/src/compiler-src.md index 67940d367..a9d708f1a 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -5,6 +5,9 @@ > top-level directory has separate directories for the compiler, build-system, > std libs, etc, rather than one huge `src/` directory. +Now that we have [seen what the compiler does](./overview.md), let's take a +look at the structure of the contents of the rust-lang/rust repo. + ## Workspace structure The `rust-lang/rust` repository consists of a single large cargo workspace @@ -35,6 +38,10 @@ built in a special way because it can use unstable features. ## Compiler +> You may find it helpful to read [The Overview Chapter](./overview.md) first, +> which gives an overview of how the compiler works. The crates mentioned in +> this section implement the compiler. + The compiler crates all have names starting with `librustc_*`. These are a large collection of interdependent crates. There is also the `rustc` crate which is the actual binary. It doesn't actually do anything besides calling the compiler From 8a89cb20338a770efb43e730eb1a4ad9e613cd2b Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Fri, 24 Jul 2020 18:13:25 -0500 Subject: [PATCH 08/10] Typo Co-authored-by: Joshua Nelson --- src/compiler-src.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index a9d708f1a..a976701aa 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -106,7 +106,7 @@ parallel compilation. However, since the query system is defined in [`rustc_middle`], nearly all subsequent parts of the compiler depend on this crate. It is a really large crate, leading to long compile times. Some efforts have been made to move stuff -out of it with limited success. Another unfortunate sideffect is that sometimes +out of it with limited success. Another unfortunate side effect is that sometimes related functionality gets scattered across different crates. For example, linting functionality is scattered across earlier parts of the crate, [`rustc_lint`], [`rustc_middle`], and other places. From 5d994e8bb39512b3d46c0d60853b390db977d824 Mon Sep 17 00:00:00 2001 From: mark Date: Sun, 2 Aug 2020 18:45:03 -0500 Subject: [PATCH 09/10] some updates after std libs move --- src/compiler-src.md | 47 ++++++++++++++++++++++++++++----------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index a976701aa..55eea49e5 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -4,6 +4,9 @@ > transitions. In particular, we want to get to a point eventually where the > top-level directory has separate directories for the compiler, build-system, > std libs, etc, rather than one huge `src/` directory. +> +> As of this writing, the std libs have been moved to `library/` and there is +> an ongoing MCP to move the compiler to `compiler/`. Now that we have [seen what the compiler does](./overview.md), let's take a look at the structure of the contents of the rust-lang/rust repo. @@ -11,27 +14,25 @@ look at the structure of the contents of the rust-lang/rust repo. ## Workspace structure The `rust-lang/rust` repository consists of a single large cargo workspace -containing the compiler, the standard library (core, alloc, std, etc), and -`rustdoc`, along with the build system and bunch of tools and submodules for -building a full Rust distribution. +containing the compiler, the standard libraries (`core`, `alloc`, `std`, +`proc_macro`, etc), and `rustdoc`, along with the build system and bunch of +tools and submodules for building a full Rust distribution. As of this writing, this structure is gradually undergoing some transformation to make it a bit less monolithic and more approachable, especially to newcommers. -> Eventually, the hope is for the standard library to live in a `stdlib/` -> directory, while the compiler lives in `compiler/`. However, as of this -> writing, both live in `src/`. - The repository consists of a `src` directory, under which there live many -crates, which are the source for the compiler, standard library, etc, as -mentioned above. +crates, which are the source for the compiler, build system, tools, etc. This +directory is currently being broken up to be less monolithic. There is also a +`library/` directory, where the standard libraries (`core`, `alloc`, `std`, +`proc_macro`, etc) live. ## Standard library -The standard library crates are obviously named `libstd`, `libcore`, -`liballoc`, etc. There is also `libproc_macro`, `libtest`, and other runtime -libraries. +The standard library crates are all in `library/`. They have intuitive names +like `std`, `core`, `alloc`, etc. There is also `proc_macro`, `test`, and +other runtime libraries. This code is fairly similar to most other Rust crates except that it must be built in a special way because it can use unstable features. @@ -41,11 +42,16 @@ built in a special way because it can use unstable features. > You may find it helpful to read [The Overview Chapter](./overview.md) first, > which gives an overview of how the compiler works. The crates mentioned in > this section implement the compiler. +> +> NOTE: As of this writing, the crates all live in `src/`, but there is an MCP +> to move them to a new `compiler/` directory. -The compiler crates all have names starting with `librustc_*`. These are a large -collection of interdependent crates. There is also the `rustc` crate which is -the actual binary. It doesn't actually do anything besides calling the compiler -main function elsewhere. +The compiler crates all have names starting with `librustc_*`. These are a +collection of around 50 interdependent crates ranging in size from tiny to +huge. There is also the `rustc` crate which is the actual binary (i.e. the +`main` function); it doesn't actually do anything besides calling the +`rustc_driver` crate, which drives the various parts of compilation in other +crates. The dependency structure of these crates is complex, but roughly it is something like this: @@ -55,7 +61,7 @@ something like this: [`rustc_interface`]. - [`rustc_interface`] depends on most of the other compiler crates. It is a fairly generic interface for driving the whole compilation. - - The most of the other `rustc_*` crates depend on [`rustc_middle`], + - Most of the other `rustc_*` crates depend on [`rustc_middle`], which defines a lot of central data structures in the compiler. - [`rustc_middle`] and most of the other crates depend on a handful of crates representing the early parts of the @@ -75,12 +81,17 @@ something like this: You can see the exact dependencies by reading the `Cargo.toml` for the various crates, just like a normal Rust crate. -One final thing: [`src/llvm-project`] is a submodule for our fork of LLVM. +One final thing: [`src/llvm-project`] is a submodule for our fork of LLVM +During bootstrapping, LLVM is built and the [`src/librustc_llvm`] and +[`src/rustllvm`] crates contain rust wrappers around LLVM (which is written in +C++), so that the compiler can interface with it. Most of this book is about the compiler, so we won't have any further explanation of these crates here. [`src/llvm-project`]: https://github.com/rust-lang/rust/tree/master/src +[`src/librustc_llvm`]: https://github.com/rust-lang/rust/tree/master/src +[`src/rustllvm`]: https://github.com/rust-lang/rust/tree/master/src ### Big picture From 514560876e84491570fb6ed0865c75445478a0e7 Mon Sep 17 00:00:00 2001 From: Who? Me?! Date: Sun, 2 Aug 2020 19:38:20 -0500 Subject: [PATCH 10/10] Typo Co-authored-by: Joshua Nelson --- src/compiler-src.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index 55eea49e5..df8bc2bd1 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -103,7 +103,7 @@ The dependency structure is influenced strongly by two main factors: 2. Compile time. By breaking the compiler into multiple crates, we can take better advantage of incremental/parallel compilation using cargo. In particular, we try to have as few dependencies between crates as possible so - that we dont' have to rebuild as many crates if you change one. + that we don't have to rebuild as many crates if you change one. At the very bottom of the dependency tree are a handful of crates that are used by the whole compiler (e.g. [`rustc_span`]). The very early parts of the