Skip to content

Commit 67791cc

Browse files
varungandhi-applevarungandhi-apple
authored
[docs] Add a how-to guide on running IWYU on the Swift project. (swiftlang#32151)
1 parent e9548b0 commit 67791cc

File tree

2 files changed

+289
-0
lines changed

2 files changed

+289
-0
lines changed

‎docs/HowToGuides/RunningIncludeWhatYouUse.md

Lines changed: 286 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,286 @@
1+
# How to run include-what-you-use (IWYU) on the Swift project
2+
3+
[include-what-you-use (IWYU)](https://include-what-you-use.org) is a
4+
Clang-based tool that analyzes `#include`s in a file and makes suggestions to
5+
add or remove `#include`s based on usage in the code. This has two key benefits:
6+
7+
- Removing unused `#include` statements reduces work for the compiler.
8+
- Adding `#include` statements for usage avoids a refactoring in a header
9+
file from breaking downstream implementation files due to accidental
10+
transitive usage.
11+
12+
Running IWYU is a bit tricky, so this how-to guide provides the steps for how
13+
to get it up and running on the Swift project for macOS.
14+
If you get IWYU working on a different platform and some steps need to be
15+
changed, please update this document with platform-specific steps.
16+
17+
- [Pre-requisites](#pre-requisites)
18+
- [Cloning and branch checkout](#cloning-and-branch-checkout)
19+
- [Building IWYU](#building-iwyu)
20+
- [Running IWYU](#running-iwyu)
21+
- [Debugging](#debugging)
22+
23+
## Pre-requisites
24+
25+
- A built Swift project with exported compilation commands.
26+
By default, compilation commands are generated in the file
27+
`build/[BuildSystem]-[BuildVariant]/swift-[target]/compile_commands.json`.
28+
Check that this file is present before proceeding.
29+
- If this file is missing, try building with
30+
`CMAKE_EXPORT_COMPILATION_COMMANDS=ON`. If you use `build-script` to
31+
manage your builds, you can do this with
32+
```
33+
swift/utils/build-script <other options> \
34+
--extra-cmake-options='-DCMAKE_EXPORT_COMPILATION_COMMANDS=ON'
35+
```
36+
- Install [`jq`](https://stedolan.github.io/jq/). It's not strictly necessary,
37+
but we will use it for some JSON munging.
38+
39+
## Cloning and branch checkout
40+
41+
The directory structure we will be using is
42+
43+
```
44+
swift-project/
45+
|--- build/
46+
| |--- [BuildSystem]-[BuildVariant]/
47+
| | |--- swift-[target]/
48+
| | | |--- compile_commands.json
49+
| | | `--- ...
50+
| | |--- iwyu-[target]/
51+
| | `--- ...
52+
| `--- ...
53+
|--- swift/
54+
|--- iwyu/
55+
| |--- src/
56+
| |--- logs/
57+
| `--- scripts/
58+
`--- ...
59+
```
60+
61+
As a running example, the description below uses `[BuildSystem] = Ninja`,
62+
`[BuildVariant] = ReleaseAssert` and `[target] = macosx-x86_64`.
63+
64+
Start with `swift-project` as the working directory.
65+
66+
1. Check out IWYU.
67+
```
68+
mkdir -p iwyu/src
69+
git clone https://github.com/include-what-you-use/include-what-you-use.git iwyu/src
70+
```
71+
2. Find out the version of the `clang` built recently.
72+
```
73+
build/Ninja-ReleaseAssert/llvm-macosx-x86_64/bin/clang --version
74+
```
75+
This should say something like `clang version 10.0.0` or similar.
76+
3. Based on the `clang` version, make sure you check out the correct branch.
77+
```
78+
git -C iwyu/src checkout clang_10
79+
```
80+
81+
## Building IWYU
82+
83+
1. Configure IWYU with CMake.
84+
```
85+
cmake -G Ninja \
86+
-DCMAKE_PREFIX_PATH=build/Ninja-ReleaseAssert/llvm-macosx-x86_64 \
87+
-DCMAKE_CXX_STANDARD=14 \
88+
-B build/Ninja-ReleaseAssert/iwyu-macosx-x86_64 \
89+
iwyu/src
90+
```
91+
2. Build IWYU
92+
```
93+
cmake --build build/Ninja-ReleaseAssert/iwyu-macosx-x86_64
94+
```
95+
3. Create an extra symlink so IWYU can find necessary Clang headers:
96+
```
97+
ln -sF build/Ninja-ReleaseAssert/llvm-macosx-x86_64/lib build/Ninja-ReleaseAssert/iwyu-macosx-x86_64/lib
98+
```
99+
4. Spot check IWYU for a basic C example.
100+
```
101+
echo '#include <stdint.h>' > tmp.c
102+
./bin/include-what-you-use tmp.c -E -o /dev/null \
103+
-I "$(xcrun --show-sdk-path)/usr/include"
104+
rm tmp.c
105+
```
106+
You should see output like:
107+
```
108+
tmp.c should add these lines:
109+
110+
tmp.c should remove these lines:
111+
- #include <stdint.h> // lines 1-1
112+
113+
The full include-list for tmp.c:
114+
---
115+
```
116+
5. Spot check IWYU for a basic C++ example. Notice the extra C++-specific
117+
include path.
118+
```
119+
echo '#include <string>\n#include <cmath>' > tmp.cpp
120+
./bin/include-what-you-use tmp.cpp -E -o /dev/null \
121+
-I "$(clang++ -print-resource-dir)/../../../include/c++/v1" \
122+
-I "$(xcrun --show-sdk-path)/usr/include"
123+
rm tmp.cpp
124+
```
125+
You should see output like:
126+
```
127+
tmp.cpp should add these lines:
128+
129+
tmp.cpp should remove these lines:
130+
- #include <cmath> // lines 2-2
131+
- #include <string> // lines 1-1
132+
133+
The full include-list for tmp.cpp:
134+
---
135+
```
136+
137+
## Running IWYU
138+
139+
1. Create a directory, say `iwyu/scripts`, and copy the following script there.
140+
141+
```
142+
#!/usr/bin/env bash
143+
144+
# iwyu_run.sh
145+
set -eu
146+
147+
SWIFT_PROJECT_DIR="$HOME/swift-project"
148+
SWIFT_BUILD_DIR="$SWIFT_PROJECT_DIR/build/Ninja-ReleaseAssert/swift-macosx-x86_64"
149+
150+
pushd "$SWIFT_BUILD_DIR"
151+
152+
if [ -f original_compile_commands.json ]; then
153+
mv original_compile_commands.json compile_commands.json
154+
fi
155+
156+
# HACK: The additional include path needs to be added before other include
157+
# paths, it doesn't seem to work if we add it at the end.
158+
# It is ok to rely on the presence of `-D__STDC_LIMIT_MACROS` flag, since
159+
# it is added by the LLVM CMake configuration for all compilation commands.
160+
( EXTRA_CXX_INCLUDE_DIR="$(clang++ -print-resource-dir)/../../../include/c++/v1";
161+
cat compile_commands.json \
162+
| jq '[.[] | select(.file | test("\\.mm" | "\\.m") | not) | {directory: .directory, command: (.command + " -Wno-everything -ferror-limit=1"), file: .file}]' \
163+
| sed -e "s|-D__STDC_LIMIT_MACROS |-D__STDC_LIMIT_MACROS -I $EXTRA_CXX_INCLUDE_DIR |" \
164+
) > filtered_compile_commands.json
165+
166+
mv compile_commands.json original_compile_commands.json
167+
mv filtered_compile_commands.json compile_commands.json
168+
169+
mkdir -p "$SWIFT_PROJECT_DIR/iwyu/logs"
170+
171+
( PATH="$SWIFT_PROJECT_DIR/iwyu/build/bin:$PATH"; \
172+
"$SWIFT_PROJECT_DIR/iwyu/include-what-you-use/iwyu_tool.py" -p "$SWIFT_BUILD_DIR"
173+
) | tee "$SWIFT_PROJECT_DIR/iwyu/logs/suggestions.log"
174+
175+
popd
176+
177+
```
178+
179+
We filter out Objective-C files because IWYU does not support Objective-C.
180+
If that step is missed, you might hit errors like:
181+
```
182+
iwyu.cc:2097: Assertion failed: TODO(csilvers): for objc and clang lang extensions
183+
```
184+
185+
2. Update the `SWIFT_PROJECT_DIR` and `SWIFT_BUILD_DIR` variables based on
186+
your project and build directories.
187+
188+
3. Run the script.
189+
```
190+
chmod +x iwyu/scripts/iwyu_run.sh
191+
iwyu/scripts/iwyu_run.sh
192+
```
193+
This will generate a log file under `iwyu/logs/suggestions.log`.
194+
Note that IWYU might take several hours to run, depending on your system.
195+
196+
NOTE: The IWYU README suggests several different ways of running IWYU on a
197+
CMake project, including using the `CMAKE_CXX_INCLUDE_WHAT_YOU_USE` and
198+
`CMAKE_C_INCLUDE_WHAT_YOU_USE` variables. At the time of writing, those did
199+
not reliably work on macOS; suggestions were generated only for specific
200+
subprojects (e.g. the stdlib) and not others (e.g. the compiler).
201+
Using CMake variables also requires reconfiguring and rebuilding, which makes
202+
debugging much more time-consuming.
203+
204+
## Debugging
205+
206+
While the above steps should work, in case you run into issues, you might find
207+
the following steps for debugging helpful.
208+
209+
### Try different include path ordering
210+
211+
If you see errors with `<cmath>`, or similar system headers, one thing that might
212+
be happening is that the include paths are in the wrong order. Try moving the
213+
include paths for the corresponding header before/after all other include paths.
214+
215+
### Iterate on files one at a time
216+
217+
Instead of trying to make changes to the CMake configuration and recompiling
218+
the whole project, first try working on individual compilation commands as
219+
emitted in `compile_commands.json` and see if IWYU works as expected.
220+
221+
For each command, try replacing the compiler with the `include-what-you-use`
222+
binary or `iwyu_stub.py` (below) to see if the behavior is as expected.
223+
You may need to manually add some include paths as in `iwyu_run.sh` above.
224+
Make sure you update paths in the script before it works.
225+
226+
```
227+
#!/usr/bin/env python3
228+
229+
# iwyu_stub.py
230+
231+
import os
232+
import re
233+
import subprocess
234+
import sys
235+
236+
clang_path = "/usr/bin/clang"
237+
clangxx_path = "/usr/bin/clang++"
238+
project_dir = "/Users/username/swift-project/"
239+
iwyu_bin_path = project_dir + "iwyu/build/bin/include-what-you-use"
240+
log_dir = project_dir + "iwyu/logs/"
241+
242+
log_file = open(log_dir + "passthrough.log", "a+")
243+
244+
argv = sys.argv
245+
246+
def call_with_args(executable_path, args=argv):
247+
new_argv = args[:]
248+
new_argv[0] = executable_path
249+
log_file.write("# about to run:\n{}\n#---\n".format(' '.join(new_argv)))
250+
sys.exit(subprocess.call(new_argv))
251+
252+
# HACK: Relies on the compilation commands generated by CMake being
253+
# of the form:
254+
#
255+
# /path/to/compiler <other options> -c MyFile.ext
256+
#
257+
def try_using_iwyu(argv):
258+
return (argv[-2] == "-c") and ("/swift/" in argv[-1])
259+
260+
# Flag for quickly switching between IWYU and Clang for iteration.
261+
# Useful for checking behavior for different include path combinations.
262+
if argv[1] == "--forward-to-clangxx":
263+
call_with_args(clangxx_path, args=([argv[0]] + argv[2:]))
264+
265+
# Check that we are getting a compilation command.
266+
if try_using_iwyu(argv):
267+
_, ext = os.path.splitext(argv[-1])
268+
if ext == ".m":
269+
call_with_args(clang_path)
270+
elif ext == ".mm":
271+
call_with_args(clangxx_path)
272+
elif ext in [".cxx", ".cc", ".cpp", ".c"]:
273+
call_with_args(iwyu_bin_path)
274+
log_file.write(
275+
"# Got a strange file extension.\n{}\n#---\n".format(' '.join(argv)))
276+
call_with_args(iwyu_bin_path)
277+
else:
278+
# got something else, just forward to clang/clang++
279+
log_file.write(
280+
"# Not going to try using iwyu.\n{}\n#---\n".format(' '.join(argv)))
281+
_, ext = os.path.splitext(argv[-1])
282+
if ext == ".m" or ext == ".c":
283+
call_with_args(clang_path)
284+
else:
285+
call_with_args(clangxx_path)
286+
```

‎docs/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,9 @@ documentation, please create a thread on the Swift forums under the
6969
How to build Swift on Windows using Visual Studio.
7070
- [WindowsCrossCompile.md](/docs/WindowsCrossCompile.md):
7171
How to cross compile Swift for Windows on a non-Windows host OS.
72+
- [RunningIncludeWhatYouUse.md](/docs/RunningIncludeWhatYouUse.md):
73+
Describes how to run [include-what-you-use](https://include-what-you-use.org)
74+
on the Swift project.
7275

7376
## Explanations
7477

0 commit comments

Comments
 (0)