Skip to content

Commit f97f23e

Browse files
chenBrightchenBright
authored
Hide libunwind's _Unwind_* symbols under Bazel to fix bthread_tracer crash (#3297)
libunwind ships its `src/unwind/*.c` (the GCC `_Unwind_*` ABI compatibility layer) as exported symbols of `libexternal_S~libunwind.so`. At runtime the dynamic loader resolves `_Unwind_*` lookups (from `pthread_exit`, libstdc++'s `__gxx_personality_v0`, etc.) to libunwind's DWARF-based implementation instead of `libgcc_s.so.1`, hitting an uninitialized internal context and crashing on the no-return cleanup chain triggered by pthread_exit / C++ exception unwinding -- e.g. it makes BthreadTest.bthread_exit segfault deterministically when `--define=with_bthread_tracer=true` is on. This is purely an ELF runtime symbol-resolution-order issue and reproduces identically on GCC and Clang, since both default to `libstdc++ + libgcc_s` on Linux.
1 parent a47e349 commit f97f23e

16 files changed

Lines changed: 1909 additions & 2 deletions

File tree

‎.bazelignore‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,8 @@
1-
./example/build_with_bazel
1+
./example/build_with_bazel
2+
3+
# `registry/` is brpc's self-maintained Bzlmod registry. Its overlay
4+
# BUILD.bazel files reference sources from the libunwind tarball that is
5+
# only materialized at module resolution time, not in the source tree.
6+
# Without this entry, `bazel build //...` would try to evaluate them as
7+
# regular packages and fail with "missing input file ...".
8+
./registry

‎.licenserc.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -218,4 +218,7 @@ header:
218218
# Fuzzing seed
219219
- 'test/fuzzing/fuzz_*_seed_corpus/*'
220220

221+
# bazel-central-registry
222+
- 'registry/**'
223+
221224
comment: on-failure

‎LICENSE‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -959,3 +959,41 @@ copyright The Chromium Authors and licensed under the 3-clause BSD license:
959959
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
960960
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
961961
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
962+
963+
--------------------------------------------------------------------------------
964+
965+
registry/modules/libunwind/**: licensed under the following terms:
966+
967+
Forked from the Bazel Central Registry (BCR):
968+
https://github.com/bazelbuild/bazel-central-registry/tree/main/modules/libunwind
969+
970+
The upstream files are distributed under the Apache License, Version 2.0,
971+
the same license as brpc itself (the full text of which is reproduced at
972+
the top of this LICENSE file). brpc has modified the forked files; per the
973+
Apache License 2.0 §4(b), the modifications are summarized below:
974+
975+
- registry/modules/libunwind/<version>/MODULE.bazel
976+
- registry/modules/libunwind/<version>/overlay/MODULE.bazel
977+
Append the suffix `.brpc-no-unwind` to the `version` field, marking
978+
these as brpc's variant of the libunwind module.
979+
980+
- registry/modules/libunwind/<version>/overlay/BUILD.bazel
981+
Add the `hide_unwind_symbols` config_setting (gated by
982+
`--define=libunwind_hide_unwind_symbols=true`); when the switch is
983+
on, drop `src/unwind/*.c` (the GCC `_Unwind_*` ABI compatibility
984+
layer) from `unwind_srcs` so the resulting libunwind does not export
985+
`_Unwind_*` symbols. See docs/cn/bthread_tracer.md for the
986+
rationale.
987+
988+
- registry/modules/libunwind/<version>/source.json
989+
Update overlay file SHA-256 hashes to match the modified
990+
BUILD.bazel / MODULE.bazel.
991+
992+
- registry/modules/libunwind/metadata.json
993+
Replace the `versions` array with brpc's renamed versions.
994+
995+
The above only governs the build/source files brpc redistributes inside
996+
registry/modules/libunwind/. The libunwind source code itself, downloaded
997+
by Bazel from the URL pinned in source.json, remains governed by the
998+
libunwind project's own license; see https://github.com/libunwind/libunwind
999+
for details.

‎bazel/config/BUILD.bazel‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,7 @@ config_setting(
136136
define_values = {
137137
"with_bthread_tracer": "true",
138138
},
139+
visibility = ["//visibility:public"],
139140
)
140141

141142
config_setting(

‎docs/cn/bthread_tracer.md‎

Lines changed: 140 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ jump_stack是bthread挂起或者运行的必经之路，也是STB的拦截点。
5858

5959
# 使用方法
6060

61-
1. 下载安装libunwind和abseil-cpp。
61+
1. 下载安装libunwind和abseil-cpp。**注意：libunwind 必须从源码编译，不要使用系统包管理器安装的 `libunwind-dev` / `libunwind-devel`**，否则会触发本文末尾「[已知问题：libunwind 与 libgcc_s 的 `_Unwind_*` 符号冲突](#已知问题libunwind-与-libgcc_s-的-_unwind_-符号冲突)」中描述的崩溃。bazel 构建可以跳过此步，直接使用 brpc 仓库自维护的 libunwind 版本。
6262
2. 给config_brpc.sh增加`--with-bthread-tracer`选项或者给cmake增加`-DWITH_BTHREAD_TRACER=ON`选项或者给bazel（Bzlmod模式）增加`--define with_bthread_tracer=true`选项。
6363
3. 访问服务的内置服务：`http://ip:port/bthreads/<bthread_id>?st=1`或者代码里调用`bthread::stack_trace()`函数。
6464
4. 如果希望追踪pthread的调用栈，在对应pthread上调用`bthread::init_for_pthread_stack_trace()`函数获取一个伪bthread_t，然后使用步骤3即可获取pthread调用栈。
@@ -73,6 +73,145 @@ jump_stack是bthread挂起或者运行的必经之路，也是STB的拦截点。
7373
#5 0x00007fdbbfa58dc0 bthread::TaskGroup::task_runner()
7474
```
7575

76+
# 已知问题
77+
78+
## libunwind 与 libgcc_s 的 `_Unwind_*` 符号冲突
79+
80+
### 现象
81+
82+
启用 bthread tracer 后，可能在 `bthread_exit` / `pthread_exit` 或者 C++ 异常处理路径上偶发段错误，类似如下调用栈：
83+
84+
```text
85+
#0 0x0000000000000000 in ?? ()
86+
#1 0x00007fa2b5d6458a in _ULx86_64_dwarf_find_proc_info ()
87+
from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so
88+
#2 0x00007fa2b5d6668d in fetch_proc_info ()
89+
from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so
90+
#3 0x00007fa2b5d681a1 in _ULx86_64_dwarf_make_proc_info ()
91+
from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so
92+
#4 0x00007fa2b5d70cfd in _ULx86_64_get_proc_info ()
93+
from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so
94+
#5 0x00007fa2b5d6c775 in __libunwind_Unwind_GetLanguageSpecificData ()
95+
from /root/.cache/bazel/_bazel_root/743b333b2429a1dbd390ef66b59c771d/execroot/_main/bazel-out/k8-fastbuild/bin/test/../_solib_k8/libexternal_Slibunwind~_Slibunwind.so
96+
#6 0x00007fa2b503c6df in __gxx_personality_v0 () from /lib/x86_64-linux-gnu/libstdc++.so.6
97+
#7 0x00007fa2b5452ce5 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
98+
#8 0x00007fa2b54533c0 in _Unwind_ForcedUnwind () from /lib/x86_64-linux-gnu/libgcc_s.so.1
99+
#9 0x00007fa2b4ca57a4 in __GI___pthread_unwind (buf=<optimized out>) at ./nptl/unwind.c:130
100+
#10 0x00007fa2b4c9dd22 in __do_cancel () at ../sysdeps/nptl/pthreadP.h:271
101+
#11 __GI___pthread_exit (value=0x0) at ./nptl/pthread_exit.c:36
102+
#12 0x0000000000000000 in ?? ()
103+
```
104+
105+
### 根因
106+
107+
libunwind 的 `src/unwind/*.c` 实现了 GCC 的 `_Unwind_*` ABI 兼容层（`_Unwind_GetLanguageSpecificData`、`_Unwind_ForcedUnwind`、`_Unwind_Resume` 等），与 `libgcc_s.so.1` 提供同名的全局符号。当 libunwind 以**动态库**形式被链接，并且在最终二进制的 `DT_NEEDED` 列表中位置比 `libgcc_s.so.1` 靠前时，运行时动态链接器 `ld.so` 会把 `pthread_exit` / 异常处理触发的 `_Unwind_*` 调用解析到 libunwind 中的 DWARF 实现。该实现需要的内部上下文在 `pthread_exit` 路径上未被 bRPC 初始化好，从而触发空指针访问。
108+
109+
这是一个 ELF **运行时符号解析顺序**问题，与编译器（GCC / Clang）无关 —— Clang 默认运行时同样使用 `libstdc++ + libgcc_s`，会复现完全一致的崩溃。
110+
111+
### 解决方案
112+
113+
> **重要：不要使用系统包管理器安装的 libunwind**（例如 `apt install libunwind-dev`、`yum install libunwind-devel`）。多数发行版打包的 `libunwind.so` 仍把 `_Unwind_*` 暴露在动态符号表中，会触发本节描述的崩溃。
114+
>
115+
> 必须使用**从源码编译的 libunwind**。上游 `./configure` + `make` 默认会通过 `-Wl,--version-script` 把 `_Unwind_*` 标为 local，不导出到动态符号表，从而避免冲突。
116+
117+
下表汇总了三种构建方式的推荐方案：
118+
119+
| 构建方式 | 推荐方案 |
120+
|---|---|
121+
| `config_brpc.sh` + `make` | 从源码编译并安装 libunwind，把头文件与库目录显式传给 `config_brpc.sh` |
122+
| `cmake` | 从源码编译并安装 libunwind，把头文件与库目录显式传给 `cmake` |
123+
| `bazel`（Bzlmod） | 直接使用 brpc 仓库自维护的 libunwind 版本 |
124+
125+
### `make`（`config_brpc.sh`）
126+
127+
源码编译并安装 libunwind 到一个独立目录（避免污染系统目录），然后让 `config_brpc.sh` 显式从该目录查找 libunwind。
128+
129+
```bash
130+
# 1) 源码编译 libunwind（推荐 v1.8.1 或以上版本）
131+
git clone https://github.com/libunwind/libunwind.git
132+
cd libunwind && git checkout tags/v1.8.1
133+
mkdir -p /opt/libunwind
134+
autoreconf -i
135+
./configure --prefix=/opt/libunwind
136+
make -j$(nproc) && make install
137+
cd ..
138+
139+
# 2) 让 config_brpc.sh 使用 /libunwind 下的头文件与库（不要让它自动找到系统的 libunwind-dev）
140+
cd brpc
141+
sh config_brpc.sh \
142+
--with-bthread-tracer \
143+
--headers="/opt/libunwind/include /usr/include" \
144+
--libs="/opt/libunwind/lib /usr/lib /usr/lib64"
145+
make -j$(nproc)
146+
```
147+
148+
构建完成后可用以下命令确认 libunwind.so 没有导出 `_Unwind_*`：
149+
150+
```bash
151+
nm -D /libunwind/lib/libunwind.so | grep ' T _Unwind_' \
152+
&& echo "WARN: _Unwind_* exported" \
153+
|| echo "OK: _Unwind_* hidden"
154+
```
155+
156+
### `cmake`
157+
158+
[`CMakeLists.txt`](../../CMakeLists.txt:90-100) 通过 `find_library(... NAMES unwind unwind-x86_64)` 查找 libunwind。同样需要先源码编译 libunwind 到独立前缀，再用 `CMAKE_PREFIX_PATH` 让 cmake 优先在该前缀下查找：
159+
160+
```bash
161+
# 1) 源码编译 libunwind（同 make 章节）
162+
163+
# 2) 让 cmake 在 /libunwind 下优先查找头文件和库
164+
cd brpc
165+
mkdir build && cd build
166+
cmake -DWITH_BTHREAD_TRACER=ON \
167+
-DCMAKE_PREFIX_PATH=/opt/libunwind \
168+
..
169+
make -j$(nproc)
170+
```
171+
172+
> 提示：如果系统已经装了 `libunwind-dev`，`find_library` 仍可能优先匹配到 `/usr/lib`。可在 cmake 命令上额外指定
173+
> `-DLIBUNWIND_LIB=/libunwind/lib/libunwind.so -DLIBUNWIND_X86_64_LIB=/libunwind/lib/libunwind-x86_64.so -DLIBUNWIND_INCLUDE_PATH=/libunwind/include`
174+
> 强制走自编译版本，避免系统包混入。
175+
176+
### `bazel`（Bzlmod）
177+
178+
bRPC 仓库已经在 [`registry/modules/libunwind/`](../../registry/modules/libunwind/) 维护了一份 libunwind 的 Bzlmod overlay，并通过 [`.bazelrc`](../../.bazelrc) 中的 `--registry=https://github.com/apache/brpc/registry` 使用 bRPC 维护的 overlay。版本号采用 `<base>.brpc-no-unwind` 后缀（例如 `1.8.3.brpc-no-unwind`），用以区别于 BCR 上的同基版本。该 overlay 增加了一个开关：
179+
180+
```
181+
--define libunwind_hide_unwind_symbols=true
182+
```
183+
184+
开启后，libunwind 的 `src/unwind/*.c`（即 GCC `_Unwind_*` 兼容层）整体不参与编译，等效于上游 autoconf 默认效果。bRPC 只使用 libunwind 的 `unw_*` 原生 API（`unw_getcontext`、`unw_init_local`、`unw_step` 等），不依赖 `_Unwind_*` 兼容层，因此该开关安全无副作用。
185+
186+
`.bazelrc` 已默认在 `build:test` / `test` 配置下打开此开关：
187+
188+
```
189+
build:test --define libunwind_hide_unwind_symbols=true
190+
test --define libunwind_hide_unwind_symbols=true
191+
```
192+
193+
用户按文档使用方法第 2 步加上 `--define=with_bthread_tracer=true` 即可：
194+
195+
```bash
196+
# 测试场景，.bazelrc 中 test 配置已自动带上 hide 开关
197+
bazel test //test:bthread_unittest
198+
199+
# 非测试构建（生产部署），需要显式同时带上两个 define
200+
bazel build --define=with_bthread_tracer=true \
201+
--define=libunwind_hide_unwind_symbols=true \
202+
//...
203+
```
204+
205+
> **特别注意**：如果在生产构建中只启用 `--define=with_bthread_tracer=true` 而漏掉 `--define=libunwind_hide_unwind_symbols=true`，binary 在 `pthread_exit` / 异常路径上会概率性崩溃。
206+
207+
构建后可用以下命令验证 libunwind 共享库没有导出 `_Unwind_*`：
208+
209+
```bash
210+
nm -D bazel-bin/external/_solib_*/libexternal*libunwind*.so 2>/dev/null \
211+
| grep ' T _Unwind_' || echo "OK: no _Unwind_* exported by libunwind.so"
212+
```
213+
214+
76215
# 相关flag
77216

78217
- `signal_trace_timeout_ms`：信号追踪模式的超时时间，默认为50ms。

‎registry/modules/libunwind/1.8.1.brpc-no-unwind/MODULE.bazel‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Forked from the Bazel Central Registry:
2+
# https://github.com/bazelbuild/bazel-central-registry/blob/main/modules/libunwind/1.8.1/MODULE.bazel
3+
# Distributed under the Apache License, Version 2.0. See the `registry/modules/
4+
# libunwind/**` entry near the bottom of brpc's LICENSE file.
5+
#
6+
# brpc modification (Apache License 2.0 §4(b)):
7+
# - `version` suffixed with `.brpc-no-unwind` to distinguish this brpc
8+
# variant from the upstream BCR version.
9+
10+
module(
11+
name = "libunwind",
12+
version = "1.8.1.brpc-no-unwind",
13+
bazel_compatibility = [">=7.2.1"], # need support for "overlay" directory
14+
compatibility_level = 0,
15+
)
16+
17+
bazel_dep(name = "bazel_skylib", version = "1.7.1")
18+
bazel_dep(name = "platforms", version = "0.0.11")
19+
bazel_dep(name = "rules_cc", version = "0.1.1")
20+
bazel_dep(name = "xz", version = "5.4.5.bcr.5")
21+
bazel_dep(name = "zlib", version = "1.3.1.bcr.5")

0 commit comments

Comments
 (0)