Debugging linking errors when wasm exceptions are enabled

[jlb6740]

Hi, I am trying to build and use an example wasm filter in envoy that needs exception support. I want to preface by saying solving this problem may need to be done 100% in the envoy build system, but I filed an issue here because I am simply looking to better understand what is causing the errors emscripten is reporting during linking. I am also hoping someone can point to an emscripten zulip ro slack channel to share these build experiences and discuss in real time.

But back to the question ... envoy uses bazel for it's build system. The code that I am building builds standalone with exception support using a cmake configuration and runs as expected. In that case the "-fwasm_exceptions" flag is passed as a CMAKE_CXX_FLAGS and that pretty much is it as long as you are using the latest emscripten and node. When building the same code in envoy things become more complicated. Building details are more abstracted, but compiling is more complicated, linking is more complicated, and potentially conflicting build flags sneak in from all over the place.

The best combination of flags I can find is:

With bazel build command:
bazel build --subcommands --verbose_failures --explain --sandbox_debug --features wasm_simd --features exceptions --define libhs_compilation_target=wasm //src:example-plugin.wasm

And build flags
proxy_wasm_cc_binary(
name = "example-plugin.wasm",
srcs = [
"hyperscan_plugin.cc",
],
deps = [
"@hyperscan//:libhs",
],
copts = [
"-fwasm-exceptions",
],
linkopts = [
"-fwasm-exceptions",
],
)

And then commenting out code in building.py in the emsdk/upstream/emscripten/tools directory near lines (231-232):

#if not settings.DISABLE_EXCEPTION_CATCHING:
#   args += ['-enable-emscripten-cxx-exceptions']

Obviously I don't want to comment this code out without understanding the ramifications but I found this flag to be incompatible in the linking phase for emscripten and I couldn't figure out a flag combination to pass in bazel to get rid of it. A wasm filter created in this way and run as a standalone in node seems to work and have exceptions caught properly but a more complex example run in envoy did not. Specifically it compiles and runs, but the exception is not caught and handled in the wasm code at runtime. I can not comment out this code and try other flag combinations including introducing the C++ (non-wasm) exception handling code flags: DISABLE_EXCEPTION_CATCHING, EXCEPTION_CATCHING_ALLOWED, and DISABLE_EXCEPTION_THROWING, but applying these combinations in various levels leads to playing wackamole with linking and even compile errors:

In particular I am interested in understanding the "ReferenceError: addCxaCatch is not defined" error. (1) Can someone tell me how to make sure emscripten generates this function? I thought it was done when passing "-s DISABLE_EXCEPTION_THROWING=0" but I guess that just causes it to be expected. I am also not clear on if the -03 options is potentially stripping out exception support. Note during linking bazel/envoy is linking in three files and using these parameters:

--sysroot=external/emscripten_bin_linux/emscripten/cache/sysroot
-fdiagnostics-color
-fno-strict-aliasing
-funsigned-char
-no-canonical-prefixes
-msimd128
-s
PRINTF_LONG_DOUBLE=1
--oformat=js
-O0
bazel-out/wasm-fastbuild-ST-2ed02aadfb4a/bin/src/_objs/proxy_wasm_example-plugin/example_plugin.o
bazel-out/wasm-fastbuild-ST-2ed02aadfb4a/bin/external/example/libhs/lib/libhs.a
bazel-out/wasm-fastbuild-ST-2ed02aadfb4a/bin/external/proxy_wasm_cpp_sdk/libproxy_wasm_intrinsics.a
-fwasm-exceptions
-s
DISABLE_EXCEPTION_THROWING=0
--no-entry
--js-library=external/proxy_wasm_cpp_sdk/proxy_wasm_intrinsics.js
-sSTANDALONE_WASM
-sEXPORTED_FUNCTIONS=_malloc
-o
bazel-out/wasm-fastbuild-ST-2ed02aadfb4a/bin/src/proxy_wasm_example-plugin

I've tried to investigate the building of each of those 3 files but so far haven't found any solution. So my question (2) is when linking is various files where wasm exception support is required, are there certain functions that I can look for when doing wasm2wat or certain build flags or just certain things to look out for when building and linking. (3) Are there any flags in the linking phase above that are missing?

[jlb6740]

Note also, I've asked a related question in the proxy-wasm-cpp-sdk repo proxy-wasm/proxy-wasm-cpp-sdk#140

[kripken]

cc @walkingeyerobot

[walkingeyerobot]

I don't know a lot about exceptions themselves; the codebase the bazel toolchain was originally written for largely disallowed their use. I know exceptions are not simply turned on or off; there are options within exceptions themselves that may affect your build, such as differences in setjmp/longjmp handling. Hopefully someone like @kripken can speak more to what combination of flags might be best, or even point you to which flags are relevant. However, I may be able to point you to a few things that might make it easier to debug.

First, the linkopts you pasted in are not necessarily the linkopts being passed to lld. Those are the flags that are passed to emcc.py during link. You might want to inspect what emscripten itself is passing to lld in the function link_lld in tools/building.py.

Second, passing copts on bazel rules doesn't always do what you expect. I'm not familiar with the inner workings of proxy_wasm_cc_binary, so it's entirely possible they're doing something different here, but I can speak to what happens when you use copts on a normal cc_binary: they only apply to the files specific to that rule (i.e. the ones in srcs) and NOT to any of the dependencies. So if you're depending on the other files via a cc_library and that rule doesn't have copts on it, when bazel goes to build that cc_library, those copts will not be used. For linkopts on a cc_binary, they do apply to the entire build, because there is only one link action while there are many compile actions

If you want to test copts or linkopts and have them apply to the entire build, you can modify crosstool.bzl directly. Take this block for example which defines the exceptions feature:

        # Set if enabling exceptions.
        feature(name = "exceptions"),

        flag_set(
            actions = all_cpp_compile_actions,
            flags = [
                "-fno-exceptions",
            ],
            not_features = ["exceptions"],
        ),
        flag_set(
            actions = all_cpp_compile_actions,
            flags = [
                "-fexceptions",
            ],
            features = ["exceptions"],
        ),

If you wanted to have the feature instead pass -fexceptions / -fno-exceptions to all compile and link actions, you could change actions = all_cpp_compile_actions to actions = all_cpp_compile_actions + all_link_actions. Adding your own features for testing here may also make debugging easier.

[jlb6740]

@walkingeyerobot Thanks for the tips! These insights have come in handy. I was able to resolve the issue in envoy by passing -fwasm-exceptions at an appropriate point in the build files where it was needed. I still have to comment out lines in building.py in the cache:

  #if not settings.DISABLE_EXCEPTION_CATCHING:
  #  args += ['-enable-emscripten-cxx-exceptions']

and this is something that still needs to be resolved in a better way. Manipulation of setting or unsetting DISABLE_EXCEPTION_CATCHING and DISABLE_EXCEPTION_THROWING as either compile or link flags does not work. Any ideas here? The point is to get rid of the "-enable-emscripten-cxx-exceptions" flag as it conflicts with the wasm exceptions flags.

[sbc100]

What version of emsdk are you using?

In recent versions -fwasm-exceptions (needed at both link time and compile time) should set DISABLE_EXCEPTION_CATCHING to be true:\

  if settings.WASM_EXCEPTIONS:    
    if 'DISABLE_EXCEPTION_CATCHING' in user_settings:    
      exit_with_error('DISABLE_EXCEPTION_CATCHING is not compatible with -fwasm-exceptions')    
    if 'DISABLE_EXCEPTION_THROWING' in user_settings:                                            
      exit_with_error('DISABLE_EXCEPTION_THROWING is not compatible with -fwasm-exceptions')    
    settings.DISABLE_EXCEPTION_CATCHING = 1                                                      
    settings.DISABLE_EXCEPTION_THROWING = 1     

Which means -enable-emscripten-cxx-exception should never be passed to llvm.