Make Memory Sanitizer actually usable

[Shnatsel]

Memory Sanitizer, the tool to detect use of uninitialized memory, is ostensibly supported but has no documentation on actually using it. Simply trying to use it like any other sanitizer produces a false positive on startup.

There is currently no reliable alternative, which led people to build fragile custom tooling. Use of uninitialized memory can pose a security vulnerability, so it is important to have a tool that can detect it.

There have been rumors along the lines of "rebuild libstd with memory sanitizer using xargo and then you can use it on your binaries", but the exact process is not documented anywhere, and xargo is not really maintained but has a number of forks.

We need to figure out and document the exact process for using Memory Sanitizer, and build the missing tooling for doing so along the way.

[alex]

Even in C using msan is very difficult, since it requires you to have every single library compiled using it, this would include both libstd and libc, as well as any C libraries a project uses. It's not like ASAN/UBSAN where it's possible to have only some of a project compiled with it. So while I agree we should make it as easy as possible, it'll probably never be as easy as ASAN.

[Shnatsel]

The hub for work on LLVM sanitizers in Rust is https://github.com/japaric/rust-san

[Shnatsel]

If someone can concoct an alternative to Memory Sanitizer that does not require all of the code to be instrumented and merely requires Rust code to be, that would be great too. Perhaps Rust's custom allocator API can enable such use cases.

[Shnatsel]

MSAN is now usable on Nightly with -Z build-std option, see https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/sanitizer.html#memorysanitizer

If C code is being linked into the binary, you will also need to pass some C compiler flags: rust-lang/rust#39610 (comment)

MSAN in Rust could really use a tutorial and I fear rust-lang/rust#53945 still requires workarounds, but other than that we should be good to go.

[Shnatsel]

I've opened rust-fuzz/cargo-fuzz#233 to make cargo-fuzz pass the required flags behind the scenes and make Memory Sanitizer "just work".

[alex]

Idea that I don't know why it just occurred to me: Make it a build time error to link a vanilla libstd with an MSAN project. This should dramatically reduce the volume of false positives people hit (and then file bugs with, to the frustration and confusion of maintainers).

I cannot imagine any circumstance where vanilla libstd linked to an MSAN project would be desirable or useful.