investigate and fix water tests segfault on mi350

[ftynse]

I noticed this exclusion that makes us not run certain tests on MI350

wave/.github/workflows/ci-gpu.yaml

Lines 249 to 250 in 9810652

	# TODO: mlir_converter tests segfault on mi35x
	export WAVE_TEST_WATER=0

it should be investigated and fixed.

[ftynse]

Looks a problem when importing from water_mlir

[ftynse]

On one of the tests run, I see this:

# | FATAL: failed to import water_mlir: [Errno 2] No such file or directory: '/home/runner/_work/wave/wave/.wave-venv/lib/python3.11/site-packages/_rocm_sdk_devel/share/amd_smi/amdsmi/libamd_smi.so'

though for some reason it wasn't showing up before. This may be thanks to using python -v in the lit RUN line, but not sure.

[ftynse]

My current suspicion is that due to modified LIBRARY_PATH on 350 specifically, water libraries somehow pickup an incorrect dependency.

[Hardcode84]

You can try hide llvm symbols like I did for ExecutionEngine. ExecutionEngine seems to work fine.

[ftynse]

I'd like to understand the root cause...

…

On Wed, Dec 24, 2025, 16:29 Ivan Butygin ***@***.***> wrote: *Hardcode84* left a comment (iree-org/wave#633) <#633 (comment)> You can try hide llvm symbols like I did for ExecutionEngine. ExecutionEngine seems to work fine. — Reply to this email directly, view it on GitHub <#633 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALRG26O4KI72WIBILJPWUL4DKWL3AVCNFSM6AAAAACP5P24NWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMOJQGEYDANRZGE> . You are receiving this because you were assigned.Message ID: ***@***.***>

[ftynse]

Head of the stack trace

0x00007fffed1e7830 in llvm::SmallPtrSetImpl<llvm::cl::SubCommand*>::makeIterator(void const* const*) const ()
   from /home/azinenko/git/.wave_venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/llvm/lib/libLLVM.so.22.0git
(gdb) bt
#0  0x00007fffed1e7830 in llvm::SmallPtrSetImpl<llvm::cl::SubCommand*>::makeIterator(void const* const*) const ()
   from /home/azinenko/git/.wave_venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/llvm/lib/libLLVM.so.22.0git
#1  0x00007fffed1e9f38 in (anonymous namespace)::CommandLineParser::forEachSubCommand(llvm::cl::Option&, llvm::function_ref<void (llvm::cl::SubCommand&)>) [clone .constprop.2] ()
   from /home/azinenko/git/.wave_venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/llvm/lib/libLLVM.so.22.0git
#2  0x00007fffed1ea3d4 in llvm::cl::Option::addArgument() () from /home/azinenko/git/.wave_venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/llvm/lib/libLLVM.so.22.0git
#3  0x00007ffba7d0cd6c in _GLOBAL__sub_I_X86TargetMachine.cpp () from /home/azinenko/git/wave/wave_lang/kernel/wave/water_mlir/water_mlir/_mlir_libs/libWaterPythonCAPI.so

it appears indeed that somehow there is an LLVM version clash between the one installed with ROCm and the one that comes with water. This doesn't appear on machines other than 350 because we install ROCm differently on those.

[ftynse]

This weird binary pulls in LLVM:

   1821680:     file=libLLVM.so.22.0git [0];  needed by /home/azinenko/git/.wave_venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/llvm/bin/offload-arch [0]

$ ldd /home/azinenko/git/.wave_venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/llvm/bin/offload-arch
        linux-vdso.so.1 (0x000070d2d8010000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x000070d2d7fed000)
        libLLVM.so.22.0git => /home/azinenko/git/.wave_venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/llvm/bin/../lib/libLLVM.so.22.0git (0x000070d2d1600000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x000070d2d1200000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000070d2d1517000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000070d2d7fbd000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000070d2d0e00000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x000070d2d7fb8000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x000070d2d7fb3000)
        librocm_sysdeps_z.so.1 => /home/azinenko/git/.wave_venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/llvm/bin/../lib/../../rocm_sysdeps/lib/librocm_sysdeps_z.so.1 (0x000070d2d7f92000)
        librocm_sysdeps_zstd.so.1 => /home/azinenko/git/.wave_venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/llvm/bin/../lib/../../rocm_sysdeps/lib/librocm_sysdeps_zstd.so.1 (0x000070d2d1131000)
        /lib64/ld-linux-x86-64.so.2 (0x000070d2d8012000)

no idea so far why it is done in the python process

[ftynse]

This is not the only dependency, here's another one after I circumvent the need for that binary

$ ldd /home/azinenko/git/.wave_venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/libamd_comgr.so.3
        linux-vdso.so.1 (0x000071e8a6b4b000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x000071e8a6b28000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x000071e8a6b23000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x000071e8a6b1e000)
        librocm_sysdeps_z.so.1 => /home/azinenko/git/.wave_venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/rocm_sysdeps/lib/librocm_sysdeps_z.so.1 (0x000071e8a6aff000)
        librocm_sysdeps_zstd.so.1 => /home/azinenko/git/.wave_venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/rocm_sysdeps/lib/librocm_sysdeps_zstd.so.1 (0x000071e8a2f31000)
        libLLVM.so.22.0git => /home/azinenko/git/.wave_venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/llvm/lib/libLLVM.so.22.0git (0x000071e89c400000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x000071e89c000000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000071e8a2e48000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000071e89bc00000)
        /lib64/ld-linux-x86-64.so.2 (0x000071e8a6b4d000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000071e8a6acd000)

[ftynse]

So this library is apparently preloaded by pytorch when rocm is available, in torch/_rocm_init.py. So anything any python process that loaded torch has some LLVM symbols in it. This sounds dangerous... Unclear yet why IREE is not hitting this problem.

[ftynse]

I added the linker script as a workaround #641. Arguably torch rocm shouldn't just pull libLLVM.so into the torch process space, but that may take time and effort to change.