rocmPackages: 6.0.2 -> 6.3.3, and various ROCm build fixes and new packages #367695
+4,556
−3,009
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
LunNova
force-pushed
the
rocm-update
branch
2 times, most recently
from
76e05f1 to
c2abb37
Compare
|
not knowing much about the rocm stack, mainly here as i am using btop with rocm support enabled. I receive the following error when compiling |
|
@Shawn8901 Should be fixed now, was broken when I first opened the PR. |
|
One thing that I've wanted to do for a long time is to completely remove the ROCm LLVM as an stdEnv, which should solve a lot of these weird compilation errors. Solus's ROCm stack does this, and we compile every non-HIP code with GCC. In this way, the entire ROCm LLVM can be compacted into a single derivation and the complexity of packaging/updating ROCm LLVM is drastically reduced. That is, you should be able to use just the default stdenv with GCC to compile non-HIP code and tell CMake/HIPCC to use the ROCm LLVM only when compiling HIP code. You can achieve this entirely through environment variables. It doesn't make sense that because a portion of the codebase contains HIP code, any C/C++ in the codebase needs to be compiled with ROCm LLVM's C compiler. |
|
Ok I just noticed the "Contemplate trying to make a normal Nix style CC wrapper work again" section, so it seems like you've already experienced the pain of the ROCm LLVM 😅 I will give my idea a try in the next few days and get back. |
|
It looks like upstream are moving away from a separate hipcc and using clang (now Maintaining a separate HIP only compiler might require maintaining significant cmakefile patches to get it to be used, but if you can work out a way to do this that isn't maintenance hell that's great. |
Please see https://github.com/GZGavinZhao/rocm-llvm-project/commits/solus-rocm-6.2.x for the patches and https://lists.debian.org/debian-ai/2024/12/msg00042.html for more details. I hope they apply cleanly on v6.3, but if not I think the changes are easy enough to manually rewrite them. If you need patches for other components, please see |
Solus does this and we didn't have to use any patches. Most of the work done was figuring out the environment variables to tell CMake and/or HIPCC what our intended HIP compiler is. The only thing I'm worrying about is locating sysroots due to non-standard installation prefix, but other than that Solus's experience shows that this is definitely doable. |
github-actions
bot
added
6.topic: python
10.rebuild-darwin: 101-500
and removed
10.rebuild-darwin: 11-100
labels
LunNova
force-pushed
the
rocm-update
branch
2 times, most recently
from
22f00e1 to
c05b8cb
Compare
|
This should be testable (including torch) if you have an instinct MI50 or newer, Radeon VII, W/RX 6800 or similar, W/RX 78/900 or similar. Cards which relied on the ISA compat patches like iGPUs or 66/7xx aren't going to work, haven't applied @GZGavinZhao's updated patches yet. If you're going to try to test it you want >200G free space so hipblaslt can spew an unfathomable amount of asm into the build temp directory and a lot of cores or you'll be here all month. |
|
I'm having trouble getting this to build. I get Failed Tests (2): during the triton-llvm-19.1.0-rc1 test phase. RX6800XT. X86_64-linux on nixos. No overlays or config or anything, just trying to create a devshell with python312Packages.torch. I can post a (nearly) minimum reproducible flake: |
|
@henryrgithub Your test flake is using non-rocm torch which is broken in master with the same error. If you use Recommend updating inputs before starting a build of torchWithRocm, just fixed some issues. |
|
GZGavinZhao
reviewed
Dec 25, 2024
| }; | ||
|
|
||
| rocm-opencl-runtime = symlinkJoin { | ||
| name = "rocm-opencl-runtime-meta"; | ||
| # FIXME: we have compressed code objects now, may be able to skip two stages? |
There was a problem hiding this comment.
CCOB (compressed code object bundle) is on by default for composable_kernel for ROCm v6.2+, so yes no need for two-stage builds now.
|
I tried to use this PR via overlay But got a collision when building ollama But overall it seems to be building without errors. When not using ollama I was able to rebuild my system without errors. |
GZGavinZhao
reviewed
Dec 26, 2024
|
I got a separate HIP compiler working and successfully compiled |
|
ROCm's standard toolchain is clang + GNU libs including libstdc++. There are a few packages which don't compile with Clashing error is because I added a /llvm link to clr for use by other ROCm packages and ollama also already adds one to its ROCm env internally, can be resolved by dropping the /llvm link from ollama. |
|
Added exact GPU targets as pkgs.rocmPackages_6.gfx908, gfx1030 etc. |
|
I think the remaining failures aren't newly broken comparing with master. Old magma versions are broken due to rocm 5 clr failing, dbx is broken due to a pydantic version mismatch, pymc has a source hash mismatch.
|
|
Zluda is still on version 3 in nixpkgs, which expects ROCm 5.x.x |
|
Will look at zluda 4 this evening, likely broken due to build parallelism hacks that aren't yet cleaned up in this PR that were removed in @GZGavinZhao 's branch. |
|
JFYI, I've just built my system from your latest commit (without rocmSupport = true, that takes ages, just rocm itself and ollama). Finally it works and I can schedule models across several GPUs without resorting to Docker. Thank you. Zluda support would be very nice to have, it makes many things much easier. If you are open to donations I would be happy to chip in. |
|
I repaired exo to properly detect AMD last night, however tinygrad needs Was unable to rebuild tinygrad w/ ROCMSupport because of Pytorch being marked as broken (overriding fails building magma-2.9.0). This morning I repointed my nixpkgs to your branch to attempt to build against, it spent a very long time to compiling on my 13900K (w/ 6900XT) before eventually dying on Curious how you got composable_kernel to build successfully on your set-up? Also I want to thank you for your Herculean efforts on this PR! 🍻 |
It needs all the RAM. There's supposed to be code to limit the parallelism if you don't have enough but I guess it isn't tuned right. |
I confirm that I was able to successfully build @LunNova maybe let's mark this ready for review and I'll merge? we can still do further fixes in follow-up fixups, I am just afraid we will run into merge conflicts again if we do not merge this PR soon enough. |
🤣... now that you mention it, I did notice the system over 80GB of my 128GB of mem in btop at one point. I didn't see how high it got before it finally died, if I have another go I'll try to track that. Edit: I'm rerunning, I think I might have misread available for in use, 78GB available now w/ 43 in use. |
|
I've gotten it to build a couple of times on 64GB with Nix limited to one build at a time, with nothing else significant running the lowest available RAM I got to was 4GB IIRC. I made a little python script just to track it 😬. Still haven't gotten models to cooperate with it, but things have been moving fast. Hope it gets merged soon, it's been fun following the progress! |
|
Is there anyone we need to give heads up to in terms of hydra resource usage for the slow core and memory hungry builds? |
Good point. Maybe this should be targeted at staging instead of master? |
use zram swap, limit jobs count and core count in nix settings, the higher the parallelism, the more ram it needs. Use remote builders to speed things up. In my experiments, a 128Gb machine should be limited to 24-28 cores with zram on in order to be able to finish the build. On how high it might get... While running with 64 cores it ate 230Gb. This might also help (add your VRAM as a swap, it's very efficient): There is a neat calculator on this page: https://www.foroelectro.net/english-f34/using-excess-video-memory-as-swap-in-linux-t497.html but essentially you just have to sum two numbers. Source: https://wiki.archlinux.org/title/Swap_on_video_RAM Test your swap before you build, otherwise there might be nasty surprises. Turn off swaps on your nvme/ssd, if you leave them on, that almost guarantees 12309-ish freeze. |
github-actions
bot
added
6.topic: nixos
LunNova
changed the title
LunNova
commented
Mar 1, 2025
| rocsolver | ||
| # hipblaslt |
There was a problem hiding this comment.
Suggested change
| # hipblaslt |
| @@ -39,20 +44,44 @@ stdenv.mkDerivation (finalAttrs: { | |||
| owner = "ROCm"; | |||
| repo = "hipBLAS"; | |||
| rev = "rocm-${finalAttrs.version}"; | |||
| hash = "sha256-Fq7o2sMmHlHIv9UKJw+u/h9K/ZhKVJWwosYTdYIsscA="; | |||
| #rev = "a4b23dec749d9d623f0e7699045f381ec3eddfab"; | |||
There was a problem hiding this comment.
Suggested change
| #rev = "a4b23dec749d9d623f0e7699045f381ec3eddfab"; |
| @@ -106,16 +108,32 @@ stdenv.mkDerivation (finalAttrs: { | |||
| sqlite | |||
| oneDNN_2 | |||
| blaze | |||
| cppcheck | |||
| # cppcheck | |||
There was a problem hiding this comment.
Suggested change
| # cppcheck |
| miopen, | ||
| protobuf, | ||
| abseil-cpp, | ||
| half, | ||
| nlohmann_json, | ||
| msgpack, | ||
| sqlite, | ||
| oneDNN_2, | ||
| blaze, | ||
| cppcheck, |
There was a problem hiding this comment.
Suggested change
| cppcheck, |
| url = "https://github.com/GZGavinZhao/MIOpen/commit/416088b534618bd669a765afce59cfc7197064c1.patch"; | ||
| hash = "sha256-OwONCA68y8s2GqtQj+OtotXwUXQ5jM8tpeM92iaD4MU="; | ||
| }) | ||
| # (fetchpatch { |
There was a problem hiding this comment.
Missed this arch compat patch. Might cause missing kernels for some ops for GPUs that aren't officially supported like iGPUs.
| --replace "unpack_db(\"\''${CMAKE_SOURCE_DIR}/src/kernels/\''${FILE_NAME}.kdb.bz2\")" "" \ | ||
| --replace "MIOPEN_HIP_COMPILER MATCHES \".*clang\\\\+\\\\+$\"" "true" \ | ||
| --replace "set(MIOPEN_TIDY_ERRORS ALL)" "" # error: missing required key 'key' | ||
| #--replace "unpack_db(\"\''${CMAKE_SOURCE_DIR}/src/kernels/\''${FILE_NAME}.kdb.bz2\")" "" \ |
There was a problem hiding this comment.
Need to remove these commented lines in postPatch
| (lib.cmakeFeature "CMAKE_C_COMPILER" "hipcc") | ||
| (lib.cmakeFeature "CMAKE_CXX_COMPILER" "hipcc") | ||
| "-DCMAKE_BUILD_TYPE=Release" | ||
| # "-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON" |
There was a problem hiding this comment.
Suggested change
| # "-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON" |
Comment on lines
88
to
90
| (writeShellScriptBin "amdclang++" '' | ||
| exec clang++ "$@" | ||
| '') |
There was a problem hiding this comment.
Should be able to drop this, clr has amdclang++ bin in PATH
Suggested change
| (writeShellScriptBin "amdclang++" '' | |
| exec clang++ "$@" | |
| '') |
Comment on lines
56
to
59
| preConfigure = '' | ||
| makeFlagsArray+=("-l$(((NIX_BUILD_CORES * 2) / 3))") | ||
| ''; | ||
|
|
There was a problem hiding this comment.
Missed a parallelism bodge, needs removed before merge
Suggested change
| preConfigure = '' | |
| makeFlagsArray+=("-l$(((NIX_BUILD_CORES * 2) / 3))") | |
| ''; |
Includes patch suggested by @shuni64 which fixes half precision ABI issues Includes hipblaslt compression patch Includes configurable hipblaslt support in rocblas rocmPackages_6.hipblaslt: respect NIX_BUILD_CORES in tensilelite rocmPackages_6.hipblas: propagate hipblas-common rocmPackages_6.clr: avoid confusion with hipClangPath Co-authored-by: Gavin Zhao <[email protected]>
rocmPackages.hsa-amd-aqlprofile-bin: 6.3.0 -> 6.3.3 rocmPackages.composable_kernel: unsplit build and clean up remove unused disable-amdgpu-inline.patch update documentation for build flags composable_kernel's library files are compressed now so no need to apply our own compression Co-authored-by: Gavin Zhao <[email protected]> rocmPackages.llvm: comment out -orig attrs and document usage rocmPackages.amdsmi: fix build rocmPackages_6.rocm-docs-core: fix build rocmPackages.rpp-opencl: mark broken rocmPackages.rocgdb: fix build rocmPackages.migraphx: fix build rocmPackages.llvm: remove unneeded duplicate patch rocmPackages.rdc: fix build
Signed-off-by: Gavin Zhao <[email protected]>
Fixes the build against rocmPackages_6 Can't easily apply the PR as a patch because we rely on the tarball with pregenerated hipified files ∴ fetchpatch of the PR will apply cleanly but fail to build
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #337159
Fixes #383836
Fixes #379354
WIP bump to 6.3 for rocmPackages_6 package set.
Upstream PRs/issues Raised
TODO List
and msgpackworking for hipblaslt. 10GB derivation is not ok.Make use of working LLVM packages .override to simplify LLVMoverrideScope isn't present and is needed.Allow better build parallelism by creating -minimal versions of some of the huge packages built for no gfx archesToo difficult, not doing in this PRThings done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.