Is CUDA Toolkit a Prerequisite? #1075
Replies: 4 comments
-
I'm using LLamasharp with CPU on a very old notebook with simple AVX support^^'. It is very slow ~ 1Token/sec but it works fine. LLamasharp is able to select an installed backend automatically, but you can also select what ever backend you like with the following call. It need to be called before you are using any other LLamasharp method. Example: You also need to install the same version of the backend as the version of llamasharp you are using. If you want to use only CPU, then you don't need to install the cuda backend, just the CPU backend. My notebook is able to use cuda8, but currently you can only download cuda 11 or 12 as a nuget package for Llamasharp. |
Beta Was this translation helpful? Give feedback.
-
I am using it with Vulkan which uses my GPU |
Beta Was this translation helpful? Give feedback.
-
Not sure why but it doesnt work on GPU for me, any ideas? ` class Program
|
Beta Was this translation helpful? Give feedback.
-
To answer the original question: yes and no 😅 Technically, the dependency is cudart (the CUDA runtime). We don't ship that as part of the CUDA nuget packages, so you need to supply it yourself. The easiest way to do that is usually to install the CUDA toolkit. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I can't make LlamaSharp work with my GPU. Is CUDA Toolkit a prerequisite? It's not mentioned anywhere.
Beta Was this translation helpful? Give feedback.
All reactions