Is CUDA Toolkit a Prerequisite?

[ievgeniilarin]

I can't make LlamaSharp work with my GPU. Is CUDA Toolkit a prerequisite? It's not mentioned anywhere.

[hswlab]

I'm using LLamasharp with CPU on a very old notebook with simple AVX support^^'. It is very slow ~ 1Token/sec but it works fine. LLamasharp is able to select an installed backend automatically, but you can also select what ever backend you like with the following call. It need to be called before you are using any other LLamasharp method.

Example:
NativeLibraryConfig.All
.WithAvx(LLama.Native.AvxLevel.Avx)
.WithCuda(false)
.SkipCheck(true)
.WithAutoFallback(false)

You also need to install the same version of the backend as the version of llamasharp you are using. If you want to use only CPU, then you don't need to install the cuda backend, just the CPU backend.

My notebook is able to use cuda8, but currently you can only download cuda 11 or 12 as a nuget package for Llamasharp.
I also tested LLamasharp on a CPU with AVX2 support, it is much faster. I can live with that speed, but if I could I would use a GPU with cuda support :)

[iejeecee]

I am using it with Vulkan which uses my GPU

[Aurimas-TheDev]

Not sure why but it doesnt work on GPU for me, any ideas?

` class Program
{
static async Task Main(string[] args)
{
// Path to your model
string modelPath = @"C:\Users\aurim.cache\lm-studio\models\bartowski\Meta-Llama-3.1-8B-Instruct-GGUF\Meta-Llama-3.1-8B-Instruct-Q4_K_S.gguf";
NativeLibraryConfig.All.WithLogCallback(delegate (LLamaLogLevel level, string message) { Console.Write($"{level}: {message}"); });

        //            NativeLibraryConfig.All
        //.WithCuda(true)
        //.SkipCheck(true)
        //.WithAutoFallback(false);

        var parameters = new ModelParams(modelPath)
        {
            //ContextSize = 4096,
            //Threads = 6,
            //UseMemorymap = false,
            //GpuLayerCount = 32,
            MainGpu = 0,          // Ensure it’s set to the first GPU (0 if only one GPU)
        };

        using var model = LLamaWeights.LoadFromFile(parameters);
        using var context = model.CreateContext(parameters);
        var executor = new InteractiveExecutor(context);

        // Add chat histories as prompt to tell AI how to act.
        var chatHistory = new ChatHistory();
        string systemPrompt = "Always answer as specific as possible and to the point. Always respond only with answer, do not provide introductions. ";

        chatHistory.AddMessage(AuthorRole.System, systemPrompt);
        chatHistory.AddMessage(AuthorRole.User, "Hello, Bob.");
        chatHistory.AddMessage(AuthorRole.Assistant, "Hello. How may I help you today?");

        ChatSession session = new(executor, chatHistory);
        // add the default templator. If llama.cpp doesn't support the template by default, 
        // you'll need to write your own transformer to format the prompt correctly
        session.WithHistoryTransform(new PromptTemplateTransformer(model, withAssistant: true));

        // Add a transformer to eliminate printing the end of turn tokens, llama 3 specifically has an odd LF that gets printed sometimes
        session.WithOutputTransform(new LLamaTransforms.KeywordTextOutputStreamTransform(
            ["User:", "�"],
            redundancyLength: 5));

        var inferenceParams = new InferenceParams
        {
            SamplingPipeline = new DefaultSamplingPipeline
            {
                Temperature = 0.6f
            },

            MaxTokens = -1, // keep generating tokens until the anti prompt is encountered
            AntiPrompts = ["User:"] // model specific end of turn string (or default)
        };

        Console.ForegroundColor = ConsoleColor.Yellow;
        Console.WriteLine("The chat session has started.");

        // show the prompt
        Console.ForegroundColor = ConsoleColor.Green;
        Console.Write("User> ");
        var userInput = Console.ReadLine() ?? "";

        while (userInput != "exit")
        {
            Console.ForegroundColor = ConsoleColor.White;
            Console.Write("Assistant> ");

            // as each token (partial or whole word is streamed back) print it to the console, stream to web client, etc
            await foreach (
                var text
                in session.ChatAsync(
                    new ChatHistory.Message(AuthorRole.User, userInput),
                    inferenceParams))
            {
                Console.ForegroundColor = ConsoleColor.White;
                Console.Write(text);
            }
            Console.WriteLine();

            Console.ForegroundColor = ConsoleColor.Green;
            Console.Write("User> ");
            userInput = Console.ReadLine() ?? "";
        }
    }
}`

[martindevans]

To answer the original question: yes and no 😅

Technically, the dependency is cudart (the CUDA runtime). We don't ship that as part of the CUDA nuget packages, so you need to supply it yourself. The easiest way to do that is usually to install the CUDA toolkit.