Llama-server functionality with rpc support?

[justinjja]

Trying to do multi node inference,
Have it working with llama-cli, but is there a way to do it with llama-server / an openai endpoint?

[ggerganov]

I believe it should just work - what command are you trying?

[justinjja]

Not sure what I was doing before, but I went to recreate this and it just worked.
Thanks!

[segmond]

I read that both systems need to have the model weights, so transfer over network can be a huge bottleneck. Would it be possible for both models to have a copy of their own weight, do a checksum to make sure it's the same, then the RPC server will just be communicating which parts of the weight each client needs to load? This should allow for faster loading across even a basic network instead of possibly an hour wait time for a very large model.

[lexasub]

@segmond full model downloading - it's bad) disks in remote hosts is not infinity, but split model on files & load in model - it's global pull request with type project)

[weikinhuang]

Was the rpc functionality removed from llama-server?

[rgerganov]

No.

You can offload to an RPC server like this: llama-server -m gemma-3-4b-it-q4_0.gguf -ngl 99 --rpc 192.168.88.30:50052

[weikinhuang]

That doesn't seem to be the case, I'm using the latest :server docker image, do I need to use the full one? It's also no documented anywhere for llama-server.

/app/llama-server --no-webui --port 11434 --metrics --slots --jinja --no-warmup --alias mistral-small-3-1 --model /models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ4_XS.gguf --chat-template-file /models/chat_template.txt --ctx-size 24576 --flash-attn --cache-type-k q8_0 --cache-type-v q8_0 --rpc localhost:12000
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA RTX 2000 Ada Generation, compute capability 8.9, VMM: yes
load_backend: loaded CUDA backend from /app/libggml-cuda.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
error: invalid argument: --rpc

[slaren]

The docker images are built without RPC support, but that should be changed.

[weikinhuang]

That would be great!

[rgerganov]

Submitted PR #13474