[serve] [llm] Known issues in Ray Serve LLM

[akshay-anyscale]

• Rename VLLMService deployment to VLLMServer
• Improve auto-scaling behavior for LLMRouter
• Fix multi-LoRA downloads from cloud storage - [serve.llm] Use pyarrow FS for cloud remote storage interaction #50820

[GeneDer]

Common pitfalls:

• /home/ray/.cache/torch_extensions/py311_cu124/xgrammar/xgrammar.so: cannot open shared object file: No such file or directory' when request json responses
  • Follow recommended requirements to pip install xgrammar==0.1.11 in your environment/ image
• RuntimeError: Failed to infer device type on vllm 0.7.2
  • Follow recommended requirements to pip install pynvml==12.0.0 in your environment/ image
• ModuleNotFoundError: No module named 'ray.serve.llm...
  • vllm 0.7.3 pins Ray to 2.40.0 and is incompatible with the latest Ray version which containing llm serving code. Please use vllm 0.7.2 for now until the incompatibility is resolve in a later vllm release

[SongGuyang]

Will we support kv-cache aware load balance in LLMRouter?

[akshay-anyscale]

Will we support kv-cache aware load balance in LLMRouter?

yes, we will publish a github issue with roadmap soon. KV cache aware LB is in design

[kouroshHakha]

all the issues here are closed and the roadmap is published at #51313