|
1 | | -<h1 align="center"> |
2 | | -Breathing Life into Language |
3 | | -</h1> |
| 1 | +<h1 align="center">Sonar</h1> |
4 | 2 |
|
5 | | - |
| 3 | + |
6 | 4 |
|
7 | | -Sonar (formerly Aphrodite Engine) is an inference engine that optimizes the serving of HuggingFace-compatible models at scale. Built on vLLM's Paged Attention technology, it delivers high-performance model inference for multiple concurrent users. Sonar serves as the backend engine powering the [Dolphin Inference Network](https://datagen.dphn.ai) and [PygmalionAI](https://pygmalion.chat)'s chat platforms and API infrastructure. |
| 5 | +Sonar is an inference engine for Hugging Face-compatible language and |
| 6 | +multimodal models. It provides continuous batching, paged KV-cache management, |
| 7 | +optimized kernels, quantization, speculative decoding, and distributed |
| 8 | +serving. |
8 | 9 |
|
9 | | -Sonar builds upon and integrates the exceptional work from [various projects](#acknowledgements), primarily [vLLM](https://vllm.ai). |
| 10 | +Sonar is based on [vLLM](https://github.com/vllm-project/vllm). It includes |
| 11 | +additional model and quantization formats, sampling methods, kernels, platforms, |
| 12 | +and deployment features. |
| 13 | +It serves production workloads for the |
| 14 | +[Dolphin Inference Network](https://datagen.dphn.ai) and |
| 15 | +[PygmalionAI](https://pygmalion.chat). |
10 | 16 |
|
11 | | -## Features |
| 17 | +## Documentation |
12 | 18 |
|
13 | | -- Continuous Batching |
14 | | -- Efficient K/V management with [PagedAttention](https://vllm.ai) from vLLM |
15 | | -- Optimized CUDA kernels for improved inference |
16 | | -- Quantization support via [AQLM](https://arxiv.org/abs/2401.06118), [AutoRound](https://arxiv.org/abs/2309.05516), [AWQ](https://arxiv.org/abs/2306.00978), [BitNet](https://arxiv.org/abs/2310.11453), [Bitsandbytes](https://arxiv.org/abs/2208.07339), [ExLlamaV3](https://github.com/turboderp-org/exllamav3), [GGUF](https://github.com/ggml-org/llama.cpp), [GPTQ](https://arxiv.org/abs/2210.17323), [QuIP#](https://arxiv.org/abs/2402.04396), [SqueezeLLM](https://arxiv.org/abs/2306.07629), [Marlin](https://arxiv.org/abs/2408.11743), [[2]](https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html) [[3]](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/), [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer), [TorchAO](https://github.com/pytorch/ao), [VPTQ](https://arxiv.org/abs/2409.17066), [compressed_tensors](https://github.com/vllm-project/llm-compressor), [MXFP4](https://huggingface.co/blog/RakshitAralimatti/learn-ai-with-me), and more. |
17 | | -- Distributed inference |
18 | | -- Quantized KV cache using scaled and scale-less FP8, and TurboQuant |
19 | | -- Support for modern samplers such as DRY, XTC, Mirostat, and more |
20 | | -- Disaggregated inference |
21 | | -- Speculative decoding, including EAGLE, DFlash, ngram, MTP, and more |
22 | | -- Multimodal support |
23 | | -- Multi-LoRA support |
| 19 | +Read the documentation at [sonar.dphn.ai](https://sonar.dphn.ai/). |
24 | 20 |
|
25 | | -## Quickstart |
| 21 | +- [Installation](https://sonar.dphn.ai/getting-started/installation/) |
| 22 | +- [Supported models](https://sonar.dphn.ai/reference/models/) |
| 23 | +- [Quantization support](https://sonar.dphn.ai/reference/quantization/) |
| 24 | +- [Server arguments](https://sonar.dphn.ai/reference/server-arguments/) |
| 25 | +- [Optimization](https://sonar.dphn.ai/deployment/optimization/) |
| 26 | +- [Parallelism](https://sonar.dphn.ai/deployment/parallelism/) |
| 27 | +- [Production deployment](https://sonar.dphn.ai/deployment/production/) |
26 | 28 |
|
27 | | -Install the engine (the Python package and CLI keep the historical `aphrodite` name for now): |
| 29 | +The model, quantization, and server-argument references are generated from the |
| 30 | +current source tree. |
28 | 31 |
|
29 | | -```sh |
30 | | -pip install -U aphrodite-engine |
31 | | -``` |
32 | | - |
33 | | -Then launch a model: |
34 | | - |
35 | | -```sh |
36 | | -aphrodite run Qwen/Qwen3.5-0.8B |
37 | | -``` |
38 | | - |
39 | | -This will create a [OpenAI](https://platform.openai.com/docs/api-reference/)-compatible API server that can be accessed at port 2242 of the localhost. You can plug in the API into a UI that supports OpenAI, such as [SillyTavern](https://github.com/SillyTavern/SillyTavern). |
40 | | - |
41 | | -## Requirements |
42 | | - |
43 | | -- Operating System: Linux, Windows (WSL2) |
44 | | -- Python: 3.10 to 3.13 (build from source for 3.14) |
45 | | - |
46 | | -#### Build Requirements |
| 32 | +## Install |
47 | 33 |
|
48 | | -- CUDA >= 12 |
| 34 | +The published NVIDIA wheel supports Linux x86-64, Python 3.10-3.13, and NVIDIA |
| 35 | +GPUs with compute capability 8.0 or newer. |
49 | 36 |
|
50 | | -### Notes |
| 37 | +Install [uv](https://docs.astral.sh/uv/getting-started/installation/), then |
| 38 | +create an environment: |
51 | 39 |
|
52 | | -1. By design, Sonar takes up 92% of your GPU's VRAM. If you're not serving an LLM at scale, you may want to limit the amount of memory it takes up. You can do this in the API example by launching the server with the `--gpu-memory-utilization 0.6` (0.6 means 60%). |
53 | | - |
54 | | -2. You can view the full list of commands by running `aphrodite run --help`. |
55 | | - |
56 | | -## Acknowledgements |
57 | | - |
58 | | -Sonar would have not been possible without the phenomenal work of other open-source projects. A (non-exhaustive) list: |
59 | | - |
60 | | -- [vLLM](https://github.com/vllm-project/vllm) |
61 | | -- [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) |
62 | | -- [xFormers](https://github.com/facebookresearch/xformers) |
63 | | -- [Flash Attention](https://github.com/Dao-AILab/flash-attention) |
64 | | -- [llama.cpp](https://github.com/ggerganov/llama.cpp) |
65 | | -- [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) |
66 | | -- [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) |
67 | | -- [SqueezeLLM](https://github.com/SqueezeAILab/SqueezeLLM/) |
68 | | -- [Exllamav2](https://github.com/turboderp/exllamav2) |
69 | | -- [TabbyAPI](https://github.com/theroyallab/tabbyAPI) |
70 | | -- [AQLM](https://github.com/Vahe1994/AQLM) |
71 | | -- [KoboldAI](https://github.com/henk717/KoboldAI) |
72 | | -- [Text Generation WebUI](https://github.com/oobabooga/text-generation-webui) |
73 | | -- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) |
74 | | -- [Ray](https://github.com/ray-project/ray) |
75 | | - |
76 | | -### Sponsors |
| 40 | +```bash |
| 41 | +uv venv --python 3.12 --seed |
| 42 | +source .venv/bin/activate |
| 43 | +uv pip install aphrodite-engine --torch-backend=cu130 |
| 44 | +``` |
77 | 45 |
|
78 | | -Past and present, in alphabetical order: |
| 46 | +Use the [complete installation guide](https://sonar.dphn.ai/getting-started/installation/) |
| 47 | +for AMD ROCm, Intel XPU, CPU, Apple silicon, Google TPU, Docker, WSL 2, source |
| 48 | +builds, and nightly wheels. |
79 | 49 |
|
80 | | -- [Arc Compute](https://www.arccompute.io/) |
81 | | -- [Lium](https://lium.io) |
82 | | -- [Prime Intellect](https://www.primeintellect.ai/) |
83 | | -- [PygmalionAI](https://pygmalion.chat) |
84 | | -- [Ruliad AI](https://ruliad.ai) |
| 50 | +## Serve a model |
85 | 51 |
|
86 | | -## Contributing |
| 52 | +```bash |
| 53 | +aphrodite serve Qwen/Qwen3-0.6B \ |
| 54 | + --served-model-name qwen3 |
| 55 | +``` |
87 | 56 |
|
88 | | -Everyone is welcome to contribute. You can support the project by opening Pull Requests for new features, fixes, or general UX improvements. |
| 57 | +The server listens on `http://127.0.0.1:2242` by default. It provides |
| 58 | +OpenAI-compatible APIs, health checks, metrics, and an OpenAPI schema. |
| 59 | + |
| 60 | +See the [OpenAI-compatible API guide](https://sonar.dphn.ai/serving/openai/) |
| 61 | +for streaming, embeddings, tool calls, reasoning output, and Sonar request |
| 62 | +parameters. |
| 63 | + |
| 64 | +## Key features |
| 65 | + |
| 66 | +- Continuous batching and paged KV-cache management |
| 67 | +- Prefix caching enabled by default |
| 68 | +- Tensor, pipeline, data, and expert parallelism |
| 69 | +- Multi-node multiprocessing without a Ray cluster |
| 70 | +- Prefill/decode disaggregation through NIXL and other KV connectors |
| 71 | +- Quantized weights and FP8 KV cache |
| 72 | +- Speculative decoding with MTP, EAGLE, DSpark, DFlash, n-gram, and other |
| 73 | + methods |
| 74 | +- Structured output, reasoning parsers, and automatic tool calling |
| 75 | +- Image, audio, and video model support |
| 76 | +- LoRA adapter serving |
| 77 | +- Prometheus metrics and health endpoints |
| 78 | +- OpenAI, Anthropic, pooling, scoring, reranking, transcription, and Kobold |
| 79 | + APIs |
| 80 | + |
| 81 | +Support depends on the model, device, data type, and quantization method. Check |
| 82 | +the generated [model matrix](https://sonar.dphn.ai/reference/models/) and |
| 83 | +[quantization matrix](https://sonar.dphn.ai/reference/quantization/) before |
| 84 | +deployment. |
| 85 | + |
| 86 | +Read the [optimization guide](https://sonar.dphn.ai/deployment/optimization/) |
| 87 | +before you tune scheduler, cache, compilation, quantization, or speculative |
| 88 | +decoding settings. |
| 89 | + |
| 90 | +## Development |
| 91 | + |
| 92 | +```bash |
| 93 | +git clone https://github.com/dphnAI/sonar.git |
| 94 | +cd sonar |
| 95 | +uv venv --python 3.13 --seed --prompt sonar |
| 96 | +source .venv/bin/activate |
| 97 | +APHRODITE_USE_PRECOMPILED=1 \ |
| 98 | + uv pip install --editable . --torch-backend=cu130 |
| 99 | +``` |
0 commit comments