Skip to content

Commit ff50e29

Browse files
AlpinDaleAlpinDale
committed
docs: add Rust frontend deployment guide
Signed-off-by: AlpinDale <alpindale@gmail.com>
1 parent 0b37108 commit ff50e29

4 files changed

Lines changed: 642 additions & 455 deletions

File tree

‎docs/astro.config.mjs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ export default defineConfig({
5353
{ label: 'Choose parallelism', slug: 'deployment/parallelism' },
5454
{ label: 'Distributed deployment', slug: 'deployment/distributed' },
5555
{ label: 'Production deployment', slug: 'deployment/production' },
56+
{ label: 'Rust frontend', slug: 'deployment/rust-frontend' },
5657
{ label: 'Deployment recipes', slug: 'deployment/recipes' },
5758
{ label: 'Model loading and storage', slug: 'deployment/model-storage' },
5859
{ label: 'Security', slug: 'deployment/security' },

‎docs/src/content/docs/deployment/production.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ See [Choose parallelism](/deployment/parallelism/) before you split one replica
2424
across GPUs. See [Distributed deployment](/deployment/distributed/) for
2525
multi-node launch commands.
2626

27+
Use the [Rust frontend](/deployment/rust-frontend/) when you want to evaluate
28+
the experimental Rust HTTP serving layer. Check its route and option
29+
compatibility before production use.
30+
2731
## Start a private server
2832

2933
```bash

‎docs/src/content/docs/deployment/rust-frontend.md‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
---
2+
title: Rust frontend
3+
description: Run Sonar with the experimental Rust HTTP frontend.
4+
---
5+
6+
The Rust frontend provides an alternative HTTP serving layer for Sonar. The
7+
Python supervisor still starts and manages the model engine. It passes the
8+
listening socket and engine transport addresses to the `aphrodite-rs`
9+
subprocess.
10+
11+
The Rust frontend is experimental. It does not implement every route or
12+
frontend option from the Python server. Test the required API surface before
13+
production use.
14+
15+
## Enable the packaged frontend
16+
17+
Set `APHRODITE_USE_RUST_FRONTEND=1`:
18+
19+
```bash
20+
APHRODITE_USE_RUST_FRONTEND=1 \
21+
aphrodite serve Qwen/Qwen3-0.6B \
22+
--served-model-name qwen3
23+
```
24+
25+
Sonar searches for an executable named `aphrodite-rs` inside the installed
26+
`aphrodite` package. Startup fails with a file-path error when the package does
27+
not contain the binary.
28+
29+
Set an explicit path when you build the binary from a source checkout:
30+
31+
```bash
32+
cargo build \
33+
--release \
34+
--manifest-path rust/Cargo.toml \
35+
--bin aphrodite-rs
36+
37+
APHRODITE_USE_RUST_FRONTEND=1 \
38+
APHRODITE_RUST_FRONTEND_PATH="$PWD/rust/target/release/aphrodite-rs" \
39+
aphrodite serve Qwen/Qwen3-0.6B \
40+
--served-model-name qwen3
41+
```
42+
43+
Build the binary from the same commit as the Python package. The two processes
44+
share an internal transport protocol.
45+
46+
## Architecture
47+
48+
The integrated launch has three parts:
49+
50+
```text
51+
client
52+
|
53+
v
54+
aphrodite-rs HTTP frontend
55+
|
56+
| ZMQ and MessagePack
57+
v
58+
Python engine process
59+
|
60+
v
61+
model workers
62+
```
63+
64+
Python binds the requested HTTP socket before it starts the Rust process. The
65+
Rust frontend inherits that socket. This keeps `--host`, `--port`, TLS, and
66+
process supervision in the normal `aphrodite serve` lifecycle.
67+
68+
Only one Rust API process can run for one integrated launch.
69+
`--api-server-count` values greater than one are rejected. Use complete Sonar
70+
replicas behind a load balancer when you need more frontend capacity.
71+
72+
## Supported routes
73+
74+
The current Rust router provides these public routes:
75+
76+
| Route | Purpose |
77+
| --- | --- |
78+
| `/health` | Process and engine health |
79+
| `/metrics` | Prometheus metrics |
80+
| `/load` | Current frontend load |
81+
| `/version` | Sonar and Rust frontend versions |
82+
| `/v1/models` | Served model list |
83+
| `/v1/completions` | OpenAI-compatible completions |
84+
| `/v1/chat/completions` | OpenAI-compatible chat completions |
85+
| `/tokenize` and `/detokenize` | Tokenizer operations |
86+
| `/inference/v1/generate` | Token-input and token-output generation |
87+
88+
Runtime LoRA routes are available when
89+
`APHRODITE_ALLOW_RUNTIME_LORA_UPDATING` is enabled. Development and profiling
90+
routes require their corresponding server settings.
91+
92+
The Rust frontend supports streaming chat and completion responses. It also
93+
supports API-key authentication, CORS, request-ID headers, TLS, chat
94+
templates, reasoning parsers, and tool parsers.
95+
96+
## Check option compatibility
97+
98+
Run this command from a source checkout:
99+
100+
```bash
101+
rust/target/release/aphrodite-rs serve --help
102+
```
103+
104+
The help output contains a section named
105+
`Options not implemented in Rust frontend yet`. The Rust process rejects these
106+
options with a startup error instead of silently applying different behavior.
107+
108+
Current limitations include multiple frontend-owned model and tokenizer
109+
overrides. Some multimodal controls and tracing options are also unavailable.
110+
The exact list can change with each commit, so use the binary's help output as
111+
the source of truth.
112+
113+
Engine-owned options still pass to the Python engine. Examples include
114+
`--tensor-parallel-size`, `--max-model-len`, `--gpu-memory-utilization`, and
115+
compilation settings.
116+
117+
## Validate a deployment
118+
119+
Check health, version, and model discovery:
120+
121+
```bash
122+
curl --fail http://127.0.0.1:2242/health
123+
curl --fail http://127.0.0.1:2242/version
124+
curl --fail http://127.0.0.1:2242/v1/models
125+
```
126+
127+
Send a non-streaming request:
128+
129+
```bash
130+
curl --fail http://127.0.0.1:2242/v1/chat/completions \
131+
-H "Content-Type: application/json" \
132+
-d '{
133+
"model": "qwen3",
134+
"messages": [
135+
{"role": "user", "content": "Reply with one short sentence."}
136+
],
137+
"max_tokens": 32
138+
}'
139+
```
140+
141+
Check the complete streaming path:
142+
143+
```bash
144+
curl --fail --no-buffer \
145+
http://127.0.0.1:2242/v1/chat/completions \
146+
-H "Content-Type: application/json" \
147+
-d '{
148+
"model": "qwen3",
149+
"messages": [
150+
{"role": "user", "content": "Count from one to three."}
151+
],
152+
"max_tokens": 32,
153+
"stream": true
154+
}'
155+
```
156+
157+
The response must contain one or more `data:` events and end with
158+
`data: [DONE]`.
159+
160+
Run [`aphrodite bench serve`](/deployment/benchmarking/) through the Rust
161+
frontend before production use. Use the same model, parsers, authentication,
162+
and streaming mode as the target workload.
163+
164+
## Diagnose startup
165+
166+
If Sonar cannot find the binary, set `APHRODITE_RUST_FRONTEND_PATH` to an
167+
absolute executable path. If the Rust process rejects an option, remove the
168+
option or use the Python frontend for that deployment.
169+
170+
Check both Python and Rust log lines during startup. A successful launch reports
171+
that the engines connected before it reports the HTTP bind address.
172+
173+
Unset `APHRODITE_USE_RUST_FRONTEND` to return to the Python frontend:
174+
175+
```bash
176+
unset APHRODITE_USE_RUST_FRONTEND
177+
unset APHRODITE_RUST_FRONTEND_PATH
178+
aphrodite serve MODEL
179+
```

0 commit comments

Comments
 (0)