Merge pull request #3492 from modelscope/codex/product-site-vllm-native-20260813

docs: validate native FunASR on vLLM 0.27.1

Author: LauraGPT

docs/vllm_native_funasr_validation.md

@@ -0,0 +1,76 @@
+# Native FunASR on vLLM 0.27.1 validation
+
+Verified on 2026-08-13. This record is a reproducible compatibility and
+concurrency probe, not an accuracy benchmark or production capacity claim.
+
+## Pinned stack
+
+- GPU: NVIDIA H100 80GB HBM3; driver 550.127.08
+- vLLM: 0.27.1+cu129, official x86_64 release wheel
+- vLLM wheel SHA-256: `bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b`
+- Torch: 2.13.0+cu129; CUDA available
+- Model: `allendou/Fun-ASR-Nano-2512-vllm`
+- Model revision: `e718b36e2578203ec893e9b488239225f8d668e2`
+- Model weight SHA-256: `96dfbec48282dd24d3334369a01e9e909f321ee39a1b0003c528c5379f68c1a6`
+- Audio extra: av 18.1.0, scipy 1.18.0, soundfile 0.14.0, soxr 1.1.0
+
+The checkpoint above is a community conversion. vLLM's FunASR architecture,
+transcription endpoint, hotword support, and initialization fix are upstream,
+but the converted checkpoint is not an official FunAudioLLM weight release.
+Use the [official split-engine guide](vllm_guide.md) when an official-weight
+chain is required.
+
+## Server
+
+```bash
+CUDA_VISIBLE_DEVICES=0 .venv/bin/vllm serve \
+  allendou/Fun-ASR-Nano-2512-vllm \
+  --revision e718b36e2578203ec893e9b488239225f8d668e2 \
+  --served-model-name fun-asr-nano \
+  --host 127.0.0.1 --port 8899 \
+  --dtype float32 \
+  --gpu-memory-utilization 0.40 \
+  --enforce-eager
+```
+
+The engine resolved `FunASRForConditionalGeneration`, retained the 40,960
+maximum model length, allocated 17.52 GiB of KV cache (82,016 tokens), and
+reported 2.00x maximum concurrency at that length. The cached second startup
+spent 20.30 seconds in engine profile, KV-cache creation, and warmup. API
+requests were sent only after `/health` returned HTTP 200.
+
+`--gpu-memory-utilization 0.20` is insufficient for the full model length: it
+left 1.70 GiB for KV cache while one 40,960-token request required 8.75 GiB.
+Do not copy the 0.40 setting blindly; size it against the target GPU and load.
+
+## Inputs and results
+
+All timing is client wall time over localhost after the server was healthy.
+Audio loading, model execution, and decoding are included. Model download and
+server startup are excluded.
+
+| Probe | Input | Result |
+| --- | --- | --- |
+| Chinese baseline | 6 s `example/zh.mp3`, SHA-256 `0e64de19e4ff9a02e682955c9112f32d2317cfdbb5bc2f3504664044c993f195` | HTTP 200 in 0.968 s: `开饭时间早上九点至下午五点。` |
+| Chinese hotword | Same input; `hotwords=开放时间,开放时间,开放时间` | HTTP 200 in 0.214 s: `开放时间早上九点至下午五点。` |
+| Two concurrent requests | 8 s English plus 8 s Japanese examples | Both HTTP 200; 1.123 s total wall time (English 1.036 s, Japanese 1.111 s) |
+
+The English result was `The tribal chieftain called for the boy, and presented
+him with fifty pieces of gold.` The Japanese result was
+`うちの中学は弁当制で、持っていけない場合は、五十円の学校販売のパンを買う。`
+
+A single `开放时间` hotword did not alter the baseline output. Repeating it
+three times changed the ambiguous phrase. This proves the request parameter
+reached the generation prompt, but also shows that hotword strength is a policy
+to validate on representative audio; it is not a deterministic correction.
+
+## Operational boundaries
+
+- Install `vllm[audio]`; without the audio extra, valid MP3 uploads return HTTP
+  400 with `Invalid or unsupported audio file`.
+- Pass `language` explicitly for non-English audio. The current vLLM FunASR
+  adapter defaults an omitted language to English.
+- Keep workers private and put authentication, TLS, rate limits, and audio
+  size/duration limits at the gateway.
+- Re-run accuracy, latency, concurrency, memory, and hotword tests on the exact
+  production GPU, driver, languages, and traffic distribution.

web-pages/product-site/data/deployments.json

@@ -10,59 +10,78 @@
       "workloads": ["batch", "private-api"],
       "hardware": ["nvidia-gpu", "kubernetes"],
       "priorities": ["throughput", "latency"],
-      "models": ["Fun-ASR-Nano-2512"],
+      "models": ["Fun-ASR-Nano-2512 (community vLLM conversion)"],
       "operating_systems": ["Linux"],
-      "interfaces": ["Python", "HTTP REST", "OpenAI-compatible HTTP", "WebSocket"],
-      "tested": {"funasr": "1.3.29", "runtime": "vLLM 0.19.1 guide", "verified": "2026-07-26"},
+      "interfaces": ["OpenAI-compatible HTTP"],
+      "tested": {"funasr": "Fun-ASR-Nano-2512 conversion@e718b36e", "runtime": "vLLM 0.27.1+cu129 / Torch 2.13.0+cu129", "verified": "2026-08-13"},
       "commands": {
-        "install": ["pip install funasr", "pip install \"vllm==0.19.1\""],
-        "launch": ["CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/serve_vllm.py --port 8899 --model FunAudioLLM/Fun-ASR-Nano-2512 --gpu-memory-utilization 0.5"],
-        "health": ["curl -fsS http://localhost:8899/health"],
-        "smoke": ["curl -X POST http://localhost:8899/v1/audio/transcriptions -F file=@audio.wav -F model=fun-asr-nano -F response_format=verbose_json"]
+        "install": [
+          "curl -fL https://github.com/vllm-project/vllm/releases/download/v0.27.1/vllm-0.27.1%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl -o vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl",
+          "echo \"bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b  vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl\" | sha256sum -c -",
+          "uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python --torch-backend=auto \"vllm[audio] @ file://$PWD/vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl\""
+        ],
+        "launch": [
+          "CUDA_VISIBLE_DEVICES=0 .venv/bin/vllm serve allendou/Fun-ASR-Nano-2512-vllm --revision e718b36e2578203ec893e9b488239225f8d668e2 --served-model-name fun-asr-nano --host 127.0.0.1 --port 8899 --dtype float32 --gpu-memory-utilization 0.40 --enforce-eager"
+        ],
+        "health": [
+          "curl -fsS http://127.0.0.1:8899/health",
+          "curl -fsS http://127.0.0.1:8899/v1/models"
+        ],
+        "smoke": [
+          "curl -fL https://huggingface.co/allendou/Fun-ASR-Nano-2512-vllm/resolve/e718b36e2578203ec893e9b488239225f8d668e2/example/zh.mp3 -o zh.mp3 && echo \"0e64de19e4ff9a02e682955c9112f32d2317cfdbb5bc2f3504664044c993f195  zh.mp3\" | sha256sum -c -",
+          "curl -fsS http://127.0.0.1:8899/v1/audio/transcriptions -F file=@zh.mp3 -F model=fun-asr-nano -F language=zh -F response_format=json",
+          "curl -fsS http://127.0.0.1:8899/v1/audio/transcriptions -F file=@zh.mp3 -F model=fun-asr-nano -F language=zh -F 'hotwords=开放时间,开放时间,开放时间' -F response_format=json"
+        ]
       },
       "evidence": [
-        {"label": "vLLM guide", "url": "https://github.com/modelscope/FunASR/blob/main/docs/vllm_guide.md"},
-        {"label": "deployment matrix", "url": "https://github.com/modelscope/FunASR/blob/main/docs/deployment_matrix.md"}
+        {"label": "H100 native FunASR validation record", "url": "https://github.com/modelscope/FunASR/blob/main/docs/vllm_native_funasr_validation.md"},
+        {"label": "vLLM 0.27.1 release and CUDA 12.9 wheel", "url": "https://github.com/vllm-project/vllm/releases/tag/v0.27.1"},
+        {"label": "native FunASR model integration", "url": "https://github.com/vllm-project/vllm/pull/33247"},
+        {"label": "native FunASR hotword support", "url": "https://github.com/vllm-project/vllm/pull/39674"},
+        {"label": "FunASR initialization fix", "url": "https://github.com/vllm-project/vllm/pull/44215"},
+        {"label": "validated community conversion at e718b36e", "url": "https://huggingface.co/allendou/Fun-ASR-Nano-2512-vllm/tree/e718b36e2578203ec893e9b488239225f8d668e2"},
+        {"label": "official FunASR split-engine guide", "url": "https://github.com/modelscope/FunASR/blob/main/docs/vllm_guide.md"},
+        {"label": "FunASR deployment matrix", "url": "https://github.com/modelscope/FunASR/blob/main/docs/deployment_matrix.md"}
       ],
       "benchmarks": [
         {
-          "model": "Fun-ASR-Nano-2512",
-          "runtime": "vLLM batch",
-          "hardware": "GPU model not recorded in the cited public table",
-          "workload": "Offline batch",
-          "audio": "184 long-form files; 11,541 seconds",
-          "settings": "Dynamic VAD; the public table does not record batch size or the full software and hardware stack",
-          "timing_scope": "Offline throughput; the cited table does not state whether warmup, file I/O, and decoding are excluded",
-          "result": "RTFx 340; CER 8.20%",
-          "qualification": "Incomplete hardware and timing record. Use only as public reference evidence, not a capacity promise.",
-          "source": "https://github.com/modelscope/FunASR/blob/main/docs/vllm_guide.md",
-          "verified": "2026-07-26"
+          "model": "allendou/Fun-ASR-Nano-2512-vllm@e718b36e",
+          "runtime": "vLLM 0.27.1+cu129 / Torch 2.13.0+cu129",
+          "hardware": "NVIDIA H100 80GB",
+          "workload": "OpenAI-compatible transcription; Chinese baseline and hotword probes plus two concurrent requests (English and Japanese)",
+          "audio": "Pinned model-repository examples: 6 s Chinese, 8 s English, and 8 s Japanese",
+          "settings": "FP32; eager mode; gpu-memory-utilization 0.40; max model length 40,960; explicit language; warmed server",
+          "timing_scope": "Client wall time after /health became ready; local HTTP and decoding included; cold model download and 20.3 s engine warmup excluded",
+          "result": "Chinese 200 in 0.968 s; repeated hotword corrected 开饭时间 to 开放时间 in 0.214 s; English and Japanese two-request batch completed in 1.123 s wall time",
+          "qualification": "Single-H100 correctness and concurrency probe with a community-converted checkpoint, not an accuracy study or production capacity promise. Revalidate the checkpoint, target GPU, languages, traffic, and hotword policy before rollout.",
+          "source": "https://github.com/modelscope/FunASR/blob/main/docs/vllm_native_funasr_validation.md",
+          "verified": "2026-08-13"
         }
       ],
       "translations": {
         "zh": {
-          "name": "vLLM GPU 高吞吐",
-          "summary": "用 Fun-ASR-Nano 在 NVIDIA GPU 上处理批量文件或提供兼容 OpenAI 的转写接口。",
-          "fit": ["批量录音转写", "需要提高 LLM 解码吞吐", "已有 NVIDIA GPU 服务环境"],
-          "not_fit": ["纯 CPU 或边缘设备", "Paraformer 等非自回归模型", "尚未完成显存与并发压测的公网服务"],
-          "selection_reason": "GPU 批处理和 vLLM 调度适合以吞吐为首要目标的 Fun-ASR-Nano 工作负载。",
-          "primary_limitation": "吞吐和显存占用取决于音频切分、批量、并发、GPU 型号与 vLLM 版本，必须按目标流量复测。",
+          "name": "vLLM 原生 FunASR 服务",
+          "summary": "通过 vLLM 内置的 FunASR 架构，在 NVIDIA GPU 上提供兼容 OpenAI 的多语言转写与热词接口。",
+          "fit": ["需要标准 /v1/audio/transcriptions 接口", "Fun-ASR-Nano 批量或并发转写", "已有 NVIDIA GPU 和 Linux 服务环境"],
+          "not_fit": ["只接受官方发布权重、不能使用社区转换 checkpoint", "纯 CPU 或边缘设备", "尚未完成显存、并发与业务音频压测的公网服务"],
+          "selection_reason": "vLLM 已原生注册 FunASR 架构，并提供 OpenAI 兼容转写、显存调度、并发和热词参数。",
+          "primary_limitation": "当前原生 vLLM 路径依赖社区转换 checkpoint；要求官方权重链路时使用官方 FunASR split-engine。实测仅覆盖单张 H100，必须按目标 GPU、音频、语言、热词与流量复测。",
           "status_label": "社区验证",
-          "operations": ["固定 FunASR、vLLM、Torch 与 CUDA 组合", "预热模型后再采集容量数据", "记录队列等待、首结果和最终结果延迟"],
-          "security": ["在 API 网关实现认证、限流和请求大小限制", "服务端不直接暴露到公网", "隔离模型缓存和上传临时目录"],
-          "troubleshooting": ["先核对 NVIDIA 驱动与 vLLM 所带 CUDA", "重复输出时记录两次完整结果和精确版本", "长音频遗漏时检查 VAD 最大分段时长"]
+          "operations": ["固定 vLLM wheel、Torch、CUDA、模型 revision 与音频依赖", "等待 /health 就绪并完成预热后再采集容量数据", "记录队列等待、最终结果延迟、显存和热词命中率"],
+          "security": ["在 API 网关实现认证、TLS、限流和音频大小/时长限制", "worker 仅绑定内网地址，不直接暴露公网", "校验 wheel、模型 revision 和示例音频摘要，隔离模型缓存与上传临时目录"],
+          "troubleshooting": ["音频返回 400 时确认安装了 vllm[audio]", "启动时 KV cache 不足则提高显存利用率或在业务允许时降低 max-model-len", "非英文请求必须显式传 language；热词用逗号分隔并以真实业务样本验证偏置强度"]
         },
         "en": {
-          "name": "vLLM GPU throughput",
-          "summary": "Run Fun-ASR-Nano on NVIDIA GPUs for file batches or an OpenAI-compatible transcription endpoint.",
-          "fit": ["Batch recording transcription", "LLM decoder throughput is the priority", "An NVIDIA GPU serving environment already exists"],
-          "not_fit": ["CPU-only or edge devices", "Non-autoregressive models such as Paraformer", "Internet-facing service before memory and load tests"],
-          "selection_reason": "GPU batching and vLLM scheduling fit Fun-ASR-Nano workloads that prioritize throughput.",
-          "primary_limitation": "Throughput and memory depend on segmentation, batch size, concurrency, GPU, and vLLM version; retest with target traffic.",
+          "name": "Native FunASR on vLLM",
+          "summary": "Use vLLM's built-in FunASR architecture for multilingual and hotword-aware transcription through an OpenAI-compatible API on NVIDIA GPUs.",
+          "fit": ["A standard /v1/audio/transcriptions endpoint", "Concurrent or batch Fun-ASR-Nano transcription", "An NVIDIA GPU Linux serving environment"],
+          "not_fit": ["Deployments that accept only officially published weights and cannot use a community conversion", "CPU-only or edge devices", "Internet-facing service before memory, concurrency, and workload tests"],
+          "selection_reason": "vLLM natively registers the FunASR architecture and provides OpenAI-compatible transcription, memory scheduling, concurrency, and hotword parameters.",
+          "primary_limitation": "The native vLLM path currently depends on a community-converted checkpoint; use the official FunASR split-engine when an official-weight chain is required. Evidence covers one H100 only, so retest the target GPU, audio, languages, hotwords, and traffic.",
           "status_label": "Community verified",
-          "operations": ["Pin the FunASR, vLLM, Torch, and CUDA combination", "Warm the model before capacity measurement", "Measure queue, first-result, and final-result latency"],
-          "security": ["Put authentication, rate limits, and upload limits at an API gateway", "Do not expose the worker directly to the internet", "Isolate model cache and temporary upload storage"],
-          "troubleshooting": ["Match the NVIDIA driver to vLLM's CUDA build", "Capture two complete outputs and exact versions for repetition", "Check the VAD maximum segment length when long audio loses content"]
+          "operations": ["Pin the vLLM wheel, Torch, CUDA, model revision, and audio dependencies", "Wait for /health and warm the model before measuring capacity", "Measure queue time, final-result latency, memory, and hotword hit rate"],
+          "security": ["Put authentication, TLS, rate limits, and audio size/duration limits at the gateway", "Bind workers to a private address instead of exposing them directly", "Verify wheel, model revision, and sample digests; isolate model cache and temporary uploads"],
+          "troubleshooting": ["For audio HTTP 400 responses, confirm vllm[audio] is installed", "If startup reports insufficient KV cache, raise GPU utilization or lower max-model-len only when the workload permits", "Pass language explicitly for non-English audio; comma-separate hotwords and validate bias strength on production samples"]
         }
       }
     },

web-pages/product-site/tests/test_registry.py

@@ -45,6 +45,56 @@ def test_language_pairs_have_identical_fields(valid_registry):
     assert all(set(zh) == set(en) for zh, en in deployment_pairs(valid_registry))
 
 
+def test_vllm_contract_tracks_native_funasr_release_and_h100_validation(valid_registry):
+    entry = next(item for item in valid_registry['deployments'] if item['id'] == 'vllm')
+
+    assert entry['maturity'] == 'community-verified'
+    assert entry['tested'] == {
+        'funasr': 'Fun-ASR-Nano-2512 conversion@e718b36e',
+        'runtime': 'vLLM 0.27.1+cu129 / Torch 2.13.0+cu129',
+        'verified': '2026-08-13',
+    }
+    assert entry['models'] == ['Fun-ASR-Nano-2512 (community vLLM conversion)']
+    install = '\n'.join(entry['commands']['install'])
+    assert 'vllm[audio]' in install
+    assert 'vllm-0.27.1%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl' in install
+    assert 'bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b' in install
+    launch = '\n'.join(entry['commands']['launch'])
+    for marker in (
+        'vllm serve allendou/Fun-ASR-Nano-2512-vllm',
+        '--revision e718b36e2578203ec893e9b488239225f8d668e2',
+        '--served-model-name fun-asr-nano',
+        '--dtype float32',
+        '--gpu-memory-utilization 0.40',
+    ):
+        assert marker in launch
+    smoke = '\n'.join(entry['commands']['smoke'])
+    assert '/v1/audio/transcriptions' in smoke
+    assert 'language=zh' in smoke
+    assert 'hotwords=开放时间,开放时间,开放时间' in smoke
+    evidence_urls = {item['url'] for item in entry['evidence']}
+    for url in (
+        'https://github.com/modelscope/FunASR/blob/main/docs/vllm_native_funasr_validation.md',
+        'https://github.com/vllm-project/vllm/releases/tag/v0.27.1',
+        'https://github.com/vllm-project/vllm/pull/33247',
+        'https://github.com/vllm-project/vllm/pull/39674',
+        'https://github.com/vllm-project/vllm/pull/44215',
+        'https://huggingface.co/allendou/Fun-ASR-Nano-2512-vllm/tree/e718b36e2578203ec893e9b488239225f8d668e2',
+    ):
+        assert url in evidence_urls
+    assert any(
+        benchmark['hardware'] == 'NVIDIA H100 80GB'
+        and 'two concurrent requests' in benchmark['workload']
+        and '1.123 s wall time' in benchmark['result']
+        and 'community-converted checkpoint' in benchmark['qualification']
+        and benchmark['source'] == 'https://github.com/modelscope/FunASR/blob/main/docs/vllm_native_funasr_validation.md'
+        for benchmark in entry['benchmarks']
+    )
+    limitation = entry['translations']['en']['primary_limitation'].lower()
+    assert 'community-converted checkpoint' in limitation
+    assert 'official funasr split-engine' in limitation
+
+
 def test_audio_cpp_contract_tracks_mainline_nano_and_sensevoice(valid_registry):
     entry = next(item for item in valid_registry['deployments'] if item['id'] == 'audio-cpp')
     llama_cpp = next(item for item in valid_registry['deployments'] if item['id'] == 'llama-cpp')