Skip to content

Commit 992e328

Browse files
authored
docs: correct Nano timestamp capability scope (#3232)
Co-authored-by: LauraGPT <LauraGPT@users.noreply.github.com>
1 parent d5502d3 commit 992e328

4 files changed

Lines changed: 4 additions & 4 deletions

File tree

benchmark.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ <h1>Benchmark</h1>
2828
<table><tr><th>Metric</th><th>Result</th></tr><tr><td>Dataset</td><td>184 long-form Chinese audio files, 11,539 s total, 192.3 min.</td></tr><tr><td>GPU</td><td>NVIDIA H100 80GB HBM3.</td></tr><tr><td>Best GPU speed</td><td>SenseVoice-Small: 169.6x realtime in the full benchmark, 211.8x in the initial run.</td></tr><tr><td>Best CPU speed</td><td>SenseVoice-Small: 17.2x realtime; Paraformer-Large: 15.6x realtime.</td></tr><tr><td>Baseline</td><td>OpenAI Whisper-large-v3: 13.4x realtime on GPU.</td></tr></table>
2929
</section>
3030
<section id="table"><h2>Results</h2>
31-
<table><tr><th>Model</th><th>Device</th><th>RTF</th><th>Speed</th><th>CER</th><th>Notes</th></tr><tr><td><b>SenseVoice-Small</b></td><td>GPU</td><td>0.005896</td><td><b>169.6x</b></td><td>7.81%</td><td>ASR + language / emotion / event tags; CER after tag stripping.</td></tr><tr><td><b>Paraformer-Large</b></td><td>GPU</td><td>0.008359</td><td><b>119.6x</b></td><td>10.18%</td><td>Fast non-autoregressive Chinese ASR with VAD/punctuation pipeline.</td></tr><tr><td><b>Fun-ASR-Nano</b></td><td>GPU</td><td>0.058803</td><td>17.0x</td><td>8.06%</td><td>LLM-based ASR for Chinese, English, Japanese, seven Chinese dialect groups, and 26 regional accents; supports timestamps and hotwords.</td></tr><tr><td>GLM-ASR-Nano</td><td>GPU</td><td>0.026974</td><td>37.1x</td><td>31.07%</td><td>LLM-based multilingual ASR.</td></tr><tr><td>Whisper-large-v3-turbo (OpenAI)</td><td>GPU</td><td>0.021708</td><td>46.1x</td><td>21.71%</td><td>OpenAI Whisper implementation.</td></tr><tr><td>Whisper-large-v3 (OpenAI)</td><td>GPU</td><td>0.074694</td><td>13.4x</td><td>20.02%</td><td>Baseline for large Whisper quality.</td></tr><tr><td><b>SenseVoice-Small</b></td><td>CPU</td><td>0.057988</td><td><b>17.2x</b></td><td>7.81%</td><td>CPU run from the remaining benchmark script.</td></tr><tr><td><b>Paraformer-Large</b></td><td>CPU</td><td>0.064056</td><td><b>15.6x</b></td><td>10.18%</td><td>CPU viable for batch jobs.</td></tr><tr><td>Fun-ASR-Nano</td><td>CPU</td><td>0.274318</td><td>3.6x</td><td>8.06%</td><td>LLM-based model is heavier but still above realtime.</td></tr></table>
31+
<table><tr><th>Model</th><th>Device</th><th>RTF</th><th>Speed</th><th>CER</th><th>Notes</th></tr><tr><td><b>SenseVoice-Small</b></td><td>GPU</td><td>0.005896</td><td><b>169.6x</b></td><td>7.81%</td><td>ASR + language / emotion / event tags; CER after tag stripping.</td></tr><tr><td><b>Paraformer-Large</b></td><td>GPU</td><td>0.008359</td><td><b>119.6x</b></td><td>10.18%</td><td>Fast non-autoregressive Chinese ASR with VAD/punctuation pipeline.</td></tr><tr><td><b>Fun-ASR-Nano</b></td><td>GPU</td><td>0.058803</td><td>17.0x</td><td>8.06%</td><td>LLM-based ASR for Chinese, English, Japanese, seven Chinese dialect groups, and 26 regional accents; supports hotwords. Reliable checkpoint-native timestamps are not available (<a href="https://github.com/FunAudioLLM/Fun-ASR/issues/106">#106</a>).</td></tr><tr><td>GLM-ASR-Nano</td><td>GPU</td><td>0.026974</td><td>37.1x</td><td>31.07%</td><td>LLM-based multilingual ASR.</td></tr><tr><td>Whisper-large-v3-turbo (OpenAI)</td><td>GPU</td><td>0.021708</td><td>46.1x</td><td>21.71%</td><td>OpenAI Whisper implementation.</td></tr><tr><td>Whisper-large-v3 (OpenAI)</td><td>GPU</td><td>0.074694</td><td>13.4x</td><td>20.02%</td><td>Baseline for large Whisper quality.</td></tr><tr><td><b>SenseVoice-Small</b></td><td>CPU</td><td>0.057988</td><td><b>17.2x</b></td><td>7.81%</td><td>CPU run from the remaining benchmark script.</td></tr><tr><td><b>Paraformer-Large</b></td><td>CPU</td><td>0.064056</td><td><b>15.6x</b></td><td>10.18%</td><td>CPU viable for batch jobs.</td></tr><tr><td>Fun-ASR-Nano</td><td>CPU</td><td>0.274318</td><td>3.6x</td><td>8.06%</td><td>LLM-based model is heavier but still above realtime.</td></tr></table>
3232
</section>
3333
<section id="method"><h2>Methodology</h2>
3434
<p>Measurements were collected with the benchmark scripts in the workspace on 184 audio files. RTF is <code>total inference time / total audio duration</code>; speed is <code>1 / RTF</code>. CER is computed after model-specific text cleanup, especially for SenseVoice tags.</p>

ja/benchmark.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ <h1>ベンチマーク</h1>
2828
<table><tr><th>指標</th><th>結果</th></tr><tr><td>データセット</td><td>中国語の長時間音声 184 ファイル、合計 11,539 秒、192.3 分。</td></tr><tr><td>GPU</td><td>NVIDIA H100 80GB HBM3.</td></tr><tr><td>最高 GPU 速度</td><td>SenseVoice-Small: 169.6x realtime in the full benchmark, 211.8x in the initial run.</td></tr><tr><td>最高 CPU 速度</td><td>SenseVoice-Small: 17.2x realtime; Paraformer-Large: 15.6x realtime.</td></tr><tr><td>ベースライン</td><td>OpenAI Whisper-large-v3: 13.4x realtime on GPU.</td></tr></table>
2929
</section>
3030
<section id="table"><h2>結果</h2>
31-
<table><tr><th>モデル</th><th>デバイス</th><th>RTF</th><th>速度</th><th>CER</th><th>メモ</th></tr><tr><td><b>SenseVoice-Small</b></td><td>GPU</td><td>0.005896</td><td><b>169.6x</b></td><td>7.81%</td><td>ASR + language / emotion / event tags; CER after tag stripping.</td></tr><tr><td><b>Paraformer-Large</b></td><td>GPU</td><td>0.008359</td><td><b>119.6x</b></td><td>10.18%</td><td>Fast non-autoregressive Chinese ASR with VAD/punctuation pipeline.</td></tr><tr><td><b>Fun-ASR-Nano</b></td><td>GPU</td><td>0.058803</td><td>17.0x</td><td>8.06%</td><td>中国語・英語・日本語、7つの中国語方言グループ、26の地域アクセントに対応する LLM-based ASR。timestamp と hotword に対応。</td></tr><tr><td>GLM-ASR-Nano</td><td>GPU</td><td>0.026974</td><td>37.1x</td><td>31.07%</td><td>LLM-based multilingual ASR.</td></tr><tr><td>Whisper-large-v3-turbo (OpenAI)</td><td>GPU</td><td>0.021708</td><td>46.1x</td><td>21.71%</td><td>OpenAI Whisper implementation.</td></tr><tr><td>Whisper-large-v3 (OpenAI)</td><td>GPU</td><td>0.074694</td><td>13.4x</td><td>20.02%</td><td>ベースライン for large Whisper quality.</td></tr><tr><td><b>SenseVoice-Small</b></td><td>CPU</td><td>0.057988</td><td><b>17.2x</b></td><td>7.81%</td><td>CPU run from the remaining benchmark script.</td></tr><tr><td><b>Paraformer-Large</b></td><td>CPU</td><td>0.064056</td><td><b>15.6x</b></td><td>10.18%</td><td>CPU viable for batch jobs.</td></tr><tr><td>Fun-ASR-Nano</td><td>CPU</td><td>0.274318</td><td>3.6x</td><td>8.06%</td><td>LLM-based model is heavier but still above realtime.</td></tr></table>
31+
<table><tr><th>モデル</th><th>デバイス</th><th>RTF</th><th>速度</th><th>CER</th><th>メモ</th></tr><tr><td><b>SenseVoice-Small</b></td><td>GPU</td><td>0.005896</td><td><b>169.6x</b></td><td>7.81%</td><td>ASR + language / emotion / event tags; CER after tag stripping.</td></tr><tr><td><b>Paraformer-Large</b></td><td>GPU</td><td>0.008359</td><td><b>119.6x</b></td><td>10.18%</td><td>Fast non-autoregressive Chinese ASR with VAD/punctuation pipeline.</td></tr><tr><td><b>Fun-ASR-Nano</b></td><td>GPU</td><td>0.058803</td><td>17.0x</td><td>8.06%</td><td>中国語・英語・日本語、7つの中国語方言グループ、26の地域アクセントに対応する LLM-based ASR。hotword に対応。信頼できる checkpoint-native timestamp は未対応(<a href="https://github.com/FunAudioLLM/Fun-ASR/issues/106">#106</a>)。</td></tr><tr><td>GLM-ASR-Nano</td><td>GPU</td><td>0.026974</td><td>37.1x</td><td>31.07%</td><td>LLM-based multilingual ASR.</td></tr><tr><td>Whisper-large-v3-turbo (OpenAI)</td><td>GPU</td><td>0.021708</td><td>46.1x</td><td>21.71%</td><td>OpenAI Whisper implementation.</td></tr><tr><td>Whisper-large-v3 (OpenAI)</td><td>GPU</td><td>0.074694</td><td>13.4x</td><td>20.02%</td><td>ベースライン for large Whisper quality.</td></tr><tr><td><b>SenseVoice-Small</b></td><td>CPU</td><td>0.057988</td><td><b>17.2x</b></td><td>7.81%</td><td>CPU run from the remaining benchmark script.</td></tr><tr><td><b>Paraformer-Large</b></td><td>CPU</td><td>0.064056</td><td><b>15.6x</b></td><td>10.18%</td><td>CPU viable for batch jobs.</td></tr><tr><td>Fun-ASR-Nano</td><td>CPU</td><td>0.274318</td><td>3.6x</td><td>8.06%</td><td>LLM-based model is heavier but still above realtime.</td></tr></table>
3232
</section>
3333
<section id="method"><h2>測定方法</h2>
3434
<p>測定はワークスペース内の benchmark scripts を使い、184 音声ファイルで実施しました。RTF は <code>総推論時間 / 総音声時間</code>、速度は <code>1 / RTF</code> です。CER は SenseVoice タグなどモデル固有の出力を整理した後に計算しています。</p>

ko/benchmark.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ <h1>Benchmark</h1>
2828
<table><tr><th>항목</th><th>결과</th></tr><tr><td>Dataset</td><td>184개의 중국어 장문 오디오, 총 11,539초, 192.3분.</td></tr><tr><td>GPU</td><td>NVIDIA H100 80GB HBM3.</td></tr><tr><td>최고 GPU 속도</td><td>SenseVoice-Small: full benchmark에서 169.6x realtime, initial run에서 211.8x.</td></tr><tr><td>최고 CPU 속도</td><td>SenseVoice-Small: 17.2x realtime; Paraformer-Large: 15.6x realtime.</td></tr><tr><td>Baseline</td><td>OpenAI Whisper-large-v3: GPU에서 13.4x realtime.</td></tr></table>
2929
</section>
3030
<section id="table"><h2>결과</h2>
31-
<table><tr><th>Model</th><th>Device</th><th>RTF</th><th>Speed</th><th>CER</th><th>Notes</th></tr><tr><td><b>SenseVoice-Small</b></td><td>GPU</td><td>0.005896</td><td><b>169.6x</b></td><td>7.81%</td><td>ASR + language / emotion / event tags; tag 제거 후 CER 계산.</td></tr><tr><td><b>Paraformer-Large</b></td><td>GPU</td><td>0.008359</td><td><b>119.6x</b></td><td>10.18%</td><td>VAD/punctuation pipeline과 잘 맞는 빠른 non-autoregressive 중국어 ASR.</td></tr><tr><td><b>Fun-ASR-Nano</b></td><td>GPU</td><td>0.058803</td><td>17.0x</td><td>8.06%</td><td>중국어·영어·일본어, 7개 중국어 방언군, 26개 지역 억양을 지원하는 LLM-based ASR. timestamp와 hotword 지원.</td></tr><tr><td>GLM-ASR-Nano</td><td>GPU</td><td>0.026974</td><td>37.1x</td><td>31.07%</td><td>LLM-based multilingual ASR.</td></tr><tr><td>Whisper-large-v3-turbo (OpenAI)</td><td>GPU</td><td>0.021708</td><td>46.1x</td><td>21.71%</td><td>OpenAI Whisper implementation.</td></tr><tr><td>Whisper-large-v3 (OpenAI)</td><td>GPU</td><td>0.074694</td><td>13.4x</td><td>20.02%</td><td>large Whisper quality 기준 baseline.</td></tr><tr><td><b>SenseVoice-Small</b></td><td>CPU</td><td>0.057988</td><td><b>17.2x</b></td><td>7.81%</td><td>remaining benchmark script에서 수집한 CPU run.</td></tr><tr><td><b>Paraformer-Large</b></td><td>CPU</td><td>0.064056</td><td><b>15.6x</b></td><td>10.18%</td><td>CPU batch job에도 활용 가능.</td></tr><tr><td>Fun-ASR-Nano</td><td>CPU</td><td>0.274318</td><td>3.6x</td><td>8.06%</td><td>LLM-based model은 더 무겁지만 realtime보다 빠릅니다.</td></tr></table>
31+
<table><tr><th>Model</th><th>Device</th><th>RTF</th><th>Speed</th><th>CER</th><th>Notes</th></tr><tr><td><b>SenseVoice-Small</b></td><td>GPU</td><td>0.005896</td><td><b>169.6x</b></td><td>7.81%</td><td>ASR + language / emotion / event tags; tag 제거 후 CER 계산.</td></tr><tr><td><b>Paraformer-Large</b></td><td>GPU</td><td>0.008359</td><td><b>119.6x</b></td><td>10.18%</td><td>VAD/punctuation pipeline과 잘 맞는 빠른 non-autoregressive 중국어 ASR.</td></tr><tr><td><b>Fun-ASR-Nano</b></td><td>GPU</td><td>0.058803</td><td>17.0x</td><td>8.06%</td><td>중국어·영어·일본어, 7개 중국어 방언군, 26개 지역 억양을 지원하는 LLM-based ASR. hotword 지원. 신뢰할 수 있는 checkpoint-native timestamp는 미지원 (<a href="https://github.com/FunAudioLLM/Fun-ASR/issues/106">#106</a>).</td></tr><tr><td>GLM-ASR-Nano</td><td>GPU</td><td>0.026974</td><td>37.1x</td><td>31.07%</td><td>LLM-based multilingual ASR.</td></tr><tr><td>Whisper-large-v3-turbo (OpenAI)</td><td>GPU</td><td>0.021708</td><td>46.1x</td><td>21.71%</td><td>OpenAI Whisper implementation.</td></tr><tr><td>Whisper-large-v3 (OpenAI)</td><td>GPU</td><td>0.074694</td><td>13.4x</td><td>20.02%</td><td>large Whisper quality 기준 baseline.</td></tr><tr><td><b>SenseVoice-Small</b></td><td>CPU</td><td>0.057988</td><td><b>17.2x</b></td><td>7.81%</td><td>remaining benchmark script에서 수집한 CPU run.</td></tr><tr><td><b>Paraformer-Large</b></td><td>CPU</td><td>0.064056</td><td><b>15.6x</b></td><td>10.18%</td><td>CPU batch job에도 활용 가능.</td></tr><tr><td>Fun-ASR-Nano</td><td>CPU</td><td>0.274318</td><td>3.6x</td><td>8.06%</td><td>LLM-based model은 더 무겁지만 realtime보다 빠릅니다.</td></tr></table>
3232
</section>
3333
<section id="method"><h2>측정 방법</h2>
3434
<p>workspace의 benchmark script로 184개 오디오 파일에서 측정했습니다. RTF는 <code>total inference time / total audio duration</code>, speed는 <code>1 / RTF</code>입니다. CER는 SenseVoice tag처럼 model-specific output을 정리한 뒤 계산합니다.</p>

zh/benchmark.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ <h1>性能评测</h1>
2828
<table><tr><th>指标</th><th>结果</th></tr><tr><td>数据集</td><td>184 条中文长音频,总时长 11,539 秒,约 192.3 分钟。</td></tr><tr><td>GPU</td><td>NVIDIA H100 80GB HBM3.</td></tr><tr><td>最佳 GPU 速度</td><td>SenseVoice-Small: 169.6x realtime in the full benchmark, 211.8x in the initial run.</td></tr><tr><td>最佳 CPU 速度</td><td>SenseVoice-Small: 17.2x realtime; Paraformer-Large: 15.6x realtime.</td></tr><tr><td>基线</td><td>OpenAI Whisper-large-v3:GPU 上 13.4 倍实时。</td></tr></table>
2929
</section>
3030
<section id="table"><h2>结果</h2>
31-
<table><tr><th>模型</th><th>设备</th><th>RTF</th><th>速度</th><th>CER</th><th>说明</th></tr><tr><td><b>SenseVoice-Small</b></td><td>GPU</td><td>0.005896</td><td><b>169.6x</b></td><td>7.81%</td><td>ASR + 语种 / 情感 / 事件标签;CER 已去除标签后计算。</td></tr><tr><td><b>Paraformer-Large</b></td><td>GPU</td><td>0.008359</td><td><b>119.6x</b></td><td>10.18%</td><td>高速非自回归中文 ASR,适合 VAD/标点生产流水线。</td></tr><tr><td><b>Fun-ASR-Nano</b></td><td>GPU</td><td>0.058803</td><td>17.0x</td><td>8.06%</td><td>LLM-based 中/英/日 ASR,另覆盖 7 类中文方言和 26 种地域口音,支持时间戳和热词。</td></tr><tr><td>GLM-ASR-Nano</td><td>GPU</td><td>0.026974</td><td>37.1x</td><td>31.07%</td><td>LLM-based 多语种 ASR。</td></tr><tr><td>Whisper-large-v3-turbo (OpenAI)</td><td>GPU</td><td>0.021708</td><td>46.1x</td><td>21.71%</td><td>OpenAI Whisper 实现。</td></tr><tr><td>Whisper-large-v3 (OpenAI)</td><td>GPU</td><td>0.074694</td><td>13.4x</td><td>20.02%</td><td>基线 for large Whisper quality.</td></tr><tr><td><b>SenseVoice-Small</b></td><td>CPU</td><td>0.057988</td><td><b>17.2x</b></td><td>7.81%</td><td>CPU 结果来自 remaining benchmark 脚本。</td></tr><tr><td><b>Paraformer-Large</b></td><td>CPU</td><td>0.064056</td><td><b>15.6x</b></td><td>10.18%</td><td>CPU 上可用于批量任务。</td></tr><tr><td>Fun-ASR-Nano</td><td>CPU</td><td>0.274318</td><td>3.6x</td><td>8.06%</td><td>LLM-based 模型更重,但仍高于实时。</td></tr></table>
31+
<table><tr><th>模型</th><th>设备</th><th>RTF</th><th>速度</th><th>CER</th><th>说明</th></tr><tr><td><b>SenseVoice-Small</b></td><td>GPU</td><td>0.005896</td><td><b>169.6x</b></td><td>7.81%</td><td>ASR + 语种 / 情感 / 事件标签;CER 已去除标签后计算。</td></tr><tr><td><b>Paraformer-Large</b></td><td>GPU</td><td>0.008359</td><td><b>119.6x</b></td><td>10.18%</td><td>高速非自回归中文 ASR,适合 VAD/标点生产流水线。</td></tr><tr><td><b>Fun-ASR-Nano</b></td><td>GPU</td><td>0.058803</td><td>17.0x</td><td>8.06%</td><td>LLM-based 中/英/日 ASR,另覆盖 7 类中文方言和 26 种地域口音,支持热词;不提供可靠的 checkpoint 原生时间戳(<a href="https://github.com/FunAudioLLM/Fun-ASR/issues/106">#106</a>)。</td></tr><tr><td>GLM-ASR-Nano</td><td>GPU</td><td>0.026974</td><td>37.1x</td><td>31.07%</td><td>LLM-based 多语种 ASR。</td></tr><tr><td>Whisper-large-v3-turbo (OpenAI)</td><td>GPU</td><td>0.021708</td><td>46.1x</td><td>21.71%</td><td>OpenAI Whisper 实现。</td></tr><tr><td>Whisper-large-v3 (OpenAI)</td><td>GPU</td><td>0.074694</td><td>13.4x</td><td>20.02%</td><td>基线 for large Whisper quality.</td></tr><tr><td><b>SenseVoice-Small</b></td><td>CPU</td><td>0.057988</td><td><b>17.2x</b></td><td>7.81%</td><td>CPU 结果来自 remaining benchmark 脚本。</td></tr><tr><td><b>Paraformer-Large</b></td><td>CPU</td><td>0.064056</td><td><b>15.6x</b></td><td>10.18%</td><td>CPU 上可用于批量任务。</td></tr><tr><td>Fun-ASR-Nano</td><td>CPU</td><td>0.274318</td><td>3.6x</td><td>8.06%</td><td>LLM-based 模型更重,但仍高于实时。</td></tr></table>
3232
</section>
3333
<section id="method"><h2>评测方法</h2>
3434
<p>评测使用工作区中的 benchmark 脚本,在 184 条音频上收集结果。RTF 为 <code>总推理时间 / 总音频时长</code>,速度为 <code>1 / RTF</code>。CER 会先做模型相关文本清理,尤其会去除 SenseVoice 标签。</p>

0 commit comments

Comments
 (0)