Original-Silero vs Faster-Whisper "Variant"

[nilslacroix]

There is an big difference in the performance of the implementation of faster-whisper silero and this repo, somestimes up to 10% in downstream tasks such as ASR. I wonder how this can be, I look throughly into this but I could not figure out the reason for this behaviour.

Background:
I did some extensive ASR benchmarking on multiple models/datasets and I used Silero to handle all audio files before they went into the ASR model. Basically for two reasons: 1. Handle the few outliers which are too long 2. Inference pipelines require segmentation, it makes sense to test it end-to-end.

I mostly tested on arabic languages for this benchmark, datasets included: MASC, FLEURS, MGB2,3 and 5, SADA22, Casablanca and more.

In general, models performed better without any VAD - that can be expected to some degree. Why? These datasets are relatively clean, even though i observed some cases where laughter for example was present in the audio, and the transcription was better with VAD. However in general it was 5-10% better across all models and datasets without Silero VAD.

Now comes the catch:
I also tested the faster whisper implementation (v1.2.1), which also uses Silero for segmentation, but its own onnx/batch variant. So i made some experiments faster-whisper silero vad vs original silero vad, where I fed the clip_timestamps from original silero vad into faster whisper and disabled their implementation. And for some reason the faster whisper silero implementation was always around 5% better than the original implementation here. I made sure and double checked that all vad_parameters were the same and looked through the whole code, but there was no real difference.

The only thing I found was, that they append 64 context samples in the vad.py:

 batched_audio = audio.reshape(-1, num_samples)
        context = batched_audio[..., -context_size_samples:]
        context[-1] = 0
        context = np.roll(context, 1, 0)
        batched_audio = np.concatenate([context, batched_audio], 1)

in order to use batching, but beside that there was no real difference.

So I wonder ... how can results be so different, given they use the same model and the same code (even though its onnx). I also checked the numpy array, but they were identicial. The resulting speech_probs were slightly different, timestamps within 30-1000 samples of each other in both implementations.

[snakers4]

What caught my attention in your description was:

The only thing I found was, that they append 64 context samples in the vad.py:
in order to use batching, but beside that there was no real difference.

In one of latest versions of the VAD this context padding was added.
Maybe you are comparing an older model, that had no such feature to a newer one, which has?

[AliOsm]

I investigated this while preparing draft PR #784, which adds a bounded temporal-sequence ONNX path to this repository.

The controlled same-version result is important here: using the packaged Silero v6.2.1 frame graph and sequence graphs reconstructed from exactly the same v6.2.1 weights, CPU ONNX Runtime probabilities and final hidden/cell states were array-exact at both 8 kHz and 16 kHz. On the 69m20.679s Arabic benchmark recording, both paths produced the same 378 speech spans and identical serialized timestamps.

This supports @snakers4's version/preprocessing suspicion: the sequence optimization itself does not explain a WER or timestamp difference when model weights, context construction, padding, and timestamp options are held constant. faster-whisper 1.2.1 predates its v6.2 graph update, which landed later in faster-whisper #1390, so that earlier comparison was not necessarily between the same model release.

The PR also covers partial-frame edges, exact block boundaries, nonzero recurrent states, and every public timestamp option through Silero's existing canonical state machine. Its detailed parity evidence may help separate graph execution from the remaining version and preprocessing variables in this discussion.