pocketsphinx_batch scores depend on control-file order when noise removal is enabled

[lenzo-ka]

User-visible symptom

When pocketsphinx_batch decodes multiple independent recordings with spectral-subtraction noise removal enabled, an utterance's acoustic score can change according to which recordings precede it in the control file. This makes batch output dependent on input order even though each entry names a separate audio file.

Using five audio files shipped in PocketSphinx's Librivox test data, reversing the control file changed the reported score for all five utterances in a v5.1.1 build. The word hypotheses happened to remain identical for those fixtures. Hypothesis changes have been observed through a separate API-based multi-utterance caller, but have not been reproduced in pocketsphinx_batch with the public fixtures used here.

Mechanism in v5.1.1

These references are to commit 511126b492dcb267cf30d49d631946d7b61a9530 (v5.1.1).

• programs/pocketsphinx_batch.c:425-492 processes each control-file entry with one shared decoder. For raw audio it calls ps_decode_raw() at line 472; there is no ps_start_stream() call.
• src/pocketsphinx.c:1030-1069 shows that ps_decode_raw() calls ps_start_utt() and ps_end_utt() around each file.
• src/pocketsphinx.c:1091-1122 shows what ps_start_utt() resets before calling acmod_start_utt(); it does not reset front-end noise statistics.
• include/pocketsphinx.h:769-780 states that noise-removal statistics are retained across utterances and that ps_start_stream() resets them.
• programs/pocketsphinx_batch.c:756-887 iterates the control file using the same decoder instance, so retained statistics from earlier entries become input state for later entries.

Retaining a stream-level estimate is reasonable for segmented continuous audio. The problem here is that batch entries can also be independent recordings, and the tool provides no per-entry stream reset or selectable policy.

Minimal reproduction

Build v5.1.1. Convert the five shipped Librivox WAV fixtures to 16 kHz, mono, signed 16-bit little-endian raw audio; for example with FFmpeg:

mkdir -p /tmp/ps-libri-raw
for f in test/data/librivox/*.wav; do
  ffmpeg -loglevel error -y -i "$f" -ar 16000 -ac 1 -f s16le \
    "/tmp/ps-libri-raw/$(basename "${f%.wav}").raw"
done

Create /tmp/forward.ctl:

sense_and_sensibility_01_austen_64kb-0870
sense_and_sensibility_01_austen_64kb-0880
sense_and_sensibility_01_austen_64kb-0890
sense_and_sensibility_01_austen_64kb-0920
sense_and_sensibility_01_austen_64kb-0930

Create /tmp/reverse.ctl with those five lines reversed. Run both orders:

pocketsphinx_batch \
  -hmm model/en-us/en-us \
  -dict model/en-us/cmudict-en-us.dict \
  -lm model/en-us/en-us.lm.bin \
  -ctl /tmp/forward.ctl -cepdir /tmp/ps-libri-raw -cepext .raw \
  -adcin yes -hyp /tmp/forward.hyp -loglevel WARN

pocketsphinx_batch \
  -hmm model/en-us/en-us \
  -dict model/en-us/cmudict-en-us.dict \
  -lm model/en-us/en-us.lm.bin \
  -ctl /tmp/reverse.ctl -cepdir /tmp/ps-libri-raw -cepext .raw \
  -adcin yes -hyp /tmp/reverse.hyp -loglevel WARN

Match rows by utterance ID rather than line number. In one build of the pinned commit, the score pairs for forward versus reverse order were:

0870  -30531  -30435
0880  -11225  -11374
0890  -22486  -22245
0920  -23555  -23566
0930  -13812  -13809

All five hypothesis strings were identical between these two runs. Exact scores may vary with build configuration, but the comparison is made between two orderings of the same executable and fixtures.

What this does and does not establish

The shared decoder, per-file ps_start_utt() path, absence of ps_start_stream(), and documented cross-utterance retention are established directly in pinned source. The score changes above were measured in pocketsphinx_batch built from that source using shipped PocketSphinx audio.

The broader finding originated in application usage, not in PocketSphinx's test suite. In that separate usage path, reversing 55 independent utterances changed 6 hypothesis strings, and resetting the stream before each utterance eliminated order differences. Those measurements establish that retained front-end state can reach hypotheses in that caller, but they do not establish the same magnitude—or any hypothesis change—in pocketsphinx_batch.

This report does not establish a WER change on the five public fixtures, does not quantify corpus-level impact, and does not claim that partition-count behavior was directly reproduced. Partitioning a control file can change predecessor state at partition boundaries, but its practical effect remains an inference until measured through the relevant partitioning workflow.

It is also not claimed that retaining noise state is wrong for a continuous recording split into segments. The ambiguity is that pocketsphinx_batch has no way to distinguish that case from independent files.

Smallest addressing change

For batches of independent recordings, call ps_start_stream() before decoding each raw-audio control entry when noise removal is enabled. A regression test can decode at least two independent fixtures in both orders and compare per-utterance outputs.

Whether this should be the unconditional batch default or an option is a design choice: preserving current behavior is useful for segmented continuous audio, while resetting is appropriate for independent recordings. The smallest compatibility-preserving change would add an explicit option and document both modes.

[akamensh]

pocketsphinx_batch allows to get the desirable result without any changes to the sources, with placing independent recordings to separate batches (control files), which has some advantages over the way introduced in #495:

• processing the recordings can be then parallelised on multi-threaded machines via separate OS processes - while single-threaded pocketsphinx_batch would process the batch sequentially;
• no need to introduce yet another (unnecessary) argument, on top of those 140 which are already there.

If there is a preference for #495, it would be nice to guarantee identity of results compared to the case of split control files - otherwise, possible discrepancies would be counter-intuitive an confusing for users.

[lenzo-ka]

it is then order-dependent within batch files though? I don't think breaking it into batches fixes it unless they are batch size 1. with the stream-reset it does guarantee order independence. I am sympathetic to not adding more config options... I would think the best approach would be to make stream-reset standard, to be honest.

[akamensh]

Yes, results for an entry within a control file generally depend on previous entries there. And indeed, a single entry per batch if recordings are to be processed independently. The ASR systems of Sphinx's type change their state while working, for instance with extracting corrections for convolutional effects from the data processed in the past, which appears as the "dependence on order" observed by you while playing with a control file. With that in mind, the control file shouldn't be taken as a naive list of independent recording to process, but rather a set of coherent data with same audio characteristics: e.g. if the recordings were performed with different microphones, there is very little sense in having them together in a batch. Probably, documentation could be more clear about it... So there are the next schemas of precessing data:

A. Consciously grouping recordings in batches; it can be useful for some researches and checks, but implies good knowledge of the system.
B. Having independent processing with 1 recording per control file (and gaining benefits mentioned in #498 (comment)).

What you suggest with #495 ("C") seems trying to reach the same as the schema "B", but doesn't guarantee the exact same results. It would be good to avoid that variance.

[lenzo-ka]

I'm am not convinced that we need to be concerned about the noise removal being deterministic (as opposed to holding hysteresis, as is the case of the bug) over deliberately ordered files. The reverse, we do, we don't want to have unimportant changes making a difference in our results.

[dhdaines]

Missed the discussion on this but as I mention in #495 I think that we should just unconditionally reset the statistics since it doesn't make much sense for a batch process to not treat its inputs independently.

In theory, we also default to "live" as the CMN mode which is maybe even worse than this in terms of order-dependent effects, but this actually gets overriden nearly all the time. Unfortunately the way this happens is really suboptimal - while it should be specific to batch vs. live mode operation and thus, probably, set as a default in pocketsphinx_batch, we're not doing that, instead it's set as a default in feat.params for a specific acoustic model.

Anyway I think we should stop doing that sort of thing, and just make batch mode do batch mode things. The only question I have is, if we reset the noise removal statistics, does this make noise removal essentially useless? Or can we compute them ahead of time in batch mode (similarly to how we do CMN)?

[lenzo-ka]

no it doesn't make the noise removal useless, it still applies but within file. what we get though is identical artifacts (deterministic outputs) irrespective of how batches are organized, which is a big win. if we have 100 files, we should get the same results whether it's 1 big batch or 10 parallel ones.

[akamensh]

As per the description of #495, enabled dithering will still make the results position-dependent and hence non-deterministic, won't it? Other than that, sure - if 1 batch of N files would produce the exact same results as N batches of 1 file (or any other breakdown) - regardless of other options applied - that would be the best.

[lenzo-ka]

You're right about the position-dependence -- adding a seed reset to the PR