Delays when transcribing streaming audio

[scott-vector]

First of all, excellent work. Vosk is great as it is, and this library makes it even better.

I am experiencing a heavy delay on transcription when pulling in a stream from webRTC (partials and fulls).

I suspect maybe it is because of the deprecated "createScriptProcessor" and "onaudioprocess" pieces, but I am unsure.

Here is how I am processing things. If you have any ideas as to why things would be delayed, please let me know. Thank you.

this.recognizeSpeech = async () => {
    console.log("starting recognizeSpeech");
    let audioContext = this.remoteAudioContext;
    let remoteStream = this.incomingAudioStream;
    //
    const recognizerNode = audioContext.createScriptProcessor(4096, 1, 1);
    const model = await createModel("./softphone/model.tar.gz");
    const recognizer = new model.KaldiRecognizer(48000);
    recognizer.setWords(true);
    recognizer.on("partialresult", function (message) {
      console.log("PARTIAL: " + message.result.partial);
    });
    recognizerNode.onaudioprocess = async (event) => {
      try {
        recognizer.acceptWaveform(event.inputBuffer);
      } catch (error) {
        console.error("acceptWaveform failed", error);
      }
    };
    this.remoteTrack.connect(recognizerNode).connect(audioContext.destination);
  };

[ccoreilly]

Hi @scott-vector, interesting use case!
Please take into account that transcription speed depends on the CPU of the user as it is running locally. I have experienced low speed on an Android phone vs a Macbook.
You might try using AudioWorklets to see if it helps. Otherwise it'd be interesting to analyze where the latencies are coming from.

[scott-vector]

Yeah I was going to convert to that style. Do you have any interest in that?

…

On Sat, Nov 12, 2022 at 3:38 AM ccoreilly ***@***.***> wrote: Hi @scott-vector <https://github.com/scott-vector>, interesting use case! Please take into account that transcription speed depends on the CPU of the user as it is running locally. I have experienced low speed on an Android phone vs a Macbook. You might try using AudioWorklets <https://github.com/ccoreilly/vosk-browser/tree/master/examples/modern-vanilla> to see if it helps. Otherwise it'd be interesting to analyze where the latencies are coming from. — Reply to this email directly, view it on GitHub <#58 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASOV5EQPL7KQCQWTSMOGR6TWH5JRFANCNFSM6AAAAAAR4AHBSM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ccoreilly]

Generally speaking, yes, it' be nice to integrate AudioWorklets seamlessly in the library but allow developers to use the underlying API directly.

[korabelnikov]

@scott-vector Hey, have you managed to fix delays?