Speech to text: Repeat lines in output with whisper-large for Dutch

[hgoelen]

Hello,

Using whisper-large as a model, I noticed that sometimes the last (part of the last) line of an audio file gets repeated. As a fictional English example, the following actual audio:

I try to go for a run three times a week. However, sometimes I skip a day.

... would yield this output:

I try to go for a run three times a week. However, sometimes I skip a day. Sometimes I skip a day.

The code looks like this:

with open(filename, 'rb') as file:
    buffer_data = file.read()

payload: FileSource = {
    'buffer': buffer_data,
}

options = PrerecordedOptions(
    model='whisper-large',
    smart_format=True,
    language='nl'
)

response = deepgram.listen.rest.v('1').transcribe_file(payload, options, timeout=httpx.Timeout(300.0, connect=10.0))
response.to_dict()['results']['channels'][0]['alternatives'][0]['transcript']

Does anyone have an idea why this might be the case? The file I am looking at has a duration of 17 seconds.

Thanks in advance

[team-deepgram]

Thanks for asking your question about Deepgram! If you didn't already include it in your post, please be sure to add as much detail as possible so we can assist you efficiently, such as:

• The request_id if you have a question about your requests or transcription responses.
• The features you used or the full api.deepgram.com URL you sent your request to, including parameters.
• Any code snippets you can share.

[jkroll-deepgram]

Hi @hgoelen, we host the open-source Whisper model, but we don't maintain or alter the model.

Have you tried our in-house Nova-2 Dutch model? That would be model=nova-2 with your same language parameter language=nl.

[hgoelen]

Hello, I have tried the nova-2 model as well and it doesn not yield these duplicate outputs. However, in calling nova-2, I don't provide the timeout argument to the call, so I was thinking maybe that would influence the output. If not, then it is purely because of whisper.

[jkroll-deepgram]

Hi @hgoelen, the timeout argument should not affect the model results at all. It would just affect whether you receive a valid/200 response or a timeout error.

Glad to hear the issue does not persist on Nova-2!

[hgoelen]

Alright, thanks for checking in and clarifying.