Diarization Beta Testing

[incubatize]

For capturing feedback on testing Deepgram's Diarization Beta.

We would like to invite you to test our latest Speaker Diarization in beta and share your valuable feedback with us.

To access the beta Speaker Diarization feature, please use the following beta endpoint:

https://api.beta.deepgram.com/v1/listen?diarize=true

Please share your thoughts on speaker labeling accuracy, speed, and anything else you'd like to note!

[ujlm]

I tried it with a Dutch podcast and an interview, and these are my findings:

• Accuracy: disappointing, unfortunately: it starts out with speaker 0, correctly detects the first switch to speaker 1, but then labels everything after it as speaker 1
• Speed: fast, but I wouldn't mind it being slower if it produced a more accurate results
• The regular API also returns different diarizations for different models. It sometimes hallucinates a third speaker when there are only 2. In general, the labeling is not working well for my Dutch audio files

[incubatize]

thanks @ujlm - appreciate the detailed feedback. Did you use our enhanced or base model?
We've also released this in GA yesterday. Do you see the same result with our GA endpoint as well? https://api.deepgram.com/v1/listen?diarize=true?

[ujlm]

I used the Whisper and Nova base models. Is diarization performance expected to be better for the enhanced models? I tested the files with the GA endpoint and got the same results. Sometimes it works quite well, sometimes it hallucinates additional speakers, randomly assigns labels to audio fragments, or doesn't recognize different speakers at all.

[incubatize]

@ujlm -Is it possible to share the audio file? It'd be helpful for us to review it internally.

[ujlm]

Sure, this file for example seems hard to diarize: https://d2xk4y16cry3iz.cloudfront.net/uploads/6410ac807ea9c7de03689947/645ce0a9311c6f8aac994abf/henry_full_audio_only.mp3

[ujlm]

Another example: The Joe Rogan x Elon Musk podcast: https://d2xk4y16cry3iz.cloudfront.net/uploads/6410ac807ea9c7de03689947/645ce7985bdc3afc35807b6b/Joe_Rogan_Experience_1470_Elon_Musk_RcYjXbSJBN8.mp3

[marscrusader]

Hi I just tested the both the beta(api.beta.deepgram.com) and production endpoint with multiple different audio files:

curl --request POST \
  --url 'https://api.deepgram.com/v1/listen?model=whisper-large&diarize=true&punctuate=true' \
  --header 'Authorization: Token {TOKEN}' \
  --header 'Content-Type: application/json' \
  --data '{"url": "https://voiceage.com/wbsamples/in_mono/Conference.wav"}'

I used both nova and whisper models and both were unfortunately disappointing. The endpoint is unable to identify different speakers and keeps returning "speaker": 0 for all audio files.
Additionally, the nova model is inaccurate compared to whisper. You can try it out using the curl request above to see the difference.