Live Transcription from Twilio Voice Stream (Nodejs) - Please help debug

[kaliq-i]

I am struggling with getting a transcription from Deepgram, I am attempting to stream raw audio from Twilio. It is encoded in base64. I can't get any transcription and seem to get an unusual error which I am struggling to debug. Any help would be appreciated and if there is an example of live streaming Twilio with a nodejs client - do point it out please! Only found this python one

I have managed to successfully live transcribe twilio with Google STT following the deepgram example provided here as well as transcribing BBC radio also with Google STT as shown here so I am not really sure whats up.

I have double checked the following:

1. There are no websocket related issues. Twilio is definitely streaming the encoded audio.
2. There are definitely no authentication issues with the deepgram client
3. Deepgram send is definitely getting buffers and sending them

This is what ends up being outputted into the console after running the code. The stream is passing data for like 5-10 seconds with no output to the console and then "here" / the object is outputted. I then manually end the call to trigger "Call Has Ended".

WebSocket connection established
A new call has connected.
Starting Media Stream MZa6449abd429fb58034209cd2393461f9
here
{"transaction_key":"deprecated","request_id":"4a7ee77d-3aa0-48ef-81d4-98f4e76d9a49","sha256":"8a819a3063b0d4a1b7f631931756f65bdfa752097d4834059bdfaf9df8043543","created":"2023-04-16T04:51:08.768Z","duration":0.0,"channels":0}
Call Has Ended
WebSocket connection closed

Here is my Websocket Handler in Nodejs

import Websocket from "ws";
import { Logger } from "../../core/logger";
import http from "http";
import { Server, WebSocketServer } from "ws";
import { PassThrough, Readable } from "stream";
import { transcriber } from "../transcriber";
import { Buffer } from "buffer";
export interface Dependencies {
  logger: Logger;
  server: http.Server;
}

export const createWebSocketServer = ({
  logger,
  server,
}: Dependencies): WebSocketServer => {
 
  const wss = new Server({ server });

  // Set up WebSocket connection event handling
  wss.on("connection", (ws) => {
    console.log("WebSocket connection established");

    const deepgramLive = transcriber.transcription.live({
      punctuate: true,
      endpointing: true,
      language: "en-GB",
    });

    // create stream
    const stream = new PassThrough();

    stream.on("data", (chunk) => {
      if (deepgramLive.getReadyState() == 1) {
        deepgramLive.send(chunk);
      }
    });

    ws.on("message", function incoming(message: any) {
      const msg = JSON.parse(message);
      switch (msg.event) {
        case "connected":
          console.log(`A new call has connected.`);

          break;
        case "start":
          console.log(`Starting Media Stream ${msg.streamSid}`);
          break;
        case "media":
          const buffer = Buffer.from(msg.media.payload, "base64");
          stream.write(buffer);
          break;
        case "stop":
          console.log(`Call Has Ended`);
          deepgramLive.finish();
          break;
      }
    });

    deepgramLive.addListener("transcriptReceived", (transcription) => {
      console.log("here");
      console.log(transcription);
    });

    ws.on("close", () => {
      console.log("WebSocket connection closed");
    });
  });

  return wss;
};

For those who are familiar with Twilio, Here is my Twiml Bin For reference.

<?xml version="1.0" encoding="UTF-8"?>
<Response> 
  <Start>
 <Stream url="wss://0e21-92-1-185-12.eu.ngrok.io/"/>
  </Start>
   <Say>Hello, this is an introductory message. Please share your response.</Say>
  <Pause length='60' />
</Response>```

[swiecki]

Have you checked to make sure the encoding types match what the APIs want? Twilio spits out audio/x-mulaw at a sample rate of 8000 hz. Deepgram can accept that but you need to specify:

const deepgramLive = transcriber.transcription.live({
      punctuate: true,
      endpointing: true,
      language: "en-GB",
      encoding: "mulaw",
      sample_rate: 8000,
    });

My understanding is Twilio's audio will be audio/x-mulaw once decoded from base64. If that's the case then you may need to decode the strings before passing them with the above encoding.

[kaliq-i]

Ah perfect, works now! Thanks for your help 🙂.

[SandraRodgers]

Hi @kaliq-i ,

I'm so glad that @swiecki was able to help you!

I want to link to a resource in our docs we have for others who might encounter this issue:

https://developers.deepgram.com/documentation/guides/determine-audio-format/

In the Deepgram blog Python example you referenced, there is a line of code when making the request to Deepgram:

deepgram_ws = websockets.connect('wss://api.deepgram.com/v1/listen?encoding=mulaw&sample_rate=8000&channels=2&multichannel=true', extra_headers = extra_headers)

You can see in that request the encoding=mulaw&sample_rate=8000. So @swiecki was exactly right that this information is what was missing from your request. In node, you just add it as a property in the request object.

Here's the moral of the story:

When streaming raw audio to Deepgram, be sure to send the encoding and the sample rate.

Sandra