The websocket works but no transcriptions are returned after it is reestablished

[benripka]

Hi there! The below code works great on the first startRecording() call, but after calling stopRecording() and then starting it again with startRecording() the websocket is opened back up and valid audio data appears to be getting streamed to Deepgram, however Deepgram returns nothing back... then after a few second Deepgram closes the conection with:

{ "type": "Metadata", "transaction_key": "deprecated", "request_id": "f72438cc-681c-4443-abab-03bbf00384a9", "sha256": "01ebbdc099da1562e201779dd86614d08df898bc9ab5529e4bed88658a0733d2", "created": "2024-01-23T02:21:49.928Z", "duration": 0, "channels": 0 }

I am connecting to the base url: wss://api.deepgram.com/v1/listen

class AudioStreamer {
  private mediaRecorder: MediaRecorder | null = null;
  private stream: MediaStream | null = null;
  private websocket: WebSocket | null = null;
  private readonly timeslice: number = 200; // Time slice for audio chunks in milliseconds
  private buffer: Blob[] = [];
  private websocketUrl: string;

  constructor(websocketUrl: string) {
    this.websocketUrl = websocketUrl;
  }

  private connectWebSocket(): void {
    this.websocket = new WebSocket(this.websocketUrl, [
      'token',
      DEEPGRAM_API_KEY,
    ]);

    this.websocket.onopen = () => {
      console.log('WebSocket connected.');
      this.flushBuffer();
      if (this.mediaRecorder) {
        this.mediaRecorder.addEventListener(
          'dataavailable',
          (event: BlobEvent) => {
            if (
              this.websocket &&
              this.websocket.readyState === WebSocket.OPEN
            ) {
              console.log(event.data);
              this.websocket.send(event.data);
            } else {
              this.buffer.push(event.data);
            }
          },
        );
        this.mediaRecorder.start(this.timeslice);
      }
    };

    this.websocket.onclose = () => {
      console.log('WebSocket closed.');
    };

    this.websocket.onerror = (error: Event) => {
      console.error('WebSocket error:', error);
    };
  }

  private flushBuffer(): void {
    while (
      this.buffer.length > 0 &&
      this.websocket &&
      this.websocket.readyState === WebSocket.OPEN
    ) {
      const data = this.buffer.shift();
      if (data) {
        this.websocket.send(data);
      }
    }
  }

  public async startRecording(): Promise<void> {
    try {
      this.stream = await navigator.mediaDevices.getUserMedia({ audio: true });
      this.mediaRecorder = new MediaRecorder(this.stream, {
        mimeType: 'audio/webm',
      });
      this.connectWebSocket();
    } catch (error) {
      console.error('Error starting recording:', error);
    }
  }

  public stopRecording(): void {
    if (!this.mediaRecorder || !this.stream) {
      console.warn('Recording has not been started or already stopped.');
      return;
    }

    this.mediaRecorder.stop();
    this.stream.getTracks().forEach((track) => track.stop());
    this.mediaRecorder = null;
    this.stream = null;
    console.log('Recording stopped.');

    if (this.websocket) {
      this.websocket.send(JSON.stringify({ type: 'CloseStream' }));
      this.websocket.close();
      this.websocket = null;
    }
  }
}

export default AudioStreamer;```

Does anyone know what might be happening? I believe after some time startRecording() will work again... might be some sort of  timeout that transcends the  actual socket connection? All thoughts are welcome.

[team-deepgram]

Thanks for asking your question about Deepgram! If you didn't already include it in your post, please be sure to add as much detail as possible so we can assist you efficiently, such as:

• The request_id if you have a question about your requests or transcription responses.
• The features you used or the full api.deepgram.com URL you sent your request to, including parameters.
• Any code snippets you can share.

[benripka]

Ah, I sorted it out for the most part. Looks like I was very "ungracefully" closing the WebSocket and Deepgram didn't like that. Graceful closure as shown in the docs and code below seems to work.

I'm curious though, does anyone know why forcing the WebSocket to close isn't good enough? It's surprising to me that the Deepgram API keeps some state about my webapp instance and "knows it's me" when I reconnect after destroying socket. Is this a browser thing? Interestingly there's no issue when I close and reopen the tab... I guess Chrome handles the WebSocket closure gracefully as well? Or can deepgram just not "tell that it's me" in that case?

Or just MAYBE I just need to go and learn how websockets actually work lol Anyways here's the updated code. Thoughts always welcome:

class AudioStreamer {
  private mediaRecorder: MediaRecorder | null = null;
  private stream: MediaStream | null = null;
  private websocket: WebSocket | null = null;
  private readonly timeslice: number = 200; // Time slice for audio chunks in milliseconds
  private buffer: Blob[] = [];
  private websocketUrl: string;
  private keepAliveInterval: NodeJS.Timeout | null = null;

  constructor(websocketUrl: string) {
    this.websocketUrl = websocketUrl;
  }

  private connectWebSocket(): void {
    this.websocket = new WebSocket(this.websocketUrl, [
      'token',
      DEEPGRAM_API_KEY,
    ]);

    this.websocket.onopen = () => {
      console.log('WebSocket connected.');
      this.flushBuffer();
      if (this.mediaRecorder) {
        this.mediaRecorder.addEventListener(
          'dataavailable',
          (event: BlobEvent) => {
            if (
              this.websocket &&
              this.websocket.readyState === WebSocket.OPEN
            ) {
              console.log(event.data);
              this.websocket.send(event.data);
            } else {
              this.buffer.push(event.data);
            }
          },
        );
        this.mediaRecorder.start(this.timeslice);
      }

      if (this.keepAliveInterval) {
        clearInterval(this.keepAliveInterval);
      }
      this.keepAliveInterval = setInterval(() => {
        if (this.websocket) {
          this.websocket.send(JSON.stringify({ type: 'KeepAlive' }));
        }
      }, 8000);
    };

    this.websocket.onclose = () => {
      console.log('WebSocket closed.');
      if (this.keepAliveInterval) {
        clearInterval(this.keepAliveInterval);
      }
      this.websocket = null;
    };

    this.websocket.onerror = (error: Event) => {
      console.error('WebSocket error:', error);
    };
  }

  private flushBuffer(): void {
    while (
      this.buffer.length > 0 &&
      this.websocket &&
      this.websocket.readyState === WebSocket.OPEN
    ) {
      const data = this.buffer.shift();
      if (data) {
        this.websocket.send(data);
      }
    }
  }

  public async startRecording(): Promise<void> {
    try {
      this.stream = await navigator.mediaDevices.getUserMedia({ audio: true });
      this.mediaRecorder = new MediaRecorder(this.stream, {
        mimeType: 'audio/webm',
      });
      this.connectWebSocket();
    } catch (error) {
      console.error('Error starting recording:', error);
    }
  }

  public stopRecording(): void {
    if (!this.mediaRecorder || !this.stream) {
      console.warn('Recording has not been started or already stopped.');
      return;
    }

    this.mediaRecorder.stop();
    this.stream.getTracks().forEach((track) => track.stop());
    this.mediaRecorder = null;
    this.stream = null;

    if (this.websocket) {
      this.websocket.send(JSON.stringify({ type: 'CloseStream' }));
    }
  }
}

export default AudioStreamer;

[jkroll-deepgram]

Hi @benripka, thanks for sharing your code, glad you were able to work through this for the most part.

Were you seeing any error messages on reconnection, or rather it seemed to be connecting properly, but you were getting a blank transcript?

I see the request ID f72438cc-681c-4443-abab-03bbf00384a9 that you shared from a reconnection attempt. Do you have any request IDs to share from the initial connection that worked well, before the websocket closure and then reconnection?

[weiwarren]

I have similar issue after reconnecting to deepgram. I have tried everything to terminate the existing connections using .finalize(), .finish() but nothing seems work. Get the metadata response back, but no new transcription message.

[gergomiklos]

There are similar issues here that slow down the implementation of any integration.