A method to retrieve finished transcripts via request_ids after call backs?

[jaxomlotus]

I am currently on a serverless setup and trying to process multiple hours of transcripts. The problem is that using Callbacks I will sometimes run into 413 errors (payload too large) and can't retrieve my transcripts from deepgram.

I would love a method to use the request_id that deepgram provides for me to query deepgram to retrieve past transcripts.

I could then just use the callback to alert my app that the transcript is ready, but not run into issues where the data posted to the callback url is too large and errors out.

Is there something like this already in place? Or will it be enabled? Thanks in advance?

[jcdyer]

At present, deepgram doesn't store any transcripts locally. You have a few options:

1. Ensure that your callback receivers can accept post bodies large enough for the transcripts of the audio you expect to send to it.
2. Split your audio into chunks that you expect to produce transcripts that will fit your callbacks.
3. I don't remember the details on this, but I've heard about customers configuring their callbacks to PUT the transcript to an s3 bucket. With that sort of setup, you can listen for changes on that s3 bucket to kick off other parts of your serverless workflow. That of course assumes you are on aws. If you're using a different provider, there may be a similar flow that works with your provider's object/blob storage service.

[jaxomlotus]

Thanks for responding. That last option could work for me (I’m on AWS). Do you remember where relevant threads on this might be?

…

On Thu, May 25, 2023 at 2:06 PM Cliff Dyer ***@***.***> wrote: At present, deepgram doesn't store any transcripts locally. You have a few options: 1. Ensure that your callback receivers can accept post bodies large enough for the transcripts of the audio you expect to send to it. 2. Split your audio into chunks that you expect to produce transcripts that will fit your callbacks. 3. I don't remember the details on this, but I've heard about customers configuring their callbacks to PUT the transcript to an s3 bucket. With that sort of setup, you can listen for changes on that s3 bucket to kick off other parts of your serverless workflow. That of course assumes you are on aws. If you're using a different provider, there may be a similar flow that works with your provider's object/blob storage service. — Reply to this email directly, view it on GitHub <#162 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADWSSSDML7LNV5KQEGKQ53XH6NUFANCNFSM6AAAAAAYPDNW74> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jcdyer]

I don't see a thread on the issue, but in short, you would generate a presigned s3 url to put a new object as described at https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html, and then pass that url in a callback={url}&callback_method=put request to the deepgram api.

[jcdyer]

@jaxomlotus :

This works as a proof of concept lambda:

Whenever an audio file is uploaded to AUDIO_BUCKET, a deepgram request is made with a presigned url pointing to that object, and a PUT callback pointing to a matching filename in the TRANSCRIPT_BUCKET.

import boto3
import json
import os
from base64 import b64decode
from urllib.request import Request, urlopen
from urllib.parse import urlencode

DEEPGRAM_API_KEY = os.environ.get("DEEPGRAM_API_KEY")
AUDIO_BUCKET = os.environ["AUDIO_BUCKET"]
TRANSCRIPT_BUCKET = os.environ["TRANSCRIPT_BUCKET"]

def lambda_handler(event, context):
    if not DEEPGRAM_API_KEY:
        return {
            "statusCode": 401,
            "body": "no DEEPGRAM_API_KEY provided",
        }

    print(context)

    object_name = ''
    for record in event['Records']:
        print(record)
        if record['eventName'] == 'ObjectCreated:Put':
            object_name = record['s3']['object']['key']
            break
    else:
        return {
            'statusCode': 404,
            'body': 'no ObjectCreated:Put event found',
        }
    
    s3_client = boto3.client('s3')
    audio_url = generate_get_url(s3_client, object_name)
    callback_url = generate_put_url(s3_client, object_name)
    
    query = urlencode({"callback": callback_url, "callback_method": "put", "model": "nova"})
    print("query", query)
    dg_request = Request(
        f"https://api.deepgram.com/v1/listen?{query}", 
        headers={"Authorization": f"Token {DEEPGRAM_API_KEY}", "Content-Type": "application/json"},
        method="POST",
    )
    response = urlopen(dg_request, data=json.dumps({"url": audio_url}).encode('utf-8'))
    body = response.read().decode('utf-8')
    print("deepgram response:", body)
    if response.status >= 400:
        return {
            "statusCode": response.status,
            "body": body,
        }
    else:
        return {
            'statusCode': 200,
            'body': body,
        }


def generate_get_url(s3_client, object_name):
    response = s3_client.generate_presigned_url(
        ClientMethod='get_object',
        Params={
            'Bucket': AUDIO_BUCKET,
            'Key': object_name,
        },
        ExpiresIn=3600,
    )
    return response
    
def generate_put_url(s3_client, object_name):
    key = f"{object_name}.json"
    response = s3_client.generate_presigned_url(
        ClientMethod='put_object',
        Params={
            'Bucket': TRANSCRIPT_BUCKET,
            'Key': key,
            'ContentType': 'application/json',
        },
        ExpiresIn=3600,
    )
    return response    

You'd need to grant the lambda read access on the AUDIO_BUCKET and write access on the TRANSCRIPT_BUCKET, and provide your own DEEPGRAM_API_KEY. Then create a trigger when objects are created in the audio bucket. You might also want to update it to handle all ObjectCreated:Put events found in events["Records"] instead of just the first one.

[jcdyer]

Some gotchas I encountered putting this together:

• the put callback url needs to include the ContentType param, or else the signature won't match
• Depending on your region, you might need to configure your s3 client with a different signature_version, like Config(signature_version='s3v4').
• Don't forget to urlencode your callback url if the library you're using doesn't do it for you.

[jaxomlotus]

OK thanks very much for the detailed walkthrough and example code. I'll need to try this out.

It does add some complexity to my set up though, and I can't imagine I'm the only one with serverless architecture who will bump into this moving forward.

I hope the deepgram team offers a way to retrieve past transcripts in the future - if nothing else it's another feature to charge for. :)