[EPIC] Prototype workflow for generating and accessioning speech-to-text extraction

[jmartin-sul]

Some admonishments/guidelines for the below investigations, based on 2024-09-10 planning meeting

model choice/configuration

• We are assuming that we will use Whisper, or some variant (e.g. WhisperX), because it provided what we felt was the best combination of output quality and performance from the tools that were evaluated earlier in 2024 (@edsu may be able to link to that analysis for context?). If we determine that we want to go with a completely different model, we need to write up our reasoning for approval.
• We want our solution to provide access to Whisper's tuning parameters so that we can tweak them as needed, so completely blackbox solutions that run Whisper with no access to configuration aren't acceptable.

terminology

After some discussion, we settled on the term "speech to text" to encompass text extraction from speech in audio, whether it has video or not (there was lack of consensus/confusion about whether "caption" applies to audio-only, and it does also apply to still-image descriptions; and "transcript" doesn't quite encompass what captions do for video).

So e.g. speechToTextWF, speech_to_text as a snake-case var name, "speech to text" or "speech-to-text generation" as a human readable term, etc.

infrastructure provisioning

• We would like to avoid (or at least minimize as much as possible) vendor lock-in. We're highly likely to go with AWS to start, since we have more departmental expertise there, but GCP isn't out of the question. The cloud vendor has to be an org with which Stanford has a business agreement, and which is available through Cardinal Cloud, so that might rule out anything other than Amazon and Google? But as much as possible, we should use building blocks that have analogs in multiple major cloud vendors
• related, but somewhat standalone point: ultimately, we should define and deploy the cloud infrastructure using Terraform. It's meant to be platform agnostic and the department already uses it. But also, all of our permanent prod/stage/qa cloud infrastructure is deployed assuming that Terraform is the source of truth, so things that were created manually (e.g. using the AWS web console or one off aws CLI commands) will cause confusion in the future. It's totally fine to experiment with building blocks by manually spinning them up that way, but once the experiment is done, those should be torn down and defined formally in Terraform.

model usage

• ⚠️ It is unacceptable for our data to be used to train the models of other orgs. This rules out, for example, OpenAI's hosted Whisper service. This is a SUL-wide rule, at the moment.

todo

• provision S3 buckets and credentials for temp space to hold 1) the incoming content to run through speech-to-text generation, and 2) the generated transcript/caption files. #6
• Finish skeleton common-accessioning robot and workflow def for speechToTextWF common-accessioning#1341
• Choose an approach for producing speech-to-text output, given media file input (let's call this speech_to_text_generation_service for now?) #2
  • [investigate/prototype] speech_to_text_generation_service approach 1: Define a Docker container for running open source Whisper in a container that we define and for which we manage deployment (lives in this repo?) #3
  • [investigate/prototype] speech_to_text_generation_service approach 2: Explore AWS SageMaker #4
• Questions surrounding speech_to_text_generation_service (e.g., do we need a speech_to_text_request_service REST API?) #5
• investigate expected cost of cloud deployment, and as well as possible approaches for measuring cost common-accessioning#1425
• Productionize speech-to-text pipeline #7
  • @andrewjbtw's Putting the Whisper pipeline into production document
  • https://github.com/sul-dlss/terraform-aws/issues/1170
  • https://github.com/sul-dlss/terraform-aws/issues/1177
  • add speech-to-text to weekly dependency updates #65
  • Build and deploy speech-to-text #46
  • write DevOpsDocs for speech-to-text infrastructure #32
  • spike: can we notify Honeybadger when the speech-to-text container exits with a non-zero error code? #61
  • CI: replace current action for ruff with officially supported version #58
  • Wire speech_to_text_generation_service (and speech_to_text_request_service or whatever is used for signalling between SDR and cloud speech_to_text_generation_service deployment) up to workflows. common-accessioning#1363
    • Add the ability to send files to an AWS S3 bucket from fetch-files robot common-accessioning#1350
    • speechToTextWF: notify speech_to_text_generation_service that there is content to be STTed common-accessioning#1356
    • common-accessioning daemon watches SQS for notifications that content has been STTed, or that an error was encountered in processing common-accessioning#1358
    • if speech-to-text generation succeeds, fetch speech-to-text output common-accessioning#1359
    • speechToTextWF stage-files: stage transcription files generated by speech_to_text_generation_service common-accessioning#1360
    • speechToTextWF update-cocina: update cocina structural to reference files generated by speech_to_text_generation_service common-accessioning#1361
    • speechToTextWF end-stt: hand off to accessionWF common-accessioning#1362
• testing improvements
  • investigate localstack for S3 upload and SQS testing common-accessioning#1364
  • spec/lib/dor/text_extraction/file_fetcher_spec.rb: consider mocking HTTP request to pres cat, instead of preservation-client content retrieval method common-accessioning#1379
• Other miscellaneous questions and areas of uncertainty
  • Tooling for introspecting on what work is waiting or underway
  • Quality Control
    • hallucination tracking spreadsheet
      • Characterizing hallucinations is another motivator (besides model updates) for having tooling for regression testing and comparison between settings, models, etc
      • more notes and discussion in 2024-12-06 storytime notes
    • caption review tool and caption review workflow
    • Whether any uncorrected speech-to-text output is presented publicly. If so, which, and what do the warnings look like about output reliability?
    • Pre-processing of input to reduce hallucinations
      • add pre-processing step that omits files with only silence from being sent to whisper for speech to text extraction common-accessioning#1436
      • Investigate pre-processing options to detect long silences in recordings to be transcribed via Whisper common-accessioning#1435
    • Post-processing of output
      • to flag content with likely hallucinations for review
      • to automatically remove known hallucinations from output for presentation without further review
      • known markers of hallucination in output (see also Dataset bias ("❤️ Translated by Amara.org Community") openai/whisper#928 and Share your hallucinations here openai/whisper#1873)
        • long runs of “thank you” ("thank you" is often hallucinated when there's silence, and often many times for long stretches of silence)
        • inhumanly high speaking rates (one could, regardless of whisper output segment length, calculate a WPM rate in the whisper output for the heuristic's own segment length, say WPM for each 5 second block in the transcript; segments that cross a certain WPM threshold could be flagged). possible false positives: recordings that contain screen readers running at a typically high word rate, the micro machines guy
        • “translated by Amara.org” and other known common hallucinated phrases, often marks from other translation companies, since those are so common in training data. possible false positives: recordings of people talking about Whisper hallucinations
        • English-language recordings that contain non-English characters in the text. possible false positives: sometimes this is correct, e.g. accented characters for loan words, brief phrases spoken in another language.