demo_part1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b6ee1ede",
   "metadata": {},
   "source": [
    "## Voice Style Control Demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7f043ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import torch\n",
    "import se_extractor\n",
    "from api import BaseSpeakerTTS, ToneColorConverter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15116b59",
   "metadata": {},
   "source": [
    "### Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aacad912",
   "metadata": {},
   "outputs": [],
   "source": [
    "ckpt_base = 'checkpoints/base_speaker'\n",
    "ckpt_converter = 'checkpoints/converter'\n",
    "device = 'cuda:0'\n",
    "output_dir = 'outputs'\n",
    "\n",
    "base_speaker_tts = BaseSpeakerTTS(f'{ckpt_base}/config.json', device=device)\n",
    "base_speaker_tts.load_ckpt(f'{ckpt_base}/checkpoint.pth')\n",
    "\n",
    "tone_color_converter = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)\n",
    "tone_color_converter.load_ckpt(f'{ckpt_converter}/checkpoint.pth')\n",
    "\n",
    "os.makedirs(output_dir, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f67740c",
   "metadata": {},
   "source": [
    "### Obtain Tone Color Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8add279",
   "metadata": {},
   "source": [
    "The `source_se` is the tone color embedding of the base speaker. \n",
    "It is an average for multiple sentences with multiple emotions\n",
    "of the base speaker. We directly provide the result here but\n",
    "the readers feel free to extract `source_se` by themselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63ff6273",
   "metadata": {},
   "outputs": [],
   "source": [
    "source_se = torch.load(f'{ckpt_base}/source_se.pth').to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f71fcc3",
   "metadata": {},
   "source": [
    "The `reference_speaker.mp3` below points to the short audio clip of the reference whose voice we want to clone. We provide an example here. If you use your own reference speakers, please **make sure each speaker has a unique filename.** The `se_extractor` will save the `targeted_se` using the filename of the audio and **will not automatically overwrite.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55105eae",
   "metadata": {},
   "outputs": [],
   "source": [
    "reference_speaker = 'resources/example_reference.mp3'\n",
    "target_se, audio_name = se_extractor.get_se(reference_speaker, tone_color_converter)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a40284aa",
   "metadata": {},
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73dc1259",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_path = f'{output_dir}/output_friendly.wav'\n",
    "\n",
    "# Run the base speaker tts\n",
    "text = \"This audio is generated by open voice.\"\n",
    "src_path = f'{output_dir}/tmp.wav'\n",
    "base_speaker_tts.tts(text, src_path, speaker='friendly', language='English', speed=1.0)\n",
    "\n",
    "# Run the tone color converter\n",
    "encode_message = \"@MyShell\"\n",
    "tone_color_converter.convert(\n",
    "    audio_src_path=src_path, \n",
    "    src_se=source_se, \n",
    "    tgt_se=target_se, \n",
    "    output_path=save_path,\n",
    "    message=encode_message)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e3ea28a",
   "metadata": {},
   "source": [
    "**Try with different styles and speed.** The style can be controlled by the `speaker` parameter in the `base_speaker_tts.tts` method. Available choices: friendly, cheerful, excited, sad, angry, terrified, shouting, whispering. The speed can be controlled by the `speed` parameter. Let's try whispering with speed 0.9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd022d38",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_path = f'{output_dir}/output_whispering.wav'\n",
    "\n",
    "# Run the base speaker tts\n",
    "text = \"This audio is generated by open voice with a half-performance model.\"\n",
    "src_path = f'{output_dir}/tmp.wav'\n",
    "base_speaker_tts.tts(text, src_path, speaker='whispering', language='English', speed=0.9)\n",
    "\n",
    "# Run the tone color converter\n",
    "encode_message = \"@MyShell\"\n",
    "tone_color_converter.convert(\n",
    "    audio_src_path=src_path, \n",
    "    src_se=source_se, \n",
    "    tgt_se=target_se, \n",
    "    output_path=save_path,\n",
    "    message=encode_message)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e513094",
   "metadata": {},
   "source": [
    "**Tech for good.** For people who will deploy OpenVoice for public usage: We offer you the option to add watermark to avoid potential misuse. Please see the ToneColorConverter class. **MyShell reserves the ability to detect whether an audio is generated by OpenVoice**, no matter whether the watermark is added or not."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "9d70c38e1c0b038dbdffdaa4f8bfa1f6767c43760905c87a9fbe7800d18c6c35"
  },
  "kernelspec": {
   "display_name": "Python 3.9.18 ('openvoice')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}