NER_using_DSPy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "444b61a0-a5af-4e92-bb87-f8ad31429068",
   "metadata": {},
   "source": [
    "# NER using `DSPy`\n",
    "In this notebook, we want to try to evaluate and use [DSPy](https://dspy-docs.vercel.app/) for *Named Entity Recognition* (NER) use cases. This uses LLMs.\n",
    "\n",
    "DSPy:\n",
    "> DSPy is a framework for algorithmically **optimizing** LM **prompts** and **weights**, mainly when LMs are used one or more times within a pipeline\n",
    "\n",
    "Some *prompt engineering* techniques that are useful for *NER* problems are:\n",
    "* Function/tools calling\n",
    "* Zero/few-shot examples using `domain` instructions\n",
    "* Chain-of-Thought (CoT) prompting\n",
    "* Prompt-CHaining\n",
    "\n",
    "We want to build a data extraction system using NER to extract food-related entities from online recipes. \n",
    "\n",
    "We are interested in the following entities:\n",
    "* `FOOD`\n",
    "* `QUANTITY`\n",
    "* `UNIT`\n",
    "* `PHYSICAL_QUALITY`\n",
    "* `COLOR`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e433a850-f859-4807-9ee2-fa690ce9a45d",
   "metadata": {},
   "source": [
    "# 0. Setup environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "b316f6ba-2c4f-4912-9f49-64c9f873fb31",
   "metadata": {},
   "outputs": [],
   "source": [
    "import dspy\n",
    "import nest_asyncio\n",
    "from pydantic import BaseModel, Field\n",
    "from dspy.functional import TypedPredictor\n",
    "from IPython.display import Markdown, display\n",
    "from typing import List, Optional, Union\n",
    "from dotenv import load_dotenv\n",
    "from devtools import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "472b3c11-a5b3-493c-83ab-c110736da1f9",
   "metadata": {},
   "source": [
    "Load env variables see `.env.example`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "00cddf69-2f6f-4622-84f2-a0d52e5d7235",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert load_dotenv() == True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4ad9ed0-6668-4d4c-bcb4-e69a695e02e4",
   "metadata": {},
   "source": [
    "Init some selected LLMs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5034d414-a627-4ded-bbb1-d7b1e3ff5dc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt4 = dspy.OpenAI(model=\"gpt-4-turbo-preview\", max_tokens=4096, model_type=\"chat\")\n",
    "gpt_turbo = dspy.OpenAI(model=\"gpt-3.5-turbo\", max_tokens=4096, model_type=\"chat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "668f67b3-27ca-41a3-95be-7eb8f1e3da13",
   "metadata": {},
   "source": [
    "Configure `DSPy` to use `gpt-3.5-turbo` as default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 551,
   "id": "69c6f971-357c-4274-bb0a-681566753ce7",
   "metadata": {},
   "outputs": [],
   "source": [
    "dspy.settings.configure(lm=gpt4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "20df097e-51d8-4c76-aeeb-425de8790738",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The number 42 being seen as the meaning of life comes from Douglas Adams\\' science fiction series \"The Hitchhiker\\'s Guide to the Galaxy.\" In the series, a supercomputer named Deep Thought is asked to find the answer to the ultimate question of life, the universe, and everything. After much calculation, Deep Thought reveals that the answer is simply the number 42, but the actual question is unknown.\\n\\nThe significance of 42 as the meaning of life has since become a popular cultural reference and meme, often used humorously to suggest that life\\'s ultimate purpose is unknowable or absurd. It has also been interpreted as a commentary on the futility of seeking a single, definitive answer to the mysteries of existence.']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpt_turbo(\"Why is 42 seen as the meaning of life?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2c4ec090-f68f-4c5f-a6fd-df0fce1c2efc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The notion that 42 is the \"meaning of life\" comes from Douglas Adams\\' science fiction series, \"The Hitchhiker\\'s Guide to the Galaxy.\" In the story, a supercomputer named Deep Thought is asked to find the \"Answer to the Ultimate Question of Life, the Universe, and Everything.\" After much computation—7.5 million years\\' worth—Deep Thought reveals that the answer is, somewhat anticlimactically, 42.\\n\\nHowever, Deep Thought also points out that the problem is actually that nobody knows what the Ultimate Question is. Therefore, while 42 is the Answer, it doesn\\'t make much sense without knowing the corresponding Question. This humorous and absurd take on the search for meaning in life has led to 42 becoming a cultural reference point and a symbol for an elusive or absurd answer to a complex or impossible question.\\n\\nDouglas Adams later explained that he chose the number 42 as the answer simply because it was a funny number, not intending any deeper meaning, which in itself became a part of the joke and the enduring appeal of the concept.']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpt4(\"Why is 42 seen as the meaning of life?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f10be9d8-6a1e-476c-90f1-1123cce4f9a1",
   "metadata": {},
   "source": [
    "# 1. Loading data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7af89deb-4c80-4c0e-8696-36ff82c64e1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"./data/recipe.md\", \"r\") as f:\n",
    "    train_data = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ac5b0f5c-af67-44ef-b59a-07472319fb66",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### Chashu pork (for ramen and more)\n",
       "Chashu pork is a classic way to prepare pork belly for Japanese dishes such as ramen. While it takes a little time, it's relatively hands off and easy, and the result is delicious.\n",
       "\n",
       "### Prep Time\n",
       "15 minutes\n",
       "\n",
       "### Cook Time\n",
       "2hours 5minutes\n",
       "\n",
       "### Rest time (approx)\n",
       "12 hours hrs\n",
       "\n",
       "### Total Time\n",
       "2 hours 20 minutes \n",
       "\n",
       "### Course\n",
       "Main Course\n",
       "\n",
       "###\n",
       "Cuisine\n",
       "Japanese \n",
       "\n",
       "### Servings\n",
       "4 or more, depending how used \n",
       "\n",
       "### Calories: 1378kcal \n",
       "\n",
       "### Author: Caroline's Cooking \n",
       "\n",
       "\n",
       "### Ingredients\n",
       "* 2 lb pork belly or a little more/less\n",
       "* 2 green onions spring onions, or 3 if small\n",
       "* 1 in fresh ginger (a chunk that will give around 4 - 6 slices)\n",
       "* 2 cloves garlic\n",
       "* ⅔ cup sake\n",
       "* ⅔ cup soy sauce\n",
       "* ¼ cup mirin\n",
       "* ½ cup sugar\n",
       "* 2 cups water or a little more as needed\n",
       "\n",
       "US Customary - Metric\n",
       "\n",
       "### Instructions\n",
       "1. Have some string/kitchen twine ready and a pair of scissors before you prepare the pork. If needed, trim excess fat from the outside of the pork but you still want a later over the meat itself. Roll the piece of pork belly with the fatter end in the middle, and the flatter part making the roll around it. It may be a little tricky to hold together, but try your best as you tie it. Loop a piece of string around one end of the rolled pork and tie it in a double knot so that it holds the pork together. Then, taker the long end of the string and make additional loops around the pork, around ⅔in/2cm apart (or closer), looping through as you go and tightening to make a row of loops, linked together, along the roll. Then loop through over both ends so that it holds the ends in a little better as well and tie off and cut the string. (See pictures and video if you need a bit more guidance on how.)\n",
       "   \n",
       "2. Cut the piece of ginger into slices - there's no need to peel it. Peel and trim the ends from the garlic. Trim both ends off the green/spring onions and cut into long lengths.\n",
       "\n",
       "3. Warm a skillet/frying pan large enough to hold the pork but not much bigger, ideally cast iron or at least heavy based. Then sear the pork on all sides to gently brown and seal the pork and fat, including the ends. Start with a fatty side so that some of the fat releases to help with the searing. Once browned all over, remove from skillet and set aside. You can discard the excess fat that has come out.\n",
       "\n",
       "4. Meanwhile, place the sake, soy sauce, mirin, sugar, water and the prepared ginger, garlic and green onions in a pot. You want a pot that is just a little wider than the piece of rolled pork so that it will be mostly covered by the liquid once added. Place over a medium heat, covered, and bring to a simmer, stirring now and then to ensure sugar dissolves.\n",
       "\n",
       "5. Once the sake-soy mixture is simmering, and you have seared the pork, add the pork into the liquid. The liquid should be at least ⅔ up the pork, if not a little more - you can add a little more water if needed. Cover with the lid and bring back to a simmer then reduce the heat so that it simmers gently. Cook for about two hours, turning roughly every 30 minutes so that each side is submerged half the time. If the liquid gets below halfway, then add a little more water.\n",
       "\n",
       "6. After two hours, the pork will be cooked through and should be fairly tender. Turn off the heat and leave the pork to cool in the cooking liquid at least 10 minutes, but you can leave until room temperature. Once cool, remove the pork and set aside a minute. Strain the cooking liquid into a measuring jug to remove the ginger etc. Let it sit a few minutes so that the fat floats to the top then spoon off the fat and discard.\n",
       "\n",
       "7. Place the pork inside a freezer bag, just a little bigger than the pork itself, and sit it in a dish in case of any spills. Then, pour some of the cooking liquid into the bag so that the pork is submerged. Carefully remove excess air from the bag, seal up and then place in the fridge overnight. Don't skip this step, as it helps the pork firm up, but also tenderize further and take in more flavor.\n",
       "\n",
       "8. The next day, when ready to use, carefully take the pork out of the liquid and remove any fat that has solidified on it (that's not park of the pork belly itself). Cut off the string then cut the pork into relatively thin slices, so that you get coiled pieces of the pork belly.\n",
       "\n",
       "9. To prepare it to top ramen, warm some of the cooking liquid in a skillet and then add the pork, a few slices at a time so that they form a single layer. Simmer a couple minutes, carefully turning as needed, so that the pork warms through and gently caramelizes in the liquid. For chashu fried rice, you will instead want to chop the pork further and then fry, along with the rice."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(train_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "547d6e50-7d2a-498e-afc7-66dffc90ede4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "### Chashu pork (for ramen and more)\n",
      "Chashu pork is a classic way to prepare pork belly for Japanese\n"
     ]
    }
   ],
   "source": [
    "print(train_data[0:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74463b70-3359-4769-852b-622983623814",
   "metadata": {},
   "source": [
    "# 2. Define `data models`\n",
    "We use `pydantic` to define the output schema of the data that we want to extract using the `DSPy` program. This in order to be able to use the [TypedPredictor](https://dspy-docs.vercel.app/docs/building-blocks/typed_predictors). The data model we will define will be called `FoodEntity` and `FoodEntities`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "644a4243-afbd-4f0e-9af0-21c83931feed",
   "metadata": {},
   "outputs": [],
   "source": [
    "class FoodMetaData(BaseModel):\n",
    "    reasoning: str = Field(description=\"Reasoning for why the entity is correct\")\n",
    "    value: Union[str, int] = Field(description=\"Value of the entity\")\n",
    "    entity: str = Field(description=\"The actual entity i.e. pork, onions etc\")\n",
    "\n",
    "class FoodMetaDatas(BaseModel):\n",
    "    context: List[FoodMetaData]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "86c0682e-e715-4480-aa6f-62734fb291c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'properties': {'reasoning': {'description': 'Reasoning for why the entity is correct',\n",
       "   'title': 'Reasoning',\n",
       "   'type': 'string'},\n",
       "  'value': {'anyOf': [{'type': 'string'}, {'type': 'integer'}],\n",
       "   'description': 'Value of the entity',\n",
       "   'title': 'Value'},\n",
       "  'entity': {'description': 'The actual entity i.e. pork, onions etc',\n",
       "   'title': 'Entity',\n",
       "   'type': 'string'}},\n",
       " 'required': ['reasoning', 'value', 'entity'],\n",
       " 'title': 'FoodMetaData',\n",
       " 'type': 'object'}"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FoodMetaData.model_json_schema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "a04c6297-c3b7-47f1-8845-e572e58221b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'$defs': {'FoodMetaData': {'properties': {'reasoning': {'description': 'Reasoning for why the entity is correct',\n",
       "     'title': 'Reasoning',\n",
       "     'type': 'string'},\n",
       "    'value': {'anyOf': [{'type': 'string'}, {'type': 'integer'}],\n",
       "     'description': 'Value of the entity',\n",
       "     'title': 'Value'},\n",
       "    'entity': {'description': 'The actual entity i.e. pork, onions etc',\n",
       "     'title': 'Entity',\n",
       "     'type': 'string'}},\n",
       "   'required': ['reasoning', 'value', 'entity'],\n",
       "   'title': 'FoodMetaData',\n",
       "   'type': 'object'}},\n",
       " 'properties': {'metadatas': {'items': {'$ref': '#/$defs/FoodMetaData'},\n",
       "   'title': 'Metadatas',\n",
       "   'type': 'array'}},\n",
       " 'required': ['metadatas'],\n",
       " 'title': 'FoodMetaDatas',\n",
       " 'type': 'object'}"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FoodMetaDatas.model_json_schema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 565,
   "id": "fb36e31c-8899-449b-ac35-83abf22c085e",
   "metadata": {},
   "outputs": [],
   "source": [
    "class FoodEntity(BaseModel):\n",
    "    food: str = Field(description=\"This can be both liquid and solid food such as meat, vegetables, alcohol, etc\")\n",
    "    quantity: int = Field(description=\"The exact quantity or amount of the food that should be used in the recipe\")\n",
    "    unit: str = Field(description=\"The unit being used e.g. grams, milliliters, pounds, etc\")\n",
    "    physical_quality: Optional[str] = Field(description=\"The characteristic of the ingredient\")\n",
    "    color: str = Field(description=\"The color of the food\")\n",
    "\n",
    "class FoodEntities(BaseModel):\n",
    "    entities: List[FoodEntity]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a9001ed-35ea-4b31-8e5a-d596fdd1447e",
   "metadata": {},
   "source": [
    "The schemas above define our desired output format, a JSON object with one or several `FoodEntities`. Notice that we set some validation checks as part of the schema. This is mainly for demonstration purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 566,
   "id": "584844d7-840e-4a41-ae56-77a3d8742595",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'properties': {'food': {'description': 'This can be both liquid and solid food such as meat, vegetables, alcohol, etc',\n",
       "   'title': 'Food',\n",
       "   'type': 'string'},\n",
       "  'quantity': {'description': 'The exact quantity or amount of the food that should be used in the recipe',\n",
       "   'title': 'Quantity',\n",
       "   'type': 'integer'},\n",
       "  'unit': {'description': 'The unit being used e.g. grams, milliliters, pounds, etc',\n",
       "   'title': 'Unit',\n",
       "   'type': 'string'},\n",
       "  'physical_quality': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "   'description': 'The characteristic of the ingredient',\n",
       "   'title': 'Physical Quality'},\n",
       "  'color': {'description': 'The color of the food',\n",
       "   'title': 'Color',\n",
       "   'type': 'string'}},\n",
       " 'required': ['food', 'quantity', 'unit', 'physical_quality', 'color'],\n",
       " 'title': 'FoodEntity',\n",
       " 'type': 'object'}"
      ]
     },
     "execution_count": 566,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FoodEntity.model_json_schema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 567,
   "id": "bf6f0b0f-ddb4-413e-9f73-6e888a11f2da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'$defs': {'FoodEntity': {'properties': {'food': {'description': 'This can be both liquid and solid food such as meat, vegetables, alcohol, etc',\n",
       "     'title': 'Food',\n",
       "     'type': 'string'},\n",
       "    'quantity': {'description': 'The exact quantity or amount of the food that should be used in the recipe',\n",
       "     'title': 'Quantity',\n",
       "     'type': 'integer'},\n",
       "    'unit': {'description': 'The unit being used e.g. grams, milliliters, pounds, etc',\n",
       "     'title': 'Unit',\n",
       "     'type': 'string'},\n",
       "    'physical_quality': {'anyOf': [{'type': 'string'}, {'type': 'null'}],\n",
       "     'description': 'The characteristic of the ingredient',\n",
       "     'title': 'Physical Quality'},\n",
       "    'color': {'description': 'The color of the food',\n",
       "     'title': 'Color',\n",
       "     'type': 'string'}},\n",
       "   'required': ['food', 'quantity', 'unit', 'physical_quality', 'color'],\n",
       "   'title': 'FoodEntity',\n",
       "   'type': 'object'}},\n",
       " 'properties': {'entities': {'items': {'$ref': '#/$defs/FoodEntity'},\n",
       "   'title': 'Entities',\n",
       "   'type': 'array'}},\n",
       " 'required': ['entities'],\n",
       " 'title': 'FoodEntities',\n",
       " 'type': 'object'}"
      ]
     },
     "execution_count": 567,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FoodEntities.model_json_schema()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c86e04b-0108-4a37-bf3f-e7a566d9451b",
   "metadata": {},
   "source": [
    "# 3. DSPy Program for NER\n",
    "A `DSPy` program consists of `Signatures` and `Modules.` These can then be optimized using `Teleprompters` and finally be compiled by the `Compiler.` This by using data for your program.\n",
    "\n",
    "A **signature** consists of three simple elements:\n",
    "\n",
    "> A minimal description of the sub-task the LM is supposed to solve.\n",
    "A description of one or more input fields (e.g., input question) that we will give to the LM.\n",
    "A description of one or more output fields (e.g., the question's answer) that we will expect from the LM\n",
    "\n",
    "In our case we will be using a `Typed Signature` as we want the LLM to follow our pydantic schema."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a843c23c-b909-4757-95f8-13eba614ba71",
   "metadata": {},
   "source": [
    "### 1. Create `Signature`\n",
    "Looking in to the code of `DSPy` the `InputField` and `OutPutFields` some to be wrappers for the `Field` object in `Pydantic`. See more [here](https://github.com/stanfordnlp/dspy/blob/main/dspy/signatures/field.py#L29)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 568,
   "id": "cbcd92dc-06c7-4fe2-84cf-bbd7855cd166",
   "metadata": {},
   "outputs": [],
   "source": [
    "class RecipeToFoodContext(dspy.Signature):\n",
    "    \"\"\"You are a food AI assistant. Your task is to extract the entity, the value of the entity and the reasoning \n",
    "    for why the extracted value is the correct value. If you cannot extract the entity, add null\"\"\"\n",
    "    recipe: str = dspy.InputField()\n",
    "    context: FoodMetaDatas = dspy.OutputField()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 569,
   "id": "fee24789-ac56-43d7-a46b-64a811dcd9a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "class RecipeToFoodEntities(dspy.Signature):\n",
    "    \"\"\"You are a food AI assistant. Your task is to extract food-related metadata from recipes.\"\"\"\n",
    "    recipe: str = dspy.InputField()\n",
    "    entities: FoodEntities = dspy.OutputField()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c76e649b-11f2-4899-91c3-59c1b8ef5053",
   "metadata": {},
   "source": [
    "Let's try the `TypedPredictor` with some dummy example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 570,
   "id": "86cceb20-b6dd-4593-8f86-7d4ee537bdc6",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor_context = dspy.TypedPredictor(RecipeToFoodContext)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 571,
   "id": "4bc216c9-3ff4-4a1e-a585-1064b412bcb3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FoodMetaDatas(\n",
      "    context=[\n",
      "        FoodMetaData(\n",
      "            reasoning='The recipe specifies the quantity of orange dutch cheese as ten grams.',\n",
      "            value='10 grams',\n",
      "            entity='orange dutch cheese',\n",
      "        ),\n",
      "        FoodMetaData(\n",
      "            reasoning='The recipe specifies the quantity of water as 2 liters.',\n",
      "            value='2 liters',\n",
      "            entity='water',\n",
      "        ),\n",
      "        FoodMetaData(\n",
      "            reasoning='The recipe specifies the quantity of ice as 5 ml, indicating the volume of ice used.',\n",
      "            value='5 ml',\n",
      "            entity='ice',\n",
      "        ),\n",
      "    ],\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Run with GPT-4 instead\n",
    "with dspy.context(lm=gpt4):\n",
    "    dummy_context = predictor_context(recipe=\"Ten grams of orange dutch cheese, 2 liters of water and 5 ml of ice\")\n",
    "    pprint(dummy_context.context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 572,
   "id": "2b3a8872-6963-482a-9663-7a9ce90bc92f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "You are a food AI assistant. Your task is to extract the entity, the value of the entity and the reasoning \n",
      "    for why the extracted value is the correct value. If you cannot extract the entity, add null\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Recipe: ${recipe}\n",
      "Context: ${context}. Respond with a single JSON object. JSON Schema: {\"$defs\": {\"FoodMetaData\": {\"properties\": {\"reasoning\": {\"description\": \"Reasoning for why the entity is correct\", \"title\": \"Reasoning\", \"type\": \"string\"}, \"value\": {\"anyOf\": [{\"type\": \"string\"}, {\"type\": \"integer\"}], \"description\": \"Value of the entity\", \"title\": \"Value\"}, \"entity\": {\"description\": \"The actual entity i.e. pork, onions etc\", \"title\": \"Entity\", \"type\": \"string\"}}, \"required\": [\"reasoning\", \"value\", \"entity\"], \"title\": \"FoodMetaData\", \"type\": \"object\"}}, \"properties\": {\"context\": {\"items\": {\"$ref\": \"#/$defs/FoodMetaData\"}, \"title\": \"Context\", \"type\": \"array\"}}, \"required\": [\"context\"], \"title\": \"FoodMetaDatas\", \"type\": \"object\"}\n",
      "\n",
      "---\n",
      "\n",
      "Recipe: Ten grams of orange dutch cheese, 2 liters of water and 5 ml of ice\n",
      "Context:\u001b[32m ```json\n",
      "{\n",
      "  \"context\": [\n",
      "    {\n",
      "      \"entity\": \"orange dutch cheese\",\n",
      "      \"value\": \"10 grams\",\n",
      "      \"reasoning\": \"The recipe specifies the quantity of orange dutch cheese as ten grams.\"\n",
      "    },\n",
      "    {\n",
      "      \"entity\": \"water\",\n",
      "      \"value\": \"2 liters\",\n",
      "      \"reasoning\": \"The recipe specifies the quantity of water as 2 liters.\"\n",
      "    },\n",
      "    {\n",
      "      \"entity\": \"ice\",\n",
      "      \"value\": \"5 ml\",\n",
      "      \"reasoning\": \"The recipe specifies the quantity of ice as 5 ml, indicating the volume of ice used.\"\n",
      "    }\n",
      "  ]\n",
      "}\n",
      "```\u001b[0m\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gpt4.inspect_history(n=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 573,
   "id": "e85c478a-275d-4628-95cb-170ef1abee76",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = dspy.TypedPredictor(RecipeToFoodEntities)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 574,
   "id": "e601e159-5aec-48f6-89a3-40638b1fd29f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FoodEntities(\n",
      "    entities=[\n",
      "        FoodEntity(\n",
      "            food='dutch cheese',\n",
      "            quantity=10,\n",
      "            unit='grams',\n",
      "            physical_quality=None,\n",
      "            color='orange',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='water',\n",
      "            quantity=2,\n",
      "            unit='liters',\n",
      "            physical_quality=None,\n",
      "            color='clear',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='ice',\n",
      "            quantity=5,\n",
      "            unit='ml',\n",
      "            physical_quality=None,\n",
      "            color='clear',\n",
      "        ),\n",
      "    ],\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Run with GPT-4 instead\n",
    "with dspy.context(lm=gpt4):\n",
    "    dummy_recipe = predictor(recipe=\"Ten grams of orange dutch cheese, 2 liters of water and 5 ml of ice\")\n",
    "    pprint(dummy_recipe.entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e060aa52-70ea-407b-ac75-a55f521e83d1",
   "metadata": {},
   "source": [
    "Using `inspect_history` below, we can take a look a the resulting prompt that is created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 575,
   "id": "dd6f8e03-440a-45fa-b413-bce22f9c3119",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "You are a food AI assistant. Your task is to extract food-related metadata from recipes.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Recipe: ${recipe}\n",
      "\n",
      "Past Error (entities): An error to avoid in the future\n",
      "\n",
      "Entities:\n",
      "${entities}. Respond with a single JSON object. \n",
      "You MUST use this format: ```json\n",
      "{\n",
      "  \"entities\": [\n",
      "    {\n",
      "      \"food\": \"chicken breast\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"pieces\",\n",
      "      \"physical_quality\": \"boneless\",\n",
      "      \"color\": \"white\"\n",
      "    }\n",
      "  ]\n",
      "}\n",
      "```\n",
      "JSON Schema: {\"$defs\": {\"FoodEntity\": {\"properties\": {\"food\": {\"description\": \"This can be both liquid and solid food such as meat, vegetables, alcohol, etc\", \"title\": \"Food\", \"type\": \"string\"}, \"quantity\": {\"description\": \"The exact quantity or amount of the food that should be used in the recipe\", \"title\": \"Quantity\", \"type\": \"integer\"}, \"unit\": {\"description\": \"The unit being used e.g. grams, milliliters, pounds, etc\", \"title\": \"Unit\", \"type\": \"string\"}, \"physical_quality\": {\"anyOf\": [{\"type\": \"string\"}, {\"type\": \"null\"}], \"description\": \"The characteristic of the ingredient\", \"title\": \"Physical Quality\"}, \"color\": {\"description\": \"The color of the food\", \"title\": \"Color\", \"type\": \"string\"}}, \"required\": [\"food\", \"quantity\", \"unit\", \"physical_quality\", \"color\"], \"title\": \"FoodEntity\", \"type\": \"object\"}}, \"properties\": {\"entities\": {\"items\": {\"$ref\": \"#/$defs/FoodEntity\"}, \"title\": \"Entities\", \"type\": \"array\"}}, \"required\": [\"entities\"], \"title\": \"FoodEntities\", \"type\": \"object\"}\n",
      "\n",
      "---\n",
      "\n",
      "Recipe: Ten grams of orange dutch cheese, 2 liters of water and 5 ml of ice\n",
      "\n",
      "Past Error (entities): Input should be a valid string: entities, 1, color (error type: string_type); Input should be a valid string: entities, 2, color (error type: string_type)\n",
      "\n",
      "Entities:\u001b[32m ```json\n",
      "{\n",
      "  \"entities\": [\n",
      "    {\n",
      "      \"food\": \"dutch cheese\",\n",
      "      \"quantity\": 10,\n",
      "      \"unit\": \"grams\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"orange\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"water\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"liters\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"clear\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"ice\",\n",
      "      \"quantity\": 5,\n",
      "      \"unit\": \"ml\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"clear\"\n",
      "    }\n",
      "  ]\n",
      "}\n",
      "```\u001b[0m\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gpt4.inspect_history(n=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fdf0955-b431-473f-b7b7-0cbd1cb29179",
   "metadata": {},
   "source": [
    "## 2. Create the `Module`\n",
    "A DSPy module is a building block for programs that use LMs.\n",
    "\n",
    "> Each built-in module abstracts a prompting technique (like chain of thought or ReAct). Crucially, they are generalized to handle any [DSPy Signature].\n",
    "\n",
    "> Your __init__ method declares the modules you will use.\n",
    "\n",
    "> Your forward method expresses any computation you want to do with your modules"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9ec7403-6b1a-4e26-ac5c-671353763607",
   "metadata": {},
   "source": [
    "#### Using `dspy.Module`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d56d611c-9722-42d2-921a-90534fedf020",
   "metadata": {},
   "source": [
    "Let's define a helper method i.e. `parse_context` to present the parsed context in a pretty format for the LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 576,
   "id": "bf00990d-5d82-4711-8aea-4f9c33b316da",
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_context(food_context: FoodMetaDatas) -> str:\n",
    "    context_str = \"\"\n",
    "    for context in food_context:\n",
    "        context: FoodMetaData\n",
    "        context_str += f\"{context.entity}:\\n\" + context.model_dump_json(indent=4) + \"\\n\"\n",
    "    return context_str"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 588,
   "id": "10212329-524d-4dd7-989b-ca4b15b1815f",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ExtractFoodEntities(dspy.Module):\n",
    "    def __init__(self, temperature: int = 0, seed: int = 123):\n",
    "        super().__init__()\n",
    "        self.temperature = temperature\n",
    "        self.seed = seed\n",
    "        self.extract_food_context = dspy.TypedPredictor(RecipeToFoodContext)\n",
    "        self.extract_food_context_cot = dspy.TypedChainOfThought(RecipeToFoodContext)\n",
    "        self.extract_food_entities = dspy.TypedPredictor(RecipeToFoodEntities)\n",
    "        \n",
    "    def forward(self, recipe: str) -> FoodEntities:\n",
    "        food_context = self.extract_food_context(recipe=recipe).context\n",
    "        parsed_context = parse_context(food_context.context)\n",
    "        food_entities = self.extract_food_entities(recipe=parsed_context)\n",
    "        return food_entities.entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 580,
   "id": "3e9f8da0-8e6c-4008-8a16-2744b670a579",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FoodEntities(\n",
      "    entities=[\n",
      "        FoodEntity(\n",
      "            food='orange dutch cheese',\n",
      "            quantity=10,\n",
      "            unit='grams',\n",
      "            physical_quality=None,\n",
      "            color='orange',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='water',\n",
      "            quantity=2000,\n",
      "            unit='milliliters',\n",
      "            physical_quality=None,\n",
      "            color='clear',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='ice',\n",
      "            quantity=5,\n",
      "            unit='milliliters',\n",
      "            physical_quality=None,\n",
      "            color='clear',\n",
      "        ),\n",
      "    ],\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "extract_food_entities = ExtractFoodEntities()\n",
    "\n",
    "with dspy.context(lm=gpt4):\n",
    "    entities = extract_food_entities(recipe=\"Ten grams of orange dutch cheese, 2 liters of water and 5 ml of ice\")\n",
    "    pprint(entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47eac0ec-a58a-4f1e-a0cc-9028d2eb066e",
   "metadata": {},
   "source": [
    "Test with `train` data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 591,
   "id": "e9ab1009-642b-41ab-8f06-690d4b6e9cb7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FoodEntities(\n",
      "    entities=[\n",
      "        FoodEntity(\n",
      "            food='pork belly',\n",
      "            quantity=2,\n",
      "            unit='lb',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='green onions',\n",
      "            quantity=2,\n",
      "            unit='count',\n",
      "            physical_quality='or 3 if small',\n",
      "            color='green',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='fresh ginger',\n",
      "            quantity=1,\n",
      "            unit='inch',\n",
      "            physical_quality='chunk',\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='garlic',\n",
      "            quantity=2,\n",
      "            unit='cloves',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='sake',\n",
      "            quantity=158,\n",
      "            unit='milliliters',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='soy sauce',\n",
      "            quantity=158,\n",
      "            unit='milliliters',\n",
      "            physical_quality=None,\n",
      "            color='dark',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='mirin',\n",
      "            quantity=59,\n",
      "            unit='milliliters',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='sugar',\n",
      "            quantity=118,\n",
      "            unit='milliliters',\n",
      "            physical_quality=None,\n",
      "            color='white',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='water',\n",
      "            quantity=473,\n",
      "            unit='milliliters',\n",
      "            physical_quality='or a little more as needed',\n",
      "            color='clear',\n",
      "        ),\n",
      "    ],\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "with dspy.context(lm=gpt4):\n",
    "    entities = extract_food_entities(recipe=train_data)\n",
    "    pprint(entities)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 593,
   "id": "9e573ee1-324f-4e8b-805d-dc773d01cc4f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[FoodEntity(food='pork belly', quantity=2, unit='lb', physical_quality=None, color=''),\n",
       " FoodEntity(food='green onions', quantity=2, unit='count', physical_quality='or 3 if small', color='green'),\n",
       " FoodEntity(food='fresh ginger', quantity=1, unit='inch', physical_quality='chunk', color=''),\n",
       " FoodEntity(food='garlic', quantity=2, unit='cloves', physical_quality=None, color=''),\n",
       " FoodEntity(food='sake', quantity=158, unit='milliliters', physical_quality=None, color=''),\n",
       " FoodEntity(food='soy sauce', quantity=158, unit='milliliters', physical_quality=None, color='dark'),\n",
       " FoodEntity(food='mirin', quantity=59, unit='milliliters', physical_quality=None, color=''),\n",
       " FoodEntity(food='sugar', quantity=118, unit='milliliters', physical_quality=None, color='white'),\n",
       " FoodEntity(food='water', quantity=473, unit='milliliters', physical_quality='or a little more as needed', color='clear')]"
      ]
     },
     "execution_count": 593,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entities.entities"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7813bf7-838a-479c-a579-df16ebb183a3",
   "metadata": {},
   "source": [
    "#### Using `FunctionalModule`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 603,
   "id": "5cc8a40c-069e-46d6-8e3b-d62b4045f914",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dspy.functional import FunctionalModule, predictor, cot\n",
    "\n",
    "class ExtractFoodEntitiesV2(FunctionalModule):\n",
    "    def __init__(self, temperature: int = 0, seed: int = 123):\n",
    "        super().__init__()\n",
    "        self.temperature = temperature\n",
    "        self.seed = seed\n",
    "\n",
    "    @predictor\n",
    "    def extract_food_context(self, recipe: str) -> FoodMetaDatas:\n",
    "        \"\"\"You are a food AI assistant. Your task is to extract the entity, the value of the entity and the reasoning \n",
    "        for why the extracted value is the correct value. If you cannot extract the entity, add null\"\"\"\n",
    "        pass\n",
    "\n",
    "    @cot\n",
    "    def extract_food_context_cot(self, recipe: str) -> FoodMetaDatas:\n",
    "        \"\"\"You are a food AI assistant. Your task is to extract the entity, the value of the entity and the reasoning \n",
    "        for why the extracted value is the correct value. If you cannot extract the entity, add null\"\"\"\n",
    "        pass\n",
    "    \n",
    "    @predictor\n",
    "    def extract_food_entities(self, recipe: str) -> FoodEntities:\n",
    "        \"\"\"You are a food AI assistant. Your task is to extract food entities from a recipe.\"\"\"\n",
    "        pass\n",
    "        \n",
    "    def forward(self, recipe: str) -> FoodEntities:\n",
    "        food_context = self.extract_food_context(recipe=recipe)\n",
    "        parsed_context = parse_context(food_context.context)\n",
    "        food_entities = self.extract_food_entities(recipe=parsed_context)\n",
    "        return food_entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 604,
   "id": "81564ca2-4f03-42cd-844a-66a5a774ec9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FoodEntities(\n",
      "    entities=[\n",
      "        FoodEntity(\n",
      "            food='orange dutch cheese',\n",
      "            quantity=10,\n",
      "            unit='grams',\n",
      "            physical_quality=None,\n",
      "            color='orange',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='water',\n",
      "            quantity=2000,\n",
      "            unit='milliliters',\n",
      "            physical_quality=None,\n",
      "            color='clear',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='ice',\n",
      "            quantity=5,\n",
      "            unit='milliliters',\n",
      "            physical_quality='solid',\n",
      "            color='clear',\n",
      "        ),\n",
      "    ],\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "extract_food_entities_v2 = ExtractFoodEntitiesV2()\n",
    "\n",
    "with dspy.context(lm=gpt4):\n",
    "    entities = extract_food_entities_v2(recipe=\"Ten grams of orange dutch cheese, 2 liters of water and 5 ml of ice\")\n",
    "    pprint(entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88726a6f-8049-42cd-b1af-084224c7d390",
   "metadata": {},
   "source": [
    "Test on the `training` data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 605,
   "id": "f4dd2126-0773-44b2-9993-3b3fc6e4b5c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FoodEntities(\n",
      "    entities=[\n",
      "        FoodEntity(\n",
      "            food='pork belly',\n",
      "            quantity=2,\n",
      "            unit='lb',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='green onions',\n",
      "            quantity=2,\n",
      "            unit='items',\n",
      "            physical_quality='or 3 if small',\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='fresh ginger',\n",
      "            quantity=1,\n",
      "            unit='inch',\n",
      "            physical_quality='chunk',\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='garlic',\n",
      "            quantity=2,\n",
      "            unit='cloves',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='sake',\n",
      "            quantity=2,\n",
      "            unit='⅔ cup',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='soy sauce',\n",
      "            quantity=2,\n",
      "            unit='⅔ cup',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='mirin',\n",
      "            quantity=1,\n",
      "            unit='¼ cup',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='sugar',\n",
      "            quantity=1,\n",
      "            unit='½ cup',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='water',\n",
      "            quantity=2,\n",
      "            unit='cups',\n",
      "            physical_quality='or a little more as needed',\n",
      "            color='',\n",
      "        ),\n",
      "    ],\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "with dspy.context(lm=gpt4):\n",
    "    entities = extract_food_entities_v2(recipe=train_data)\n",
    "    pprint(entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "626b7cfe-827d-4557-8436-0f78949de31b",
   "metadata": {},
   "source": [
    "And below we have some of the resulting `entities`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 607,
   "id": "069c4611-e20b-4ca5-a565-1fa7d4f4c539",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FoodEntities(\n",
      "    entities=[\n",
      "        FoodEntity(\n",
      "            food='pork belly',\n",
      "            quantity=2,\n",
      "            unit='lb',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='green onions',\n",
      "            quantity=2,\n",
      "            unit='items',\n",
      "            physical_quality='or 3 if small',\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='fresh ginger',\n",
      "            quantity=1,\n",
      "            unit='inch',\n",
      "            physical_quality='chunk',\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='garlic',\n",
      "            quantity=2,\n",
      "            unit='cloves',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='sake',\n",
      "            quantity=2,\n",
      "            unit='⅔ cup',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='soy sauce',\n",
      "            quantity=2,\n",
      "            unit='⅔ cup',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='mirin',\n",
      "            quantity=1,\n",
      "            unit='¼ cup',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='sugar',\n",
      "            quantity=1,\n",
      "            unit='½ cup',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='water',\n",
      "            quantity=2,\n",
      "            unit='cups',\n",
      "            physical_quality='or a little more as needed',\n",
      "            color='',\n",
      "        ),\n",
      "    ],\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "pprint(entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8f1ae56-8c76-45fd-8317-4017512cbd9b",
   "metadata": {},
   "source": [
    "Using the `inspect_history` we can look at the underlying prompt that is being used"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 608,
   "id": "0f49a8db-5bad-4c31-a362-0fc526bb3a38",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "You are a food AI assistant. Your task is to extract food entities from a recipe.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Recipe: ${recipe}\n",
      "Extract Food Entities: ${extract_food_entities}. Respond with a single JSON object. JSON Schema: {\"$defs\": {\"FoodEntity\": {\"properties\": {\"food\": {\"description\": \"This can be both liquid and solid food such as meat, vegetables, alcohol, etc\", \"title\": \"Food\", \"type\": \"string\"}, \"quantity\": {\"description\": \"The exact quantity or amount of the food that should be used in the recipe\", \"title\": \"Quantity\", \"type\": \"integer\"}, \"unit\": {\"description\": \"The unit being used e.g. grams, milliliters, pounds, etc\", \"title\": \"Unit\", \"type\": \"string\"}, \"physical_quality\": {\"anyOf\": [{\"type\": \"string\"}, {\"type\": \"null\"}], \"description\": \"The characteristic of the ingredient\", \"title\": \"Physical Quality\"}, \"color\": {\"description\": \"The color of the food\", \"title\": \"Color\", \"type\": \"string\"}}, \"required\": [\"food\", \"quantity\", \"unit\", \"physical_quality\", \"color\"], \"title\": \"FoodEntity\", \"type\": \"object\"}}, \"properties\": {\"entities\": {\"items\": {\"$ref\": \"#/$defs/FoodEntity\"}, \"title\": \"Entities\", \"type\": \"array\"}}, \"required\": [\"entities\"], \"title\": \"FoodEntities\", \"type\": \"object\"}\n",
      "\n",
      "---\n",
      "\n",
      "Recipe:\n",
      "pork belly:\n",
      "{\n",
      "    \"reasoning\": \"The recipe specifies using 2 lb of pork belly as the main ingredient for the chashu pork.\",\n",
      "    \"value\": \"2 lb\",\n",
      "    \"entity\": \"pork belly\"\n",
      "}\n",
      "green onions:\n",
      "{\n",
      "    \"reasoning\": \"The recipe calls for 2 green onions, or 3 if they are small, to be used in the cooking process.\",\n",
      "    \"value\": \"2 or 3\",\n",
      "    \"entity\": \"green onions\"\n",
      "}\n",
      "fresh ginger:\n",
      "{\n",
      "    \"reasoning\": \"A 1 inch chunk of fresh ginger is required, which will give around 4 - 6 slices for the recipe.\",\n",
      "    \"value\": \"1 in\",\n",
      "    \"entity\": \"fresh ginger\"\n",
      "}\n",
      "garlic:\n",
      "{\n",
      "    \"reasoning\": \"2 cloves of garlic are needed as part of the ingredients.\",\n",
      "    \"value\": \"2 cloves\",\n",
      "    \"entity\": \"garlic\"\n",
      "}\n",
      "sake:\n",
      "{\n",
      "    \"reasoning\": \"⅔ cup of sake is used in the cooking liquid for flavor.\",\n",
      "    \"value\": \"⅔ cup\",\n",
      "    \"entity\": \"sake\"\n",
      "}\n",
      "soy sauce:\n",
      "{\n",
      "    \"reasoning\": \"⅔ cup of soy sauce is added to the cooking liquid, contributing to the dish's savory taste.\",\n",
      "    \"value\": \"⅔ cup\",\n",
      "    \"entity\": \"soy sauce\"\n",
      "}\n",
      "mirin:\n",
      "{\n",
      "    \"reasoning\": \"¼ cup of mirin is included in the recipe for sweetness and depth of flavor.\",\n",
      "    \"value\": \"¼ cup\",\n",
      "    \"entity\": \"mirin\"\n",
      "}\n",
      "sugar:\n",
      "{\n",
      "    \"reasoning\": \"½ cup of sugar is used to sweeten the cooking liquid.\",\n",
      "    \"value\": \"½ cup\",\n",
      "    \"entity\": \"sugar\"\n",
      "}\n",
      "water:\n",
      "{\n",
      "    \"reasoning\": \"2 cups of water (or a little more as needed) are required to create the cooking liquid for the pork.\",\n",
      "    \"value\": \"2 cups\",\n",
      "    \"entity\": \"water\"\n",
      "}\n",
      "\n",
      "Extract Food Entities:\u001b[32m ```json\n",
      "{\n",
      "  \"entities\": [\n",
      "    {\n",
      "      \"food\": \"pork belly\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"lb\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"green onions\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"items\",\n",
      "      \"physical_quality\": \"or 3 if small\",\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"fresh ginger\",\n",
      "      \"quantity\": 1,\n",
      "      \"unit\": \"inch\",\n",
      "      \"physical_quality\": \"chunk\",\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"garlic\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"cloves\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"sake\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"⅔ cup\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"soy sauce\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"⅔ cup\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"mirin\",\n",
      "      \"quantity\": 1,\n",
      "      \"unit\": \"¼ cup\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"sugar\",\n",
      "      \"quantity\": 1,\n",
      "      \"unit\": \"½ cup\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"water\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"cups\",\n",
      "      \"physical_quality\": \"or a little more as needed\",\n",
      "      \"color\": \"\"\n",
      "    }\n",
      "  ]\n",
      "}\n",
      "```\u001b[0m\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gpt4.inspect_history(n=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1152e59b-e051-4f97-83d5-9fe178c98476",
   "metadata": {},
   "source": [
    "### 3. Optimize program using `teleprompter` and `compile` it\n",
    "\n",
    "> For large LMs, this is primarily in the form of creating and validating good demonstrations for inclusion in your prompt(s).\n",
    "\n",
    "**Teleprompters**:\n",
    "> Teleprompters act as optimizers for DSPy programs. They take a metric and, together with the DSPy compiler, learn to bootstrap and select effective prompts for a DSPy program’s modules\n",
    "\n",
    "**Compiler**:\n",
    "> The DSPy compiler will internally trace your program and then optimize it using an optimizer (teleprompter) to maximize a given metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 609,
   "id": "54a7a202-1f1f-486c-a75f-8dc6451bdb3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate_entities(example, pred, trace=None):\n",
    "    \"\"\"Check if both objects are equal\"\"\"\n",
    "    return example.entities == pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 610,
   "id": "bd7c1410-3215-468e-8fb3-07c8b7bf1aff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create some dummy data for training\n",
    "trainset = [\n",
    "    dspy.Example(\n",
    "        recipe=\"French omelett with 2 eggs, 500grams of butter and 10 grams gruyere\", \n",
    "        entities=[\n",
    "            FoodEntity(food=\"eggs\", quantity=2, unit=\"\", physical_quality=\"\", color=\"white\"),\n",
    "            FoodEntity(food=\"butter\", quantity=500, unit=\"grams\", physical_quality=\"\", color=\"yellow\"),\n",
    "            FoodEntity(food=\"cheese\", quantity=10, unit=\"grams\", physical_quality=\"gruyer\", color=\"yellow\")\n",
    "        ]\n",
    "    ).with_inputs(\"recipe\"),\n",
    "    dspy.Example(\n",
    "        recipe=\"200 grams of Ramen noodles bowel with one pickled egg, 500grams of pork, and 1 spring onion\", \n",
    "        entities=[\n",
    "            FoodEntity(food=\"egg\", quantity=1, unit=\"\", physical_quality=\"pickled\", color=\"ivory\"),\n",
    "            FoodEntity(food=\"ramen nudles\", quantity=200, unit=\"grams\", physical_quality=\"\", color=\"yellow\"),\n",
    "            FoodEntity(food=\"spring onion\", quantity=1, unit=\"\", physical_quality=\"\", color=\"white\")\n",
    "        ]\n",
    "    ).with_inputs(\"recipe\"),\n",
    "    dspy.Example(\n",
    "        recipe=\"10 grams of dutch orange cheese, 2 liters of water, and 5 ml of ice\", \n",
    "        entities=[\n",
    "            FoodEntity(food=\"cheese\", quantity=10, unit=\"grams\", physical_quality=\"\", color=\"orange\"),\n",
    "            FoodEntity(food=\"water\", quantity=2, unit=\"liters\", physical_quality=\"translucent\", color=\"\"),\n",
    "            FoodEntity(food=\"ice\", quantity=5, unit=\"militers\", physical_quality=\"cold\", color=\"white\")\n",
    "        ]\n",
    "    ).with_inputs(\"recipe\"),\n",
    "    dspy.Example(\n",
    "        recipe=\"Pasta carbonara, 250 grams of pasta 300 grams of pancetta, \\\n",
    "        150 grams pecorino romano, 150grams parmesan cheese, 3 egg yolks\", \n",
    "        entities=[\n",
    "            FoodEntity(food=\"pasta\", quantity=250, unit=\"grams\", physical_quality=\"dried\", color=\"yellow\"),\n",
    "            FoodEntity(food=\"egg yolk\", quantity=3, unit=\"\", physical_quality=\"\", color=\"orange\"),\n",
    "            FoodEntity(food=\"pancetta\", quantity=300, unit=\"grams\", physical_quality=\"pork\", color=\"\"),\n",
    "            FoodEntity(food=\"pecorino\", quantity=150, unit=\"grams\", physical_quality=\"goat chese\", color=\"yellow\"),\n",
    "            FoodEntity(food=\"parmesan\", quantity=150, unit=\"grams\", physical_quality=\"chese\", color=\"yellow\"),\n",
    "        ]\n",
    "    ).with_inputs(\"recipe\"),\n",
    "    dspy.Example(\n",
    "        recipe=\"American pancakes with 250g flour, 1 tsp baking powder, 1 gram salt, 10g sugar, 100ml fat milk\", \n",
    "        entities=[\n",
    "            FoodEntity(food=\"flour\", quantity=250, unit=\"grams\", physical_quality=\"\", color=\"white\"),\n",
    "            FoodEntity(food=\"baking powder\", quantity=1, unit=\"tsp\", physical_quality=\"\", color=\"white\"),\n",
    "            FoodEntity(food=\"salt\", quantity=1, unit=\"grams\", physical_quality=\"salty\", color=\"white\"),\n",
    "            FoodEntity(food=\"milk\", quantity=100, unit=\"mil\", physical_quality=\"fat\", color=\"white\"),\n",
    "        ]\n",
    "    ).with_inputs(\"recipe\")\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc493060-9f93-4100-9c3c-1e6e2d3ba3ec",
   "metadata": {},
   "source": [
    "For optimizer we are using `BoostrapFewShoot`:\n",
    "> BootstrapFewShot: Uses your program to self-generate complete demonstrations for every stage of your program. Will simply use the generated demonstrations (if they pass the metric) without any further optimization. Advanced: Supports using a teacher program (a different DSPy program that has compatible structure) and a teacher LM, for harder tasks.\n",
    "\n",
    "And when to use this `optimizer`:\n",
    "> If you have very little data, e.g. 10 examples of your task, use BootstrapFewShot.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 616,
   "id": "93bc091b-a09b-4ce1-b37d-d02d2c81684d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 155.87it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 0 full traces after 5 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from dspy.teleprompt import BootstrapFewShot\n",
    "\n",
    "teleprompter = BootstrapFewShot(metric=validate_entities)\n",
    "\n",
    "compiled_ner = teleprompter.compile(ExtractFoodEntitiesV2(), trainset=trainset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 619,
   "id": "6f5c1e6a-16b1-4e6e-afd7-267f67c14382",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FoodEntities(\n",
      "    entities=[\n",
      "        FoodEntity(\n",
      "            food='pork belly',\n",
      "            quantity=2,\n",
      "            unit='lb',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='green onions',\n",
      "            quantity=2,\n",
      "            unit='items',\n",
      "            physical_quality='or 3 if small',\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='fresh ginger',\n",
      "            quantity=1,\n",
      "            unit='inch',\n",
      "            physical_quality='chunk',\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='garlic',\n",
      "            quantity=2,\n",
      "            unit='cloves',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='sake',\n",
      "            quantity=2,\n",
      "            unit='⅔ cup',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='soy sauce',\n",
      "            quantity=2,\n",
      "            unit='⅔ cup',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='mirin',\n",
      "            quantity=1,\n",
      "            unit='¼ cup',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='sugar',\n",
      "            quantity=1,\n",
      "            unit='½ cup',\n",
      "            physical_quality=None,\n",
      "            color='',\n",
      "        ),\n",
      "        FoodEntity(\n",
      "            food='water',\n",
      "            quantity=2,\n",
      "            unit='cups',\n",
      "            physical_quality='or a little more as needed',\n",
      "            color='',\n",
      "        ),\n",
      "    ],\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "pprint(compiled_ner(recipe=train_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 622,
   "id": "ae3c91e4-0f26-493b-98e3-99c209afc2fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "You are a food AI assistant. Your task is to extract food entities from a recipe.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Recipe: ${recipe}\n",
      "Extract Food Entities: ${extract_food_entities}. Respond with a single JSON object. JSON Schema: {\"$defs\": {\"FoodEntity\": {\"properties\": {\"food\": {\"description\": \"This can be both liquid and solid food such as meat, vegetables, alcohol, etc\", \"title\": \"Food\", \"type\": \"string\"}, \"quantity\": {\"description\": \"The exact quantity or amount of the food that should be used in the recipe\", \"title\": \"Quantity\", \"type\": \"integer\"}, \"unit\": {\"description\": \"The unit being used e.g. grams, milliliters, pounds, etc\", \"title\": \"Unit\", \"type\": \"string\"}, \"physical_quality\": {\"anyOf\": [{\"type\": \"string\"}, {\"type\": \"null\"}], \"description\": \"The characteristic of the ingredient\", \"title\": \"Physical Quality\"}, \"color\": {\"description\": \"The color of the food\", \"title\": \"Color\", \"type\": \"string\"}}, \"required\": [\"food\", \"quantity\", \"unit\", \"physical_quality\", \"color\"], \"title\": \"FoodEntity\", \"type\": \"object\"}}, \"properties\": {\"entities\": {\"items\": {\"$ref\": \"#/$defs/FoodEntity\"}, \"title\": \"Entities\", \"type\": \"array\"}}, \"required\": [\"entities\"], \"title\": \"FoodEntities\", \"type\": \"object\"}\n",
      "\n",
      "---\n",
      "\n",
      "Recipe:\n",
      "pork belly:\n",
      "{\n",
      "    \"reasoning\": \"The recipe specifies using 2 lb of pork belly as the main ingredient for the chashu pork.\",\n",
      "    \"value\": \"2 lb\",\n",
      "    \"entity\": \"pork belly\"\n",
      "}\n",
      "green onions:\n",
      "{\n",
      "    \"reasoning\": \"The recipe calls for 2 green onions, or 3 if they are small, to be used in the cooking process.\",\n",
      "    \"value\": \"2 or 3\",\n",
      "    \"entity\": \"green onions\"\n",
      "}\n",
      "fresh ginger:\n",
      "{\n",
      "    \"reasoning\": \"A 1 inch chunk of fresh ginger is required, which will give around 4 - 6 slices for the recipe.\",\n",
      "    \"value\": \"1 in\",\n",
      "    \"entity\": \"fresh ginger\"\n",
      "}\n",
      "garlic:\n",
      "{\n",
      "    \"reasoning\": \"2 cloves of garlic are needed as part of the ingredients.\",\n",
      "    \"value\": \"2 cloves\",\n",
      "    \"entity\": \"garlic\"\n",
      "}\n",
      "sake:\n",
      "{\n",
      "    \"reasoning\": \"⅔ cup of sake is used in the cooking liquid for flavor.\",\n",
      "    \"value\": \"⅔ cup\",\n",
      "    \"entity\": \"sake\"\n",
      "}\n",
      "soy sauce:\n",
      "{\n",
      "    \"reasoning\": \"⅔ cup of soy sauce is added to the cooking liquid, contributing to the dish's savory taste.\",\n",
      "    \"value\": \"⅔ cup\",\n",
      "    \"entity\": \"soy sauce\"\n",
      "}\n",
      "mirin:\n",
      "{\n",
      "    \"reasoning\": \"¼ cup of mirin is included in the recipe for sweetness and depth of flavor.\",\n",
      "    \"value\": \"¼ cup\",\n",
      "    \"entity\": \"mirin\"\n",
      "}\n",
      "sugar:\n",
      "{\n",
      "    \"reasoning\": \"½ cup of sugar is used to sweeten the cooking liquid.\",\n",
      "    \"value\": \"½ cup\",\n",
      "    \"entity\": \"sugar\"\n",
      "}\n",
      "water:\n",
      "{\n",
      "    \"reasoning\": \"2 cups of water (or a little more as needed) are required to create the cooking liquid for the pork.\",\n",
      "    \"value\": \"2 cups\",\n",
      "    \"entity\": \"water\"\n",
      "}\n",
      "\n",
      "Extract Food Entities:\u001b[32m ```json\n",
      "{\n",
      "  \"entities\": [\n",
      "    {\n",
      "      \"food\": \"pork belly\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"lb\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"green onions\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"items\",\n",
      "      \"physical_quality\": \"or 3 if small\",\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"fresh ginger\",\n",
      "      \"quantity\": 1,\n",
      "      \"unit\": \"inch\",\n",
      "      \"physical_quality\": \"chunk\",\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"garlic\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"cloves\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"sake\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"⅔ cup\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"soy sauce\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"⅔ cup\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"mirin\",\n",
      "      \"quantity\": 1,\n",
      "      \"unit\": \"¼ cup\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"sugar\",\n",
      "      \"quantity\": 1,\n",
      "      \"unit\": \"½ cup\",\n",
      "      \"physical_quality\": null,\n",
      "      \"color\": \"\"\n",
      "    },\n",
      "    {\n",
      "      \"food\": \"water\",\n",
      "      \"quantity\": 2,\n",
      "      \"unit\": \"cups\",\n",
      "      \"physical_quality\": \"or a little more as needed\",\n",
      "      \"color\": \"\"\n",
      "    }\n",
      "  ]\n",
      "}\n",
      "```\u001b[0m\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gpt4.inspect_history(n=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d62f3e53-2a7d-49e9-aaf8-63e60538a229",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}