examples/notebooks/GPT4RegressionTesting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0a13ddc8",
   "metadata": {},
   "source": [
    "# GPT-4 regression testing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "780dc3bf",
   "metadata": {},
   "source": [
    "Did GPT-4 get worse? Run this notebook to find out.\n",
    "\n",
    "We'll use a structured output validation to compare the most recent version of GPT-4 to older frozen versions on a few tasks.\n",
    "\n",
    "If you want to run this notebook in Google Colab, see [here](https://colab.research.google.com/github/hegelai/prompttools/blob/main/examples/notebooks/GPT4RegressionTesting.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "623f0cfe",
   "metadata": {},
   "source": [
    "## Installations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52881369",
   "metadata": {},
   "source": [
    "You can setup prompttools either by installing via `pip` or using `python setup.py develop` in the root of this repo. Either way, you'll need to restart the kernel after the package is installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "885dabeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install --quiet --force-reinstall prompttools"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eac35f8",
   "metadata": {},
   "source": [
    "## Setup imports and API keys"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5edba05a",
   "metadata": {},
   "source": [
    "Next, we'll need to set our API keys. Since we want to use GPT-4 for auto-eval, we need to set that one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ed4e635e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"DEBUG\"] = \"\"\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "842f1e47",
   "metadata": {},
   "source": [
    "Then we'll import the relevant `prompttools` modules to setup our experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "beaa70a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Dict, List, Tuple\n",
    "from prompttools.experiment import OpenAIChatExperiment\n",
    "from prompttools.selector.prompt_selector import PromptSelector"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "622dea9a",
   "metadata": {},
   "source": [
    "## Run an experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cd0bae8",
   "metadata": {},
   "source": [
    "To set up this experiment, we'll use a `PromptSelector`. This makes it easy to format prompts for chat models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2788d49f",
   "metadata": {},
   "outputs": [],
   "source": [
    "instructions = [\n",
    "    \"\"\"\n",
    "You are a helpful assistant.\n",
    "\"\"\",\n",
    "    \"\"\"\n",
    "You are a math tutor. When given a math question, talk through each step \n",
    "of your thought process before arriving at the answer. \n",
    "\"\"\",\n",
    "]\n",
    "inputs = [\n",
    "    \"\"\"\n",
    "Is 17 a prime number?\n",
    "\"\"\",\n",
    "    \"\"\"\n",
    "Is 45 a prime number?\n",
    "\"\"\",\n",
    "    \"\"\"\n",
    "Is 17077 a prime number?\n",
    "\"\"\",\n",
    "    \"\"\"\n",
    "Is 26 a prime number?\n",
    "\"\"\",\n",
    "]\n",
    "selectors = [PromptSelector(instructions[i], inputs[j]) for i in range(len(instructions)) for j in range(len(inputs))]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3babfe5a",
   "metadata": {},
   "source": [
    "Next, we create our test inputs. We can iterate over models, inputs, and configurations like temperature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fe83830",
   "metadata": {},
   "outputs": [],
   "source": [
    "models = [\"gpt-4-0314\", \"gpt-4-0613\", \"gpt-4\"]\n",
    "temperatures = [0.0]\n",
    "openai_experiment = OpenAIChatExperiment(models, selectors, temperature=temperatures)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c3162e6",
   "metadata": {},
   "source": [
    "After that - we can run our experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f22ebd7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "openai_experiment.prepare()\n",
    "openai_experiment.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b545e33",
   "metadata": {},
   "outputs": [],
   "source": [
    "openai_experiment.visualize()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ceb662a",
   "metadata": {},
   "source": [
    "Finally, we define an evaluation function that can be used to evaluate outputs across different models. Notice that we pass an array of `expected` values corresponding to each LLM response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ddbb951",
   "metadata": {},
   "outputs": [],
   "source": [
    "from prompttools.utils import autoeval_from_expected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e80dfeec",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "openai_experiment.evaluate(\"auto-evaluation\", autoeval_from_expected.evaluate, expected=[\"YES\", \"NO\"] * 12)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "974d6065",
   "metadata": {},
   "source": [
    "Finally, we can visualize the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d09c18e",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "openai_experiment.visualize()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36fa7a07",
   "metadata": {},
   "source": [
    "Finally, we can aggregate the metrics for each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ee8dc43",
   "metadata": {},
   "outputs": [],
   "source": [
    "openai_experiment.aggregate(metric_name=\"auto-evaluation\", column_name=\"model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c90958d",
   "metadata": {},
   "outputs": [],
   "source": [
    "openai_experiment.aggregate(metric_name=\"latency\", column_name=\"model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c6b9e58f",
   "metadata": {},
   "outputs": [],
   "source": [
    "instructions = [\n",
    "    \"\"\"\n",
    "You are a math tutor. When given a math question, talk through each step \n",
    "of your thought process before arriving at the answer. \n",
    "\"\"\"\n",
    "]\n",
    "inputs = [\n",
    "    \"\"\"\n",
    "Is 17077 a prime number?\n",
    "\"\"\",\n",
    "    \"\"\"\n",
    "Is 26 a prime number?\n",
    "\"\"\",\n",
    "]\n",
    "selectors = [PromptSelector(instructions[i], inputs[j]) for i in range(len(instructions)) for j in range(len(inputs))]\n",
    "models = [\"gpt-4-0314\", \"gpt-4-0613\", \"gpt-4\"]\n",
    "temperatures = [0.0]\n",
    "openai_experiment = OpenAIChatExperiment(models, selectors, temperature=temperatures)\n",
    "openai_experiment.prepare()\n",
    "openai_experiment.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6d85051d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>messages</th>\n",
       "      <th>response(s)</th>\n",
       "      <th>latency</th>\n",
       "      <th>model</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[{'role': 'system', 'content': '\n",
       "You are a math tutor. When given a math question, talk through each step \n",
       "of your thought process before arriving at the answer. \n",
       "'}, {'role': 'user', 'content': '\n",
       "Is 17077 a prime number?\n",
       "'}]</td>\n",
       "      <td>To determine if 17077 is a prime number, we need to check if it has any divisors other than 1 and itself. A prime number is a number that can only be divided by 1 and itself without leaving a remainder.\\n\\nStep 1: Check for divisibility by 2\\nSince 17077 is an odd number (it ends in 7), it is not divisible by 2.\\n\\nStep 2: Check for divisibility by 3\\nTo check for divisibility by 3, we can add up the digits of the number (1 + 7 + 0 + 7 + 7 = 22) and see if the sum is divisible by 3. Since 22 is not divisible by 3, 17077 is not divisible by 3.\\n\\nStep 3: Check for divisibility by other prime numbers\\nWe can now check for divisibility by other prime numbers up to the square root of 17077. The square root of 17077 is approximately 130.6, so we only need to check for divisibility by prime numbers up to 130.\\n\\nWe've already checked for divisibility by 2 and 3, so we'll continue with the next prime numbers: 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127.\\n\\nStep 4: Check for divisibility by each prime number\\n17077 is not divisible by 5 (it doesn't end in 0 or 5).\\n17077 is not divisible by 7 (17077 ÷ 7 ≈ 2439.57).\\n17077 is not divisible by 11 (17077 ÷ 11 ≈ 1552.45).\\n17077 is not divisible by 13 (17077 ÷ 13 ≈ 1313.62).\\n\\nHowever, when we check for divisibility by 17, we find that 17077 ÷ 17 = 1004.53, which is very close to a whole number. In fact, 17077 ÷ 17 = 1005, with a small error in the previous calculation. So, 17077 is divisible by 17 (17 * 1005 = 17077).\\n\\nSince 17077 has divisors other than 1 and itself (17 and 1005), it is not a prime number.</td>\n",
       "      <td>22.004887</td>\n",
       "      <td>gpt-4-0314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[{'role': 'system', 'content': '\n",
       "You are a math tutor. When given a math question, talk through each step \n",
       "of your thought process before arriving at the answer. \n",
       "'}, {'role': 'user', 'content': '\n",
       "Is 26 a prime number?\n",
       "'}]</td>\n",
       "      <td>To determine if 26 is a prime number, we need to check if it has any factors other than 1 and itself. A prime number is a number that can only be divided by 1 and itself without leaving a remainder.\\n\\nStep 1: Identify the smallest prime number, which is 2.\\n\\nStep 2: Check if 26 is divisible by 2. To do this, divide 26 by 2:\\n26 ÷ 2 = 13\\n\\nSince 26 can be divided by 2 without leaving a remainder, it means that 26 has factors other than 1 and itself (2 and 13).\\n\\nTherefore, 26 is not a prime number.</td>\n",
       "      <td>5.219352</td>\n",
       "      <td>gpt-4-0314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[{'role': 'system', 'content': '\n",
       "You are a math tutor. When given a math question, talk through each step \n",
       "of your thought process before arriving at the answer. \n",
       "'}, {'role': 'user', 'content': '\n",
       "Is 17077 a prime number?\n",
       "'}]</td>\n",
       "      <td>To determine if a number is prime, we need to check if it has any divisors other than 1 and itself. A prime number is a number that has exactly two distinct positive divisors: 1 and itself.\\n\\nThe first step in determining if a number is prime is to find its square root. The square root of 17077 is approximately 130.68. This is because any factor of 17077 must be less than or equal to its square root. \\n\\nNext, we check if 17077 can be divided evenly by any prime numbers less than 130.68. We start with the smallest prime number, 2, and continue with 3, 5, 7, 11, 13, and so on.\\n\\nIf 17077 can be divided evenly (i.e., with no remainder) by any of these numbers, then it is not a prime number. If it cannot be divided evenly by any of these numbers, then it is a prime number.\\n\\nLet's start with 2. Since 17077 is an odd number, it cannot be divided evenly by 2. \\n\\nNext, let's try 3. The sum of the digits of 17077 is 1+7+0+7+7=22, which is not divisible by 3, so 17077 is not divisible by 3.\\n\\nWe continue this process with the prime numbers 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127. \\n\\nIf 17077 is not divisible by any of these prime numbers, then it is a prime number. However, when we reach 127, we find that 17077 divided by 127 equals 134 with no remainder. \\n\\nTherefore, 17077 is not a prime number because it has divisors other than 1 and itself (namely, 127 and 134).</td>\n",
       "      <td>29.593874</td>\n",
       "      <td>gpt-4-0613</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[{'role': 'system', 'content': '\n",
       "You are a math tutor. When given a math question, talk through each step \n",
       "of your thought process before arriving at the answer. \n",
       "'}, {'role': 'user', 'content': '\n",
       "Is 26 a prime number?\n",
       "'}]</td>\n",
       "      <td>To determine if a number is prime, we need to understand what a prime number is. A prime number is a number that has exactly two distinct positive divisors: 1 and itself. In other words, the only way to multiply two positive numbers to get a prime number is to multiply 1 by the number itself.\\n\\nSo, let's consider the number 26. \\n\\nThe first step is to see if it can be divided evenly by 2, which is the smallest prime number. If it can, then it's not a prime number. \\n\\n26 divided by 2 equals 13, which is a whole number. This means that 26 can be divided evenly by numbers other than 1 and itself, specifically 2 and 13.\\n\\nTherefore, 26 is not a prime number because it has more than two distinct positive divisors: 1, 2, 13, and 26.</td>\n",
       "      <td>12.799341</td>\n",
       "      <td>gpt-4-0613</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[{'role': 'system', 'content': '\n",
       "You are a math tutor. When given a math question, talk through each step \n",
       "of your thought process before arriving at the answer. \n",
       "'}, {'role': 'user', 'content': '\n",
       "Is 17077 a prime number?\n",
       "'}]</td>\n",
       "      <td>To determine if a number is prime, we need to check if it has any divisors other than 1 and itself. A prime number is a number that has exactly two distinct positive divisors: 1 and itself. \\n\\nFor the number 17077, we can start by checking if it's divisible by the smallest prime numbers like 2, 3, 5, 7, 11, 13, and so on. \\n\\n1. Check if it's divisible by 2: All even numbers are divisible by 2. Since 17077 is an odd number, it's not divisible by 2.\\n\\n2. Check if it's divisible by 3: A number is divisible by 3 if the sum of its digits is divisible by 3. The sum of the digits in 17077 is 1+7+0+7+7 = 22, which is not divisible by 3.\\n\\n3. Check if it's divisible by 5: A number is divisible by 5 if its last digit is either 0 or 5. The last digit of 17077 is 7, so it's not divisible by 5.\\n\\n4. Check if it's divisible by 7: There's no simple rule for divisibility by 7. We would have to perform the actual division to check. However, before we do that, there's a more efficient way to check for prime numbers.\\n\\nWe only need to check divisibility up to the square root of the number. The square root of 17077 is approximately 130.67. So we only need to check divisibility by primes up to 131.\\n\\n5. Check if it's divisible by primes up to 131: This would involve checking divisibility by 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, and 131. \\n\\nUpon checking, we find that 17077 is divisible by 131 (17077 ÷ 131 = 130). Therefore, 17077 is not a prime number because it has more divisors than just 1 and itself.</td>\n",
       "      <td>35.819862</td>\n",
       "      <td>gpt-4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[{'role': 'system', 'content': '\n",
       "You are a math tutor. When given a math question, talk through each step \n",
       "of your thought process before arriving at the answer. \n",
       "'}, {'role': 'user', 'content': '\n",
       "Is 26 a prime number?\n",
       "'}]</td>\n",
       "      <td>To determine if a number is prime, we need to understand what a prime number is. A prime number is a number that has exactly two distinct positive divisors: 1 and itself. In other words, the only way to multiply two numbers to get a prime number is to multiply 1 by the number itself.\\n\\nNow, let's consider the number 26. \\n\\nThe first step is to see if 26 can be divided evenly by 2, which is the smallest prime number. If it can, then 26 is not a prime number. \\n\\n26 divided by 2 equals 13, which is an integer with no remainder. This means that 26 can be divided evenly by numbers other than 1 and itself (in this case, 2 and 13). \\n\\nTherefore, 26 is not a prime number.</td>\n",
       "      <td>12.498451</td>\n",
       "      <td>gpt-4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                            messages  \\\n",
       "0  [{'role': 'system', 'content': '\n",
       "You are a math tutor. When given a math question, talk through each step \n",
       "of your thought process before arriving at the answer. \n",
       "'}, {'role': 'user', 'content': '\n",
       "Is 17077 a prime number?\n",
       "'}]   \n",
       "1  [{'role': 'system', 'content': '\n",
       "You are a math tutor. When given a math question, talk through each step \n",
       "of your thought process before arriving at the answer. \n",
       "'}, {'role': 'user', 'content': '\n",
       "Is 26 a prime number?\n",
       "'}]      \n",
       "2  [{'role': 'system', 'content': '\n",
       "You are a math tutor. When given a math question, talk through each step \n",
       "of your thought process before arriving at the answer. \n",
       "'}, {'role': 'user', 'content': '\n",
       "Is 17077 a prime number?\n",
       "'}]   \n",
       "3  [{'role': 'system', 'content': '\n",
       "You are a math tutor. When given a math question, talk through each step \n",
       "of your thought process before arriving at the answer. \n",
       "'}, {'role': 'user', 'content': '\n",
       "Is 26 a prime number?\n",
       "'}]      \n",
       "4  [{'role': 'system', 'content': '\n",
       "You are a math tutor. When given a math question, talk through each step \n",
       "of your thought process before arriving at the answer. \n",
       "'}, {'role': 'user', 'content': '\n",
       "Is 17077 a prime number?\n",
       "'}]   \n",
       "5  [{'role': 'system', 'content': '\n",
       "You are a math tutor. When given a math question, talk through each step \n",
       "of your thought process before arriving at the answer. \n",
       "'}, {'role': 'user', 'content': '\n",
       "Is 26 a prime number?\n",
       "'}]      \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          response(s)  \\\n",
       "0  To determine if 17077 is a prime number, we need to check if it has any divisors other than 1 and itself. A prime number is a number that can only be divided by 1 and itself without leaving a remainder.\\n\\nStep 1: Check for divisibility by 2\\nSince 17077 is an odd number (it ends in 7), it is not divisible by 2.\\n\\nStep 2: Check for divisibility by 3\\nTo check for divisibility by 3, we can add up the digits of the number (1 + 7 + 0 + 7 + 7 = 22) and see if the sum is divisible by 3. Since 22 is not divisible by 3, 17077 is not divisible by 3.\\n\\nStep 3: Check for divisibility by other prime numbers\\nWe can now check for divisibility by other prime numbers up to the square root of 17077. The square root of 17077 is approximately 130.6, so we only need to check for divisibility by prime numbers up to 130.\\n\\nWe've already checked for divisibility by 2 and 3, so we'll continue with the next prime numbers: 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127.\\n\\nStep 4: Check for divisibility by each prime number\\n17077 is not divisible by 5 (it doesn't end in 0 or 5).\\n17077 is not divisible by 7 (17077 ÷ 7 ≈ 2439.57).\\n17077 is not divisible by 11 (17077 ÷ 11 ≈ 1552.45).\\n17077 is not divisible by 13 (17077 ÷ 13 ≈ 1313.62).\\n\\nHowever, when we check for divisibility by 17, we find that 17077 ÷ 17 = 1004.53, which is very close to a whole number. In fact, 17077 ÷ 17 = 1005, with a small error in the previous calculation. So, 17077 is divisible by 17 (17 * 1005 = 17077).\\n\\nSince 17077 has divisors other than 1 and itself (17 and 1005), it is not a prime number.   \n",
       "1  To determine if 26 is a prime number, we need to check if it has any factors other than 1 and itself. A prime number is a number that can only be divided by 1 and itself without leaving a remainder.\\n\\nStep 1: Identify the smallest prime number, which is 2.\\n\\nStep 2: Check if 26 is divisible by 2. To do this, divide 26 by 2:\\n26 ÷ 2 = 13\\n\\nSince 26 can be divided by 2 without leaving a remainder, it means that 26 has factors other than 1 and itself (2 and 13).\\n\\nTherefore, 26 is not a prime number.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "2  To determine if a number is prime, we need to check if it has any divisors other than 1 and itself. A prime number is a number that has exactly two distinct positive divisors: 1 and itself.\\n\\nThe first step in determining if a number is prime is to find its square root. The square root of 17077 is approximately 130.68. This is because any factor of 17077 must be less than or equal to its square root. \\n\\nNext, we check if 17077 can be divided evenly by any prime numbers less than 130.68. We start with the smallest prime number, 2, and continue with 3, 5, 7, 11, 13, and so on.\\n\\nIf 17077 can be divided evenly (i.e., with no remainder) by any of these numbers, then it is not a prime number. If it cannot be divided evenly by any of these numbers, then it is a prime number.\\n\\nLet's start with 2. Since 17077 is an odd number, it cannot be divided evenly by 2. \\n\\nNext, let's try 3. The sum of the digits of 17077 is 1+7+0+7+7=22, which is not divisible by 3, so 17077 is not divisible by 3.\\n\\nWe continue this process with the prime numbers 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127. \\n\\nIf 17077 is not divisible by any of these prime numbers, then it is a prime number. However, when we reach 127, we find that 17077 divided by 127 equals 134 with no remainder. \\n\\nTherefore, 17077 is not a prime number because it has divisors other than 1 and itself (namely, 127 and 134).                                                                                                                                                                                           \n",
       "3  To determine if a number is prime, we need to understand what a prime number is. A prime number is a number that has exactly two distinct positive divisors: 1 and itself. In other words, the only way to multiply two positive numbers to get a prime number is to multiply 1 by the number itself.\\n\\nSo, let's consider the number 26. \\n\\nThe first step is to see if it can be divided evenly by 2, which is the smallest prime number. If it can, then it's not a prime number. \\n\\n26 divided by 2 equals 13, which is a whole number. This means that 26 can be divided evenly by numbers other than 1 and itself, specifically 2 and 13.\\n\\nTherefore, 26 is not a prime number because it has more than two distinct positive divisors: 1, 2, 13, and 26.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \n",
       "4  To determine if a number is prime, we need to check if it has any divisors other than 1 and itself. A prime number is a number that has exactly two distinct positive divisors: 1 and itself. \\n\\nFor the number 17077, we can start by checking if it's divisible by the smallest prime numbers like 2, 3, 5, 7, 11, 13, and so on. \\n\\n1. Check if it's divisible by 2: All even numbers are divisible by 2. Since 17077 is an odd number, it's not divisible by 2.\\n\\n2. Check if it's divisible by 3: A number is divisible by 3 if the sum of its digits is divisible by 3. The sum of the digits in 17077 is 1+7+0+7+7 = 22, which is not divisible by 3.\\n\\n3. Check if it's divisible by 5: A number is divisible by 5 if its last digit is either 0 or 5. The last digit of 17077 is 7, so it's not divisible by 5.\\n\\n4. Check if it's divisible by 7: There's no simple rule for divisibility by 7. We would have to perform the actual division to check. However, before we do that, there's a more efficient way to check for prime numbers.\\n\\nWe only need to check divisibility up to the square root of the number. The square root of 17077 is approximately 130.67. So we only need to check divisibility by primes up to 131.\\n\\n5. Check if it's divisible by primes up to 131: This would involve checking divisibility by 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, and 131. \\n\\nUpon checking, we find that 17077 is divisible by 131 (17077 ÷ 131 = 130). Therefore, 17077 is not a prime number because it has more divisors than just 1 and itself.                                                           \n",
       "5  To determine if a number is prime, we need to understand what a prime number is. A prime number is a number that has exactly two distinct positive divisors: 1 and itself. In other words, the only way to multiply two numbers to get a prime number is to multiply 1 by the number itself.\\n\\nNow, let's consider the number 26. \\n\\nThe first step is to see if 26 can be divided evenly by 2, which is the smallest prime number. If it can, then 26 is not a prime number. \\n\\n26 divided by 2 equals 13, which is an integer with no remainder. This means that 26 can be divided evenly by numbers other than 1 and itself (in this case, 2 and 13). \\n\\nTherefore, 26 is not a prime number.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \n",
       "\n",
       "     latency       model  \n",
       "0  22.004887  gpt-4-0314  \n",
       "1  5.219352   gpt-4-0314  \n",
       "2  29.593874  gpt-4-0613  \n",
       "3  12.799341  gpt-4-0613  \n",
       "4  35.819862  gpt-4       \n",
       "5  12.498451  gpt-4       "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "openai_experiment.visualize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6747248e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAaz0lEQVR4nO3de2zV9f348VcRKKC0WG6loVy84g1cnCiiqBMFthhEohPnAs656UDFxgtseGPfyHRGxAwlTieaDN1IFLyNBXGCF3CKEucNFSXggM7LaIFpQfr5/bEfZ1ZRKbTvcsrjkXwSzue8z+fzPuRNefb0c3oKsizLAgAgkRZNPQEAYM8iPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AIKmWTT2BL6utrY01a9ZE+/bto6CgoKmnAwDsgCzLYsOGDVFWVhYtWnzzaxu7XXysWbMmysvLm3oaAMBOWL16dXTv3v0bx9QrPqZMmRIPPfRQvPXWW9G2bds47rjj4qabboqDDz44N+akk06KhQsX1nncz3/+85gxY8YOnaN9+/a5yRcVFdVnegBAE6muro7y8vLc/+PfpF7xsXDhwhg7dmwcffTR8fnnn8cvf/nLOO200+KNN96IvffeOzfuwgsvjMmTJ+dut2vXbofPse1HLUVFReIDAPLMjlwyUa/4mDdvXp3bM2fOjC5dusTSpUtj0KBBuf3t2rWL0tLS+hwaANhD7NK7XaqqqiIioqSkpM7+P/7xj9GpU6c4/PDDY+LEifGf//zna49RU1MT1dXVdTYAoPna6QtOa2trY/z48TFw4MA4/PDDc/vPPffc6NmzZ5SVlcWrr74aV199dSxfvjweeuih7R5nypQpccMNN+zsNACAPFOQZVm2Mw+8+OKL4y9/+Us8++yz33hV61NPPRWnnHJKvPvuu7H//vt/5f6ampqoqanJ3d52wUpVVZVrPgAgT1RXV0dxcfEO/f+9U698jBs3Lh577LFYtGjRt76d5phjjomI+Nr4KCwsjMLCwp2ZBgCQh+oVH1mWxSWXXBIPP/xwPP3009G7d+9vfcyyZcsiIqJbt247NUEAoHmpV3yMHTs2Zs2aFXPnzo327dvHunXrIiKiuLg42rZtGytWrIhZs2bF97///ejYsWO8+uqrcfnll8egQYOib9++jfIEAID8Uq9rPr7uvbv33ntvjBkzJlavXh3nnXdevPbaa7Fp06YoLy+PESNGxKRJk3b4+o36/MwIANg9NNo1H9/WKeXl5V/57aYAAF/kU20BgKTEBwCQlPgAAJISHwBAUuIDAEhqpz/bJV/1mvB4U0+BJrbyNz9o6ikA7NG88gEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKRaNvUEYE/Ta8LjTT0FmtjK3/ygqacATcorHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACRVr/iYMmVKHH300dG+ffvo0qVLnHHGGbF8+fI6Yz777LMYO3ZsdOzYMfbZZ58YOXJkVFZWNuikAYD8Va/4WLhwYYwdOzaWLFkS8+fPjy1btsRpp50WmzZtyo25/PLL49FHH43Zs2fHwoULY82aNXHmmWc2+MQBgPzUsj6D582bV+f2zJkzo0uXLrF06dIYNGhQVFVVxT333BOzZs2K733vexERce+998YhhxwSS5YsiWOPPbbhZg4A5KVduuajqqoqIiJKSkoiImLp0qWxZcuWGDx4cG5Mnz59okePHrF48eLtHqOmpiaqq6vrbABA87XT8VFbWxvjx4+PgQMHxuGHHx4REevWrYvWrVtHhw4d6ozt2rVrrFu3brvHmTJlShQXF+e28vLynZ0SAJAHdjo+xo4dG6+99lo8+OCDuzSBiRMnRlVVVW5bvXr1Lh0PANi91euaj23GjRsXjz32WCxatCi6d++e219aWhqbN2+O9evX13n1o7KyMkpLS7d7rMLCwigsLNyZaQAAeaher3xkWRbjxo2Lhx9+OJ566qno3bt3nfuPOuqoaNWqVSxYsCC3b/ny5bFq1aoYMGBAw8wYAMhr9XrlY+zYsTFr1qyYO3dutG/fPncdR3FxcbRt2zaKi4vjggsuiIqKiigpKYmioqK45JJLYsCAAd7pAgBERD3j484774yIiJNOOqnO/nvvvTfGjBkTERFTp06NFi1axMiRI6OmpiaGDBkSd9xxR4NMFgDIf/WKjyzLvnVMmzZtYvr06TF9+vSdnhQA0Hz5bBcAICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFL1jo9FixbF6aefHmVlZVFQUBBz5sypc/+YMWOioKCgzjZ06NCGmi8AkOfqHR+bNm2Kfv36xfTp0792zNChQ2Pt2rW57YEHHtilSQIAzUfL+j5g2LBhMWzYsG8cU1hYGKWlpTs9KQCg+WqUaz6efvrp6NKlSxx88MFx8cUXx8cff/y1Y2tqaqK6urrOBgA0Xw0eH0OHDo37778/FixYEDfddFMsXLgwhg0bFlu3bt3u+ClTpkRxcXFuKy8vb+gpAQC7kXr/2OXbnHPOObk/H3HEEdG3b9/Yf//94+mnn45TTjnlK+MnTpwYFRUVudvV1dUCBACasUZ/q+1+++0XnTp1infffXe79xcWFkZRUVGdDQBovho9Pj744IP4+OOPo1u3bo19KgAgD9T7xy4bN26s8yrG+++/H8uWLYuSkpIoKSmJG264IUaOHBmlpaWxYsWKuOqqq+KAAw6IIUOGNOjEAYD8VO/4eOmll+Lkk0/O3d52vcbo0aPjzjvvjFdffTXuu+++WL9+fZSVlcVpp50Wv/71r6OwsLDhZg0A5K16x8dJJ50UWZZ97f1//etfd2lCAEDz5rNdAICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASKren2oLQH7rNeHxpp4CTWzlb37QpOf3ygcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICk6h0fixYtitNPPz3KysqioKAg5syZU+f+LMvi2muvjW7dukXbtm1j8ODB8c477zTUfAGAPFfv+Ni0aVP069cvpk+fvt37b7755rj99ttjxowZ8cILL8Tee+8dQ4YMic8++2yXJwsA5L+W9X3AsGHDYtiwYdu9L8uyuO2222LSpEkxfPjwiIi4//77o2vXrjFnzpw455xzdm22AEDea9BrPt5///1Yt25dDB48OLevuLg4jjnmmFi8ePF2H1NTUxPV1dV1NgCg+WrQ+Fi3bl1ERHTt2rXO/q5du+bu+7IpU6ZEcXFxbisvL2/IKQEAu5kmf7fLxIkTo6qqKretXr26qacEADSiBo2P0tLSiIiorKyss7+ysjJ335cVFhZGUVFRnQ0AaL4aND569+4dpaWlsWDBgty+6urqeOGFF2LAgAENeSoAIE/V+90uGzdujHfffTd3+/33349ly5ZFSUlJ9OjRI8aPHx//93//FwceeGD07t07rrnmmigrK4szzjijIecNAOSpesfHSy+9FCeffHLudkVFRUREjB49OmbOnBlXXXVVbNq0KX72s5/F+vXr4/jjj4958+ZFmzZtGm7WAEDeqnd8nHTSSZFl2dfeX1BQEJMnT47Jkyfv0sQAgOapyd/tAgDsWcQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEiqwePj+uuvj4KCgjpbnz59Gvo0AECeatkYBz3ssMPiySef/N9JWjbKaQCAPNQoVdCyZcsoLS1tjEMDAHmuUa75eOedd6KsrCz222+/+NGPfhSrVq362rE1NTVRXV1dZwMAmq8Gj49jjjkmZs6cGfPmzYs777wz3n///TjhhBNiw4YN2x0/ZcqUKC4uzm3l5eUNPSUAYDfS4PExbNiwOOuss6Jv374xZMiQeOKJJ2L9+vXx5z//ebvjJ06cGFVVVblt9erVDT0lAGA30uhXgnbo0CEOOuigePfdd7d7f2FhYRQWFjb2NACA3USj/56PjRs3xooVK6Jbt26NfSoAIA80eHxcccUVsXDhwli5cmU8//zzMWLEiNhrr71i1KhRDX0qACAPNfiPXT744IMYNWpUfPzxx9G5c+c4/vjjY8mSJdG5c+eGPhUAkIcaPD4efPDBhj4kANCM+GwXACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUuIDAEhKfAAASYkPACAp8QEAJCU+AICkxAcAkJT4AACSEh8AQFLiAwBISnwAAEmJDwAgKfEBACQlPgCApMQHAJCU+AAAkhIfAEBSjRYf06dPj169ekWbNm3imGOOib///e+NdSoAII80Snz86U9/ioqKirjuuuvi5Zdfjn79+sWQIUPiX//6V2OcDgDII40SH7feemtceOGFcf7558ehhx4aM2bMiHbt2sUf/vCHxjgdAJBHWjb0ATdv3hxLly6NiRMn5va1aNEiBg8eHIsXL/7K+JqamqipqcndrqqqioiI6urqhp5aRETU1vynUY5L/mistbWjrEGsQZpaY6zBbcfMsuxbxzZ4fHz00UexdevW6Nq1a539Xbt2jbfeeusr46dMmRI33HDDV/aXl5c39NQgIiKKb2vqGbCnswZpao25Bjds2BDFxcXfOKbB46O+Jk6cGBUVFbnbtbW18cknn0THjh2joKCgCWfW/FRXV0d5eXmsXr06ioqKmno67IGsQZqaNdh4siyLDRs2RFlZ2beObfD46NSpU+y1115RWVlZZ39lZWWUlpZ+ZXxhYWEUFhbW2dehQ4eGnhZfUFRU5B8dTcoapKlZg43j217x2KbBLzht3bp1HHXUUbFgwYLcvtra2liwYEEMGDCgoU8HAOSZRvmxS0VFRYwePTq++93vRv/+/eO2226LTZs2xfnnn98YpwMA8kijxMcPf/jD+PDDD+Paa6+NdevWxZFHHhnz5s37ykWopFVYWBjXXXfdV37MBalYgzQ1a3D3UJDtyHtiAAAaiM92AQCSEh8AQFLiAwBISnwAAEmJjz1QQUFBzJkzp16PefDBB6OgoCDOOOOMRpkTu7fGXjOfffZZjB07Njp27Bj77LNPjBw58iu/qDAiYubMmdG3b99o06ZNdOnSJcaOHVvnGGPGjIkjjjgiWrZsud3zPvvsszFw4MDo2LFjtG3bNvr06RNTp06t1/OiaTT1Gvz4449j6NChUVZWFoWFhVFeXh7jxo2r8xkpa9eujXPPPTcOOuigaNGiRYwfP77B5tfciA++1cqVK+OKK66IE044oamnQp6o75q5/PLL49FHH43Zs2fHwoULY82aNXHmmWfWGXPrrbfGr371q5gwYUK8/vrr8eSTT8aQIUNy92/dujXatm0bl156aQwePHi759l7771j3LhxsWjRonjzzTdj0qRJMWnSpLjrrrt2/smyW2roNdiiRYsYPnx4PPLII/H222/HzJkz48knn4yLLrooN6ampiY6d+4ckyZNin79+jXo/JqdjLxSXV2dnXvuuVm7du2y0tLS7NZbb81OPPHE7LLLLsuyLMt69uyZTZ48OTvnnHOydu3aZWVlZdnvfve73ON79uyZRURu69mz5zee7/PPP8+OO+647O67785Gjx6dDR8+vPGeHI1id18z69evz1q1apXNnj07t+/NN9/MIiJbvHhxlmVZ9sknn2Rt27bNnnzyyR16zvVZqyNGjMjOO++8HRrLzmkOa3B7pk2blnXv3n27933x+e3q/Jojr3zkmYqKinjuuefikUceifnz58czzzwTL7/8cp0xv/3tb6Nfv37xyiuvxIQJE+Kyyy6L+fPnR0TEiy++GBER9957b6xduzZ3++tMnjw5unTpEhdccEHjPCEa3e6+ZpYuXRpbtmyp82pFnz59okePHrF48eKIiJg/f37U1tbGP//5zzjkkEOie/fucfbZZ8fq1at3+O9he1555ZV4/vnn48QTT9yl4/DNmsMa/LI1a9bEQw89tFNrx9fV3eBTbdlxGzZsiPvuuy9mzZoVp5xySkT89x/jlz9BcODAgTFhwoSIiDjooIPiueeei6lTp8app54anTt3joj/fnjf9j7o74ueffbZuOeee2LZsmUN/2RIIh/WzLp166J169Zf+UDJrl27xrp16yIi4r333ova2tq48cYbY9q0aVFcXByTJk2KU089NV599dVo3br1Dp8vIqJ79+7x4Ycfxueffx7XX399/PSnP63X49lxzWUNbjNq1KiYO3dufPrpp3H66afH3XffvcPn2dn5NUde+cgj7733XmzZsiX69++f21dcXBwHH3xwnXFf/gC/AQMGxJtvvvm1x121alXss88+ue3GG2+MDRs2xI9//OP4/e9/H506dWrYJ0Iyu9uaufHGG+s8btWqVTv0PGpra2PLli1x++23x5AhQ+LYY4+NBx54IN55553429/+tkPH+KJnnnkmXnrppZgxY0bcdttt8cADD9T7GOyY5rIGt5k6dWq8/PLLMXfu3FixYkVUVFTs8GN9Xf0fr3wQZWVldSq8pKQkVqxYEStXrozTTz89t7+2tjYiIlq2bBnLly+P/fffP/VU2U3s7Jq56KKL4uyzz65znNLS0ti8eXOsX7++zneelZWVue9yu3XrFhERhx56aO7+zp07R6dOner9n0dERO/evSMi4ogjjojKysq4/vrrY9SoUfU+Dk0n9RrcprS0NEpLS6NPnz5RUlISJ5xwQlxzzTW5NfpNfF39H/GRR/bbb79o1apVvPjii9GjR4+IiKiqqoq33347Bg0alBu3ZMmSOo9bsmRJHHLIIbnbrVq1iq1bt+Zut2zZMg444IA6j2nXrl384x//qLNv0qRJsWHDhpg2bVqUl5c32POi8exua6Z169ZRUlJSZ8xRRx0VrVq1igULFsTIkSMjImL58uWxatWq3HfDAwcOzO3v3r17RER88skn8dFHH0XPnj3r/xfzBbW1tVFTU7NLx+DrNZc1uD3bwmFH10+fPn18Xf3/xEcead++fYwePTquvPLKKCkpiS5dusR1110XLVq0iIKCgty45557Lm6++eY444wzYv78+TF79ux4/PHHc/f36tUrFixYEAMHDozCwsLYd999v3KuNm3axOGHH15n37bvCL68n91XPqyZ4uLiuOCCC6KioiJKSkqiqKgoLrnkkhgwYEAce+yxEfHfawCGDx8el112Wdx1111RVFQUEydOjD59+sTJJ5+cO9Ybb7wRmzdvjk8++SQ2bNiQ+874yCOPjIiI6dOnR48ePaJPnz4REbFo0aK45ZZb4tJLL93xv1TqpbmswSeeeCIqKyvj6KOPjn322Sdef/31uPLKK2PgwIHRq1ev3LG2rbmNGzfGhx9+GMuWLYvWrVvHoYce6uvqFzX1222on+29Za1///7ZhAkTsiz771vSbrjhhuyss87KjZk2bVqdYzzyyCPZAQcckLVs2fJb37L2RXvqW8LyXT6smU8//TT7xS9+ke27775Zu3btshEjRmRr166tM6aqqir7yU9+knXo0CErKSnJRowYka1atarOmC+/JXPbts3tt9+eHXbYYVm7du2yoqKi7Dvf+U52xx13ZFu3bt3h50T9NYc1+NRTT2UDBgzIiouLszZt2mQHHnhgdvXVV2f//ve/6xxne+vvm+a7p35dFR95buPGjVlxcXF29913Z1n233/EU6dObdpJsVuzZmhq1iB+7JJnXnnllXjrrbeif//+UVVVFZMnT46IiOHDhzfxzNhdWTM0NWuQLxMfeeiWW26J5cuXR+vWreOoo46KZ555Zo9/2xbfzJqhqVmDfFFBlmVZU08CANhz+CVjAEBS4gMASEp8AABJiQ8AICnxAQAkJT4AgKTEBwCQlPgAAJISHwBAUv8Pn60diKQv4ewAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "openai_experiment.aggregate(metric_name=\"latency\", column_name=\"model\", is_average=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d43a26ba",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdda3b71",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}