Copy_of_Langchain_Retrieval_Augmentation_using_cassio.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oLHkt-bMq4up"
      },
      "source": [
        "# Langchain Retrieval Augmentation (using Wikipedia data)\n",
        "Large Language Models (LLMs) have a data freshness problem. The most powerful LLMs in the world, like GPT-4, have no idea about recent world events or anything outside of data sets they have been trained on.\n",
        "\n",
        "The world of LLMs is frozen in time. Their world exists as a static snapshot of the world as it was within their training data.\n",
        "\n",
        "A solution to this problem is retrieval augmentation. The idea behind this is that we retrieve relevant information from an external knowledge base and give that information to our LLM. In this notebook we will learn how to do that. In this demo, external or proprietary data will be stored in Astra DB and used to provide more current and relevant LLM responses."
      ],
      "id": "oLHkt-bMq4up"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WQUeV_S5q4u0"
      },
      "source": [
        "## Colab-specific setup"
      ],
      "id": "WQUeV_S5q4u0"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z1p8iUgjq4u2"
      },
      "source": [
        "Make sure you have a Database and get ready to upload the Secure Connect Bundle and supply the Token string\n",
        "(see [Pre-requisites](https://cassio.org/start_here/#vector-database) on cassio.org for details).\n",
        "\n",
        "Likewise, ensure you have the necessary secret for the LLM provider of your choice: you'll be asked to input it shortly\n",
        "(see [Pre-requisites](https://cassio.org/start_here/#llm-access) on cassio.org for details).\n",
        "\n",
        "_Note: some portions of this notebook is part of the CassIO documentation. Visit [this page on cassIO.org](https://cassio.org/frameworks/langchain/qa-basic/)._\n"
      ],
      "id": "Z1p8iUgjq4u2"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2953d95b",
      "metadata": {
        "id": "2953d95b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "f3e1d883-536f-47c8-b9b7-e235263fef31"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting langchain\n",
            "  Cloning https://github.com/hemidactylus/langchain (to revision updated-full-preview--lab) to /tmp/pip-install-mxunnmop/langchain_ce6a2c8218b141dba2ea25ab9988ad20\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/hemidactylus/langchain /tmp/pip-install-mxunnmop/langchain_ce6a2c8218b141dba2ea25ab9988ad20\n",
            "  Running command git checkout -b updated-full-preview--lab --track origin/updated-full-preview--lab\n",
            "  Switched to a new branch 'updated-full-preview--lab'\n",
            "  Branch 'updated-full-preview--lab' set up to track remote branch 'updated-full-preview--lab' from 'origin'.\n",
            "  Resolved https://github.com/hemidactylus/langchain to commit 2ccba98643831554d673fb99f96eac80c5784c76\n",
            "  Running command git submodule update --init --recursive -q\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting cassandra-driver>=3.28.0\n",
            "  Downloading cassandra_driver-3.28.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m19.1/19.1 MB\u001b[0m \u001b[31m7.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting cassio==0.0.6\n",
            "  Downloading cassio-0.0.6-py3-none-any.whl (11 kB)\n",
            "Collecting google-cloud-aiplatform>=1.25.0\n",
            "  Downloading google_cloud_aiplatform-1.33.1-py2.py3-none-any.whl (2.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.9/2.9 MB\u001b[0m \u001b[31m54.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting jupyter>=1.0.0\n",
            "  Downloading jupyter-1.0.0-py2.py3-none-any.whl (2.7 kB)\n",
            "Collecting openai==0.27.7\n",
            "  Downloading openai-0.27.7-py3-none-any.whl (71 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m72.0/72.0 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting python-dotenv==1.0.0\n",
            "  Downloading python_dotenv-1.0.0-py3-none-any.whl (19 kB)\n",
            "Collecting tensorflow-cpu==2.12.0\n",
            "  Downloading tensorflow_cpu-2.12.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (231.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m231.8/231.8 MB\u001b[0m \u001b[31m2.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tiktoken==0.4.0\n",
            "  Downloading tiktoken-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m81.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting transformers>=4.29.2\n",
            "  Downloading transformers-4.33.3-py3-none-any.whl (7.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.6/7.6 MB\u001b[0m \u001b[31m95.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.0 in /usr/local/lib/python3.10/dist-packages (from cassio==0.0.6) (1.23.5)\n",
            "Requirement already satisfied: requests>=2.20 in /usr/local/lib/python3.10/dist-packages (from openai==0.27.7) (2.31.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from openai==0.27.7) (4.66.1)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from openai==0.27.7) (3.8.5)\n",
            "Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (1.4.0)\n",
            "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (1.6.3)\n",
            "Requirement already satisfied: flatbuffers>=2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (23.5.26)\n",
            "Requirement already satisfied: gast<=0.4.0,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (0.4.0)\n",
            "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (0.2.0)\n",
            "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (1.57.0)\n",
            "Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (3.9.0)\n",
            "Requirement already satisfied: jax>=0.3.15 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (0.4.14)\n",
            "Collecting keras<2.13,>=2.12.0 (from tensorflow-cpu==2.12.0)\n",
            "  Downloading keras-2.12.0-py2.py3-none-any.whl (1.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m78.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (16.0.6)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (3.3.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (23.1)\n",
            "Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (3.20.3)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (67.7.2)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (1.16.0)\n",
            "Collecting tensorboard<2.13,>=2.12 (from tensorflow-cpu==2.12.0)\n",
            "  Downloading tensorboard-2.12.3-py3-none-any.whl (5.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.6/5.6 MB\u001b[0m \u001b[31m98.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tensorflow-estimator<2.13,>=2.12.0 (from tensorflow-cpu==2.12.0)\n",
            "  Downloading tensorflow_estimator-2.12.0-py2.py3-none-any.whl (440 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m440.7/440.7 kB\u001b[0m \u001b[31m41.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (2.3.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (4.5.0)\n",
            "Collecting wrapt<1.15,>=1.11.0 (from tensorflow-cpu==2.12.0)\n",
            "  Downloading wrapt-1.14.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (77 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow-cpu==2.12.0) (0.33.0)\n",
            "Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken==0.4.0) (2023.6.3)\n",
            "Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (6.0.1)\n",
            "Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.0.20)\n",
            "Requirement already satisfied: anyio<4.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (3.7.1)\n",
            "Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (4.0.3)\n",
            "Collecting dataclasses-json<0.7,>=0.5.7 (from langchain)\n",
            "  Downloading dataclasses_json-0.6.1-py3-none-any.whl (27 kB)\n",
            "Collecting jsonpatch<2.0,>=1.33 (from langchain)\n",
            "  Downloading jsonpatch-1.33-py2.py3-none-any.whl (12 kB)\n",
            "Collecting langsmith<0.1.0,>=0.0.38 (from langchain)\n",
            "  Downloading langsmith-0.0.41-py3-none-any.whl (39 kB)\n",
            "Requirement already satisfied: numexpr<3.0.0,>=2.8.4 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.8.5)\n",
            "Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.10.12)\n",
            "Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (8.2.3)\n",
            "Collecting geomet<0.3,>=0.1 (from cassandra-driver>=3.28.0)\n",
            "  Downloading geomet-0.2.1.post1-py3-none-any.whl (18 kB)\n",
            "Requirement already satisfied: google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform>=1.25.0) (2.11.1)\n",
            "Requirement already satisfied: proto-plus<2.0.0dev,>=1.22.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform>=1.25.0) (1.22.3)\n",
            "Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform>=1.25.0) (2.8.0)\n",
            "Requirement already satisfied: google-cloud-bigquery<4.0.0dev,>=1.15.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform>=1.25.0) (3.10.0)\n",
            "Collecting google-cloud-resource-manager<3.0.0dev,>=1.3.3 (from google-cloud-aiplatform>=1.25.0)\n",
            "  Downloading google_cloud_resource_manager-1.10.4-py2.py3-none-any.whl (320 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m321.0/321.0 kB\u001b[0m \u001b[31m33.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting shapely<2.0.0 (from google-cloud-aiplatform>=1.25.0)\n",
            "  Downloading Shapely-1.8.5.post1-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m86.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: notebook in /usr/local/lib/python3.10/dist-packages (from jupyter>=1.0.0) (6.5.5)\n",
            "Collecting qtconsole (from jupyter>=1.0.0)\n",
            "  Downloading qtconsole-5.4.4-py3-none-any.whl (121 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.9/121.9 kB\u001b[0m \u001b[31m15.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: jupyter-console in /usr/local/lib/python3.10/dist-packages (from jupyter>=1.0.0) (6.1.0)\n",
            "Requirement already satisfied: nbconvert in /usr/local/lib/python3.10/dist-packages (from jupyter>=1.0.0) (6.5.4)\n",
            "Requirement already satisfied: ipykernel in /usr/local/lib/python3.10/dist-packages (from jupyter>=1.0.0) (5.5.6)\n",
            "Requirement already satisfied: ipywidgets in /usr/local/lib/python3.10/dist-packages (from jupyter>=1.0.0) (7.7.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers>=4.29.2) (3.12.2)\n",
            "Collecting huggingface-hub<1.0,>=0.15.1 (from transformers>=4.29.2)\n",
            "  Downloading huggingface_hub-0.17.3-py3-none-any.whl (295 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m295.0/295.0 kB\u001b[0m \u001b[31m31.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers>=4.29.2)\n",
            "  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m98.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting safetensors>=0.3.1 (from transformers>=4.29.2)\n",
            "  Downloading safetensors-0.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m57.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai==0.27.7) (23.1.0)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai==0.27.7) (3.2.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai==0.27.7) (6.0.4)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai==0.27.7) (1.9.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai==0.27.7) (1.4.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->openai==0.27.7) (1.3.1)\n",
            "Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<4.0->langchain) (3.4)\n",
            "Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<4.0->langchain) (1.3.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4.0->langchain) (1.1.3)\n",
            "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow-cpu==2.12.0) (0.41.2)\n",
            "Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json<0.7,>=0.5.7->langchain)\n",
            "  Downloading marshmallow-3.20.1-py3-none-any.whl (49 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting typing-inspect<1,>=0.4.0 (from dataclasses-json<0.7,>=0.5.7->langchain)\n",
            "  Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from geomet<0.3,>=0.1->cassandra-driver>=3.28.0) (8.1.7)\n",
            "Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform>=1.25.0) (1.60.0)\n",
            "Requirement already satisfied: google-auth<3.0.dev0,>=2.14.1 in /usr/local/lib/python3.10/dist-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform>=1.25.0) (2.17.3)\n",
            "Requirement already satisfied: grpcio-status<2.0.dev0,>=1.33.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform>=1.25.0) (1.48.2)\n",
            "Requirement already satisfied: google-cloud-core<3.0.0dev,>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform>=1.25.0) (2.3.3)\n",
            "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform>=1.25.0) (2.6.0)\n",
            "Requirement already satisfied: python-dateutil<3.0dev,>=2.7.2 in /usr/local/lib/python3.10/dist-packages (from google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform>=1.25.0) (2.8.2)\n",
            "Requirement already satisfied: grpc-google-iam-v1<1.0.0dev,>=0.12.4 in /usr/local/lib/python3.10/dist-packages (from google-cloud-resource-manager<3.0.0dev,>=1.3.3->google-cloud-aiplatform>=1.25.0) (0.12.6)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.15.1->transformers>=4.29.2) (2023.6.0)\n",
            "Requirement already satisfied: ml-dtypes>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from jax>=0.3.15->tensorflow-cpu==2.12.0) (0.2.0)\n",
            "Requirement already satisfied: scipy>=1.7 in /usr/local/lib/python3.10/dist-packages (from jax>=0.3.15->tensorflow-cpu==2.12.0) (1.11.2)\n",
            "Collecting jsonpointer>=1.9 (from jsonpatch<2.0,>=1.33->langchain)\n",
            "  Downloading jsonpointer-2.4-py2.py3-none-any.whl (7.8 kB)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai==0.27.7) (2.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.20->openai==0.27.7) (2023.7.22)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain) (2.0.2)\n",
            "Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.13,>=2.12->tensorflow-cpu==2.12.0) (1.0.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.13,>=2.12->tensorflow-cpu==2.12.0) (3.4.4)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.13,>=2.12->tensorflow-cpu==2.12.0) (0.7.1)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.13,>=2.12->tensorflow-cpu==2.12.0) (2.3.7)\n",
            "Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.10/dist-packages (from ipykernel->jupyter>=1.0.0) (0.2.0)\n",
            "Requirement already satisfied: ipython>=5.0.0 in /usr/local/lib/python3.10/dist-packages (from ipykernel->jupyter>=1.0.0) (7.34.0)\n",
            "Requirement already satisfied: traitlets>=4.1.0 in /usr/local/lib/python3.10/dist-packages (from ipykernel->jupyter>=1.0.0) (5.7.1)\n",
            "Requirement already satisfied: jupyter-client in /usr/local/lib/python3.10/dist-packages (from ipykernel->jupyter>=1.0.0) (6.1.12)\n",
            "Requirement already satisfied: tornado>=4.2 in /usr/local/lib/python3.10/dist-packages (from ipykernel->jupyter>=1.0.0) (6.3.2)\n",
            "Requirement already satisfied: widgetsnbextension~=3.6.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets->jupyter>=1.0.0) (3.6.5)\n",
            "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets->jupyter>=1.0.0) (3.0.8)\n",
            "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from jupyter-console->jupyter>=1.0.0) (3.0.39)\n",
            "Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from jupyter-console->jupyter>=1.0.0) (2.16.1)\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (4.9.3)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (4.11.2)\n",
            "Requirement already satisfied: bleach in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (6.0.0)\n",
            "Requirement already satisfied: defusedxml in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (0.7.1)\n",
            "Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (0.4)\n",
            "Requirement already satisfied: jinja2>=3.0 in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (3.1.2)\n",
            "Requirement already satisfied: jupyter-core>=4.7 in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (5.3.1)\n",
            "Requirement already satisfied: jupyterlab-pygments in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (0.2.2)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (2.1.3)\n",
            "Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (0.8.4)\n",
            "Requirement already satisfied: nbclient>=0.5.0 in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (0.8.0)\n",
            "Requirement already satisfied: nbformat>=5.1 in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (5.9.2)\n",
            "Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (1.5.0)\n",
            "Requirement already satisfied: tinycss2 in /usr/local/lib/python3.10/dist-packages (from nbconvert->jupyter>=1.0.0) (1.2.1)\n",
            "Requirement already satisfied: pyzmq<25,>=17 in /usr/local/lib/python3.10/dist-packages (from notebook->jupyter>=1.0.0) (23.2.1)\n",
            "Requirement already satisfied: argon2-cffi in /usr/local/lib/python3.10/dist-packages (from notebook->jupyter>=1.0.0) (23.1.0)\n",
            "Requirement already satisfied: nest-asyncio>=1.5 in /usr/local/lib/python3.10/dist-packages (from notebook->jupyter>=1.0.0) (1.5.7)\n",
            "Requirement already satisfied: Send2Trash>=1.8.0 in /usr/local/lib/python3.10/dist-packages (from notebook->jupyter>=1.0.0) (1.8.2)\n",
            "Requirement already satisfied: terminado>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from notebook->jupyter>=1.0.0) (0.17.1)\n",
            "Requirement already satisfied: prometheus-client in /usr/local/lib/python3.10/dist-packages (from notebook->jupyter>=1.0.0) (0.17.1)\n",
            "Requirement already satisfied: nbclassic>=0.4.7 in /usr/local/lib/python3.10/dist-packages (from notebook->jupyter>=1.0.0) (1.0.0)\n",
            "Collecting qtpy>=2.4.0 (from qtconsole->jupyter>=1.0.0)\n",
            "  Downloading QtPy-2.4.0-py3-none-any.whl (93 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m93.4/93.4 kB\u001b[0m \u001b[31m10.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform>=1.25.0) (5.3.1)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform>=1.25.0) (0.3.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform>=1.25.0) (4.9)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard<2.13,>=2.12->tensorflow-cpu==2.12.0) (1.3.1)\n",
            "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /usr/local/lib/python3.10/dist-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform>=1.25.0) (1.5.0)\n",
            "Collecting jedi>=0.16 (from ipython>=5.0.0->ipykernel->jupyter>=1.0.0)\n",
            "  Downloading jedi-0.19.0-py2.py3-none-any.whl (1.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m81.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from ipython>=5.0.0->ipykernel->jupyter>=1.0.0) (4.4.2)\n",
            "Requirement already satisfied: pickleshare in /usr/local/lib/python3.10/dist-packages (from ipython>=5.0.0->ipykernel->jupyter>=1.0.0) (0.7.5)\n",
            "Requirement already satisfied: backcall in /usr/local/lib/python3.10/dist-packages (from ipython>=5.0.0->ipykernel->jupyter>=1.0.0) (0.2.0)\n",
            "Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.10/dist-packages (from ipython>=5.0.0->ipykernel->jupyter>=1.0.0) (0.1.6)\n",
            "Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/dist-packages (from ipython>=5.0.0->ipykernel->jupyter>=1.0.0) (4.8.0)\n",
            "Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.10/dist-packages (from jupyter-core>=4.7->nbconvert->jupyter>=1.0.0) (3.10.0)\n",
            "Requirement already satisfied: jupyter-server>=1.8 in /usr/local/lib/python3.10/dist-packages (from nbclassic>=0.4.7->notebook->jupyter>=1.0.0) (1.24.0)\n",
            "Requirement already satisfied: notebook-shim>=0.2.3 in /usr/local/lib/python3.10/dist-packages (from nbclassic>=0.4.7->notebook->jupyter>=1.0.0) (0.2.3)\n",
            "Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.10/dist-packages (from nbformat>=5.1->nbconvert->jupyter>=1.0.0) (2.18.0)\n",
            "Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.10/dist-packages (from nbformat>=5.1->nbconvert->jupyter>=1.0.0) (4.19.0)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->jupyter-console->jupyter>=1.0.0) (0.2.6)\n",
            "Requirement already satisfied: ptyprocess in /usr/local/lib/python3.10/dist-packages (from terminado>=0.8.3->notebook->jupyter>=1.0.0) (0.7.0)\n",
            "Collecting mypy-extensions>=0.3.0 (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain)\n",
            "  Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
            "Requirement already satisfied: argon2-cffi-bindings in /usr/local/lib/python3.10/dist-packages (from argon2-cffi->notebook->jupyter>=1.0.0) (21.2.0)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->nbconvert->jupyter>=1.0.0) (2.5)\n",
            "Requirement already satisfied: webencodings in /usr/local/lib/python3.10/dist-packages (from bleach->nbconvert->jupyter>=1.0.0) (0.5.1)\n",
            "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from jedi>=0.16->ipython>=5.0.0->ipykernel->jupyter>=1.0.0) (0.8.3)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter>=1.0.0) (2023.7.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter>=1.0.0) (0.30.2)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=2.6->nbformat>=5.1->nbconvert->jupyter>=1.0.0) (0.10.2)\n",
            "Requirement already satisfied: websocket-client in /usr/local/lib/python3.10/dist-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter>=1.0.0) (1.6.2)\n",
            "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform>=1.25.0) (0.5.0)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard<2.13,>=2.12->tensorflow-cpu==2.12.0) (3.2.2)\n",
            "Requirement already satisfied: cffi>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from argon2-cffi-bindings->argon2-cffi->notebook->jupyter>=1.0.0) (1.15.1)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook->jupyter>=1.0.0) (2.21)\n",
            "Building wheels for collected packages: langchain\n",
            "  Building wheel for langchain (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for langchain: filename=langchain-0.0.299-py3-none-any.whl size=1738715 sha256=4c65dfe5fa15bb1ed5b518e6e0903a5ca304ed9fec9f2564be33bcda236af7b9\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-z1rlrooi/wheels/eb/74/22/c06cef0eb49622a306eff1aab8aae9bf57f4defb29c867ade4\n",
            "Successfully built langchain\n",
            "Installing collected packages: tokenizers, safetensors, wrapt, tensorflow-estimator, shapely, qtpy, python-dotenv, mypy-extensions, marshmallow, keras, jsonpointer, jedi, geomet, typing-inspect, tiktoken, langsmith, jsonpatch, huggingface-hub, cassandra-driver, transformers, openai, dataclasses-json, cassio, tensorboard, qtconsole, langchain, tensorflow-cpu, google-cloud-resource-manager, google-cloud-aiplatform, jupyter\n",
            "  Attempting uninstall: wrapt\n",
            "    Found existing installation: wrapt 1.15.0\n",
            "    Uninstalling wrapt-1.15.0:\n",
            "      Successfully uninstalled wrapt-1.15.0\n",
            "  Attempting uninstall: tensorflow-estimator\n",
            "    Found existing installation: tensorflow-estimator 2.13.0\n",
            "    Uninstalling tensorflow-estimator-2.13.0:\n",
            "      Successfully uninstalled tensorflow-estimator-2.13.0\n",
            "  Attempting uninstall: shapely\n",
            "    Found existing installation: shapely 2.0.1\n",
            "    Uninstalling shapely-2.0.1:\n",
            "      Successfully uninstalled shapely-2.0.1\n",
            "  Attempting uninstall: keras\n",
            "    Found existing installation: keras 2.13.1\n",
            "    Uninstalling keras-2.13.1:\n",
            "      Successfully uninstalled keras-2.13.1\n",
            "  Attempting uninstall: tensorboard\n",
            "    Found existing installation: tensorboard 2.13.0\n",
            "    Uninstalling tensorboard-2.13.0:\n",
            "      Successfully uninstalled tensorboard-2.13.0\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "tensorflow 2.13.0 requires keras<2.14,>=2.13.1, but you have keras 2.12.0 which is incompatible.\n",
            "tensorflow 2.13.0 requires tensorboard<2.14,>=2.13, but you have tensorboard 2.12.3 which is incompatible.\n",
            "tensorflow 2.13.0 requires tensorflow-estimator<2.14,>=2.13.0, but you have tensorflow-estimator 2.12.0 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed cassandra-driver-3.28.0 cassio-0.0.6 dataclasses-json-0.6.1 geomet-0.2.1.post1 google-cloud-aiplatform-1.33.1 google-cloud-resource-manager-1.10.4 huggingface-hub-0.17.3 jedi-0.19.0 jsonpatch-1.33 jsonpointer-2.4 jupyter-1.0.0 keras-2.12.0 langchain-0.0.299 langsmith-0.0.41 marshmallow-3.20.1 mypy-extensions-1.0.0 openai-0.27.7 python-dotenv-1.0.0 qtconsole-5.4.4 qtpy-2.4.0 safetensors-0.3.3 shapely-1.8.5.post1 tensorboard-2.12.3 tensorflow-cpu-2.12.0 tensorflow-estimator-2.12.0 tiktoken-0.4.0 tokenizers-0.13.3 transformers-4.33.3 typing-inspect-0.9.0 wrapt-1.14.1\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "google"
                ]
              }
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "# install required dependencies\n",
        "! pip install \\\n",
        "    \"git+https://github.com/hemidactylus/langchain@updated-full-preview--lab#egg=langchain&subdirectory=libs/langchain\" \\\n",
        "    \"cassandra-driver>=3.28.0\" \\\n",
        "    \"cassio==0.0.6\" \\\n",
        "    \"google-cloud-aiplatform>=1.25.0\" \\\n",
        "    \"jupyter>=1.0.0\" \\\n",
        "    \"openai==0.27.7\" \\\n",
        "    \"python-dotenv==1.0.0\" \\\n",
        "    \"tensorflow-cpu==2.12.0\" \\\n",
        "    \"tiktoken==0.4.0\" \\\n",
        "    \"transformers>=4.29.2\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "222f44ff",
      "metadata": {
        "id": "222f44ff"
      },
      "source": [
        "You will likely be asked to \"Restart the Runtime\" at this time, as some dependencies\n",
        "have been upgraded. **Please do restart the runtime now** for a smoother execution from this point onward."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Get the Wikipedia data from 20220301.simple"
      ],
      "metadata": {
        "id": "eZS2Xsy0WY-c"
      },
      "id": "eZS2Xsy0WY-c"
    },
    {
      "cell_type": "code",
      "source": [
        " !wget https://raw.githubusercontent.com/GeorgeCrossIV/Langchain-Retrieval-Augmentation-with-CASSIO/main/20220301.simple.csv"
      ],
      "metadata": {
        "id": "XrP5HE9RWhVg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0b1f1aff-d71b-40d0-a88e-de6be1316336"
      },
      "id": "XrP5HE9RWhVg",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-09-28 19:50:28--  https://raw.githubusercontent.com/GeorgeCrossIV/Langchain-Retrieval-Augmentation-with-CASSIO/main/20220301.simple.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.111.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 30939404 (30M) [text/plain]\n",
            "Saving to: ‘20220301.simple.csv’\n",
            "\n",
            "20220301.simple.csv 100%[===================>]  29.51M  58.6MB/s    in 0.5s    \n",
            "\n",
            "2023-09-28 19:50:30 (58.6 MB/s) - ‘20220301.simple.csv’ saved [30939404/30939404]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Import the 20220301.simple wikipedia from the CSV file"
      ],
      "metadata": {
        "id": "Kte4Sybn2EOu"
      },
      "id": "Kte4Sybn2EOu"
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "data = pd.read_csv('20220301.simple.csv')"
      ],
      "metadata": {
        "id": "SnIUf1Z3UD_5"
      },
      "id": "SnIUf1Z3UD_5",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "There are 10,000 entries in the Wikipedia data file. We'll reduce the dataset to 10 rows for this demo. It takes a while to process the data; however, feel free to increase the number of rows for future demo runs."
      ],
      "metadata": {
        "id": "YT7rxk7y1LK6"
      },
      "id": "YT7rxk7y1LK6"
    },
    {
      "cell_type": "code",
      "source": [
        "data = data.head(10)\n",
        "data = data.rename(columns={'text ': 'text'})\n",
        "data"
      ],
      "metadata": {
        "id": "ffhIsRs1UQYz",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        },
        "outputId": "3a41121e-a44f-4d34-ce31-7304c8a10d3f"
      },
      "id": "ffhIsRs1UQYz",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   id                                                url  \\\n",
              "0   1            https://simple.wikipedia.org/wiki/April   \n",
              "1   2           https://simple.wikipedia.org/wiki/August   \n",
              "2   6              https://simple.wikipedia.org/wiki/Art   \n",
              "3   8                https://simple.wikipedia.org/wiki/A   \n",
              "4   9              https://simple.wikipedia.org/wiki/Air   \n",
              "5  12  https://simple.wikipedia.org/wiki/Autonomous%2...   \n",
              "6  13    https://simple.wikipedia.org/wiki/Alan%20Turing   \n",
              "7  14  https://simple.wikipedia.org/wiki/Alanis%20Mor...   \n",
              "8  17  https://simple.wikipedia.org/wiki/Adobe%20Illu...   \n",
              "9  18        https://simple.wikipedia.org/wiki/Andouille   \n",
              "\n",
              "                             title  \\\n",
              "0                            April   \n",
              "1                           August   \n",
              "2                              Art   \n",
              "3                                A   \n",
              "4                              Air   \n",
              "5  Autonomous communities of Spain   \n",
              "6                      Alan Turing   \n",
              "7                Alanis Morissette   \n",
              "8                Adobe Illustrator   \n",
              "9                        Andouille   \n",
              "\n",
              "                                                text  \n",
              "0  April is the fourth month of the year in the J...  \n",
              "1  August (Aug.) is the eighth month of the year ...  \n",
              "2  Art is a creative activity that expresses imag...  \n",
              "3  A or a is the first letter of the English alph...  \n",
              "4  Air refers to the Earth's atmosphere. Air is a...  \n",
              "5  Spain is divided in 17 parts called autonomous...  \n",
              "6  Alan Mathison Turing OBE FRS (London, 23 June ...  \n",
              "7  Alanis Nadine Morissette (born June 1, 1974) i...  \n",
              "8  Adobe Illustrator is a computer program for ma...  \n",
              "9  Andouille is a type of pork sausage. It is spi...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-19234084-91cf-4099-a77a-1444b51f2617\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>url</th>\n",
              "      <th>title</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/April</td>\n",
              "      <td>April</td>\n",
              "      <td>April is the fourth month of the year in the J...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/August</td>\n",
              "      <td>August</td>\n",
              "      <td>August (Aug.) is the eighth month of the year ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>6</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Art</td>\n",
              "      <td>Art</td>\n",
              "      <td>Art is a creative activity that expresses imag...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>8</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/A</td>\n",
              "      <td>A</td>\n",
              "      <td>A or a is the first letter of the English alph...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>9</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Air</td>\n",
              "      <td>Air</td>\n",
              "      <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>12</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Autonomous%2...</td>\n",
              "      <td>Autonomous communities of Spain</td>\n",
              "      <td>Spain is divided in 17 parts called autonomous...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>13</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Alan%20Turing</td>\n",
              "      <td>Alan Turing</td>\n",
              "      <td>Alan Mathison Turing OBE FRS (London, 23 June ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>14</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Alanis%20Mor...</td>\n",
              "      <td>Alanis Morissette</td>\n",
              "      <td>Alanis Nadine Morissette (born June 1, 1974) i...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>17</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Adobe%20Illu...</td>\n",
              "      <td>Adobe Illustrator</td>\n",
              "      <td>Adobe Illustrator is a computer program for ma...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>18</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Andouille</td>\n",
              "      <td>Andouille</td>\n",
              "      <td>Andouille is a type of pork sausage. It is spi...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-19234084-91cf-4099-a77a-1444b51f2617')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-19234084-91cf-4099-a77a-1444b51f2617 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-19234084-91cf-4099-a77a-1444b51f2617');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-c9ae7bd6-58b9-490f-9786-48d10449062c\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c9ae7bd6-58b9-490f-9786-48d10449062c')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-c9ae7bd6-58b9-490f-9786-48d10449062c button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will execute queries against the [Andouille](https://simple.wikipedia.org/wiki/Andouille) Wikipedia entries later in this demo. The Wikipedia data used in this demo is from a snapshot in time, stored in a CSV file. Below is the text of the Wikipedia record that will be processed."
      ],
      "metadata": {
        "id": "4xY-8xy61dME"
      },
      "id": "4xY-8xy61dME"
    },
    {
      "cell_type": "code",
      "source": [
        "data.iloc[9]['text']"
      ],
      "metadata": {
        "id": "5a9IQujzCkJ1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "outputId": "fc18aa7b-cce7-41f0-9f05-56e835ea19d1"
      },
      "id": "5a9IQujzCkJ1",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Andouille is a type of pork sausage. It is spicy (hot in taste) and smoked. There are different kinds, all with different combinations of pork meat, fat, intestines (tubes going to the stomach), and tripe (the wall of the stomach).\\n\\nOther sorts are \"French andouille\" and \"German andouille\"; they are less spicy than Cajun. Cajun has extra salt, black pepper, and garlic. Andouille makers smoke the sausages over pecan wood and sugar cane for a maximum of seven or eight hours, at about 175 degrees Fahrenheit (80 degrees Celsius).\\n\\nSausage'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zh4P-XUDq4u9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bef70a77-6252-406e-c1d5-938e94d27216"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Your Astra DB Keyspace name: vector_preview\n"
          ]
        }
      ],
      "source": [
        "# Input your database keyspace name:\n",
        "ASTRA_DB_KEYSPACE = input('Your Astra DB Keyspace name: ')"
      ],
      "id": "zh4P-XUDq4u9"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lThGqYchq4u-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bb1d8953-bf9b-42d2-d9e5-f048700dba67"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Your Astra DB Token: ··········\n"
          ]
        }
      ],
      "source": [
        "# Input your Astra DB token string, the one starting with \"AstraCS:...\"\n",
        "import getpass\n",
        "ASTRA_DB_TOKEN_BASED_PASSWORD = getpass.getpass('Your Astra DB Token: ')"
      ],
      "id": "lThGqYchq4u-"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QNQ6T_Gjk0Oz"
      },
      "source": [
        "### Astra DB Secure Connect Bundle\n",
        "\n",
        "Please upload the Secure Connect Bundle zipfile to connect to your Astra DB instance.\n",
        "\n",
        "The Secure Connect Bundle is needed to establish a secure connection to the database.\n",
        "Click [here](https://awesome-astra.github.io/docs/pages/astra/download-scb/#c-procedure) for instructions on how to download it from Astra DB."
      ],
      "id": "QNQ6T_Gjk0Oz"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xnNziXZ1q4vD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        },
        "outputId": "cf129789-4713-4f1f-eb53-832148fad785"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Please upload your Secure Connect Bundle\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-9a55b49c-3ea5-459e-8b5e-c91be372cdf8\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-9a55b49c-3ea5-459e-8b5e-c91be372cdf8\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script>// Copyright 2017 Google LLC\n",
              "//\n",
              "// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
              "// you may not use this file except in compliance with the License.\n",
              "// You may obtain a copy of the License at\n",
              "//\n",
              "//      http://www.apache.org/licenses/LICENSE-2.0\n",
              "//\n",
              "// Unless required by applicable law or agreed to in writing, software\n",
              "// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
              "// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
              "// See the License for the specific language governing permissions and\n",
              "// limitations under the License.\n",
              "\n",
              "/**\n",
              " * @fileoverview Helpers for google.colab Python module.\n",
              " */\n",
              "(function(scope) {\n",
              "function span(text, styleAttributes = {}) {\n",
              "  const element = document.createElement('span');\n",
              "  element.textContent = text;\n",
              "  for (const key of Object.keys(styleAttributes)) {\n",
              "    element.style[key] = styleAttributes[key];\n",
              "  }\n",
              "  return element;\n",
              "}\n",
              "\n",
              "// Max number of bytes which will be uploaded at a time.\n",
              "const MAX_PAYLOAD_SIZE = 100 * 1024;\n",
              "\n",
              "function _uploadFiles(inputId, outputId) {\n",
              "  const steps = uploadFilesStep(inputId, outputId);\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  // Cache steps on the outputElement to make it available for the next call\n",
              "  // to uploadFilesContinue from Python.\n",
              "  outputElement.steps = steps;\n",
              "\n",
              "  return _uploadFilesContinue(outputId);\n",
              "}\n",
              "\n",
              "// This is roughly an async generator (not supported in the browser yet),\n",
              "// where there are multiple asynchronous steps and the Python side is going\n",
              "// to poll for completion of each step.\n",
              "// This uses a Promise to block the python side on completion of each step,\n",
              "// then passes the result of the previous step as the input to the next step.\n",
              "function _uploadFilesContinue(outputId) {\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  const steps = outputElement.steps;\n",
              "\n",
              "  const next = steps.next(outputElement.lastPromiseValue);\n",
              "  return Promise.resolve(next.value.promise).then((value) => {\n",
              "    // Cache the last promise value to make it available to the next\n",
              "    // step of the generator.\n",
              "    outputElement.lastPromiseValue = value;\n",
              "    return next.value.response;\n",
              "  });\n",
              "}\n",
              "\n",
              "/**\n",
              " * Generator function which is called between each async step of the upload\n",
              " * process.\n",
              " * @param {string} inputId Element ID of the input file picker element.\n",
              " * @param {string} outputId Element ID of the output display.\n",
              " * @return {!Iterable<!Object>} Iterable of next steps.\n",
              " */\n",
              "function* uploadFilesStep(inputId, outputId) {\n",
              "  const inputElement = document.getElementById(inputId);\n",
              "  inputElement.disabled = false;\n",
              "\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  outputElement.innerHTML = '';\n",
              "\n",
              "  const pickedPromise = new Promise((resolve) => {\n",
              "    inputElement.addEventListener('change', (e) => {\n",
              "      resolve(e.target.files);\n",
              "    });\n",
              "  });\n",
              "\n",
              "  const cancel = document.createElement('button');\n",
              "  inputElement.parentElement.appendChild(cancel);\n",
              "  cancel.textContent = 'Cancel upload';\n",
              "  const cancelPromise = new Promise((resolve) => {\n",
              "    cancel.onclick = () => {\n",
              "      resolve(null);\n",
              "    };\n",
              "  });\n",
              "\n",
              "  // Wait for the user to pick the files.\n",
              "  const files = yield {\n",
              "    promise: Promise.race([pickedPromise, cancelPromise]),\n",
              "    response: {\n",
              "      action: 'starting',\n",
              "    }\n",
              "  };\n",
              "\n",
              "  cancel.remove();\n",
              "\n",
              "  // Disable the input element since further picks are not allowed.\n",
              "  inputElement.disabled = true;\n",
              "\n",
              "  if (!files) {\n",
              "    return {\n",
              "      response: {\n",
              "        action: 'complete',\n",
              "      }\n",
              "    };\n",
              "  }\n",
              "\n",
              "  for (const file of files) {\n",
              "    const li = document.createElement('li');\n",
              "    li.append(span(file.name, {fontWeight: 'bold'}));\n",
              "    li.append(span(\n",
              "        `(${file.type || 'n/a'}) - ${file.size} bytes, ` +\n",
              "        `last modified: ${\n",
              "            file.lastModifiedDate ? file.lastModifiedDate.toLocaleDateString() :\n",
              "                                    'n/a'} - `));\n",
              "    const percent = span('0% done');\n",
              "    li.appendChild(percent);\n",
              "\n",
              "    outputElement.appendChild(li);\n",
              "\n",
              "    const fileDataPromise = new Promise((resolve) => {\n",
              "      const reader = new FileReader();\n",
              "      reader.onload = (e) => {\n",
              "        resolve(e.target.result);\n",
              "      };\n",
              "      reader.readAsArrayBuffer(file);\n",
              "    });\n",
              "    // Wait for the data to be ready.\n",
              "    let fileData = yield {\n",
              "      promise: fileDataPromise,\n",
              "      response: {\n",
              "        action: 'continue',\n",
              "      }\n",
              "    };\n",
              "\n",
              "    // Use a chunked sending to avoid message size limits. See b/62115660.\n",
              "    let position = 0;\n",
              "    do {\n",
              "      const length = Math.min(fileData.byteLength - position, MAX_PAYLOAD_SIZE);\n",
              "      const chunk = new Uint8Array(fileData, position, length);\n",
              "      position += length;\n",
              "\n",
              "      const base64 = btoa(String.fromCharCode.apply(null, chunk));\n",
              "      yield {\n",
              "        response: {\n",
              "          action: 'append',\n",
              "          file: file.name,\n",
              "          data: base64,\n",
              "        },\n",
              "      };\n",
              "\n",
              "      let percentDone = fileData.byteLength === 0 ?\n",
              "          100 :\n",
              "          Math.round((position / fileData.byteLength) * 100);\n",
              "      percent.textContent = `${percentDone}% done`;\n",
              "\n",
              "    } while (position < fileData.byteLength);\n",
              "  }\n",
              "\n",
              "  // All done.\n",
              "  yield {\n",
              "    response: {\n",
              "      action: 'complete',\n",
              "    }\n",
              "  };\n",
              "}\n",
              "\n",
              "scope.google = scope.google || {};\n",
              "scope.google.colab = scope.google.colab || {};\n",
              "scope.google.colab._files = {\n",
              "  _uploadFiles,\n",
              "  _uploadFilesContinue,\n",
              "};\n",
              "})(self);\n",
              "</script> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving secure-connect-vectordemo.zip to secure-connect-vectordemo.zip\n"
          ]
        }
      ],
      "source": [
        "# Upload your Secure Connect Bundle zipfile:\n",
        "import os\n",
        "from google.colab import files\n",
        "\n",
        "\n",
        "print('Please upload your Secure Connect Bundle')\n",
        "uploaded = files.upload()\n",
        "if uploaded:\n",
        "    astraBundleFileTitle = list(uploaded.keys())[0]\n",
        "    ASTRA_DB_SECURE_BUNDLE_PATH = os.path.join(os.getcwd(), astraBundleFileTitle)\n",
        "else:\n",
        "    raise ValueError(\n",
        "        'Cannot proceed without Secure Connect Bundle. Please re-run the cell.'\n",
        "    )"
      ],
      "id": "xnNziXZ1q4vD"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TUDw-07Iq4vE"
      },
      "outputs": [],
      "source": [
        "# colab-specific override of helper functions\n",
        "from cassandra.cluster import (\n",
        "    Cluster,\n",
        ")\n",
        "from cassandra.auth import PlainTextAuthProvider\n",
        "\n",
        "# The \"username\" is the literal string 'token' for this connection mode:\n",
        "ASTRA_DB_TOKEN_BASED_USERNAME = 'token'\n",
        "\n",
        "\n",
        "def getCQLSession(mode='astra_db'):\n",
        "    if mode == 'astra_db':\n",
        "        cluster = Cluster(\n",
        "            cloud={\n",
        "                \"secure_connect_bundle\": ASTRA_DB_SECURE_BUNDLE_PATH,\n",
        "            },\n",
        "            auth_provider=PlainTextAuthProvider(\n",
        "                ASTRA_DB_TOKEN_BASED_USERNAME,\n",
        "                ASTRA_DB_TOKEN_BASED_PASSWORD,\n",
        "            ),\n",
        "        )\n",
        "        astraSession = cluster.connect()\n",
        "        return astraSession\n",
        "    else:\n",
        "        raise ValueError('Unsupported CQL Session mode')\n",
        "\n",
        "def getCQLKeyspace(mode='astra_db'):\n",
        "    if mode == 'astra_db':\n",
        "        return ASTRA_DB_KEYSPACE\n",
        "    else:\n",
        "        raise ValueError('Unsupported CQL Session mode')\n",
        "\n",
        "def getTableCount():\n",
        "  # create a query that counts the number of records of the Astra DB table\n",
        "  query = SimpleStatement(f\"\"\"SELECT COUNT(*) FROM {keyspace}.{table_name};\"\"\")\n",
        "\n",
        "  # execute the query\n",
        "  results = session.execute(query)\n",
        "  return results.one().count"
      ],
      "id": "TUDw-07Iq4vE"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QXCQ6T_Gjk0Oz"
      },
      "source": [
        "### LLM Provider\n",
        "\n",
        "In the cell below you can choose between **GCP VertexAI** or **OpenAI** for your LLM services.\n",
        "(See [Pre-requisites](https://cassio.org/start_here/#llm-access) on cassio.org for more details).\n",
        "\n",
        "Make sure you set the `llmProvider` variable and supply the corresponding access secrets in the following cell."
      ],
      "id": "QXCQ6T_Gjk0Oz"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pGpzZc5zq4vG"
      },
      "outputs": [],
      "source": [
        "# Set your secret(s) for LLM access:\n",
        "llmProvider = 'OpenAI'  # 'GCP_VertexAI'\n"
      ],
      "id": "pGpzZc5zq4vG"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zOFStlEAq4vH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c3512074-db7c-41b7-fc5c-955fd9e01dde"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Your secret for LLM provider \"{llmProvider}\": ··········\n"
          ]
        }
      ],
      "source": [
        "if llmProvider == 'OpenAI':\n",
        "    apiSecret = getpass.getpass('Your secret for LLM provider \"{llmProvider}\": ')\n",
        "    os.environ['OPENAI_API_KEY'] = apiSecret\n",
        "elif llmProvider == 'GCP_VertexAI':\n",
        "    # we need a json file\n",
        "    getpass.getpass('Please upload your Service Account JSON for the LLM provider \"{llmProvider}\":')\n",
        "    from google.colab import files\n",
        "    uploaded = files.upload()\n",
        "    if uploaded:\n",
        "        vertexAIJsonFileTitle = list(uploaded.keys())[0]\n",
        "        os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = os.path.join(os.getcwd(), vertexAIJsonFileTitle)\n",
        "    else:\n",
        "        raise ValueError(\n",
        "            'No file uploaded. Please re-run the cell.'\n",
        "        )\n",
        "else:\n",
        "    raise ValueError('Unknown/unsupported LLM Provider')"
      ],
      "id": "zOFStlEAq4vH"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W0rYQTmwq4vI"
      },
      "source": [
        "### Colab preamble completed\n",
        "\n",
        "The following cells constitute the demo notebook proper."
      ],
      "id": "W0rYQTmwq4vI"
    },
    {
      "cell_type": "markdown",
      "id": "6715bc2b",
      "metadata": {
        "id": "6715bc2b"
      },
      "source": [
        "# Vector Similarity Search QA Quickstart"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "761d9b70",
      "metadata": {
        "id": "761d9b70"
      },
      "source": [
        "_**NOTE:** this uses Cassandra's \"Vector Similarity Search\" capability.\n",
        "Make sure you are connecting to a vector-enabled database for this demo._"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "042f832e",
      "metadata": {
        "id": "042f832e"
      },
      "outputs": [],
      "source": [
        "from langchain.indexes import VectorstoreIndexCreator\n",
        "from langchain.text_splitter import (\n",
        "    CharacterTextSplitter,\n",
        "    RecursiveCharacterTextSplitter,\n",
        ")\n",
        "from langchain.docstore.document import Document\n",
        "from langchain.document_loaders import TextLoader\n",
        "from langchain.indexes.vectorstore import VectorStoreIndexWrapper\n",
        "from cassandra.query import SimpleStatement"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4388ac1d",
      "metadata": {
        "id": "4388ac1d"
      },
      "source": [
        "The following line imports the Cassandra flavor of a LangChain vector store:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d65c46f0",
      "metadata": {
        "id": "d65c46f0"
      },
      "outputs": [],
      "source": [
        "from langchain.vectorstores.cassandra import Cassandra"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4578a87b",
      "metadata": {
        "id": "4578a87b"
      },
      "source": [
        "A database connection is needed to access Cassandra. The following assumes\n",
        "that a _vector-search-capable Astra DB instance_ is available. Adjust as needed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "11013224",
      "metadata": {
        "id": "11013224",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "005957ae-ad4e-450e-ab1f-2a95f9aa8210"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:cassandra.cluster:Downgrading core protocol version from 66 to 65 for a4c29bc6-b900-44c5-b980-584a0c4542b5-us-east1.db.astra.datastax.com:29042:05d703a1-936c-48cc-9ed8-c195f51b1c07. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version\n",
            "WARNING:cassandra.cluster:Downgrading core protocol version from 65 to 5 for a4c29bc6-b900-44c5-b980-584a0c4542b5-us-east1.db.astra.datastax.com:29042:05d703a1-936c-48cc-9ed8-c195f51b1c07. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version\n",
            "ERROR:cassandra.connection:Closing connection <AsyncoreConnection(135229325036272) a4c29bc6-b900-44c5-b980-584a0c4542b5-us-east1.db.astra.datastax.com:29042:05d703a1-936c-48cc-9ed8-c195f51b1c07> due to protocol error: Error from server: code=000a [Protocol error] message=\"Beta version of the protocol used (5/v5-beta), but USE_BETA flag is unset\"\n",
            "WARNING:cassandra.cluster:Downgrading core protocol version from 5 to 4 for a4c29bc6-b900-44c5-b980-584a0c4542b5-us-east1.db.astra.datastax.com:29042:05d703a1-936c-48cc-9ed8-c195f51b1c07. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version\n"
          ]
        }
      ],
      "source": [
        "# creation of the DB connection\n",
        "cqlMode = 'astra_db'\n",
        "session = getCQLSession(mode=cqlMode)\n",
        "keyspace = getCQLKeyspace(mode=cqlMode)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "32e2a156",
      "metadata": {
        "id": "32e2a156"
      },
      "source": [
        "Both an LLM and an embedding function are required.\n",
        "\n",
        "Below is the logic to instantiate the LLM and embeddings of choice. We choose to leave it in the notebooks for clarity."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "124e3de4",
      "metadata": {
        "id": "124e3de4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3c73c8f7-4131-45ca-d3d7-75b950e6cb00"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LLM+embeddings from OpenAI\n"
          ]
        }
      ],
      "source": [
        "# creation of the LLM resources\n",
        "\n",
        "if llmProvider == 'GCP_VertexAI':\n",
        "    from langchain.llms import VertexAI\n",
        "    from langchain.embeddings import VertexAIEmbeddings\n",
        "    llm = VertexAI()\n",
        "    myEmbedding = VertexAIEmbeddings()\n",
        "    print('LLM+embeddings from VertexAI')\n",
        "elif llmProvider == 'OpenAI':\n",
        "    from langchain.llms import OpenAI\n",
        "    from langchain.embeddings import OpenAIEmbeddings\n",
        "    llm = OpenAI(temperature=0)\n",
        "    myEmbedding = OpenAIEmbeddings()\n",
        "    print('LLM+embeddings from OpenAI')\n",
        "else:\n",
        "    raise ValueError('Unknown LLM provider.')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "285f29cf",
      "metadata": {
        "id": "285f29cf"
      },
      "source": [
        "## Langchain Retrieval Augmentation"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5cf74a31",
      "metadata": {
        "id": "5cf74a31"
      },
      "source": [
        "The following is a minimal usage of the Cassandra vector store. The store is created and filled at once, and is then queried to retrieve relevant parts of the indexed text, which are then stuffed into a prompt finally used to answer a question."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6f29fc57",
      "metadata": {
        "id": "6f29fc57"
      },
      "source": [
        "The following creates an \"index creator\", which knows about the type of vector store, the embedding to use and how to preprocess the input text:\n",
        "\n",
        "_(Note: stores built with different embedding functions will need different tables. This is why we append the `llmProvider` name to the table name in the next cell.)_"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d2cfe71b",
      "metadata": {
        "id": "d2cfe71b"
      },
      "outputs": [],
      "source": [
        "table_name = 'vs_test1_' + llmProvider\n",
        "\n",
        "index_creator = VectorstoreIndexCreator(\n",
        "    vectorstore_cls=Cassandra,\n",
        "    embedding=myEmbedding,\n",
        "    text_splitter=CharacterTextSplitter(\n",
        "        chunk_size=400,\n",
        "        chunk_overlap=0,\n",
        "    ),\n",
        "    vectorstore_kwargs={\n",
        "        'session': session,\n",
        "        'keyspace': keyspace,\n",
        "        'table_name': table_name,\n",
        "    },\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create the Cassandra Vector Store and clear entries if the table already exists"
      ],
      "metadata": {
        "id": "C7V7uPgvE3yY"
      },
      "id": "C7V7uPgvE3yY"
    },
    {
      "cell_type": "code",
      "source": [
        "myCassandraVStore = Cassandra(\n",
        "    embedding=myEmbedding,\n",
        "    session=session,\n",
        "    keyspace=keyspace,\n",
        "    table_name='vs_test1_' + llmProvider,\n",
        ")\n",
        "\n",
        "myCassandraVStore.clear()"
      ],
      "metadata": {
        "id": "SCxWxjRWl8Dg",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 366
        },
        "outputId": "9897dfad-92df-4749-931d-6dde012d4a25"
      },
      "id": "SCxWxjRWl8Dg",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "TypeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-16-f94b25e48a97>\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m myCassandraVStore = Cassandra(\n\u001b[0m\u001b[1;32m      2\u001b[0m     \u001b[0membedding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmyEmbedding\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m     \u001b[0msession\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m     \u001b[0mkeyspace\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkeyspace\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0mtable_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'vs_test1_'\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mllmProvider\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/langchain/vectorstores/cassandra.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, embedding, session, keyspace, table_name, ttl_seconds)\u001b[0m\n\u001b[1;32m     90\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_embedding_dimension\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     91\u001b[0m         \u001b[0;31m#\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 92\u001b[0;31m         self.table = VectorTable(\n\u001b[0m\u001b[1;32m     93\u001b[0m             \u001b[0msession\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     94\u001b[0m             \u001b[0mkeyspace\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkeyspace\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mTypeError\u001b[0m: VectorTable.__init__() got an unexpected keyword argument 'primary_key_type'"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "mySplitter = RecursiveCharacterTextSplitter(chunk_size=250, chunk_overlap=120)"
      ],
      "metadata": {
        "id": "6ITWD1pqmgeu"
      },
      "id": "6ITWD1pqmgeu",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The Astra DB table should be cleared from the last statement. Let's do a quick table count to make sure."
      ],
      "metadata": {
        "id": "AFQxGLE_D4Ti"
      },
      "id": "AFQxGLE_D4Ti"
    },
    {
      "cell_type": "code",
      "source": [
        "# create a query that counts the number of records of the Astra DB table\n",
        "print('Total records: ' + str(getTableCount()))"
      ],
      "metadata": {
        "id": "_v6hqPI_EDp4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "19c7d466-11f6-4096-c567-3b0f7905b053"
      },
      "id": "_v6hqPI_EDp4",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:cassandra.protocol:Server warning: Aggregation query used without partition key\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total records: 0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create the function for creating a vector index for a Wikipedia entry"
      ],
      "metadata": {
        "id": "oogCCCXcv3yG"
      },
      "id": "oogCCCXcv3yG"
    },
    {
      "cell_type": "code",
      "source": [
        "def create_vector_index(row, myCassandraVStore):\n",
        "  metadata = {\n",
        "    'url': row['url'],\n",
        "    'title': row['title']\n",
        "  }\n",
        "  page_content = row['text']\n",
        "\n",
        "  wikiDocument = Document(\n",
        "      page_content=page_content,\n",
        "      metadata=metadata\n",
        "  )\n",
        "  wikiDocs = mySplitter.transform_documents([wikiDocument])\n",
        "  myCassandraVStore.add_documents(wikiDocs)"
      ],
      "metadata": {
        "id": "mbsblXxQwAT9"
      },
      "id": "mbsblXxQwAT9",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Execute the create_vector_index function for each row in the Wikipedia dataframe."
      ],
      "metadata": {
        "id": "ZCXx5rKUxj_z"
      },
      "id": "ZCXx5rKUxj_z"
    },
    {
      "cell_type": "code",
      "source": [
        "for index, row in data.iterrows():\n",
        "  create_vector_index(row, myCassandraVStore)"
      ],
      "metadata": {
        "id": "M3K5EOR1xrdO"
      },
      "id": "M3K5EOR1xrdO",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that we've processed records by embedding them with vector search values, let's see how many records are in the Astra DB table. We'll also look at one of the rows and a snippet of it's new vector values"
      ],
      "metadata": {
        "id": "f4QNwcMuJAXh"
      },
      "id": "f4QNwcMuJAXh"
    },
    {
      "cell_type": "code",
      "source": [
        "# Count the number records of the Astra DB table after the embedding\n",
        "print('Total records: ' + str(getTableCount()))"
      ],
      "metadata": {
        "id": "luCa9nvqNdXm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7885773f-85c9-46b2-9f2b-db73bdfed9ee"
      },
      "id": "luCa9nvqNdXm",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:cassandra.protocol:Server warning: Aggregation query used without partition key\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total records: 329\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# create a query that returns the first row of the Astra DB table\n",
        "query = SimpleStatement(f\"\"\"SELECT * from {keyspace}.{table_name} limit 1;\"\"\")\n",
        "\n",
        "# execute the query and display the first row of the table\n",
        "results = session.execute(query)\n",
        "print('document_id: ' + results.one().document_id)\n",
        "print('document: ' + results.one().document)\n",
        "print('metadata_blob: ' + results.one().metadata_blob)\n",
        "# print('first 20 bytes of embedding_vector: ' + str(results.one().embedding_vector[:20]))\n",
        "# optional print of the whole vector (comment out line above)\n",
        "print('mbedding_vector: ' + str(results.one().embedding_vector))"
      ],
      "metadata": {
        "id": "pQUQd1uSNe2G",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c0f86dbf-3ccb-4fb6-b732-c6c83332b13a"
      },
      "id": "pQUQd1uSNe2G",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "document_id: 290614f0c86ee39a4c6972f156ad7c8d\n",
            "document: August 17  1945: Indonesia declares independence from the Netherlands.\n",
            " August 17  1960: Gabon becomes independent.\n",
            " August 17  1962: Peter Fechter becomes the first person to be shot dead at the Berlin Wall.\n",
            "metadata_blob: {\"url\": \"https://simple.wikipedia.org/wiki/August\", \"title\": \"August\"}\n",
            "mbedding_vector: [-0.011523437686264515, -0.008404161781072617, 0.00027421972481533885, -0.006563061382621527, -0.0009768431773409247, -0.003413983853533864, -0.02405351959168911, 0.013934087939560413, -0.02080841362476349, -0.020901132375001907, -0.0012185703963041306, 0.030278826132416725, -2.0582649540301645e-06, 0.004533214494585991, -0.0015298356302082539, 0.010722095146775246, 0.02283494919538498, 0.0057716527953743935, 0.005212037358433008, 0.0049073947593569756, -0.007967066019773483, -0.0047914981842041016, 0.010993624106049538, 0.019165992736816406, 0.021801814436912537, 0.0038543911650776863, 0.010702227242290974, -0.038967762142419815, 0.028424479067325592, -0.016848061233758926, -0.0015977179864421487, -0.0036225980147719383, -0.016397718340158463, -0.00014756126620341092, -0.02993444725871086, -0.008152500726282597, -0.026450924575328827, -0.016344737261533737, -0.0007156617939472198, 0.019060030579566956, 0.019629579037427902, 0.016543418169021606, 0.004102740902453661, 0.0013013536809012294, -0.009609486907720566, 0.03374910354614258, 0.013397651724517345, 0.010483679361641407, -0.018675915896892548, 0.028530441224575043, 0.02063622511923313, 0.027762211859226227, -0.020238865166902542, -0.00659617455676198, -0.0023924377746880054, -0.028848329558968544, -0.009622732177376747, 0.004880904220044613, -0.003235172014683485, -0.007013402413576841, -0.01597386784851551, -0.004755073692649603, -0.017033495008945465, 0.012987046502530575, -0.029696030542254448, -0.009973733685910702, -0.005563038866966963, 0.010073074139654636, -0.011556550860404968, 0.02778870426118374, 0.02961655892431736, 0.02557673305273056, 0.014596354216337204, -0.00011279226600890979, 0.020265355706214905, -0.014450655318796635, -0.009483656845986843, 0.020146148279309273, -0.017324892804026604, 0.011338002979755402, 0.01990773156285286, -0.027126437053084373, -0.013298312202095985, 0.025603223592042923, 0.026967493817210197, -0.003973599057644606, 0.010351225733757019, 0.014384428970515728, 0.017338138073682785, -0.008874371647834778, 0.0025795281399041414, 0.0362127348780632, 0.013126122765243053, 0.0033874933142215014, -0.011152568273246288, 0.0061193425208330154, -0.008046538569033146, 0.012490347027778625, -0.004665667656809092, -0.03446435183286667, 0.02192102186381817, -0.009947243146598339, -0.021974002942442894, -0.021245509386062622, -0.009132655337452888, 0.009655846282839775, 0.019046785309910774, -0.02121901884675026, 0.0013708916958421469, -0.006463721394538879, -0.015788434073328972, 0.02506016567349434, 0.03711341693997383, -0.05102101340889931, 0.010821434669196606, -0.04445132985711098, -0.034914691001176834, -0.012298289686441422, 0.012900952249765396, -0.03812006115913391, 9.10616508917883e-05, 0.025748923420906067, 0.03284842148423195, -0.013682426884770393, 0.019271956756711006, 0.0012409219052642584, -0.006139210890978575, -0.006076295394450426, -0.006324645131826401, -0.010092942044138908, 0.0006088712834753096, 0.0038643251173198223, -0.012338025495409966, -0.006973666604608297, 0.010900907218456268, 0.01728515699505806, -0.033007364720106125, 0.004142477177083492, -0.019775278866291046, -0.02353695221245289, 0.023722387850284576, 0.009523392654955387, -0.03062320500612259, -0.01859644427895546, 0.004327911883592606, 0.02245083451271057, 0.023126346990466118, -0.0049338857643306255, 0.026927756145596504, 0.016715606674551964, 0.014278465881943703, 0.011126077733933926, 0.0013129434082657099, -0.019232219085097313, 0.007801499683409929, 0.015947377309203148, -0.00429810956120491, 0.031894754618406296, 0.007914084941148758, 0.0032666295301169157, -0.005430585704743862, 0.006059738807380199, -0.013814879581332207, 0.01025850884616375, 0.021854795515537262, 0.037563756108284, -0.003957042470574379, 0.025245599448680878, -0.012920820154249668, 0.025762168690562248, 0.025338318198919296, 0.02092762291431427, -0.007516724988818169, 0.02437140792608261, 0.007145856041461229, 0.03928564861416817, 0.015232129953801632, 0.024940958246588707, -0.023589933291077614, -0.03245105966925621, -0.016636135056614876, 0.021775323897600174, 0.02263627015054226, 0.03364314138889313, -0.006334579549729824, -0.012238685041666031, 0.020755432546138763, -0.0038742590695619583, 0.004264996387064457, -0.014914242550730705, -0.0021523660980165005, -0.007629310246556997, 0.016516927629709244, -0.01414601318538189, -0.6395905017852783, -0.0037517398595809937, -0.008708804845809937, -0.005225283093750477, 0.01617254875600338, 0.025815149769186974, 0.012185703963041306, 0.03790813684463501, -0.011152568273246288, 0.00833793543279171, -0.01617254875600338, 0.013245330192148685, 0.016424210742115974, -0.023430990055203438, -0.026742322370409966, -0.010225394740700722, -0.02458333410322666, -0.00016235877410508692, -0.007265063934028149, 0.01389435213059187, 0.009079674258828163, 0.01666262559592724, -0.013629444874823093, -0.0029884777031838894, -0.01182807981967926, -0.011059851385653019, 0.02588137611746788, -0.009675714187324047, -0.012490347027778625, 0.03194773569703102, -0.01727191172540188, 0.019417654722929, -0.0038113438058644533, 0.019086521118879318, 0.037855155766010284, -0.002284819493070245, -0.03880881890654564, 0.013722162693738937, -0.006208748556673527, 0.0414578840136528, -0.0047517623752355576, -0.003917306661605835, 0.006794854532927275, -0.004738517105579376, -0.005010046064853668, 0.0009735318017192185, 0.03255702182650566, -0.0060895406641066074, 0.006645844783633947, -0.0333782322704792, 0.01980176940560341, -0.001955342013388872, -0.01839776337146759, 0.009132655337452888, -0.016040096059441566, 0.013205594383180141, -0.003543125931173563, -0.014914242550730705, 0.006443853490054607, -6.389836926246062e-05, -0.0034636538475751877, 0.017510326579213142, -0.013059895485639572, 0.019894486293196678, -0.019046785309910774, 0.017867950722575188, -0.016344737261533737, 0.02972252108156681, 0.0137486532330513, -0.006861081346869469, -0.008854503743350506, 0.015629490837454796, -0.017338138073682785, -0.004245128482580185, 0.022477325052022934, 0.018040141090750694, 0.011934042908251286, -0.010874416679143906, -0.010980378836393356, -0.005036537069827318, -0.008357803337275982, -0.0035927959252148867, -0.004182212986052036, 0.04243803769350052, 0.014331447891891003, -0.010404206812381744, -0.015271865762770176, -0.008828013204038143, 0.014318202622234821, 0.007669046521186829, 0.029881466180086136, 0.006221993826329708, 0.014424164779484272, 0.009198881685733795, 0.005255084950476885, 0.01931169256567955, -0.012443987652659416, -0.030649695545434952, -0.003973599057644606, -0.020675960928201675, -0.013139368034899235, -0.034305404871702194, -0.007993556559085846, -0.017417609691619873, -0.005427274387329817, 0.010132677853107452, -0.001129164476878941, -0.012450610287487507, 0.047338809818029404, -0.027417834848165512, -0.006337890867143869, -0.008033293299376965, -0.01728515699505806, 0.0031689454335719347, 0.001067904755473137, -0.024940958246588707, 0.03446435183286667, 0.008450521156191826, 0.0271794181317091, -0.022119702771306038, 0.0014197338605299592, -0.019656069576740265, 0.02173558808863163, -0.03751077502965927, 0.028159571811556816, 0.013549973256886005, -0.013218839652836323, -0.01769576221704483, -0.019232219085097313, 0.005235217045992613, 0.0034934559371322393, 0.0057617188431322575, 0.0338020846247673, 0.00995386578142643, 0.010900907218456268, 0.00432460056617856, -0.005152433644980192, -0.00404644850641489, -0.003851079847663641, -0.004678912926465273, -0.03610677272081375, 0.01561624463647604, 0.005066338926553726, -0.0020066676661372185, 0.0053875381126999855, -0.024729032069444656, -0.016199039295315742, -0.008477011695504189, -0.013059895485639572, -0.0018858038820326328, 0.011887684464454651, -0.017020249739289284, -0.012238685041666031, -0.00986114889383316, -0.004914017394185066, -0.007271686568856239, -0.0323186069726944, -0.023669404909014702, -0.017642781138420105, 0.0021076633129268885, 0.003437163308262825, -0.007569706533104181, -0.016503682360053062, 0.005006734747439623, 0.026424434036016464, -0.0054835667833685875, 0.0220402292907238, 0.01789444126188755, -0.012258552946150303, -0.02929867058992386, -0.018053386360406876, 0.02121901884675026, -0.0056954920291900635, -0.006258418783545494, 0.007046516053378582, 0.011046606115996838, -0.0015414253575727344, -0.008616087958216667, 0.020172638818621635, -0.0036557111889123917, 0.0022235598880797625, -0.03226562589406967, 0.01385461539030075, -0.014132767915725708, 0.014530127868056297, 0.017947422340512276, 0.0024255511816591024, 0.006599485874176025, -0.019126256927847862, 0.017735498026013374, -0.008298199623823166, -0.008026670664548874, 0.013530105352401733, -0.014503637328743935, -0.0030348363798111677, 0.04151086509227753, 0.022384608164429665, 0.014569863677024841, 0.022318381816148758, 0.03849092870950699, 0.030808638781309128, 0.012324780225753784, 0.032583512365818024, -0.018344782292842865, -0.0007102808449417353, -0.006827967707067728, -0.012198949232697487, -0.01577518880367279, 0.018781878054142, 0.026305226609110832, 0.022411098703742027, -0.007722027599811554, -0.01809312216937542, -0.014066540636122227, 0.016397718340158463, 0.0008592908270657063, -0.008675691671669483, -0.0177884791046381, -0.011463833972811699, 0.005374292843043804, 0.02356344275176525, 0.004480232950299978, 0.012119477614760399, 0.009172391146421432, -0.02921919897198677, 0.032504040747880936, 0.007622687611728907, 0.02053026296198368, 0.017007004469633102, -0.053617097437381744, -0.017960667610168457, -0.0015207295073196292, 0.003187157679349184, -0.004284864291548729, 0.01768251694738865, 0.0057716527953743935, -0.0007657456444576383, -0.02333827316761017, 0.031179508194327354, 0.02333827316761017, 0.011768476106226444, -0.006529947742819786, 0.036345187574625015, -0.009165768511593342, -0.0003174740122631192, -0.02011965773999691, 0.020861394703388214, 0.011410851962864399, -0.026411188766360283, 0.028371497988700867, -0.011689004488289356, -0.0010488645639270544, -0.009483656845986843, 0.006265041418373585, 0.0025563486851751804, -0.03912670537829399, 0.009596241638064384, 0.0009751874604262412, -0.016596399247646332, 0.029245689511299133, 0.03136494383215904, -0.02245083451271057, 0.006655778735876083, -0.02172234281897545, -0.0065465047955513, 0.022013738751411438, -0.01319234911352396, 0.00424843979999423, 0.009940620511770248, -0.01830504648387432, 0.0014818212948739529, -0.008251841180026531, 0.013357915915548801, -0.01649043709039688, 0.004738517105579376, -0.016053341329097748, 0.019576597958803177, -0.01278836652636528, 0.014384428970515728, 0.012198949232697487, -0.010000224225223064, 0.012106232345104218, 0.010642623528838158, 0.03973599150776863, -0.006294843275099993, -0.03737832233309746, -0.02435816265642643, 0.010947265662252903, -0.04461027309298515, 0.016437456011772156, -0.012424119748175144, 0.00985452625900507, 0.016238775104284286, -0.01879512332379818, -0.004920640494674444, -0.019086521118879318, 0.01902029477059841, -0.013973823748528957, 0.00819223653525114, -0.03499416261911392, 0.0006684752879664302, -0.008933975361287594, -0.0016581497620791197, -0.010887661948800087, 0.014543373137712479, -0.00047103705583140254, -0.010688981972634792, -0.017629534006118774, 0.025046920403838158, 0.0026705898344516754, 0.003976910375058651, 0.0031308650504797697, -0.028424479067325592, -0.0020116346422582865, 0.005543170962482691, -0.0016680838307365775, -0.02274223230779171, 7.869589171605185e-05, 0.005188858136534691, -0.009616109542548656, -0.020358072593808174, -0.005639199633151293, -0.021788569167256355, -0.012245307676494122, 0.11295618116855621, 0.02111305668950081, 0.007596197072416544, 0.002096073469147086, 0.003861013799905777, -0.014477145858108997, -0.012602931819856167, -0.02425220049917698, 0.016954023391008377, -0.014993714168667793, 0.008761785924434662, 0.010927397757768631, 0.004006712697446346, -0.006751807406544685, 0.016728851944208145, -0.010397584177553654, -0.0015778499655425549, -0.028742367401719093, -0.0014776821481063962, -0.008761785924434662, -0.01011280994862318, -0.009371071122586727, 0.01798715814948082, 0.03470276668667793, -0.013179103843867779, -0.005692180711776018, 0.005112697370350361, 0.002258328953757882, 0.01717919297516346, 0.001543081016279757, -0.011039982549846172, 0.003947108518332243, 0.028027119114995003, 0.019046785309910774, -0.007066383957862854, 0.01480827946215868, 0.015444055199623108, 0.029749011620879173, 0.019417654722929, -0.010351225733757019, 0.007662423886358738, -0.0028179441578686237, -0.007900839671492577, -0.014490392059087753, -0.005755096208304167, -0.019629579037427902, -0.02417272888123989, 0.014066540636122227, -0.019285202026367188, -0.0038378345780074596, 0.019854750484228134, -0.021351473405957222, -0.009887639433145523, -0.020371317863464355, -0.015483791008591652, -0.003639154601842165, 0.010503547266125679, -0.004258373752236366, -0.007470366545021534, 0.0025977403856813908, 0.016728851944208145, -0.03396102786064148, 0.0048477910459041595, -0.0037749193143099546, 0.010271754115819931, -0.033908046782016754, -0.028053609654307365, -0.0045265918597579, -0.017033495008945465, -0.03496767207980156, 0.0023129659239202738, -0.025232354179024696, -0.024689296260476112, 0.011218794621527195, 0.023470725864171982, 0.015629490837454796, 0.034199442714452744, 0.014596354216337204, -0.016530172899365425, -0.0005182235618121922, -0.005831256974488497, -0.04336521029472351, -0.006953798700124025, -0.0271794181317091, 0.024093255400657654, 0.029166217893362045, 0.040398258715867996, -0.02083490416407585, -0.020252110436558723, 0.014781788922846317, -0.014265220612287521, -0.002087795175611973, 0.01223206240683794, 0.004718649201095104, 0.006914062425494194, -0.00970220472663641, -0.006993534509092569, 0.0012309879530221224, 0.012139345519244671, -0.003940485883504152, 0.01333804801106453, -0.003466965164989233, -0.027576778084039688, -0.03062320500612259, 0.004592818208038807, 0.047974586486816406, 0.016834815964102745, 0.006642533466219902, -0.03576239198446274, 0.005824633873999119, 0.008834635838866234, -0.0221461933106184, 0.002154021989554167, -0.0015588098904117942, 0.022689251229166985, 0.018954068422317505, 0.019351428374648094, 0.008702182210981846, -0.0005827945424243808, -0.021483926102519035, -0.039179686456918716, 0.010291622020304203, 0.005596152041107416, 0.028636405244469643, -0.011927420273423195, 0.015311602503061295, -0.026344962418079376, -0.0020944178104400635, -0.012298289686441422, 0.0015844727167859674, -0.020238865166902542, 0.023735633119940758, -0.00833793543279171, -0.008205481804907322, -0.013384406454861164, 0.009828034788370132, -0.010483679361641407, 0.007649178151041269, 0.0010083008091896772, -0.013132745400071144, -0.005523303057998419, -0.01890108734369278, 0.005804765969514847, -0.012894329614937305, -0.03732534125447273, -0.02263627015054226, -0.014848015271127224, -0.010483679361641407, -0.005370981525629759, 0.0003025730256922543, -0.030384788289666176, -0.004112675320357084, -0.020848149433732033, 0.013430764898657799, -0.006953798700124025, -0.02415948361158371, -0.017046740278601646, -0.019245464354753494, 0.0234574805945158, 0.03094109147787094, 0.0049438197165727615, 0.017152702435851097, 0.044530801475048065, 0.017007004469633102, 0.01606658659875393, 0.009635978378355503, 0.007159101311117411, 0.017033495008945465, -0.027656249701976776, -0.00329808727838099, 0.016649380326271057, 0.012033382430672646, -0.010218772105872631, 0.017245421186089516, 0.013669180683791637, 0.0020646159537136555, -0.000375008414266631, 0.010887661948800087, 0.00416565639898181, -0.03586835414171219, 0.0015745386481285095, 0.01617254875600338, 0.004513346590101719, -0.0012723795371130109, -0.016596399247646332, -0.016119567677378654, 0.01970905251801014, -0.01879512332379818, 0.003076228080317378, 0.0034768993500620127, -0.017642781138420105, -0.016013605520129204, 0.030755657702684402, 0.005748473573476076, -0.01827855594456196, -0.020954113453626633, -0.00470209214836359, 0.005341179668903351, 0.01414601318538189, 0.010132677853107452, 0.010477056726813316, 6.964146450627595e-05, -0.028186064213514328, -0.02829202637076378, -0.0117287402972579, 0.016609644517302513, -0.010960510931909084, 0.004082872997969389, -0.012238685041666031, -0.021086566150188446, -0.01010618731379509, -0.0274973064661026, 0.00424181716516614, -0.017351383343338966, 0.012583063915371895, -0.000793064187746495, -0.0004817988956347108, 0.012596309185028076, 0.005692180711776018, -0.052928339689970016, 0.009364448487758636, -0.00357623933814466, 0.028662895783782005, 0.007424007635563612, 0.0325305312871933, 0.020702451467514038, -0.00030795394559390843, -0.03793462738394737, 0.0013601299142464995, 0.009477034211158752, 0.014013559557497501, 0.011159190908074379, 0.016291756182909012, 0.006122653838247061, -0.03298087418079376, -0.01737787388265133, 0.038861799985170364, -0.0274973064661026, 0.01011280994862318, 0.015801679342985153, 0.027762211859226227, -0.009920752607285976, -0.009126032702624798, 0.010715472511947155, -0.010410829447209835, 0.011801589280366898, -0.02162962406873703, -4.8557587433606386e-05, -0.019060030579566956, -0.028053609654307365, -0.014357938431203365, -0.010337980464100838, -0.034623295068740845, 0.014371183700859547, 0.004665667656809092, -0.0008592908270657063, -0.00016432488337159157, -0.0012533393455669284, -0.010781698860228062, 0.007476989179849625, -0.029854975640773773, -0.006053116172552109, 0.01132475771009922, 0.011741985566914082, -0.02496744878590107, 0.006225305143743753, -0.024914465844631195, -0.010503547266125679, 0.012298289686441422, 0.022901177406311035, -0.012450610287487507, 0.004019957967102528, -0.017656026408076286, 0.003973599057644606, 0.009510147385299206, -0.002662311540916562, 0.00424843979999423, -0.003831211943179369, 0.02498069405555725, -0.012179081328213215, 0.00042798975482583046, 0.010046583600342274, -0.009470411576330662, -0.014516882598400116, -0.00954988319426775, -0.020874641835689545, -0.00939756166189909, -0.027152927592396736, 0.007536592893302441, -0.011026737280189991, -0.0031076855957508087, 0.021404454484581947, -0.020887887105345726, -0.02275547757744789, -0.008907484821975231, 0.007622687611728907, 0.0024586645886301994, 0.010152545757591724, -0.029033765196800232, 0.002860991284251213, 0.016927532851696014, 0.018834859132766724, 1.931179576786235e-05, -0.02467605099081993, 0.010470434091985226, -0.007390894461423159, 0.018252065405249596, -0.013132745400071144, 0.011622777208685875, -0.0016747064655646682, 0.020185884088277817, 0.0035696164704859257, 0.019470635801553726, -0.005374292843043804, 0.02193426713347435, -0.0010745273903012276, 0.013384406454861164, -0.013311557471752167, -0.027046965435147285, 0.004642488434910774, -0.022702496498823166, 0.01545730046927929, 0.0054835667833685875, -0.030146371573209763, -0.0008332140860147774, -0.00980816688388586, 0.030358297750353813, 0.008377671241760254, -0.01626526564359665, -0.0024768768344074488, 0.015258620493113995, 0.010953888297080994, -0.0177884791046381, 0.009092919528484344, -0.03809357061982155, -0.02495420351624489, -0.009079674258828163, 0.009649223648011684, -0.02516612783074379, 0.0014478801749646664, 0.020993849262595177, 0.040292296558618546, 0.01596062257885933, -0.0009710483136586845, 0.007463743910193443, -0.018821613863110542, 0.01213272288441658, -0.008112764917314053, -0.014093032106757164, 0.013152613304555416, -0.0013932432048022747, 0.035841863602399826, -0.004851102363318205, 0.021974002942442894, 0.0007268374902196229, -0.041590336710214615, -0.017152702435851097, 0.006576306652277708, -0.01830504648387432, 0.00863595586270094, 0.04339170455932617, 0.0357888825237751, -0.009748563170433044, 0.016556663438677788, -0.0004151583416387439, 0.015576508827507496, -0.01309963222593069, -0.012927442789077759, 0.041192978620529175, -0.006778297945857048, 0.005672312807291746, -0.011311512440443039, -0.010404206812381744, -0.016132812947034836, 0.03565642982721329, -0.02223891019821167, 0.006960421334952116, 0.019576597958803177, -0.006483589299023151, 0.006609419826418161, 0.010695604607462883, -0.026133036240935326, 0.0013593020848929882, 0.019152747467160225, 0.015165903605520725, -0.042914871126413345, 0.014132767915725708, 0.009033315815031528, 0.0003855632967315614, -0.009344580583274364, -0.009106164798140526, -0.008417407982051373, 0.0021424321457743645, 0.007722027599811554, -0.012053250335156918, -0.0006659917999058962, 0.005129254423081875, -0.014728807844221592, 0.030384788289666176, 0.004619309213012457, -0.0015232129953801632, 0.026265490800142288, -0.02111305668950081, 0.01829180121421814, 0.0033709367271512747, -0.005758407525718212, -0.0002162714081350714, 0.0037153151351958513, -0.016636135056614876, 0.0008158295531757176, 0.030278826132416725, -0.017059985548257828, 0.03284842148423195, -0.019483881071209908, -0.007801499683409929, -0.024808503687381744, -0.004331223201006651, -0.03480872884392738, 0.018172593787312508, -0.0023808481637388468, -0.04659707099199295, -0.0038245890755206347, -0.009629355743527412, -0.002899071667343378, -0.012251930311322212, -0.01304665021598339, -0.017245421186089516, -0.015483791008591652, 0.008722050115466118, 0.006251796148717403, -0.006354447454214096, -0.00424181716516614, 0.01303340494632721, -0.017854705452919006, 0.004490166902542114, 0.006652467418462038, 0.1979382187128067, 0.007609442342072725, 0.009616109542548656, 0.023470725864171982, -0.02255679853260517, 0.0025778724811971188, 0.01780172437429428, 0.004387515597045422, 0.015563263557851315, 0.012563196010887623, -0.02475552260875702, 0.0037484285421669483, -0.027470815926790237, 0.009907507337629795, 0.012569818645715714, -0.015165903605520725, -0.021311737596988678, -0.02325879968702793, -0.026252245530486107, 0.01566922664642334, -0.011185681447386742, 0.017205683514475822, -0.006404117215424776, 0.00890086218714714, 0.019060030579566956, 0.00715247867628932, 0.009417429566383362, 0.006705448497086763, 0.022609779611229897, -0.0025166128762066364, -0.027921156957745552, -0.004053071141242981, -0.007178969215601683, 0.009761808440089226, -0.023126346990466118, -0.00740413973107934, -0.022119702771306038, 0.00370206986553967, 0.0080796517431736, 0.02161637879908085, -0.01021214947104454, -0.02123226411640644, 0.016225529834628105, -0.0161593034863472, 0.00809289701282978, 0.031974226236343384, -0.01455661840736866, -0.014993714168667793, -0.005168990232050419, 0.013523482717573643, -0.025748923420906067, -0.024993939325213432, -0.0015712273307144642, -0.0006871015066280961, 0.007106119766831398, -0.022199174389243126, 0.009126032702624798, -0.0031523886136710644, 0.007331290282309055, 0.009006824344396591, -0.007854481227695942, 0.030649695545434952, -0.04569638893008232, 0.009669091552495956, -0.0110201146453619, 0.033828575164079666, -0.0009743596310727298, 0.03827900439500809, 0.021987248212099075, -0.029775502160191536, 0.013828124850988388, -0.009086296893656254, -0.026424434036016464, -0.0037318719550967216, -0.007708782330155373, -0.03202721104025841, 0.053007811307907104, 0.010708849877119064, 0.012523460201919079, -0.011543305590748787, 0.010079696774482727, 0.006569684017449617, 0.01116581354290247, 0.00819223653525114, -0.04312679544091225, -0.004255062434822321, 0.012026759795844555, 0.00858959648758173, -0.027841685339808464, 0.0002673901035450399, 0.005245150998234749, -0.02821255475282669, -0.0008816423360258341, 0.03131196275353432, 0.01970905251801014, 0.012775121256709099, 0.03102056495845318, 0.007596197072416544, -0.01696726866066456, -0.0274973064661026, -0.008682314306497574, 0.011033359915018082, 0.03266298398375511, -0.023523706942796707, -0.017563307657837868, 0.003943797200918198, -0.004182212986052036, -0.0021904464811086655, 0.0164771918207407, -0.01278174389153719, -0.023073365911841393, -0.013059895485639572, -0.0011382705997675657, -0.006592863239347935, -0.005066338926553726, 0.017324892804026604, -0.01666262559592724, -0.02911323681473732, 0.002205347642302513, -0.014053295366466045, -0.02233162708580494, -0.03184177353978157, 0.016318246722221375, 0.0028477460145950317, -0.008549860678613186, -0.01951037161052227, -0.020742187276482582, 0.025444280356168747, -0.0117287402972579, -0.02172234281897545, -0.013510237447917461, -0.020689206197857857, 0.019245464354753494, -0.016821570694446564, 0.005596152041107416, 0.010437320917844772, -0.01101349201053381, 0.0054074060171842575, -0.014477145858108997, 0.02192102186381817, -0.01658315397799015, -0.01561624463647604, -0.001976865576580167, 0.011748608201742172, 0.027550287544727325, 0.0015761943068355322, 0.011682381853461266, -0.0005314688896760345, -0.03793462738394737, -0.004457053728401661, -0.013159235939383507, -0.01344401016831398, 0.0015016893157735467, 0.013404274359345436, 0.03083512932062149, -0.014940733090043068, -0.007318045012652874, -0.031073546037077904, -0.00599020067602396, 0.015576508827507496, -0.014331447891891003, -0.006625976413488388, 0.024119747802615166, -0.032000716775655746, -0.0033759037032723427, -0.003149077296257019, -0.1677388697862625, 0.009576373733580112, 0.03072916716337204, -0.00904656108468771, 0.001235127099789679, -0.0045265918597579, 0.031576868146657944, -0.029139727354049683, 0.009510147385299206, 0.006460410077124834, 0.021377963945269585, 0.019245464354753494, -0.031788792461156845, -0.02275547757744789, 0.0049338857643306255, 0.010801566764712334, -0.009781676344573498, 0.034623295068740845, 0.04529903084039688, 0.010861171409487724, 0.01536458358168602, -0.019338183104991913, 0.02336476370692253, 0.010232017375528812, -0.014821524731814861, 0.01707323081791401, -0.007198837120085955, -0.022874685004353523, -0.005486878100782633, -0.017841460183262825, -0.04288838058710098, -0.009437298402190208, 0.031682830303907394, -0.010960510931909084, -0.009867771528661251, -0.0026755568105727434, -0.019762033596634865, 0.001740105333738029, -0.019232219085097313, 0.022795213386416435, 0.021086566150188446, 0.021563397720456123, -0.007443875540047884, 0.010688981972634792, 0.0031937803141772747, 0.02002693898975849, 0.008119387552142143, -0.0021175972651690245, 0.01830504648387432, 0.010629377327859402, -0.017457345500588417, -0.028371497988700867, -0.0056193312630057335, 0.011258531361818314, 0.010642623528838158, 0.009986978955566883, -0.0010869449470192194, 0.0090002017095685, 0.00222190422937274, 0.010463811457157135, 0.00010658351675374433, -0.012179081328213215, 0.006642533466219902, -0.033616647124290466, -0.02133822813630104, 0.00035058733192272484, -0.022159438580274582, 0.01617254875600338, -0.0204110536724329, 0.011311512440443039, -0.01156979613006115, -0.034835219383239746, 0.017934177070856094, -0.03849092870950699, -0.007569706533104181, -0.006397494580596685, -0.009099542163312435, -0.024993939325213432, -0.0110730966553092, 0.023788614198565483, -0.016503682360053062, 0.04466325417160988, 0.011953910812735558, 0.01358970906585455, -0.007761763874441385, 0.01319234911352396, 0.009079674258828163, 0.030888110399246216, -0.0467030368745327, -0.0386233851313591, -0.0031407990027219057, -0.011258531361818314, -0.027735721319913864, -0.023417744785547256, 0.01789444126188755, 0.014384428970515728, 0.0003524499770719558, -0.013318180106580257, -0.0016879518516361713, -0.01385461539030075, -0.017920931801199913, 0.012371138669550419, -0.0054173399694263935, 0.01520563941448927, 0.01687455177307129, 0.012569818645715714, -0.010973756201565266, -0.006755118723958731, 0.02545752562582493, 0.016238775104284286, -0.03475574776530266, 0.017814969643950462, -0.0015372862108051777, 0.03102056495845318, -0.00108115014154464, 0.017205683514475822, -0.007993556559085846, -0.019788524135947227, 0.0221461933106184, 0.01850372739136219, 0.04601427912712097, -0.015934132039546967, 0.010993624106049538, -0.007377649191766977, -0.012457232922315598, -0.007589574437588453, -0.08561781793832779, -0.004685535561293364, 0.02860991470515728, 0.02213294804096222, 0.03125898167490959, 0.01790768653154373, 0.0008212105021812022, 0.02143094502389431, 0.0003795615048147738, 0.012549950741231441, -0.009424053132534027, -0.024490615352988243, 0.012556573376059532, -0.008715427480638027, 0.04590831696987152, 0.002273229882121086, 0.021589888259768486, -0.025046920403838158, -0.003927240613847971, 0.012556573376059532, -0.0002091313508572057, 0.0029189398046582937, 0.004106052685528994, -0.009536637924611568, -0.006255107466131449, -0.014993714168667793, -0.03510012477636337, 0.032212644815444946, 0.004066316410899162, -0.010304867289960384, -0.02417272888123989, -0.01308638695627451, 0.011669136583805084, -0.03215966373682022, -0.015311602503061295, -0.0205170176923275, -0.04612024128437042, 0.00028643026598729193, -0.004013335332274437, -0.017642781138420105, 0.004357713740319014, 0.023205818608403206, 0.018556708469986916, -0.013179103843867779, -0.01496722362935543, -0.02303363010287285, -0.00099257193505764, 0.011278399266302586, 0.006712071131914854, -0.025444280356168747, -0.026464169844985008, -0.02272898703813553, -0.013536727987229824, -0.02030509151518345, 0.009212126955389977, 0.003250072943046689, 0.0013882762286812067, 0.005457076244056225, -0.004890838172286749, 0.012311534956097603, -0.016305001452565193, 0.016530172899365425, -0.006980289239436388, -0.012894329614937305, -0.001986799528822303, 0.019550107419490814, -0.03284842148423195, -0.010907529853284359, 0.012238685041666031, 0.0181063674390316, -0.012914197519421577, 0.0030547042842954397, -0.00899357907474041, 0.006513391155749559, -0.03732534125447273, -0.008549860678613186, -0.042491018772125244, -0.007139232940971851, 0.017642781138420105, -0.0234574805945158, -0.030596712604165077, -0.007563083898276091, 0.0055299256928265095, 0.02394755743443966, 0.03062320500612259, 0.035444505512714386, 0.021603133529424667, 0.013669180683791637, 0.017523571848869324, -0.005414028652012348, -0.0038742590695619583, 0.03396102786064148, -0.00510607473552227, -0.020874641835689545, 0.004940508399158716, 0.027152927592396736, -0.003725249320268631, 0.003000067314133048, 0.0013684082077816129, 0.01789444126188755, -0.02495420351624489, 0.0049239518120884895, -0.08360452950000763, 0.02821255475282669, -0.0052716415375471115, 0.024702541530132294, 0.015298356302082539, 0.03136494383215904, -0.01400031428784132, 0.016424210742115974, 0.007231950294226408, -0.002654033014550805, -0.016742097213864326, 0.008377671241760254, -0.022517062723636627, -0.0018295112531632185, -0.030782148241996765, -0.029484106227755547, 0.010953888297080994, 0.03377559408545494, 0.006894194521009922, -0.01929844729602337, -0.015086431056261063, 0.001300525851547718, -0.011636022478342056, 0.0010058173211291432, 0.009523392654955387, 0.00010596264473861083, -0.0005082895513623953, 0.0029305294156074524, -0.006231928244233131, 0.0037616738118231297, 0.023815104737877846, -0.018874596804380417, -0.00890086218714714, -0.005980266723781824, -0.014609599485993385, -0.01678183302283287, -0.03173581138253212, 0.027338361367583275, 0.005698803346604109, 0.050491198897361755, -0.0061921919696033, -0.021311737596988678, 0.0015811612829566002, -0.023907821625471115, -0.0004482716612983495, -0.0009371071355417371, 0.030570222064852715, 0.01667587086558342, 0.007543215528130531, -0.005659067537635565, -0.002470254199579358, 0.02374887838959694, -0.030782148241996765, 0.007318045012652874, 0.013132745400071144, -0.012914197519421577, 0.002702047349885106, 0.011000246740877628, -0.011139323003590107, -0.035629939287900925, 0.04566989839076996, 0.00849687959998846, 0.02213294804096222, -0.010145923122763634, 0.021258754655718803, 0.011788344010710716, -0.007900839671492577, 0.02647741511464119, 0.015112922526896, -0.01879512332379818, -0.01147707924246788, -0.019550107419490814, 0.005794832017272711, 0.01769576221704483, 0.018954068422317505, 0.005314688663929701, 0.015298356302082539, 0.004079561680555344, -0.013523482717573643, -0.0004079147765878588, 0.006801477167755365, -0.014225484803318977, -0.037351831793785095, 0.008377671241760254, 0.024013783782720566, 0.004559705033898354, -0.01809312216937542, 0.005755096208304167, -0.01899380423128605, -0.015086431056261063, -0.016861306503415108, 0.004808054771274328, -0.004642488434910774, -0.01739111915230751, -0.031153017655014992, 0.009543260559439659, -0.005218660458922386, 0.001070388243533671, 0.012145968154072762, 0.01304665021598339, 0.01350361481308937, -0.007397517096251249, -0.021974002942442894, -0.01739111915230751, 0.015126167796552181, 0.012761875987052917, 0.009205504320561886, -0.02720590867102146, -0.006867703981697559, -0.001432151417247951, 0.029881466180086136, 0.027868175879120827, -0.013801634311676025, 0.02548401616513729, -0.002192102139815688, 0.015801679342985153, 0.006010068580508232, 0.00954988319426775, -0.03152388706803322, 0.01757655292749405, 0.018649425357580185, 0.0052219717763364315, 0.0036225980147719383, -0.004960376303642988, 0.020358072593808174, -0.019271956756711006, 0.014357938431203365, -0.012854592874646187, 0.002129186876118183, -0.010251886211335659, -0.012834724970161915, -0.007569706533104181, -0.00024400382244493812, 0.010450566187500954, 0.0006436402909457684, -0.010311489924788475, 0.010318112559616566, 0.022199174389243126, -0.0019735542591661215, 0.05674299597740173, 0.032106682658195496, -0.007523347623646259, 0.023603178560733795, -0.025550242513418198, 0.014053295366466045, 0.004884215537458658, -0.0005803110543638468, -0.018583199009299278, -0.027576778084039688, 0.016053341329097748, 0.014093032106757164, 0.0051855468191206455, -0.022318381816148758, -0.02478201314806938, -0.004092806950211525, -0.004755073692649603, 0.020569998770952225, -0.01931169256567955, 0.0031805350445210934, 0.04026580601930618, -0.03208019211888313, 0.0007723683374933898, 0.012828102335333824, -0.017828214913606644, -0.005082895513623953, 0.03298087418079376, -0.021603133529424667, -0.01496722362935543, -0.03989493474364281, -0.010755208320915699, 0.0048014321364462376, -0.016437456011772156, -0.008013425394892693, 0.014609599485993385, -0.014185748994350433, 0.0004366819921415299, 0.0009718761430121958, 0.018040141090750694, 0.00409611826762557, -0.006496834568679333, 0.006649156101047993, -0.015934132039546967, -0.02557673305273056, -0.0012458888813853264, 0.009364448487758636, -0.01739111915230751, 0.022185929119586945, 0.011258531361818314]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "index = VectorStoreIndexWrapper(vectorstore=myCassandraVStore)"
      ],
      "metadata": {
        "id": "fySPtKkYmAgh"
      },
      "id": "fySPtKkYmAgh",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's query our proprietory store. We'll ask \"What is Andouille?\""
      ],
      "metadata": {
        "id": "geA-eXncFwvi"
      },
      "id": "geA-eXncFwvi"
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"What is Andouille?\"\n",
        "index.query(query,llm=llm)"
      ],
      "metadata": {
        "id": "cEp-I8wd5Abv"
      },
      "id": "cEp-I8wd5Abv",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "I'm really interested in what temperature to cook my andouille."
      ],
      "metadata": {
        "id": "tRiU4IMkF6di"
      },
      "id": "tRiU4IMkF6di"
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"What temperature should Andouile be cooked?\"\n",
        "index.query(query,llm=llm)"
      ],
      "metadata": {
        "id": "S_96yaY45OFw",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "4e93afd9-cb17-4186-a08b-954e2cea49fb"
      },
      "id": "S_96yaY45OFw",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "' About 175 degrees Fahrenheit (80 degrees Celsius).'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's compare this answer to what OpenAi GPT-3 will return"
      ],
      "metadata": {
        "id": "4uJebWz9JfAu"
      },
      "id": "4uJebWz9JfAu"
    },
    {
      "cell_type": "code",
      "source": [
        "import openai\n",
        "\n",
        "openai.api_key = apiSecret\n",
        "response = openai.Completion.create(\n",
        "  engine=\"text-davinci-003\",\n",
        "  prompt=\"What temperature should Andouille be cooked?\",\n",
        "  max_tokens=100\n",
        ")\n",
        "\n",
        "print(response.choices[0].text.strip())"
      ],
      "metadata": {
        "id": "zX48Pu3-Jl9B",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d4a82ae5-6244-44e4-b16d-fd6e8fa36054"
      },
      "id": "zX48Pu3-Jl9B",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Andouille sausage should be cooked to an internal temperature of 160°F (71°C).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "You've now seen how we can use a LLM to answer the prompt from our Astra Vector Store, but notice that the answer is different from using the LLM directly."
      ],
      "metadata": {
        "id": "GmIdlZngKvtU"
      },
      "id": "GmIdlZngKvtU"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's get some information about the source for the response to the question \"What temperature should Andouille be cooked?\""
      ],
      "metadata": {
        "id": "DikzwEj0Gawb"
      },
      "id": "DikzwEj0Gawb"
    },
    {
      "cell_type": "code",
      "source": [
        "retriever = index.vectorstore.as_retriever(search_kwargs={\n",
        "    'k': 2,\n",
        "})"
      ],
      "metadata": {
        "id": "a5MrZWD7uaBa"
      },
      "id": "a5MrZWD7uaBa",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "retriever.get_relevant_documents(\n",
        "    \"What temperature should Andouille be cooked?\"\n",
        ")"
      ],
      "metadata": {
        "id": "dg1FUbYoudSr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5f0341f7-5aaa-4b06-f303-bf60772b1cad"
      },
      "id": "dg1FUbYoudSr",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(page_content='and garlic. Andouille makers smoke the sausages over pecan wood and sugar cane for a maximum of seven or eight hours, at about 175 degrees Fahrenheit (80 degrees Celsius).', metadata={'url': 'https://simple.wikipedia.org/wiki/Andouille', 'title': 'Andouille'}),\n",
              " Document(page_content='Andouille is a type of pork sausage. It is spicy (hot in taste) and smoked. There are different kinds, all with different combinations of pork meat, fat, intestines (tubes going to the stomach), and tripe (the wall of the stomach).', metadata={'url': 'https://simple.wikipedia.org/wiki/Andouille', 'title': 'Andouille'})]"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}