token_clf_HMM.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4htvPk6d-5cl"
      },
      "source": [
        "# <strong><font color=#A52A2A></span> NLP_HW4 </strong>\n",
        "Dr Asgari - Spring 2023\n",
        "<br>\n",
        "98101566 -> Mohammadreza Daviran\n",
        "<br>\n",
        "98170668 -> Sara Azarnoush\n",
        "<br>\n",
        "98171007 -> Nona Ghazizadeh\n",
        "<br>\n",
        "<strong><font color=#A52A2A>Token classification"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-vKH-ODc6jGl"
      },
      "source": [
        "# <font color='red'> Table of content\n",
        "-------"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QaVz_ztSupDb"
      },
      "source": [
        " - Intoduction\n",
        " - Requirements\n",
        "    - Download\n",
        "    - Import\n",
        " - Download and load\n",
        "   - Load dataset\n",
        "   - Example\n",
        " - Create dataset\n",
        " - Preprocess\n",
        "  - Preprocess\n",
        "  - Save\n",
        "    - Train\n",
        "    - Validation\n",
        "    - Test\n",
        " - Read data\n",
        " - Labels\n",
        " - TF_IDF  \n",
        "    - QADataset\n",
        "    - TF_IDF\n",
        " - HMM\n",
        "    - Vectorization\n",
        "    - Model\n",
        "    - Evaluation\n",
        "        - F1 score\n",
        "        - EM score\n",
        " - LSTM/CRF model\n",
        "    - Requirements\n",
        "    - Model\n",
        "      - LSTM/CRF\n",
        "      - Simple LSTM\n",
        "      - Padding labels\n",
        "    - Training\n",
        "      - Training with training data\n",
        "      - Training with training and validation data\n",
        "      - Evaluation\n",
        "\n",
        " - Transformer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XXS2d3ghtweH"
      },
      "source": [
        "# <font color='red'> Introduction </font>\n",
        "-------\n",
        "\n",
        "In this project, we Implement a model for extractive question and answer.\n",
        "\n",
        "It inputs a text and question and returns an answer from the text.\n",
        "\n",
        "We import project requirements and download the SajjadAyoubi/persian_qa dataset and preprocess it and save clean data in JSON and read it.\n",
        "\n",
        "We make labels from the start index and end index of the answer in text and vectorized them with tf-idf and make a  new dataset.\n",
        "\n",
        "More information is available in each section."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ahOaVRsg59At"
      },
      "source": [
        "# <font color='red'> Requirements </font>\n",
        "-------\n",
        "\n",
        "Download and import requirements for this project."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jHUFoQfot2B4"
      },
      "source": [
        "## <font color='green'> Download </font>\n",
        "\n",
        "Download project requirements\n",
        "\n",
        "*note: If you get an error uninstall and install numpy then restart the execution and run again."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0o8cZ8OaDe5Y"
      },
      "outputs": [],
      "source": [
        "# !pip uninstall numpy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0Taquzj2DjRe",
        "outputId": "736b4e80-3e69-4273-d244-95e0c40ae65a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (1.25.0)\n"
          ]
        }
      ],
      "source": [
        "!pip install numpy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-Eb5rZgcDWW6",
        "outputId": "b64ad88d-dacb-4a0e-e64d-105eda6e591c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (4.65.0)\n",
            "Requirement already satisfied: gdown in /usr/local/lib/python3.10/dist-packages (4.6.6)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from gdown) (3.12.2)\n",
            "Requirement already satisfied: requests[socks] in /usr/local/lib/python3.10/dist-packages (from gdown) (2.27.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from gdown) (1.16.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from gdown) (4.65.0)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from gdown) (4.11.2)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->gdown) (2.4.1)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (2023.5.7)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (3.4)\n",
            "Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.10/dist-packages (from requests[socks]->gdown) (1.7.1)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (1.5.3)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2022.7.1)\n",
            "Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.10/dist-packages (from pandas) (1.25.0)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas) (1.16.0)\n",
            "Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.30.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.12.2)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.16.2)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.25.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (23.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2022.10.31)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.27.1)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.13.3)\n",
            "Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.3.1)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.65.0)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers) (2023.6.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers) (4.6.3)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2023.5.7)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.4)\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (2.0.1+cu118)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch) (3.12.2)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch) (4.6.3)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch) (1.11.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch) (3.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch) (3.1.2)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch) (2.0.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch) (3.25.2)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch) (16.0.6)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch) (2.1.3)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch) (1.3.0)\n",
            "Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (2.13.1)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from datasets) (1.25.0)\n",
            "Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (9.0.0)\n",
            "Requirement already satisfied: dill<0.3.7,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.3.6)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (1.5.3)\n",
            "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (2.27.1)\n",
            "Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (4.65.0)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets) (3.2.0)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets) (0.70.14)\n",
            "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (2023.6.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets) (3.8.4)\n",
            "Requirement already satisfied: huggingface-hub<1.0.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.16.2)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets) (23.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (6.0)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (23.1.0)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (2.0.12)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.0.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.2)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.9.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.3)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets) (3.12.2)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets) (4.6.3)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (2023.5.7)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (3.4)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2022.7.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.16.0)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (1.2.2)\n",
            "Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.25.0)\n",
            "Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.10.1)\n",
            "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.2.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (3.1.0)\n",
            "Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.30.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.12.2)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.16.2)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.25.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (23.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2022.10.31)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.27.1)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.13.3)\n",
            "Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.3.1)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.65.0)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers) (2023.6.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers) (4.6.3)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2023.5.7)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.4)\n",
            "Requirement already satisfied: hazm in /usr/local/lib/python3.10/dist-packages (0.9.1)\n",
            "Requirement already satisfied: gensim<5.0.0,>=4.3.1 in /usr/local/lib/python3.10/dist-packages (from hazm) (4.3.1)\n",
            "Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /usr/local/lib/python3.10/dist-packages (from hazm) (3.8.1)\n",
            "Requirement already satisfied: numpy<2.0.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from hazm) (1.25.0)\n",
            "Requirement already satisfied: python-crfsuite<0.10.0,>=0.9.9 in /usr/local/lib/python3.10/dist-packages (from hazm) (0.9.9)\n",
            "Requirement already satisfied: scikit-learn<2.0.0,>=1.2.2 in /usr/local/lib/python3.10/dist-packages (from hazm) (1.2.2)\n",
            "Requirement already satisfied: scipy>=1.7.0 in /usr/local/lib/python3.10/dist-packages (from gensim<5.0.0,>=4.3.1->hazm) (1.10.1)\n",
            "Requirement already satisfied: smart-open>=1.8.1 in /usr/local/lib/python3.10/dist-packages (from gensim<5.0.0,>=4.3.1->hazm) (6.3.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk<4.0.0,>=3.8.1->hazm) (8.1.3)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk<4.0.0,>=3.8.1->hazm) (1.2.0)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk<4.0.0,>=3.8.1->hazm) (2022.10.31)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from nltk<4.0.0,>=3.8.1->hazm) (4.65.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn<2.0.0,>=1.2.2->hazm) (3.1.0)\n"
          ]
        }
      ],
      "source": [
        "! pip install tqdm\n",
        "! pip install gdown\n",
        "! pip install pandas\n",
        "! pip install transformers\n",
        "! pip install torch\n",
        "! pip install datasets\n",
        "! pip install scikit-learn\n",
        "! pip install transformers\n",
        "! pip install hazm"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V2eWzvtDt52m"
      },
      "source": [
        "## <font color='green'> Import requirements </font>\n",
        "\n",
        "Import project requirements"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7WP4CtDNDgR8"
      },
      "outputs": [],
      "source": [
        "import re\n",
        "import hazm\n",
        "import json\n",
        "import gdown\n",
        "import torch\n",
        "import pickle\n",
        "import numpy as np\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "from torch.utils.data import DataLoader, Dataset\n",
        "from collections import OrderedDict\n",
        "from pathlib import Path\n",
        "\n",
        "import pandas as pd\n",
        "from tqdm import tqdm\n",
        "from tabulate import tabulate\n",
        "from string import punctuation\n",
        "from transformers import AutoTokenizer\n",
        "\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from transformers import DataCollatorWithPadding\n",
        "\n",
        "\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from transformers import BigBirdModel, AutoTokenizer\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from transformers import Trainer, TrainingArguments, AutoModelForSequenceClassification\n",
        "\n",
        "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VNEUecurq95z"
      },
      "outputs": [],
      "source": [
        "\n",
        "from datasets import DatasetDict, Dataset\n",
        "from datasets import load_dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PBT0JWMt6qb0"
      },
      "source": [
        "# <font color='red'> Download and load </font>\n",
        "------\n",
        "\n",
        "Downlad \"SajjadAyoubi/persian_qa\"  dataset.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QNTmiXdTdpTw"
      },
      "source": [
        "## <font color='green'> Load dataset </font>\n",
        "\n",
        "For this task we use \"SajjadAyoubi/persian_qa\" dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86,
          "referenced_widgets": [
            "a9374d78c6504f0fa21b20a69b90a40c",
            "1538bad5e0ce4c1e88ffb1a2c874d965",
            "3733351f80974d78a7368f77c407617d",
            "f83ced7bdd0f411ab0f94eac6ec4c9f0",
            "1d1068b874684f1b823b518c8e24852e",
            "d8e0be76669d4870b040219f85c2b4f2",
            "88fd33c6de8f4ea9a7baef7b7a887a23",
            "09bc48ffe5d24c0692cc5c2a2ff7bc08",
            "c6cec377a8b74b37a0acb5b4ce414851",
            "0fe6777985f142e8a18103c48aea7941",
            "e7766ab28b7140808f305a29816b6341"
          ]
        },
        "id": "QFSYGurvfNAY",
        "outputId": "ed93c3ca-5d8d-480c-82ea-95fca977ecec"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "WARNING:datasets.builder:Found cached dataset persian_qa (/root/.cache/huggingface/datasets/SajjadAyoubi___persian_qa/persian_qa/1.0.0/adcc9e82d1a679ba85f7958663d8e771894c35e2fbc6a92d9ea2b6a8a72f9225)\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a9374d78c6504f0fa21b20a69b90a40c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/2 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "dataset = load_dataset(\"SajjadAyoubi/persian_qa\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dPxTHIkdsAy9"
      },
      "source": [
        "## <font color='green'> Example </font>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5ZKwVbv3zjqy",
        "outputId": "2510706d-e285-45c8-ab49-a4c917a511b9"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'id': 2,\n",
              " 'title': 'شرکت فولاد مبارکه اصفهان',\n",
              " 'context': 'شرکت فولاد مبارکۀ اصفهان، بزرگ\\u200cترین واحد صنعتی خصوصی در ایران و بزرگ\\u200cترین مجتمع تولید فولاد در کشور ایران است، که در شرق شهر مبارکه قرار دارد. فولاد مبارکه هم\\u200cاکنون محرک بسیاری از صنایع بالادستی و پایین\\u200cدستی است. فولاد مبارکه در ۱۱ دوره جایزۀ ملی تعالی سازمانی و ۶ دوره جایزۀ شرکت دانشی در کشور رتبۀ نخست را بدست آورده\\u200cاست و همچنین این شرکت در سال ۱۳۹۱ برای نخستین\\u200cبار به عنوان تنها شرکت ایرانی با کسب امتیاز ۶۵۴ تندیس زرین جایزۀ ملی تعالی سازمانی را از آن خود کند. شرکت فولاد مبارکۀ اصفهان در ۲۳ دی ماه ۱۳۷۱ احداث شد و اکنون بزرگ\\u200cترین واحدهای صنعتی و بزرگترین مجتمع تولید فولاد در ایران است. این شرکت در زمینی به مساحت ۳۵ کیلومتر مربع در نزدیکی شهر مبارکه و در ۷۵ کیلومتری جنوب غربی شهر اصفهان واقع شده\\u200cاست. مصرف آب این کارخانه در کمترین میزان خود، ۱٫۵٪ از دبی زاینده\\u200cرود برابر سالانه ۲۳ میلیون متر مکعب در سال است و خود یکی از عوامل کم\\u200cآبی زاینده\\u200cرود شناخته می\\u200cشود.',\n",
              " 'question': 'فولاد مبارکه چند بار برنده جایزه شرکت دانشی را کسب کرده است؟',\n",
              " 'answers': {'text': ['۶'], 'answer_start': [263]}}"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset['train'][1]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FeHRN8wlm2m3",
        "outputId": "7fda8037-2aea-49e7-a18b-71cd07a1b455"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "263"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset['train'][1]['context'].find(\"۶\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8FcIja1ZfTJ3",
        "outputId": "11e48855-6447-45d7-b545-5abf29897471"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'text': ['در شرق شهر مبارکه'], 'answer_start': [114]}\n"
          ]
        }
      ],
      "source": [
        "for i, x in enumerate(dataset['train']):\n",
        "  if i == 1:\n",
        "    break\n",
        "  print(x['answers'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dhSMCNhAy8c2"
      },
      "source": [
        "# <font color='red'> Creating Dataset </font>\n",
        "-----\n",
        "\n",
        "Create new proper train, test and validation dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oDccFLpry-KB",
        "outputId": "b9b9c046-e015-40a2-bd44-6a3769b8be55"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "5360"
            ]
          },
          "execution_count": 105,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "splitter = int(0.85 * len(dataset['train']))\n",
        "splitter"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lkF4mjX2sCNe"
      },
      "source": [
        "We set text, question, answer, the first index of the answer, and dic with x and y for train, test, and validation datasets. First, we set text, question, answer, and start, then concat text and answer for x value and first index and last index of answer for y value. \\\\\n",
        "At first, we calculate the splitter and then create data if I < splitter set them as our train value, else we set them as validation data. Set our test dataset from the validation dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ylu6yzESsaKu"
      },
      "outputs": [],
      "source": [
        "train_val_contexts = []\n",
        "train_val_questions = []\n",
        "train_val_answers = []\n",
        "train_val_start = []\n",
        "train_val_data = {\"x\": [], \"y\": []}\n",
        "\n",
        "train_contexts = []\n",
        "train_questions = []\n",
        "train_answers = []\n",
        "train_start = []\n",
        "train_data = {\"x\": [], \"y\": []}\n",
        "\n",
        "valid_contexts = []\n",
        "valid_questions = []\n",
        "valid_answers = []\n",
        "valid_start = []\n",
        "valid_data = {\"x\": [], \"y\": []}\n",
        "\n",
        "test_contexts = []\n",
        "test_questions = []\n",
        "test_answers = []\n",
        "test_start = []\n",
        "test_data = {\"x\": [], \"y\": []}\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6H1G_fxi2mmU"
      },
      "outputs": [],
      "source": [
        "for i, x in enumerate(dataset['train']):\n",
        "  try:\n",
        "      context = x['context']\n",
        "      question = x['question']\n",
        "      answer = x['answers']['text'][0]\n",
        "      start = x['answers']['answer_start'][0]\n",
        "\n",
        "      train_val_data['x'].append(context + question)\n",
        "      train_val_data['y'].append(answer)\n",
        "  except:\n",
        "      continue\n",
        "\n",
        "  if i < splitter:\n",
        "    context = x['context']\n",
        "    question = x['question']\n",
        "    answer = x['answers']['text'][0]\n",
        "    start = x['answers']['answer_start'][0]\n",
        "\n",
        "    train_contexts.append(context)\n",
        "    train_questions.append(question)\n",
        "    train_answers.append(answer)\n",
        "    train_start.append(start)\n",
        "\n",
        "    train_data['x'].append(context + question)\n",
        "    train_data['y'].append((start, start + len(answer)))\n",
        "\n",
        "  else:\n",
        "    context = x['context']\n",
        "    question = x['question']\n",
        "    answer = x['answers']['text'][0]\n",
        "    start = x['answers']['answer_start'][0]\n",
        "\n",
        "    valid_answers.append(x['answers']['text'][0])\n",
        "    valid_contexts.append(x['context'])\n",
        "    valid_questions.append(x['question'])\n",
        "    valid_start.append(x['answers']['answer_start'][0])\n",
        "\n",
        "    valid_data['x'].append(context + question)\n",
        "    valid_data['y'].append((start, start + len(answer)))\n",
        "\n",
        "\n",
        "for i, x in enumerate(dataset['validation']):\n",
        "  try:\n",
        "    context = x['context']\n",
        "    question = x['question']\n",
        "    answer = x['answers']['text'][0]\n",
        "    start = x['answers']['answer_start'][0]\n",
        "\n",
        "    test_answers.append(x['answers']['text'][0])\n",
        "    test_contexts.append(x['context'])\n",
        "    test_questions.append(x['question'])\n",
        "    test_start.append(x['answers']['answer_start'][0])\n",
        "\n",
        "    test_data['x'].append(context + question)\n",
        "    test_data['y'].append((start, start + len(answer)))\n",
        "  except:\n",
        "      continue"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eDingu_NUM5W"
      },
      "source": [
        "# <font color='red'> Preprocess </font>\n",
        "----\n",
        "\n",
        "Preprocess data to get clean data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NhN0dn5GrW4j"
      },
      "source": [
        "## <font color='green'> Preprocess </font>\n",
        "\n",
        "Get data to normalize it, remove links and punctuations, and tokenize and remove chosen stop words from it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1_lIab4zUMXK"
      },
      "outputs": [],
      "source": [
        "class Preprocessor:\n",
        "\n",
        "    def __init__(self, stopwords_path):\n",
        "        self.stopwords = []\n",
        "        with open(stopwords_path, encoding='utf-8') as file:\n",
        "            self.stopwords = file.read().split()\n",
        "\n",
        "    def preprocess(self, text):\n",
        "        text = self.normalize(text)\n",
        "        text = self.remove_links(text)\n",
        "        text = self.remove_punctuations(text)\n",
        "        words = self.word_tokenize(text)\n",
        "        words = self.remove_stopwords(words)\n",
        "        return words\n",
        "\n",
        "    def normalize(self, text):\n",
        "        return hazm.Normalizer().normalize(text)\n",
        "\n",
        "    def remove_links(self, text):\n",
        "        patterns = ['\\S*http\\S*', '\\S*www\\S*', '\\S+\\.ir\\S*', '\\S+\\.com\\S*', '\\S+\\.org\\S*', '\\S*@\\S*']\n",
        "        for pattern in patterns:\n",
        "            text = re.sub(pattern, ' ', text)\n",
        "        return text\n",
        "\n",
        "    def remove_punctuations(self, text):\n",
        "        return re.sub(f'[{punctuation}؟،٪×÷»«]+', '', text)\n",
        "\n",
        "    def word_tokenize(self, text):\n",
        "        return hazm.word_tokenize(text)\n",
        "\n",
        "    def remove_invalid_words(self, words):\n",
        "        return [word for word in words if len(word) > 3 or re.match('^[\\u0600-\\u06FF]{2,3}$', word)]\n",
        "\n",
        "    def remove_stopwords(self, words):\n",
        "        return [word for word in words if word not in self.stopwords]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_g7qnKlgUbvw"
      },
      "outputs": [],
      "source": [
        "def get_preprocessed_text(data, preprocessor):\n",
        "    preprocessed_contents = []\n",
        "    contents = data\n",
        "    preprocessed_contents = [preprocessor.preprocess(content) for content in tqdm(contents)]\n",
        "    return preprocessed_contents"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dpbvlz2OwWwK"
      },
      "source": [
        "Downlaod stopwords from google drive"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 159
        },
        "id": "Rmk_KFlL5ddG",
        "outputId": "1006d3dd-4e3b-4740-c513-4e558d499800"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/gdown/parse_url.py:35: UserWarning: You specified a Google Drive link that is not the correct link to download a file. You might want to try `--fuzzy` option or the following url: https://drive.google.com/uc?id=None\n",
            "  warnings.warn(\n",
            "Downloading...\n",
            "From: https://drive.google.com/file/d/1OVWfL-UubaJn-bBKW3s4CD3JrM_84mYj\n",
            "To: /content/stopwords.txt\n",
            "80.2kB [00:00, 8.77MB/s]\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'stopwords.txt'"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "url = \"https://drive.google.com/file/d/1OVWfL-UubaJn-bBKW3s4CD3JrM_84mYj\"\n",
        "output = \"stopwords.txt\"\n",
        "gdown.download(url, output, quiet=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rF3Ixfe5UeWM"
      },
      "outputs": [],
      "source": [
        "preprocessor = Preprocessor(stopwords_path='stopwords.txt')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pU8n4rQnrVkr"
      },
      "source": [
        "## <font color='green'> Save </font>\n",
        "\n",
        "Save text and answer for all three trains, validation, and test datasets after preprocessing them in JSON file."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mOSpcy6Jutvg"
      },
      "source": [
        "### <font color='blue'> Train\n",
        "\n",
        "Save text and answer for trains datasets after preprocessing them in JSON file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k-_q4qI1Ugnh"
      },
      "outputs": [],
      "source": [
        "train_preprocessed_content = get_preprocessed_text(train_data['x'], preprocessor)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T-ydmIR7_Oce",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "train_preprocessed_answers = get_preprocessed_text(train_answers, preprocessor)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-qDEQBvMUpmT"
      },
      "outputs": [],
      "source": [
        "with open('token_preprocessed_train_dataset.json', 'w') as f:\n",
        "    json.dump(train_preprocessed_content, f)\n",
        "\n",
        "with open('token_preprocessed_train_answers_dataset.json', 'w') as f:\n",
        "    json.dump(train_preprocessed_answers, f)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lFWNjnacuztl"
      },
      "source": [
        "### <font color='blue'> Validation\n",
        "\n",
        "Save text and answer for validation datasets after preprocessing them in JSON file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RTqclh4h0RNC"
      },
      "outputs": [],
      "source": [
        "val_preprocessed_content = get_preprocessed_text(valid_data['x'], preprocessor)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vcipe_eF_Och"
      },
      "outputs": [],
      "source": [
        "val_preprocessed_answers = get_preprocessed_text(valid_answers, preprocessor)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1rLBqK6JUqGV"
      },
      "outputs": [],
      "source": [
        "with open('token_preprocessed_val_dataset.json', 'w') as f:\n",
        "    json.dump(val_preprocessed_content, f)\n",
        "\n",
        "with open('token_preprocessed_val_answers_dataset.json', 'w') as f:\n",
        "    json.dump(val_preprocessed_answers, f)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_9sr8Ya4u1cO"
      },
      "source": [
        "### <font color='blue'> Test\n",
        "\n",
        "Save text and answer for test datasets after preprocessing them in JSON file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ybomI-yNZ9mY"
      },
      "outputs": [],
      "source": [
        "test_preprocessed_content = get_preprocessed_text(test_data['x'], preprocessor)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vfvPauLU_Oci"
      },
      "outputs": [],
      "source": [
        "test_preprocessed_answers = get_preprocessed_text(test_answers, preprocessor)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a0-f5_xLZ_iB"
      },
      "outputs": [],
      "source": [
        "with open('token_preprocessed_test_dataset.json', 'w') as f:\n",
        "    json.dump(test_preprocessed_content, f)\n",
        "\n",
        "with open('token_preprocessed_test_answers_dataset.json', 'w') as f:\n",
        "    json.dump(test_preprocessed_answers, f)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zpHj1RYJ_Ocj"
      },
      "source": [
        "# <font color='red'> Read data </font>\n",
        "------\n",
        "\n",
        "Download and read processed data from google drive"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PZIdiRhO2o9r"
      },
      "outputs": [],
      "source": [
        "import gdown\n",
        "import json\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sxV-RC_MHWLQ"
      },
      "outputs": [],
      "source": [
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AxVr8ctZ26M4"
      },
      "source": [
        "Download text and answer dataset for train, test and validation from google drive."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oJvwytSg85rE"
      },
      "outputs": [],
      "source": [
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)\n",
        "# train\n",
        "download = drive.CreateFile({'id': '1BYQTIAvaCbxUlHhYYDTaVvJmPyf9hX6T'})\n",
        "download.GetContentFile('token_preprocessed_train_dataset.json')\n",
        "download = drive.CreateFile({'id': '1D--9Mv22LzjXC8T59ZOEXTSzI6AqmC2Y'})\n",
        "download.GetContentFile('token_preprocessed_train_answers_dataset.json')\n",
        "# validation\n",
        "download = drive.CreateFile({'id': '1QIKf2etnVYIywhoWSRAmnj6SJAJ_fuNq'})\n",
        "download.GetContentFile('token_preprocessed_val_dataset.json')\n",
        "download = drive.CreateFile({'id': '19ft10_swfFs9TI1qH9ZTPDbFm90Xp76Y'})\n",
        "download.GetContentFile('token_preprocessed_val_answers_dataset.json')\n",
        "# test\n",
        "download = drive.CreateFile({'id': '1_PxwHovDumnssLJZAWqa6JcmOoNKRbLC'})\n",
        "download.GetContentFile('token_preprocessed_test_dataset.json')\n",
        "download = drive.CreateFile({'id': '1Ssd90fwrklFD0wiOnj6HpnYtxwJYmiBP'})\n",
        "download.GetContentFile('token_preprocessed_test_answers_dataset.json')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ankTrAs52yH9"
      },
      "source": [
        "Read and load json file"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S_N7wKP-_Ock"
      },
      "outputs": [],
      "source": [
        "# train\n",
        "with open('token_preprocessed_train_dataset.json', 'r') as f:\n",
        "    train_preprocessed_content = json.load(f)\n",
        "with open('token_preprocessed_train_answers_dataset.json', 'r') as f:\n",
        "    train_preprocessed_answers = json.load(f)\n",
        "# validation\n",
        "with open('token_preprocessed_val_dataset.json', 'r') as f:\n",
        "    val_preprocessed_content = json.load(f)\n",
        "with open('token_preprocessed_val_answers_dataset.json', 'r') as f:\n",
        "    val_preprocessed_answers = json.load(f)\n",
        "# test\n",
        "with open('token_preprocessed_test_dataset.json', 'r') as f:\n",
        "    test_preprocessed_content = json.load(f)\n",
        "with open('token_preprocessed_test_answers_dataset.json', 'r') as f:\n",
        "    test_preprocessed_answers = json.load(f)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kn4jNVhTrgMH"
      },
      "source": [
        "# <font color='red'> Labels </font>\n",
        "-----\n",
        "\n",
        "Concat processed data and create lables from start index and end index of answer in text."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MwinI9Rp3KgV"
      },
      "source": [
        "\n",
        "Search content train dataset, concat each of them answer dataset if the answer was in content save start and end index of answer in labels."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zzg_LAf6_Ocm",
        "outputId": "5bedb954-096e-44f9-e6c0-0b9de73f1e3e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "5361"
            ]
          },
          "execution_count": 69,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "train_labels = []\n",
        "for i, d in enumerate(train_preprocessed_content):\n",
        "    length = len(train_preprocessed_answers[i])\n",
        "    answer = \" \".join(train_preprocessed_answers[i])\n",
        "    start_index = 0\n",
        "    end_index = 0\n",
        "\n",
        "    labels = np.zeros(len(d))\n",
        "    for i in range(len(d)):\n",
        "        substr = \" \".join(d[i:i+length])\n",
        "        if answer.strip() == substr.strip():\n",
        "            start_index = i\n",
        "            end_index = i + length\n",
        "            break\n",
        "    labels[start_index:end_index] = 1\n",
        "    train_labels.append(labels)\n",
        "\n",
        "print(train_preprocessed_content[0][start_index:end_index])\n",
        "print(answer)\n",
        "len(train_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LnDT5M-UgGni"
      },
      "source": [
        "\n",
        "Search content validation dataset, concat each of them answer dataset if the answer was in content save start and end index of answer in labels."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zv7HXN2e_Ocm",
        "outputId": "474c6a96-4f95-4ba7-a1fb-0a1c82ff35b1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "945"
            ]
          },
          "execution_count": 70,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "val_labels = []\n",
        "for i, d in enumerate(val_preprocessed_content):\n",
        "    length = len(val_preprocessed_answers[i])\n",
        "    answer = \" \".join(val_preprocessed_answers[i])\n",
        "    start_index = 0\n",
        "    end_index = 0\n",
        "\n",
        "    labels = np.zeros(len(d))\n",
        "    for i in range(len(d)):\n",
        "        substr = \" \".join(d[i:i+length])\n",
        "        if answer.strip() == substr.strip():\n",
        "            start_index = i\n",
        "            end_index = i + length\n",
        "            break\n",
        "    labels[start_index:end_index] = 1\n",
        "    val_labels.append(labels)\n",
        "len(val_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tnI7xAjqgJYm"
      },
      "source": [
        "\n",
        "Search content test dataset, concat each of them answer dataset if the answer was in content save start and end index of answer in labels."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4NVbmJnU_Ocn",
        "outputId": "b8e681a6-cb82-4e56-a5c9-a5c0d5d6e320"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "651"
            ]
          },
          "execution_count": 71,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "test_labels = []\n",
        "for i, d in enumerate(test_preprocessed_content):\n",
        "    length = len(test_preprocessed_answers[i])\n",
        "    answer = \" \".join(test_preprocessed_answers[i])\n",
        "    start_index = 0\n",
        "    end_index = 0\n",
        "\n",
        "    labels = np.zeros(len(d))\n",
        "    for i in range(len(d)):\n",
        "        substr = \" \".join(d[i:i+length])\n",
        "        if answer.strip() == substr.strip():\n",
        "            start_index = i\n",
        "            end_index = i + length\n",
        "            break\n",
        "    labels[start_index:end_index] = 1\n",
        "    test_labels.append(labels)\n",
        "len(test_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SlL_P0IjUSR1"
      },
      "source": [
        "# <font color='red'> TF_IDF </font>\n",
        "------\n",
        "\n",
        "Vectorize train, validation, and test content with TF-IDF model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m_JRzgwzwu42"
      },
      "source": [
        "## <font color='green'> QADataset\n",
        "\n",
        "Create new QA dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oOilvGVn_JZa"
      },
      "outputs": [],
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "from datasets import DatasetDict, Dataset\n",
        "from datasets import load_dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6y-Q4g3c0Dvo"
      },
      "outputs": [],
      "source": [
        "class QADataset(Dataset):\n",
        "    def __init__(self, docs, labels):\n",
        "        self.documents = docs\n",
        "        self.labels = labels\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.labels)\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        document = self.documents[idx]\n",
        "        label = self.labels[idx]\n",
        "        return document, label"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XFXbX5oMvV16"
      },
      "source": [
        "## <font color='green'> TF_IDF\n",
        "\n",
        "Vectorize train-val, and test content with TF-IDF model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j1crHOmnJXIP"
      },
      "outputs": [],
      "source": [
        "from torch.utils.data import DataLoader, Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0BEqaMm1mr3D",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "class TF_IDF:\n",
        "    def __init__(self):\n",
        "        self.vectorizer = TfidfVectorizer()\n",
        "\n",
        "    def fit_vectorizer(self, data):\n",
        "        self.vectorizer.fit(list(map(lambda doc: ' '.join(doc), data)))\n",
        "\n",
        "    def transform_vectorizer(self, preprocessed_content):\n",
        "        embeddings = []\n",
        "        for content in preprocessed_content:\n",
        "            emb_content = self.vectorizer.transform([' '.join(content)]).toarray()[0]\n",
        "            embeddings.append(emb_content)\n",
        "        return embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LynNgEBqebmq"
      },
      "source": [
        "Concat preprocessed data and vectorized it with tf-idf model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "E8YrdwT5_Oco"
      },
      "outputs": [],
      "source": [
        "all_preprocessed_content = train_preprocessed_content + val_preprocessed_content + test_preprocessed_content"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IBfFvPEU_Ocp"
      },
      "outputs": [],
      "source": [
        "TF_IDF_model = TF_IDF()\n",
        "TF_IDF_model.fit_vectorizer(all_preprocessed_content)\n",
        "\n",
        "training_vectors = np.array(TF_IDF_model.transform_vectorizer(train_preprocessed_content))\n",
        "validation_vectors = np.array(TF_IDF_model.transform_vectorizer(val_preprocessed_content))\n",
        "test_vectors = np.array(TF_IDF_model.transform_vectorizer(test_preprocessed_content))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JAOBd1jqro8z",
        "outputId": "c7be1ef2-2aec-471b-9e8f-f203c7f9ed11"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "5361\n",
            "5361\n",
            "19151\n",
            "19151\n"
          ]
        }
      ],
      "source": [
        "print(training_vectors.shape[0])\n",
        "print(len(train_labels))\n",
        "print(validation_vectors.shape[1])\n",
        "print(test_vectors.shape[1])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oPFn2oby1330"
      },
      "outputs": [],
      "source": [
        "# Assuming you have 'train_data' and 'valid_data' containing your training and validation data respectively\n",
        "train_dataset = QADataset(training_vectors, train_labels)\n",
        "valid_dataset = QADataset(validation_vectors, val_labels)\n",
        "test_dataset = QADataset(test_vectors, test_labels)\n",
        "\n",
        "batch_size = 16\n",
        "train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n",
        "valid_loader = DataLoader(valid_dataset, batch_size=batch_size)\n",
        "test_loader = DataLoader(test_dataset, batch_size=batch_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "00apodZ3_Ocq"
      },
      "source": [
        "# <font color='red'> HMM </font>\n",
        "------\n",
        "A simple model for Extractive Question Answering could be a HMM model that gets a input as a concatenation of the text and the question together. Also the observation for each word will be the parts of the document which are in the answer too."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QZWSfLOlNwfg"
      },
      "source": [
        "For this matter first we install hmmlearn library."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xu226ebM_Ocq",
        "outputId": "23103823-31a3-47e2-ba48-e058e9dbc873"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting hmmlearn\n",
            "  Downloading hmmlearn-0.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (160 kB)\n",
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/160.4 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m160.4/160.4 kB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.10 in /usr/local/lib/python3.10/dist-packages (from hmmlearn) (1.25.0)\n",
            "Requirement already satisfied: scikit-learn!=0.22.0,>=0.16 in /usr/local/lib/python3.10/dist-packages (from hmmlearn) (1.2.2)\n",
            "Requirement already satisfied: scipy>=0.19 in /usr/local/lib/python3.10/dist-packages (from hmmlearn) (1.10.1)\n",
            "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn!=0.22.0,>=0.16->hmmlearn) (1.2.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn!=0.22.0,>=0.16->hmmlearn) (3.1.0)\n",
            "Installing collected packages: hmmlearn\n",
            "Successfully installed hmmlearn-0.3.0\n"
          ]
        }
      ],
      "source": [
        "!pip install hmmlearn"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OXfRw3pHPVzk"
      },
      "source": [
        "Then we import hmm model from hmmlearn library:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "G2VoJAJ6imxF"
      },
      "outputs": [],
      "source": [
        "from hmmlearn import hmm"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DyW61C-gRKuZ"
      },
      "source": [
        "We create data from concatenation of train data and validation data. Then we use this data to fit the model on it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9LoNRx7kimxF"
      },
      "outputs": [],
      "source": [
        "train_val_data = train_preprocessed_content + val_preprocessed_content\n",
        "train_val_answers = train_preprocessed_answers + val_preprocessed_answers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vfB5_JnuUpvf"
      },
      "source": [
        "## <font color='green'> TF_IDF </font>\n",
        "\n",
        "Vectorize train-val, and test content with TF-IDF model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rtuObbB2imxF"
      },
      "outputs": [],
      "source": [
        "TF_IDF_model_hmm = TF_IDF()\n",
        "\n",
        "TF_IDF_model_hmm = TF_IDF()\n",
        "TF_IDF_model_hmm.fit_vectorizer(train_val_data[20:50])\n",
        "\n",
        "training_vectors = np.array(TF_IDF_model_hmm.transform_vectorizer(train_val_data))\n",
        "\n",
        "training_vectors_ans = np.array(TF_IDF_model_hmm.transform_vectorizer(train_val_answers))\n",
        "\n",
        "X_train = np.concatenate((training_vectors, training_vectors_ans), axis=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9UICxdMdimxF",
        "outputId": "a2883688-d2ec-4b75-fb23-54a5d5333fff"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(6306, 1032)"
            ]
          },
          "execution_count": 53,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "X_train.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vcrsFPA4U-Fk"
      },
      "source": [
        "## <font color='green'> Model </font>\n",
        "We fit our model on the X_train which is a concatenation of the training_vectors and training_vectors_ans."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UWRsEpK-imxG"
      },
      "outputs": [],
      "source": [
        "hmm2 = hmm.GaussianHMM(n_components=2)\n",
        "hmm2.startprob_ = np.ones(2) / 2\n",
        "hmm2.transmat_ = np.ones((2, 2)) / 2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "id": "YGqa40tpimxG",
        "outputId": "acdef352-b877-4469-af2d-2a9f8438da0f"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "WARNING:hmmlearn.base:Even though the 'startprob_' attribute is set, it will be overwritten during initialization because 'init_params' contains 's'\n",
            "WARNING:hmmlearn.base:Even though the 'transmat_' attribute is set, it will be overwritten during initialization because 'init_params' contains 't'\n",
            "WARNING:hmmlearn.base:Model is not converging.  Current: 25636693.184450474 is not greater than 25636693.184456453. Delta is -5.979090929031372e-06\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<style>#sk-container-id-3 {color: black;background-color: white;}#sk-container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-3 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align: center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GaussianHMM(n_components=2)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" checked><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GaussianHMM</label><div class=\"sk-toggleable__content\"><pre>GaussianHMM(n_components=2)</pre></div></div></div></div></div>"
            ],
            "text/plain": [
              "GaussianHMM(n_components=2)"
            ]
          },
          "execution_count": 56,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "X_train = np.round(X_train).astype(int)\n",
        "X_train = np.nan_to_num(X_train)\n",
        "hmm2.fit(X_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kDgWkegSimxG"
      },
      "outputs": [],
      "source": [
        "test_vectors = np.array(TF_IDF_model_hmm.transform_vectorizer(test_preprocessed_content))\n",
        "test_vectors_ans = np.array(TF_IDF_model_hmm.transform_vectorizer(test_preprocessed_answers))\n",
        "X_test = np.concatenate((test_vectors, test_vectors_ans), axis=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PjC25qsuimxG"
      },
      "outputs": [],
      "source": [
        "hmm2.transmat_ += np.eye(hmm2.n_components) * 1e-10\n",
        "hmm2.transmat_ /= hmm2.transmat_.sum(axis=1, keepdims=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j455OSlmimxG",
        "outputId": "8908b7e8-c5b7-4f07-8ec6-555ad3cdf18e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[2.00000000e-10, 1.00000000e+00],\n",
              "       [1.13891609e-02, 9.88610839e-01]])"
            ]
          },
          "execution_count": 73,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "hmm2.transmat_"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YSpnSIDFnUAK",
        "outputId": "20fa7472-1481-4f37-9993-4c9451a5d3be"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([0., 0., 0., ..., 0., 0., 0.])"
            ]
          },
          "execution_count": 76,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "X_test[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbunES1kVYKz"
      },
      "source": [
        "## <font color='green'> Evaluation </font>\n",
        "\n",
        "Compare true lables and predicted labels to find EM and f1."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lGmrdERgimxG"
      },
      "outputs": [],
      "source": [
        "def evaluate_model(model, test_data, test_answers):\n",
        "    predictions = []\n",
        "    for i in range(len(test_data[:5])):\n",
        "\n",
        "        test_vectors = np.array(TF_IDF_model_hmm.transform_vectorizer([' '.join(test_data[i])]))\n",
        "        test_vectors_ans = np.array(TF_IDF_model_hmm.transform_vectorizer([' '.join(test_answers[i])]))\n",
        "        X_test = np.concatenate((test_vectors, test_vectors_ans), axis=1)\n",
        "        X_test = np.round(X_test).astype(int)\n",
        "        X_test = np.nan_to_num(X_test)\n",
        "        pred = model.predict(X_test)\n",
        "        print(pred)\n",
        "        predictions.append(pred)\n",
        "    return predictions\n",
        "\n",
        "predictions = evaluate_model(hmm2, test_preprocessed_content, test_preprocessed_answers)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OeX3mYUZnYMJ"
      },
      "outputs": [],
      "source": [
        "hmm_pred = []\n",
        "for idx, i in enumerate(X_test):\n",
        "    hmm_pred.append(hmm2.predict(X_test[idx].reshape(-1,1)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DgIiDZ_Joutl"
      },
      "outputs": [],
      "source": [
        "reshaped_test_labels = []\n",
        "for idx, i in enumerate(test_labels):\n",
        "    label_reshape = np.resize(test_labels[idx], (1032,))\n",
        "    reshaped_test_labels.append(label_reshape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L4bkNAg9XWfH"
      },
      "source": [
        "from evaluate we import load which ables us to use squad_metric. This metric can calculate EM and f1."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2pE8no8QXobM"
      },
      "source": [
        "### <font color='blue'> F1 score </font>\n",
        "\n",
        "we define f1_score function to calculate f1_score according to the prediction and ground_truth."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n_XEEQ8Jpsyu",
        "outputId": "6c810569-9473-4fed-c2a3-c1ac2d77784f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.06163341644009166"
            ]
          },
          "execution_count": 94,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "def f1_score(prediction, ground_truth):\n",
        "    prediction = np.array(prediction)\n",
        "    ground_truth = np.array(ground_truth)\n",
        "    tp = np.sum(prediction * ground_truth)\n",
        "    fp = np.sum(prediction * (1 - ground_truth))\n",
        "    fn = np.sum((1 - prediction) * ground_truth)\n",
        "    precision = tp / (tp + fp)\n",
        "    recall = tp / (tp + fn)\n",
        "    f1 = 2 * precision * recall / (precision + recall)\n",
        "    return f1\n",
        "\n",
        "f1_score(hmm_pred, reshaped_test_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KZKATs5RXxns"
      },
      "source": [
        "### <font color='blue'> EM score </font>\n",
        "\n",
        "Also the exact_match_score has been defined to calculate the EM metric according to the prediction and ground_truth."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TVwyr69wubkp",
        "outputId": "184e1df0-28be-4b8f-f0dd-f0a701ab45ad"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.0"
            ]
          },
          "execution_count": 97,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "def exact_match_score(prediction, ground_truth):\n",
        "    prediction = np.array(prediction)\n",
        "    ground_truth = np.array(ground_truth)\n",
        "    return np.sum(np.all(prediction == ground_truth, axis=1)) / prediction.shape[0]\n",
        "exact_match_score(hmm_pred, reshaped_test_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x4Gubw4G_Ocu"
      },
      "source": [
        "# <font color='red'> LSTM/CRF model </font>\n",
        "------\n",
        "For this model we do the same thing as HMM model with a few differences."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EwFpzM6enjR_"
      },
      "source": [
        "## <font color='green'> Requirements </font>\n",
        "\n",
        "First we install torchcrf library and import necessary libraries and functions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V08Ka2sS_Ocu",
        "outputId": "e268ecbf-1997-4e7e-bb4d-6ec33e6d823a",
        "scrolled": false
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: torchcrf in /usr/local/lib/python3.10/dist-packages (1.1.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from torchcrf) (1.25.0)\n",
            "Requirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from torchcrf) (2.0.1+cu118)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->torchcrf) (3.12.2)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->torchcrf) (4.6.3)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->torchcrf) (1.11.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->torchcrf) (3.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->torchcrf) (3.1.2)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->torchcrf) (2.0.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.0.0->torchcrf) (3.25.2)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.0.0->torchcrf) (16.0.6)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.0.0->torchcrf) (2.1.3)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.0.0->torchcrf) (1.3.0)\n"
          ]
        }
      ],
      "source": [
        "! pip install torchcrf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oqNiyY0q_Ocu",
        "outputId": "1935ebaf-ce73-4746-dd54-aa3a8e8da063"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: torchcrf in /usr/local/lib/python3.10/dist-packages (1.1.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from torchcrf) (1.25.0)\n",
            "Requirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from torchcrf) (2.0.1+cu118)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->torchcrf) (3.12.2)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->torchcrf) (4.6.3)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->torchcrf) (1.11.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->torchcrf) (3.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->torchcrf) (3.1.2)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->torchcrf) (2.0.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.0.0->torchcrf) (3.25.2)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.0.0->torchcrf) (16.0.6)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.0.0->torchcrf) (2.1.3)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.0.0->torchcrf) (1.3.0)\n"
          ]
        }
      ],
      "source": [
        "! pip install --upgrade torchcrf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nZbO7gZc_Ocu",
        "outputId": "14bdc0de-227e-4123-ca12-8a945f79a159"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting pytorch-crf\n",
            "  Downloading pytorch_crf-0.7.2-py3-none-any.whl (9.5 kB)\n",
            "Installing collected packages: pytorch-crf\n",
            "Successfully installed pytorch-crf-0.7.2\n"
          ]
        }
      ],
      "source": [
        "! pip install pytorch-crf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mbXYzs7i_Ocu"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "from TorchCRF import CRF"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vlxGIUPVbFA5"
      },
      "source": [
        "## <font color='green'> Model </font>\n",
        "\n",
        "We use LSTM and CRF models to build a LSTM/CRF model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W1n667bbrr36"
      },
      "source": [
        "### <font color='blue'> LSTM/CRF </font>\n",
        "\n",
        "This class provides a convenient way to define and use an LSTM-CRF model for sequence labeling tasks, where the LSTM layer is used to capture the sequential information, and the CRF layer helps to model the dependencies among the labels in the sequence."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "45D2qtnj_Ocv"
      },
      "outputs": [],
      "source": [
        "class LSTMCRFModel(nn.Module):\n",
        "    def __init__(self, input_dim, hidden_dim, num_labels):\n",
        "        super(LSTMCRFModel, self).__init__()\n",
        "        self.lstm = nn.LSTM(input_dim, hidden_dim, batch_first=True)\n",
        "        self.crf = CRF(num_labels)\n",
        "\n",
        "    def forward(self, x, labels, mask):\n",
        "        lstm_output, _ = self.lstm(x)\n",
        "        crf_output = self.crf(lstm_output, labels, mask=mask)\n",
        "        return crf_output"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oVkPdryVqU_w"
      },
      "source": [
        "### <font color='blue'> Simple LSTM </font>\n",
        "\n",
        "Another approach to solve Extractive Question Answering is to use LSTMModel with fully-connected layers. The LSTM layer allows the model to capture the contextual information and dependencies within the input sequence. The fully connected layer helps to predict the start and end positions of the answer based on the learned representations from the LSTM layer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JrT0DMWvEJ8-"
      },
      "outputs": [],
      "source": [
        "class LSTMModel(nn.Module):\n",
        "    def __init__(self, input_dim, hidden_dim, num_layers, dropout):\n",
        "        super(LSTMModel, self).__init__()\n",
        "        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, dropout=dropout, batch_first=True)\n",
        "        self.fc = nn.Linear(hidden_dim, 2)\n",
        "\n",
        "    def forward(self, x):\n",
        "        out, _ = self.lstm(x)\n",
        "        logits = self.fc(out)\n",
        "\n",
        "        return logits"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lYX4Gl-VFiG0"
      },
      "outputs": [],
      "source": [
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dEz79yoKr95R"
      },
      "source": [
        "### <font color='blue'> Padding labels </font>\n",
        "\n",
        "This function is designed to pad a list of labels to a specified maximum length. This function is used in in this task because the length of the labels may vary across different documents. This method is used to convert training labels to padded_train_labels."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fHl5sq73_Ocw"
      },
      "outputs": [],
      "source": [
        "def pad_labels(labels, max_length):\n",
        "    padded_labels = []\n",
        "    for label in labels:\n",
        "        if len(label) < max_length:\n",
        "            pad_length = max_length - len(label)\n",
        "            padded_label = np.pad(label, (0, pad_length), constant_values=0)\n",
        "            padded_labels.append(padded_label)\n",
        "        else:\n",
        "            padded_labels.append(label[:max_length])\n",
        "    return padded_labels"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Vdyb7xlF_Ocw"
      },
      "outputs": [],
      "source": [
        "max_train_length = max(len(label) for label in train_labels)\n",
        "padded_train_labels = pad_labels(train_labels, max_train_length)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-VnGe6wV_Ocx",
        "outputId": "1d3c99f3-1068-4a52-de3d-3a717635d798"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "<ipython-input-98-e37d34318493>:1: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:245.)\n",
            "  padded_train_labels = torch.tensor(padded_train_labels).to(device)\n"
          ]
        }
      ],
      "source": [
        "padded_train_labels = torch.tensor(padded_train_labels).to(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IcVxS8lnvtK_"
      },
      "source": [
        "## <font color='green'> Training </font>\n",
        "In the training part we train our data on the LSTM/CRF model then evaluate it on the test data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R-GrDjLWzoHZ"
      },
      "source": [
        "### <font color='blue'> Training with training data </font>\n",
        "\n",
        "In this part we only use training vectors and without using any batches or Dataloaders. We train the model on the data, two times."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wWBoo8sX_Ocx"
      },
      "outputs": [],
      "source": [
        "input_dim = 19151\n",
        "hidden_dim = 128\n",
        "num_layers = 4\n",
        "dropout = 0.2\n",
        "num_labels = 2\n",
        "\n",
        "model = LSTMCRFModel(input_dim, hidden_dim, num_labels)\n",
        "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
        "num_epochs = 2\n",
        "\n",
        "for epoch in range(num_epochs):\n",
        "    documents = torch.tensor(training_vectors, dtype=torch.float32).to(device)\n",
        "    padded_train_labels = torch.tensor(padded_train_labels).to(device)\n",
        "    optimizer.zero_grad()\n",
        "    mask = documents.gt(0).float()\n",
        "    lengths = mask.sum(dim=1).int()\n",
        "    outputs = model(documents, padded_train_labels, mask=mask)\n",
        "    loss = -outputs\n",
        "    loss.backward()\n",
        "    optimizer.step()\n",
        "    print(f\"Epoch {epoch+1}/{num_epochs}, Loss: {loss.item()}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OreVBwIIzzez"
      },
      "source": [
        "### <font color='blue'> Training with training and validation data </font>\n",
        "\n",
        "Unlike the last part, in this section we use validation and training data in the process. We also use Dataloaders and batches so our model be more accurate and more efficient. Also in this part padding labels is used so we can compare the predicted output with the acutal output."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YUKez26lENMA",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "input_dim = 19151\n",
        "hidden_dim = 128\n",
        "num_layers = 4\n",
        "num_labels = 2\n",
        "dropout = 0.2\n",
        "\n",
        "model = LSTMCRFModel(input_dim, hidden_dim, num_labels)\n",
        "criterion = model.crf.loss\n",
        "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
        "num_epochs = 10\n",
        "\n",
        "total_training_loss = 0\n",
        "avg_training_loss = 0\n",
        "\n",
        "num_epochs = 1\n",
        "for epoch in range(num_epochs):\n",
        "    model.train()\n",
        "    training_loss = 0.0\n",
        "    for batch in train_loader:\n",
        "        documents, labels = batch\n",
        "        documents_length = len(documents)\n",
        "        documents = torch.tensor(documents, dtype=torch.float32).to(device)\n",
        "        max_length = max(len(label) for label in labels)\n",
        "        padded_labels = pad_labels(labels, max_length)\n",
        "        padded_labels = torch.tensor(padded_labels).to(device)\n",
        "        optimizer.zero_grad()\n",
        "        mask = documents.gt(0).float()\n",
        "        lengths = mask.sum(dim=1).int()\n",
        "        outputs = model(documents, padded_labels, mask=mask)\n",
        "        loss = -outputs\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        ## for LSTMModel:\n",
        "        #  documents, start_indices, end_indices = batch\n",
        "        #  documents = torch.tensor(documents).to(device)\n",
        "        #  start_indices = torch.tensor(start_indices).to(device)\n",
        "        #  end_indices = torch.tensor(end_indices).to(device)\n",
        "        #  documents = documents.to(torch.float32)\n",
        "        #  start_indices = start_indices.to(torch.float32)\n",
        "        #  end_indices = end_indices.to(torch.float32)\n",
        "        #  logits = model(documents)\n",
        "        #  start_logits = logits[:, 0]\n",
        "        #  end_logits = logits[:, 1]\n",
        "        #  start_loss = criterion(start_logits, start_indices)\n",
        "        #  end_loss = criterion(end_logits, end_indices)\n",
        "        #  loss = start_loss + end_loss\n",
        "        #  total_training_loss += loss\n",
        "\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        training_loss += loss.item() * documents_length\n",
        "\n",
        "    average_loss = training_loss / len(train_dataset)\n",
        "    print(f'Epoch {epoch + 1}/{num_epochs}, Training Loss: {average_loss:.4f}')\n",
        "\n",
        "\n",
        "    model.eval()\n",
        "    with torch.no_grad():\n",
        "        total_loss = 0.0\n",
        "        for batch in valid_loader:\n",
        "\n",
        "            documents, labels = batch\n",
        "            documents_length = len(documents)\n",
        "            documents = torch.tensor(documents, dtype=torch.float32).to(device)\n",
        "            max_length = max(len(label) for label in labels)\n",
        "            padded_labels = pad_labels(labels, max_length)\n",
        "            padded_labels = torch.tensor(padded_labels).to(device)\n",
        "            mask = documents.gt(0).float()\n",
        "            lengths = mask.sum(dim=1).int()\n",
        "            outputs = model(documents, padded_labels, mask=mask)\n",
        "            loss = -outputs\n",
        "\n",
        "            ## for LSTMModel:\n",
        "            #  documents, start_indices, end_indices = batch\n",
        "            #  documents = torch.tensor(documents).to(device)\n",
        "            #  start_indices = torch.tensor(start_indices).to(device)\n",
        "            #  end_indices = torch.tensor(end_indices).to(device)\n",
        "            #  documents = documents.to(torch.float32)\n",
        "            #  start_indices = start_indices.to(torch.float32)\n",
        "            #  end_indices = end_indices.to(torch.float32)\n",
        "            #  logits = model(documents)\n",
        "            #  start_logits = logits[:, 0]\n",
        "            #  end_logits = logits[:, 1]\n",
        "            #  start_loss = criterion(start_logits, start_indices)\n",
        "            #  end_loss = criterion(end_logits, end_indices)\n",
        "            #  loss = start_loss + end_loss\n",
        "            #  total_training_loss += loss\n",
        "\n",
        "            total_loss += loss.item() * documents_length\n",
        "\n",
        "        average_loss = total_loss / len(valid_dataset)\n",
        "        print(f'Epoch {epoch + 1}/{num_epochs}, Validation Loss: {average_loss:.4f}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nPcavKwQ0YV-"
      },
      "source": [
        "### <font color='blue'> Evaluation </font>\n",
        "\n",
        "FInally we can evaluate our model on the test data and use different metrics to asses the efficieny and accuracy of that."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cmTx7ZXU_Ocy"
      },
      "outputs": [],
      "source": [
        "model.eval()\n",
        "accuracy = 0.0\n",
        "total = 0.0\n",
        "test_loss = 0\n",
        "with torch.no_grad():\n",
        "    for data in test_loader:\n",
        "        documents, labels = batch\n",
        "        documents_length = len(documents)\n",
        "        documents = torch.tensor(documents, dtype=torch.float32).to(device)\n",
        "        max_length = max(len(label) for label in labels)\n",
        "        padded_labels = pad_labels(labels, max_length)\n",
        "        padded_labels = torch.tensor(padded_labels).to(device)\n",
        "        mask = documents.gt(0).float()\n",
        "        lengths = mask.sum(dim=1).int()\n",
        "        outputs = model(documents, padded_labels, mask=mask)\n",
        "        loss = -outputs\n",
        "\n",
        "\n",
        "      # # for LSTMModel:\n",
        "      #  documents, start_indices, end_indices = batch\n",
        "      #  documents = torch.tensor(documents).to(device)\n",
        "      #  start_indices = torch.tensor(start_indices).to(device)\n",
        "      #  end_indices = torch.tensor(end_indices).to(device)\n",
        "      #  documents = documents.to(torch.float32)\n",
        "      #  start_indices = start_indices.to(torch.float32)\n",
        "      #  end_indices = end_indices.to(torch.float32)\n",
        "      #  logits = model(documents)\n",
        "      #  start_logits = logits[:, 0]\n",
        "      #  end_logits = logits[:, 1]\n",
        "      #  start_loss = criterion(start_logits, start_indices)\n",
        "      #  end_loss = criterion(end_logits, end_indices)\n",
        "      #  loss = start_loss + end_loss\n",
        "      #  total_training_loss += loss\n",
        "\n",
        "        total_loss += loss.item() * documents_length\n",
        "#         _, predicted = torch.max(outputs.data, 1)\n",
        "#         accuracy += (predicted == labels).sum().item()\n",
        "#         total += start_indices.size(0)\n",
        "\n",
        "test_loss /= len(test_loader.dataset)\n",
        "print(f\"Test Loss: {test_loss:.4f}\")\n",
        "accuracy = (100 * accuracy / total)\n",
        "print(f\"accuracy: {accuracy:.2f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "83gBvbgTKjSH"
      },
      "source": [
        "# <font color='red'> Transformer </font>\n",
        "----\n",
        "\n",
        "Available in another notebook. (NLP_HW4_token_clf_transformer)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "ahOaVRsg59At",
        "jHUFoQfot2B4",
        "V2eWzvtDt52m",
        "PBT0JWMt6qb0",
        "QNTmiXdTdpTw",
        "dhSMCNhAy8c2",
        "eDingu_NUM5W",
        "lFWNjnacuztl",
        "_9sr8Ya4u1cO",
        "zpHj1RYJ_Ocj",
        "kn4jNVhTrgMH",
        "SlL_P0IjUSR1",
        "m_JRzgwzwu42",
        "XFXbX5oMvV16"
      ],
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.8"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "09bc48ffe5d24c0692cc5c2a2ff7bc08": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0fe6777985f142e8a18103c48aea7941": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1538bad5e0ce4c1e88ffb1a2c874d965": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d8e0be76669d4870b040219f85c2b4f2",
            "placeholder": "​",
            "style": "IPY_MODEL_88fd33c6de8f4ea9a7baef7b7a887a23",
            "value": "100%"
          }
        },
        "1d1068b874684f1b823b518c8e24852e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3733351f80974d78a7368f77c407617d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_09bc48ffe5d24c0692cc5c2a2ff7bc08",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c6cec377a8b74b37a0acb5b4ce414851",
            "value": 2
          }
        },
        "88fd33c6de8f4ea9a7baef7b7a887a23": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a9374d78c6504f0fa21b20a69b90a40c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_1538bad5e0ce4c1e88ffb1a2c874d965",
              "IPY_MODEL_3733351f80974d78a7368f77c407617d",
              "IPY_MODEL_f83ced7bdd0f411ab0f94eac6ec4c9f0"
            ],
            "layout": "IPY_MODEL_1d1068b874684f1b823b518c8e24852e"
          }
        },
        "c6cec377a8b74b37a0acb5b4ce414851": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "d8e0be76669d4870b040219f85c2b4f2": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e7766ab28b7140808f305a29816b6341": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f83ced7bdd0f411ab0f94eac6ec4c9f0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0fe6777985f142e8a18103c48aea7941",
            "placeholder": "​",
            "style": "IPY_MODEL_e7766ab28b7140808f305a29816b6341",
            "value": " 2/2 [00:00&lt;00:00, 55.45it/s]"
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}