Assignment3-text-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 3 - Text Analysis\n",
    "An explanation this assignment could be found in the .pdf explanation document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Materials to review for this assignment\n",
    "<h4>From Moodle:</h4> \n",
    "<h5><u>Review the notebooks regarding the following python topics</u>:</h5>\n",
    "<div class=\"alert alert-info\">\n",
    "&#x2714; <b>Working with strings</b> (tutorial notebook)<br/>\n",
    "&#x2714; <b>Text Analysis</b> (tutorial notebook)<br/>\n",
    "&#x2714; <b>Hebrew text analysis tools (tokenizer, wordnet)</b> (moodle example)<br/>\n",
    "&#x2714; <b>(brief review) All previous notebooks</b><br/>\n",
    "</div> \n",
    "<h5><u>Review the presentations regarding the following topics</u>:</h5>\n",
    "<div class=\"alert alert-info\">\n",
    "&#x2714; <b>Text Analysis</b> (lecture presentation)<br/>\n",
    "&#x2714; <b>(brief review) All other presentations</b><br/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Personal Details:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: Tom Elia\n",
      "ID: 206378036\n"
     ]
    }
   ],
   "source": [
    "# Details Student 1:\n",
    "def student_details(name, id_num):\n",
    "    print(\"name:\", name)\n",
    "    print( \"ID:\", id_num)\n",
    "# Details Student 2:\n",
    "student_details(\"Tom Elia\", '206378036')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preceding Step - import modules (packages)\n",
    "This step is necessary in order to use external modules (packages). <br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "# --------------------------------------\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "# --------------------------------------\n",
    "\n",
    "\n",
    "# --------------------------------------\n",
    "# ------------- visualizations:\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.colors import ListedColormap\n",
    "# --------------------------------------\n",
    "\n",
    "\n",
    "# ---------------------------------------\n",
    "import sklearn\n",
    "from sklearn import preprocessing, metrics, pipeline, model_selection, feature_extraction \n",
    "from sklearn import naive_bayes, linear_model, svm, neural_network, neighbors, tree\n",
    "from sklearn import decomposition, cluster\n",
    "from sklearn.metrics import precision_score, recall_score, f1_score, make_scorer\n",
    "\n",
    "from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV \n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix\n",
    "from sklearn.metrics import precision_score, recall_score, f1_score\n",
    "from sklearn.metrics import mean_squared_error, r2_score, silhouette_score\n",
    "from sklearn.preprocessing import MinMaxScaler, StandardScaler, LabelEncoder\n",
    "from sklearn.metrics import make_scorer\n",
    "\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.linear_model import Perceptron, SGDClassifier\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.naive_bayes import MultinomialNB, GaussianNB\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "# ---------------------------------------\n",
    "\n",
    "\n",
    "# ----------------- output and visualizations: \n",
    "import warnings\n",
    "from sklearn.exceptions import ConvergenceWarning\n",
    "warnings.simplefilter(\"ignore\")\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
    "warnings.simplefilter(\"ignore\", category=ConvergenceWarning)\n",
    "# show several prints in one cell. This will allow us to condence every trick in one cell.\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "%matplotlib inline\n",
    "pd.pandas.set_option('display.max_columns', None)\n",
    "pd.set_option('display.float_format', lambda x: '%.3f' % x)\n",
    "# ---------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text analysis and String manipulation imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# --------------------------------------\n",
    "# --------- Text analysis and Hebrew text analysis imports:\n",
    "# vectorizers:\n",
    "from sklearn.feature_extraction import text\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "\n",
    "# regular expressions:\n",
    "import re\n",
    "# --------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (optional) Hebrew text analysis - WordNet (for Hebrew)\n",
    "Note: the WordNet is not a must"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) Only if you didn't install Wordnet (for Hebrew) use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting wn\n",
      "  Downloading wn-0.9.4-py3-none-any.whl (75 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.7/75.7 kB\u001b[0m \u001b[31m799.5 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: requests in /Users/tomelia/opt/anaconda3/lib/python3.9/site-packages (from wn) (2.28.1)\n",
      "Requirement already satisfied: tomli in /Users/tomelia/opt/anaconda3/lib/python3.9/site-packages (from wn) (2.0.1)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/tomelia/opt/anaconda3/lib/python3.9/site-packages (from requests->wn) (1.26.11)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/tomelia/opt/anaconda3/lib/python3.9/site-packages (from requests->wn) (3.3)\n",
      "Requirement already satisfied: charset-normalizer<3,>=2 in /Users/tomelia/opt/anaconda3/lib/python3.9/site-packages (from requests->wn) (2.0.4)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/tomelia/opt/anaconda3/lib/python3.9/site-packages (from requests->wn) (2022.9.24)\n",
      "Installing collected packages: wn\n",
      "Successfully installed wn-0.9.4\n",
      "Download [##############################] (315276/315276 bytes) Completeg\n",
      "Read [##############################] (29690/29690) 000gq/T/tmpg731zn27/omw-he/omw-he.xmll\n",
      "\u001b[KAdded omw-he:1.4 (Hebrew Wordnet)####] (29687/29687) Examplesonsionssours\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# word net installation:\n",
    "\n",
    "# unmark if you want to use and need to install\n",
    "!pip install wn\n",
    "!python -m wn download omw-he:1.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# word net import:\n",
    "\n",
    "# unmark if you want to use:\n",
    "import wn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[KCached file found: /Users/tomelia/.wn_data/downloads/7ecf10e89326bc0ac26ad94b40fe60a7b6ac3302\n",
      "\u001b[KSkipping omw-he:1.4 (Hebrew Wordnet); already added\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('/Users/tomelia/.wn_data/downloads/7ecf10e89326bc0ac26ad94b40fe60a7b6ac3302')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of total Hebrew sysets: 5448\n",
      "number of sysets: 3\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Sense('omw-he-דָּרַךְ-02091410-v')]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "['דָּרַךְ']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[Sense('omw-he-דֶּרֶךְ-02671062-n')]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "['דֶּרֶךְ']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[Sense('omw-he-צוּרָה-04928903-n'),\n",
       " Sense('omw-he-דֶּרֶךְ-04928903-n'),\n",
       " Sense('omw-he-סִגְנוֹן-04928903-n')]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "['צוּרָה', 'דֶּרֶךְ', 'סִגְנוֹן']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'הדרך שבה דברים נעשים או מוצגים'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#testing word net\n",
    "\n",
    "wn.download('omw-he:1.4')\n",
    "wordnet_he = wn.Wordnet('omw-he:1.4')\n",
    "\n",
    "synsets= wordnet_he.synsets()\n",
    "print('number of total Hebrew sysets:', len(synsets))\n",
    "\n",
    "synsets= wordnet_he.synsets('דרך')     \n",
    "print('number of sysets:', len(synsets))\n",
    "synsets[0].senses()\n",
    "synsets[0].lemmas()\n",
    "synsets[1].senses()\n",
    "synsets[1].lemmas()\n",
    "synsets[2].senses()\n",
    "synsets[2].lemmas()\n",
    "synsets[2].definition()  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (optional) Hebrew text analysis - hebrew_tokenizer (Tokenizer for Hebrew)\n",
    "Note: the hebrew_tokenizer is not a must"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) Only if you didn't install hebrew_tokenizer use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting hebrew_tokenizer\n",
      "  Downloading hebrew_tokenizer-2.3.0.tar.gz (13 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hBuilding wheels for collected packages: hebrew_tokenizer\n",
      "  Building wheel for hebrew_tokenizer (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for hebrew_tokenizer: filename=hebrew_tokenizer-2.3.0-py3-none-any.whl size=13418 sha256=305f991ad9b40641011d047e4eecd5d29ae4a30ef78686c2606d87a55f53f207\n",
      "  Stored in directory: /Users/tomelia/Library/Caches/pip/wheels/a7/2f/af/95d5eb2e7c291753542d4419e200429a22fb2d74e3f27d7711\n",
      "Successfully built hebrew_tokenizer\n",
      "Installing collected packages: hebrew_tokenizer\n",
      "Successfully installed hebrew_tokenizer-2.3.0\n"
     ]
    }
   ],
   "source": [
    "# Hebrew tokenizer installation:\n",
    "\n",
    "# unmark if you want to use and need to install:\n",
    "!pip install hebrew_tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/tomelia/Downloads/assignment3-text_analysis\n"
     ]
    }
   ],
   "source": [
    "# Hebrew tokenizer import:\n",
    "\n",
    "# unmark if you want to use:\n",
    "import hebrew_tokenizer as ht"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEBREW, אתמול\n",
      "PUNCTUATION, ,\n",
      "DATE, 8.6.2018\n",
      "PUNCTUATION, ,\n",
      "HEBREW, בשעה\n",
      "HOUR, 17:00\n",
      "HEBREW, הלכתי\n",
      "HEBREW, עם\n",
      "HEBREW, אמא\n",
      "HEBREW, למכולת\n"
     ]
    }
   ],
   "source": [
    "#Hebrew tokinizer test\n",
    "hebrew_text = \"אתמול, 8.6.2018, בשעה 17:00 הלכתי עם אמא למכולת\"\n",
    "tokens = ht.tokenize(hebrew_text)  # tokenize returns a generator!\n",
    "for grp, token, token_num, (start_index, end_index) in tokens:\n",
    "    print('{}, {}'.format(grp, token))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading input files\n",
    "Reading input files for train annotated corpus (raw text data) corpus and for the test corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_filename = 'annotated_corpus_for_train.csv'\n",
    "test_filename  = 'corpus_for_test.csv'\n",
    "df_train = pd.read_csv(train_filename, index_col=None, encoding='utf-8')\n",
    "df_test  = pd.read_csv(test_filename, index_col=None, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>story</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>כשחבר הזמין אותי לחול, לא באמת חשבתי שזה יקרה,...</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>לפני שהתגייסתי לצבא עשיתי כל מני מיונים ליחידו...</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>מאז שהתחילו הלימודים חלומו של כל סטודנט זה הפנ...</td>\n",
       "      <td>f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>כשהייתי ילד, מטוסים היה הדבר שהכי ריתק אותי. ב...</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>‏הייתי מדריכה בכפר נוער ומתאם הכפר היינו צריכי...</td>\n",
       "      <td>f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>לפני כ3 חודשים טסתי לרומא למשך שבוע. טסתי במטו...</td>\n",
       "      <td>f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>אני כבר שנתיים נשוי והשנה אני ואישתי סוף סוף י...</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>השנה התחלנו שיפוץ בדירה שלנו בתל אביב. הדירה ה...</td>\n",
       "      <td>f</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               story gender\n",
       "0  כשחבר הזמין אותי לחול, לא באמת חשבתי שזה יקרה,...      m\n",
       "1  לפני שהתגייסתי לצבא עשיתי כל מני מיונים ליחידו...      m\n",
       "2  מאז שהתחילו הלימודים חלומו של כל סטודנט זה הפנ...      f\n",
       "3  כשהייתי ילד, מטוסים היה הדבר שהכי ריתק אותי. ב...      m\n",
       "4  ‏הייתי מדריכה בכפר נוער ומתאם הכפר היינו צריכי...      f\n",
       "5  לפני כ3 חודשים טסתי לרומא למשך שבוע. טסתי במטו...      f\n",
       "6  אני כבר שנתיים נשוי והשנה אני ואישתי סוף סוף י...      m\n",
       "7  השנה התחלנו שיפוץ בדירה שלנו בתל אביב. הדירה ה...      f"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(753, 2)"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head(8)\n",
    "df_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_example_id</th>\n",
       "      <th>story</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>כל קיץ אני והמשפחה נוסעים לארצות הברית לוס אנג...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>הגעתי לשירות המדינה אחרי שנתיים כפעיל בתנועת \"...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>אחת האהבות הגדולות שלי אלו הכלבים שלי ושל אישת...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   test_example_id                                              story\n",
       "0                0  כל קיץ אני והמשפחה נוסעים לארצות הברית לוס אנג...\n",
       "1                1  הגעתי לשירות המדינה אחרי שנתיים כפעיל בתנועת \"...\n",
       "2                2  אחת האהבות הגדולות שלי אלו הכלבים שלי ושל אישת..."
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(323, 2)"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test.head(3)\n",
    "df_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your implementation:\n",
    "Write your code solution in the following code-cells"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_cleaning_text(input_text):\n",
    "    # Remove punctuation marks such as \", . ; \" and more\n",
    "    txt = re.sub(r'[.,;:!?/\"()\\-a-zA-Z]', '', input_text)\n",
    "    \n",
    "    # Delete numbers\n",
    "    txt = re.sub(r'\\d+', '', txt)\n",
    "    \n",
    "    # convert multiple spaces into one space\n",
    "    txt = re.sub(r'\\s+', ' ', txt)\n",
    "    \n",
    "\n",
    "    return txt\n",
    "\n",
    "\n",
    "\n",
    "df_train['story'] = df_train['story'].apply(data_cleaning_text)\n",
    "df_test['story'] = df_test['story'].apply(data_cleaning_text)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "def stemming_func(input_text):\n",
    "\n",
    "    # STEMING:\n",
    "        # Remove prefixes : 'כ' 'ו' 'ב' 'ש' 'ה' 'ל' 'כש' 'לכש' 'וש' 'וכש'\n",
    "    txt = re.sub(r'\\b(כ|ו|ה|ב|וש|לכש|וכש|ש|כש|ל)', '', input_text)\n",
    "    # Remove suffixes : 'תי' 'נו' 'ים' 'ו'\n",
    "    txt =  re.sub(r'(תי|ו|נו|ים)\\b', '', txt)\n",
    "    txt = re.sub(r'ות\\b', 'ה', txt) \n",
    "    \n",
    "        # fixing words - adding ending  - אות סופית \n",
    "    txt = re.sub(r'כ\\b', 'ך', txt)\n",
    "    txt = re.sub(r'מ\\b', 'ם', txt)\n",
    "    txt = re.sub(r'צ\\b', 'ץ', txt)    \n",
    "    txt = re.sub(r'נ\\b', 'ן', txt)\n",
    "    txt = re.sub(r'פ\\b', 'ף', txt)\n",
    "    return txt\n",
    "    \n",
    "df_train['story'] = df_train['story'].apply(stemming_func)\n",
    "df_test['story'] = df_test['story'].apply(stemming_func)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'שחבר זמין או חול א אמת חשב זה יקרה פשוט אמר  ן תיאר עצני זה יתבטל אחרי בועי ערך אני מקבל טלפוןם מם ומע מצא אחלה מקודה נוטכל טייל הם אז בן זה ולך קרה תחל התארגןם על דבר ציוד הליכה תיק גד חם סף דרכון מעודכן אחר תכנון נפגש שדה וא ביא י את אחד מהתיק ל י י אין תיק טוב טיול עלי מטוס איטליה טיסה עצמה א צלח ישון יה ילד קטן בכה ל דרך מעצבן שהגע לך ישר סוכנה שכרת רכב לקח את רכב הזם מראש סיטרואל צבע סגול י זה מה נשאר חצי קרא ה עלי על חצי התחל את מסע כיוון אגם גארדה שעה ייתה ערב קצת קריר חוץ חושך מוה אין  מושג אן אנח נוסע רק עם התחלה תחל חפש מקום ישון  מצא עיירה סמוכה החלט לכת שם על דרך עצר פיצה םיצה ראשונה איטליה משם משך עיירה עצמה מצא אכסנייה די נחמדה בה עצר לילה בוקר למחורת וא מצא מסלול טיול על אחד הר אזור נסע שם נסיעה ל שעה ערך תחל עלה עם רכב כיוון מסלול דרך ייתה נופית עץ יער מכל יוון עד באיזשה לב וא מבקש ממני עצור לחנה את רכב אני לא ראי חניה מסודרת לא מכיר מסלולי טיול אלה א בן מה וא רוצה עצר את רכב יצא מם מסתכל כיוון הר דיעבד הר יה גובה ל קם וא מצביע ראש הר אומר י רואה את צלב ם שם אנח מגיע אמר אוקיי איפה התחלה וא אומר י פה התחל לכת כיוון ראש הר התחלה עליה ייתה די תלולה אני אומנם לי םפחד גבה אבל ל זמן דיבר אית מה יה קורה אם אם מישה נופל מה עוש איל מזעיק עזרה עייריה כי קרובה זה עה נסיעה אני לי טלפון עם רשת הוא עם חצי וח סכם אם אחד נופל זאת פציעה קשה עוצר מובן מנס פנה מה אפשר א קרה  לום שמח אחר טיפוס ל עתי ערךף גע ראש הר נוף יה מרש שמי י מכוס ענן לא ראי יותר מידי התחלה אבל אחר מה דקה שמי תבהר קצת ראי ממש רחוק אפיל אתץ אוט לא גנב אה סוף זאת ייתה אחת מהדאגה לי אחר מכן חזר אוט נסע עיירה חדשה מטרה טייל וב אז תחילה קורונה א חשב זה יגיע אזור ל אבל עדיין ששמע על זה פעם ראשונה קצת צחק אמר אין סיכוי זה יגיע אלי אבל שהגיע אליטליה מצא את עצם מנס חזור מה יותר נמהר ביתה סוף ספק עלה את טיסה ראשטה מאיטליה פני היא נסגרה מזל יה היי עם מסיכה בר אז א נדבק'"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train['story'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_to_tokens(text):\n",
    "    return text.split(' ')\n",
    "\n",
    "X = df_train['story']\n",
    "y = df_train['gender']\n",
    "\n",
    "\n",
    "vectorizer = CountVectorizer(tokenizer=split_to_tokens, ngram_range=(1,3), min_df=5, max_df=400, max_features=10000) \n",
    "trained_vector = vectorizer.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=10, estimator=KNeighborsClassifier(),\n",
       "             param_grid={'n_neighbors': [3, 5, 7, 9, 11, 13],\n",
       "                         'weights': ['uniform', 'distance']},\n",
       "             scoring=make_scorer(f1_score, average=macro))"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean score: 0.5214, Parameters: {'n_neighbors': 3, 'weights': 'uniform'}\n",
      "Mean score: 0.5340, Parameters: {'n_neighbors': 3, 'weights': 'distance'}\n",
      "Mean score: 0.4768, Parameters: {'n_neighbors': 5, 'weights': 'uniform'}\n",
      "Mean score: 0.4895, Parameters: {'n_neighbors': 5, 'weights': 'distance'}\n",
      "Mean score: 0.4422, Parameters: {'n_neighbors': 7, 'weights': 'uniform'}\n",
      "Mean score: 0.4739, Parameters: {'n_neighbors': 7, 'weights': 'distance'}\n",
      "Mean score: 0.4326, Parameters: {'n_neighbors': 9, 'weights': 'uniform'}\n",
      "Mean score: 0.4655, Parameters: {'n_neighbors': 9, 'weights': 'distance'}\n",
      "Mean score: 0.4330, Parameters: {'n_neighbors': 11, 'weights': 'uniform'}\n",
      "Mean score: 0.4492, Parameters: {'n_neighbors': 11, 'weights': 'distance'}\n",
      "Mean score: 0.4330, Parameters: {'n_neighbors': 13, 'weights': 'uniform'}\n",
      "Mean score: 0.4492, Parameters: {'n_neighbors': 13, 'weights': 'distance'}\n",
      "estimatior: MultinomialNB\n",
      "The best parameters combination: {'n_neighbors': 3, 'weights': 'distance'}\n",
      "Highest F1 Score: 0.534022410445784\n"
     ]
    }
   ],
   "source": [
    "#first machine learning estimator (model): 'K-Nearest Neighbors'\n",
    "\n",
    "\n",
    "#GridSearchCV implements cross-validation\n",
    "model_KNN =  {\n",
    "        'model_name': 'K-Nearest Neighbors',\n",
    "        'model': KNeighborsClassifier(),\n",
    "        'parameters': {'n_neighbors': [3, 5, 7, 9, 11, 13], 'weights': ['uniform', 'distance']}\n",
    "    }\n",
    "estimator = model_KNN['model']\n",
    "    \n",
    "grid_search = GridSearchCV(estimator, model_KNN['parameters'], scoring=f1_scorer, cv=10)\n",
    "grid_search.fit(trained_vector, y)\n",
    "    \n",
    "    #getting information about the best parameter combination and the corresponding best model.\n",
    "best_model = grid_search.best_estimator_\n",
    "best_parameters = grid_search.best_params_\n",
    "best_score = grid_search.best_score_\n",
    "    \n",
    "\n",
    "cv_results = grid_search.cv_results_\n",
    "\n",
    "    # Access the mean test scores for each parameter combination\n",
    "mean_test_scores = cv_results['mean_test_score']\n",
    "\n",
    "    # Print out the mean test score for each parameter combination\n",
    "for mean_score, params in zip(mean_test_scores, cv_results['params']):\n",
    "    print(f\"Mean score: {mean_score:.4f}, Parameters: {params}\")\n",
    "    \n",
    "    \n",
    "    \n",
    "print(\"estimatior:\", model['model_name'])\n",
    "print(\"The best parameters combination:\", best_parameters)\n",
    "print(\"Highest F1 Score:\", best_score)\n",
    "     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=10, estimator=MultinomialNB(),\n",
       "             param_grid={'alpha': [0.1, 0.5, 1.0, 1.5, 2.0],\n",
       "                         'fit_prior': [True, False]},\n",
       "             scoring=make_scorer(f1_score, average=macro))"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean score: 0.6349, Parameters: {'alpha': 0.1, 'fit_prior': True}\n",
      "Mean score: 0.6439, Parameters: {'alpha': 0.1, 'fit_prior': False}\n",
      "Mean score: 0.7231, Parameters: {'alpha': 0.5, 'fit_prior': True}\n",
      "Mean score: 0.7246, Parameters: {'alpha': 0.5, 'fit_prior': False}\n",
      "Mean score: 0.7304, Parameters: {'alpha': 1.0, 'fit_prior': True}\n",
      "Mean score: 0.7420, Parameters: {'alpha': 1.0, 'fit_prior': False}\n",
      "Mean score: 0.7222, Parameters: {'alpha': 1.5, 'fit_prior': True}\n",
      "Mean score: 0.7286, Parameters: {'alpha': 1.5, 'fit_prior': False}\n",
      "Mean score: 0.6904, Parameters: {'alpha': 2.0, 'fit_prior': True}\n",
      "Mean score: 0.7094, Parameters: {'alpha': 2.0, 'fit_prior': False}\n",
      "estimatior: MultinomialNB\n",
      "The best parameters combination: {'alpha': 1.0, 'fit_prior': False}\n",
      "Highest F1 Score: 0.7420393811241368\n"
     ]
    }
   ],
   "source": [
    "#Second machine learning estimator (model): 'MultinomialNB'\n",
    "\n",
    "\n",
    "#GridSearchCV implements cross-validation\n",
    "model =       {\n",
    "        'model_name': 'MultinomialNB',\n",
    "        'model': MultinomialNB(),\n",
    "        'parameters': {'alpha': [0.1, 0.5, 1.0, 1.5, 2.0], 'fit_prior': [True, False]}\n",
    "    }\n",
    "estimator = model['model']\n",
    "    \n",
    "grid_search = GridSearchCV(estimator, model['parameters'], scoring=f1_scorer, cv=10)\n",
    "grid_search.fit(trained_vector, y)\n",
    "    \n",
    "    #getting information about the best parameter combination and the corresponding best model.\n",
    "best_model = grid_search.best_estimator_\n",
    "best_parameters = grid_search.best_params_\n",
    "best_score = grid_search.best_score_\n",
    "    \n",
    "\n",
    "cv_results = grid_search.cv_results_\n",
    "\n",
    "    # Access the mean test scores for each parameter combination\n",
    "mean_test_scores = cv_results['mean_test_score']\n",
    "\n",
    "    # Print out the mean test score for each parameter combination\n",
    "for mean_score, params in zip(mean_test_scores, cv_results['params']):\n",
    "    print(f\"Mean score: {mean_score:.4f}, Parameters: {params}\")\n",
    "    \n",
    "    \n",
    "    \n",
    "print(\"estimatior:\", model['model_name'])\n",
    "print(\"The best parameters combination:\", best_parameters)\n",
    "print(\"Highest F1 Score:\", best_score)\n",
    "     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=10, estimator=DecisionTreeClassifier(),\n",
       "             param_grid={'max_depth': [None, 5, 10, 20, 50, 100]},\n",
       "             scoring=make_scorer(f1_score, average=macro))"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean score: 0.6629, Parameters: {'max_depth': None}\n",
      "Mean score: 0.6577, Parameters: {'max_depth': 5}\n",
      "Mean score: 0.6865, Parameters: {'max_depth': 10}\n",
      "Mean score: 0.7007, Parameters: {'max_depth': 20}\n",
      "Mean score: 0.6411, Parameters: {'max_depth': 50}\n",
      "Mean score: 0.6575, Parameters: {'max_depth': 100}\n",
      "estimatior: Decision Tree\n",
      "The best parameters combination: {'max_depth': 20}\n",
      "Highest F1 Score: 0.70070715281071\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#Third machine learning estimator (model): 'Decision Tree'\n",
    "\n",
    "\n",
    "#GridSearchCV implements cross-validation\n",
    "model = {\n",
    "        'model_name': 'Decision Tree',\n",
    "        'model': DecisionTreeClassifier(),\n",
    "        'parameters': {'max_depth': [None, 5, 10, 20, 50, 100]}\n",
    "    }\n",
    "estimator = model['model']\n",
    "    \n",
    "grid_search = GridSearchCV(estimator, model['parameters'], scoring=f1_scorer, cv=10)\n",
    "grid_search.fit(trained_vector, y)\n",
    "    \n",
    "    #getting information about the best parameter combination and the corresponding best model.\n",
    "best_model = grid_search.best_estimator_\n",
    "best_parameters = grid_search.best_params_\n",
    "best_score = grid_search.best_score_\n",
    "    \n",
    "\n",
    "cv_results = grid_search.cv_results_\n",
    "\n",
    "    # Access the mean test scores for each parameter combination\n",
    "mean_test_scores = cv_results['mean_test_score']\n",
    "\n",
    "    # Print out the mean test score for each parameter combination\n",
    "for mean_score, params in zip(mean_test_scores, cv_results['params']):\n",
    "    print(f\"Mean score: {mean_score:.4f}, Parameters: {params}\")\n",
    "    \n",
    "    \n",
    "    \n",
    "print(\"estimatior:\", model['model_name'])\n",
    "print(\"The best parameters combination:\", best_parameters)\n",
    "print(\"Highest F1 Score:\", best_score)\n",
    "     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(fit_prior=False)"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "array(['m', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'f', 'm', 'm',\n",
       "       'm', 'f', 'm', 'm', 'm', 'f', 'm', 'f', 'm', 'm', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'f', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'f', 'm',\n",
       "       'f', 'm', 'm', 'm', 'm', 'm', 'f', 'm', 'm', 'm', 'm', 'f', 'f',\n",
       "       'm', 'm', 'f', 'm', 'm', 'm', 'm', 'm', 'm', 'f', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'f', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'f', 'm', 'm', 'm', 'm', 'f', 'm', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'f', 'm', 'm', 'm', 'm',\n",
       "       'm', 'f', 'm', 'm', 'm', 'm', 'm', 'f', 'f', 'm', 'f', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'f', 'f', 'm', 'm', 'm', 'm', 'm', 'm', 'f',\n",
       "       'f', 'm', 'm', 'm', 'm', 'm', 'f', 'm', 'f', 'm', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'f', 'm', 'm', 'f', 'm', 'm', 'm', 'm', 'm',\n",
       "       'f', 'f', 'm', 'm', 'm', 'm', 'm', 'm', 'f', 'm', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'f', 'm', 'm', 'm', 'm',\n",
       "       'm', 'm', 'f', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'f', 'm', 'm',\n",
       "       'm', 'f', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'm', 'm', 'm', 'f', 'm', 'm', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'm', 'f', 'm', 'm', 'm', 'm', 'm', 'm', 'f',\n",
       "       'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'f', 'm', 'm', 'm', 'm',\n",
       "       'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm', 'm'], dtype='<U1')"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_example_id</th>\n",
       "      <th>predicted_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>318</th>\n",
       "      <td>318</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>319</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>320</th>\n",
       "      <td>320</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>321</th>\n",
       "      <td>321</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>322</th>\n",
       "      <td>322</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>323 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     test_example_id predicted_category\n",
       "0                  0                  m\n",
       "1                  1                  m\n",
       "2                  2                  m\n",
       "3                  3                  m\n",
       "4                  4                  m\n",
       "..               ...                ...\n",
       "318              318                  m\n",
       "319              319                  m\n",
       "320              320                  m\n",
       "321              321                  m\n",
       "322              322                  m\n",
       "\n",
       "[323 rows x 2 columns]"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_model = MultinomialNB(fit_prior=False)\n",
    "\n",
    "best_model.fit(vectorizer.fit_transform(X), y)\n",
    "\n",
    "y_prediction = best_model.predict(vectorizer.transform(df_test['story']))\n",
    "y_prediction\n",
    "\n",
    "predict_data_frame = pd.DataFrame({'test_example_id': df_test['test_example_id'], 'predicted_category': y_prediction})\n",
    "predict_data_frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save output to csv (optional)\n",
    "After you're done save your output to the 'classification_results.csv' csv file.<br/>\n",
    "We assume that the dataframe with your results contain the following columns:\n",
    "* column 1 (left column): 'test_example_id'  - the same id associated to each of the test stories to be predicted.\n",
    "* column 2 (right column): 'predicted_category' - the predicted gender value for each of the associated story. \n",
    "\n",
    "Assuming your predicted values are in the `df_predicted` dataframe, you should save you're results as following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict_data_frame.to_csv('classification_results.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Create Assignment",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}