8_Intermediate_Case_Study_4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 10 Practical Tasks\n",
    "## KNN Classifier and Naive Bayes Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Classifying Iris Species with KNN Classifier\n",
    "In this section, we will go through a simple machine learning application and create\n",
    "our first classification model. In the process, we will introduce some core concepts and terms.\n",
    "\n",
    "Let’s assume that a hobby botanist is interested in distinguishing the species of some\n",
    "iris flowers that she has found. She has collected some measurements associated with\n",
    "each iris: the length and width of the petals and the length and width of the sepals, all\n",
    "measured in centimeters.\n",
    "\n",
    "She also has the measurements of some irises that have been previously identified by\n",
    "an expert botanist as belonging to the species setosa, versicolor, or virginica. For these\n",
    "measurements, she can be certain of which species each iris belongs to. Let’s assume\n",
    "that these are the only species our hobby botanist will encounter in the wild.\n",
    "\n",
    "Our goal is to build a machine learning model that can learn from the measurements\n",
    "of these irises whose species is known, so that we can predict the species for a new\n",
    "iris.\n",
    "\n",
    "Reference: Introduction to Machine learning with Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### Import Data\n",
    "\n",
    "The sklearn package provides some built-in real-world data sets to let users experience working on a real-world data analysis applications. The Iris data set is one of them. Please refer to https://scikit-learn.org/stable/datasets/index.html for more information about these built-in data sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature names: \n",
      " ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']\n",
      "Target names: \n",
      " ['setosa' 'versicolor' 'virginica']\n",
      "Feature data size: \n",
      " (150, 4)\n",
      "Target data size: \n",
      " (150,)\n",
      "Target values: \n",
      " [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
      " 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "float_formatter = \"{:.6f}\".format\n",
    "np.set_printoptions(formatter={'float_kind':float_formatter})\n",
    "\n",
    "from sklearn import datasets\n",
    "\n",
    "iris_data = datasets.load_iris()\n",
    "\n",
    "print(\"Feature names: \\n\", iris_data.feature_names)\n",
    "print(\"Target names: \\n\", iris_data.target_names)\n",
    "\n",
    "print(\"Feature data size: \\n\", iris_data.data.shape)\n",
    "print(\"Target data size: \\n\", iris_data.target.shape)\n",
    "print(\"Target values: \\n\", iris_data.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### Training and Testing Data\n",
    "\n",
    "We want to build a machine learning model from this data that can predict the species\n",
    "of iris for a new set of measurements. But before we can apply our model to new\n",
    "measurements, we need to know whether it actually works—that is, whether we\n",
    "should trust its predictions.\n",
    "\n",
    "To assess the model’s performance, we show it new data (data that it hasn’t seen\n",
    "before) for which we have labels. This is usually done by splitting the labeled data we\n",
    "have collected (here, our 150 flower measurements) into two parts. One part of the\n",
    "data is used to build our machine learning model, and is called the training data or\n",
    "training set. The rest of the data will be used to assess how well the model works; this\n",
    "is called the test data, test set, or hold-out set.\n",
    "\n",
    "scikit-learn contains a function that shuffles the dataset and splits it for you: the\n",
    "[train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html#sklearn.model_selection.train_test_split) function. As default, this function extracts 75% of the rows in the data as the\n",
    "training set, together with the corresponding labels for this data. The remaining 25%\n",
    "of the data, together with the remaining labels, is declared as the test set. Deciding\n",
    "how much data you want to put into the training and the test set respectively is somewhat\n",
    "arbitrary, but using a test set containing 25% of the data is a good rule of thumb.\n",
    "\n",
    "In scikit-learn, data is usually denoted with a capital X, while labels are denoted by\n",
    "a lowercase y. This is inspired by the standard formulation f(x)=y in mathematics,\n",
    "where x is the input to a function and y is the output. Following more conventions\n",
    "from mathematics, we use a capital X because the data is a two-dimensional array (a\n",
    "matrix) and a lowercase y because the target is a one-dimensional array (a vector).\n",
    "Let’s call train_test_split on our data and assign the outputs using this nomenclature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before making the split, the train_test_split function shuffles the dataset using a\n",
    "pseudorandom number generator. If we just took the last 25% of the data as a test set,\n",
    "all the data points would have the label 2, as the data points are sorted by the label\n",
    "(see the output for iris['target'] shown earlier). Using a test set containing only\n",
    "one of the three classes would not tell us much about how well our model generalizes. This will result in situation that the training data and testing data have different distributions.\n",
    "So, we shuffle our data to make sure the test data contains data from all classes.\n",
    "\n",
    "To make sure that we will get the same output if we run the same function several\n",
    "times, we provide the pseudorandom number generator with a fixed seed using the\n",
    "random_state parameter. This will make the outcome deterministic, so this line will\n",
    "always have the same outcome. We will always fix the random_state in this way when\n",
    "using randomized procedures.\n",
    "\n",
    "The output of the train_test_split function is X_train, X_test, y_train, and\n",
    "y_test, which are all NumPy arrays. X_train contains 75% of the rows of the dataset,\n",
    "and X_test contains the remaining 25%:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train shape: (112, 4)\n",
      "y_train shape: (112,)\n",
      "X_test shape: (38, 4)\n",
      "y_test shape: (38,)\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(iris_data.data, iris_data.target, random_state=142)\n",
    "\n",
    "print(\"X_train shape: {}\".format(X_train.shape))\n",
    "print(\"y_train shape: {}\".format(y_train.shape))\n",
    "print(\"X_test shape: {}\".format(X_test.shape))\n",
    "print(\"y_test shape: {}\".format(y_test.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### K-Nearest Neighbours Classifier\n",
    "\n",
    "Now we can start building the actual machine learning model. There are many classification\n",
    "algorithms in scikit-learn that we could use. Here we will use a k-nearest\n",
    "neighbors classifier, which is easy to understand. Building this model only consists of\n",
    "storing the training set. To make a prediction for a new data point, the algorithm\n",
    "finds the point in the training set that is closest to the new point. Then it assigns the\n",
    "label of this training point to the new data point.\n",
    "\n",
    "All machine learning models in scikit-learn are implemented in their own classes,\n",
    "which are called Estimator classes. The k-nearest neighbors classification algorithm\n",
    "is implemented in the [KNeighborsClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier) class in the neighbors module. Before\n",
    "we can use the model, we need to instantiate the class into an object. This is when we\n",
    "will set any parameters of the model. The most important parameter of KNeighbors\n",
    "Classifier is the number of neighbors (i.e., $K$), which we will set to 1 for our first exploration.\n",
    "\n",
    "**Model Training**: To build the model on the training set, we call the 'fit' method of the knn object,\n",
    "which takes as arguments the NumPy array X_train containing the training data and\n",
    "the NumPy array y_train of the corresponding training labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(n_neighbors=1)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import the KNN classifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "# Build a KNN classifier model\n",
    "clf_knn = KNeighborsClassifier(n_neighbors=1)\n",
    "\n",
    "# Train the model with the training data\n",
    "clf_knn.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Prediction**: We can now make predictions using this model on new data for which we might not\n",
    "know the correct labels. Imagine we found an iris in the wild with a sepal length of\n",
    "5 cm, a sepal width of 2.9 cm, a petal length of 1 cm, and a petal width of 0.2 cm.\n",
    "What species of iris would this be? We can put this data into a NumPy array, again by\n",
    "calculating the shape—that is, the number of samples (1) multiplied by the number of\n",
    "features (4):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_new.shape: (1, 4)\n",
      "The predicted class is: \n",
      " [0]\n"
     ]
    }
   ],
   "source": [
    "# Produce the features of a testing data instance\n",
    "X_new = np.array([[5, 2.9, 1, 0.2]])\n",
    "print(\"X_new.shape: {}\".format(X_new.shape))\n",
    "\n",
    "# Predict the result label of X_new:\n",
    "y_new_pred = clf_knn.predict(X_new)\n",
    "print(\"The predicted class is: \\n\", y_new_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model predicts that this new iris belongs to the class 0, meaning its species is setosa. But how do we know whether we can trust our model? We don’t know the correct species of this sample, which is the whole point of building the model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Evaluating Model**: This is where the test set that we created earlier comes in. This data was not used to build the model, but we do know what the correct species is for each iris in the test set. So, we can use the trained model to predict these data instances and calculate the accuracy to evaluate how good the model is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1 \n",
    "Write code to calculate the accuracy score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy is: 0.8947\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "y_pred = clf_knn.predict(X_test)\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy is: %.4f\\n\" % accuracy)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### Parameter Tuning with Cross Validation (CV)\n",
    "\n",
    "In this section, we’ll explore a CV method that can be used to tune the hyperparameter $K$ using the above training and test data.\n",
    "\n",
    "Scikit-learn comes in handy with its [cross_val_score](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html#sklearn.model_selection.cross_val_score) method. We specifiy that we are performing 10 folds with the cv=KFold(n_splits=10, shuffle=True) parameter and that our scoring metric should be accuracy since we are in a classification setting. In each iteration, the training data take 90% of the total data while testing data takes 10%. The average on the accuracies reported from each iteration will make the testing accuracy more robust than just a single split of the data.\n",
    "\n",
    "**Manual tuning with cross validation**: Plot the misclassification error versus $K$. You need to figure out the possible values of $K$. If the number of possible values is too big, you can take some values with a certain step, e.g., $K$ = 1, 5, 10, ... with a step of 5. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEICAYAAABS0fM3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAsdklEQVR4nO3deXzU1b3/8dcnewJZSEICJCAQdgQEwpZq3bBVW4uKt1IRuoqIWGvrvfX23mu99i7drS1asGq1ytZat/rTVnHBBQUCKIosEkAIm0kISchCtvP7Yya5ISQwgUxmJvN+Ph48yPc73/nOJ4HMe77nfM855pxDRETCV0SgCxARkcBSEIiIhDkFgYhImFMQiIiEOQWBiEiYiwp0AR2Vnp7uBg4cGOgyRERCyoYNG4qdc73beizkgmDgwIHk5+cHugwRkZBiZp+295iahkREwpyCQEQkzCkIRETCXMj1EYhI91VXV0dhYSE1NTWBLiVkxcXFkZ2dTXR0tM/PURCISNAoLCwkMTGRgQMHYmaBLifkOOcoKSmhsLCQQYMG+fw8NQ2JSNCoqakhLS1NIXCGzIy0tLQOX1EpCEQkqCgEzs6Z/PwUBCIS0q5f8i7XL3k30GWENAWBiEgrzzzzDGbGtm3bAl1Kl1AQiEhIWry6gDUFxSfsW1NQzOLVBWd97uXLl3P++eezYsWKsz5XexoaGvx27o5SEIhISBqbnczCZZsoq64DPCGwcNkmxmYnn9V5jx07xjvvvMMjjzzSHAQNDQ3ceeedjBkzhrFjx/K73/0OgPXr15OXl8e4ceOYPHkyFRUVPPbYYyxcuLD5fF/+8pd54403AOjZsyd33303U6ZM4d133+Xee+9l0qRJnHvuucybN4+mFSN37tzJ9OnTGTduHBMmTKCgoIA5c+bw3HPPNZ939uzZPP/882f1vTbR7aMiEpT+829b+PhA+SmPyUiMZfuhCqIjjbmPrGNIRk/uX/UJ96/6pM3jR/VL4sdXjT7lOZ999lkuv/xyhg0bRmpqKhs3bmTt2rXs3r2bTZs2ERUVxZEjR6itreX6669n5cqVTJo0ifLycuLj40957srKSs4991zuvfdeTz2jRnH33XcDMGfOHF544QWuuuoqZs+ezV133cU111xDTU0NjY2NfOc73+G+++5jxowZlJWVsWbNGh5//PFTvp6vdEUgIiErOT6a6EijtsGRmRRLcrzvg6jas3z5cmbNmgXArFmzWL58OatWrWL+/PlERXk+O6emprJ9+3b69u3LpEmTAEhKSmp+vD2RkZHMnDmzefv1119nypQpjBkzhtdee40tW7ZQUVHB/v37ueaaawDPALGEhAQuvPBCdu7cyWeffcby5cuZOXPmaV/PV7oiEJGgdLpP7uBpDpr7yDqyUuKormvk9ulDyctJP+PXLCkp4bXXXuOjjz7CzGhoaMDMmDhx4km3ZTrn2rxVMyoqisbGxubtlvf0x8XFERkZ2bx/wYIF5Ofn079/f+655x5qamqam4faMmfOHJYuXcqKFSt49NFHz/j7bE1XBCISkpr6BIZk9CS7VwKLbhjPwmWbTupA7oinnnqKuXPn8umnn7Jnzx727dvHoEGDmDBhAosXL6a+vh6AI0eOMGLECA4cOMD69esBqKiooL6+noEDB/L+++/T2NjIvn37WLduXZuv1RQQ6enpHDt2jKeeegrwXFlkZ2fz7LPPAnD8+HGqqqoA+MY3vsFvfvMbAEaPPn1Q+kpBINIO3Z8e3DYXlrHohvHNzUF5OeksumE8mwvLzvicy5cvb26SaTJz5kwOHDjAgAEDGDt2LOPGjWPZsmXExMSwcuVKbrvtNsaNG8dll11GTU0Nn/vc5xg0aBBjxozhzjvvZMKECW2+VkpKCjfddBNjxozh6quvbm5iAnjiiSf47W9/y9ixY8nLy+PQoUMAZGZmMnLkSL75zW+e8ffYFjvVZUgwys3NdeG2ME3Tm9HKm6cFuJLwop9719u6dSsjR47s0HPC6d+pqqqKMWPGsHHjRpKT2787qq2fo5ltcM7ltnW8364IzOxRM/vMzD5q53Ezs9+a2U4z22xmbcemhBV9CpeOWnnztLAIgVWrVjFixAhuu+22U4bAmfBnZ/FjwCLgT+08fgUw1PtnCvB7798iItLK9OnT2bt3r1/O7bcrAufcm8CRUxwyA/iT83gPSDGzvv6qp6OfNPXJtG36uYi/hVpzdbA5k59fIDuLs4B9LbYLvftOYmbzzCzfzPKLioq6pLiO0hukyNmLi4ujpKREYXCGmtYjiIuL69DzAjmOoK25Utv813fOPQQ8BJ7OYn8WFY7CpbMtXL7PUJadnU1hYSHB+oEvFDStUNYRgQyCQqB/i+1s4ECAauly/nxT0huehKro6OgOrawlnSOQTUPPA3O9dw9NBcqccwcDWI+ISFjy2xWBmS0HLgLSzawQ+DEQDeCcWwy8CFwJ7ASqgM4dIdFFFq8uOGm2wzUFxWwuLGP+hTkBqkpExHd+CwLn3NdO87gDbvXX63eVpqlwMxI9E141DXtfdMP4QJcmIuKTbj/FREcXr+jI8Q2NjnPSerDw4iHsOFzBJ4crmP/EBn567RimDU7rlHpCne6mEgl+3X720Y5+Ym95fGJcFM9t2s+/P/cRN0wewH2v7GD/0WoKS6soLK3mUFkN9Y3/dxPTkSrPAhnznthAj5hIsnslkNUrnmzvn6yUBHrERLJg6Ub6JMaSnBBzynqCqdkpmGppEsqd4qFcu3Q/3T4ImiaimvPwWhoc3PCHtcRFRXDT4+3PV9TQ6NhWWQvA+j2lACx5cxdmkJkYR1aveCae06v5zb28po5f/GM7qQnRVNc1cs34LKIijcLSavaXVpO/5wjlNfUnvMbRqjoiI+Drj67j80N788G+MkqO1TYHR++esUHV7BRMtUjnUihJtw8C8IRBao8Yio7VMr5/ChPP6XXa5yxft5fK2gYuHZHBt84fRFZKPH1T4oiNijzhuKY3xKEZPUmOj+b26UOb3yBbzoteXlPH/tJqCks9VxS//Md2KmsbSE2MZuPeUl7d9tkJ542NiiCrVzxZKXF8dKCcpNgoFjy5kQdvnHBW862fqbycdH527RjmP7mBHrFR3PSnfL5/2TDOSetBQ6MjMuLEYSHBeAUhIm0LiyBYU1BMaVUdWSlxfHqkin++fPgp30zXFBTz2Jo9ZKXEsWnfUcxgYHqPNo9tmgq3aWm8llPhtnyNpLhokvpGM7JvEmsKijle39i8mMYDsycwNjvFGxRV3uYn79el1eCgzHtFsWT1LoqP1fKFUZnERUe2WZOvTvdmfaSylnW7S3hv1xHe21XC9sMVOEfz1c1PXtjKT17YSlSE0Sc5ztsElkBWSjw1dQ3c/MQGMnrGktbz1E1g3UUwfbIOplok+HX7IGi5eMWpPrGf6fFNn25brpGal5PebtCc7vzD+ySedPzcR9aR2iOG8po6Ptp/lNU7ikiMi+LLY/syc0I2E8/p1eZKSafTurnn7x8d5M6/bCYvJ40v3LeaHYePARAXHUHuOamMyUrm6U37yUiMpaq2gQUX5ZAUH90cWIWl1bz9STGHK2pomiGgoqaeguJK5j66jpnjs4mNiqC2vpGYqG5/n4JIyOj2QeDrJ/YzPd6f9bQZGks38aMrR7DtYAXPbjrA8nX7GJTeg2vHZ1FVV88FQ3ufdI6WzTHOOYqOHaewtLr5ymLl+n1ERMDa3Z45At/eWczEc3ox47wspg5OZUxWCvmfHmm3Cexrkwec8Jq19Y0cLPMEw61LN3C0up6kuChW5u9jZf4+4qMjmXhOL6YMSmVqThpjs5P54zt71JQkEiDdPgg6+om9o8f7s542Q2O2JzR+ff153Ht1PS9+eJC/bijkV6/sAOAPb+6md89YevWI5pf/2MYjb+9hWk4a7+ws9nxqP1pNbX3jCa9jQEMj5OWkcecXhzMmK5noyIjT19JOgMVERXBOWg/2H63m2PGG5iawxTdOwDlP4Ly3q8RT8yue/pAhGT25f9Un9O4ZQ2ZSXFg0JYkEi24fBKHsdKHRMzaKr+b256u5/dl3pIqnN+5n6dpPOVhew8HyGj4+WAHAB/uOkt0rnhF9E5k+KtN7t5OnPX//0Srm/WkDmUmxbDtUQU1dw0kh4EstrZ2qCeyer3jWWi2trGXt7iOs3V3C2l1HqK5rYG9pNXtLq5n7yDq+Mq4fADV1DSf0h/i7I1od3RJuFASdJNCdcv1TE7h9+lC+e+kQJv33KoqP1XLjlAH86EsjSYhp+595TUExd/5ls8/9IR3hyxVErx4xXH5uHy4/tw8AZVV1XPzL1zlSVUevHtE8+/5+nt60n5jICM7rn8LUwalMGZzGiMxEv97KGoy3yqrzV/xJQRAg/vqFfndXCUe9d0i9+NEhrhzbt903dX/2h5xJE9uWg2WU19Q3NyUtmZNLhHmaktbuKmHR6zv57Ws7iY40BqX3YMfhCnolRHPLkxtPGV4d/YQ/bXAa/3PNudzqDYPOCkeRYKUg6Eb8fcfTmfA18E5V+4+u9CzCXVFTR/6npaz13s7a6KCk0jOa++uPrqNPUpx3QF6Ct+nL83VGYiy3Lt1IZlIcyfHRvPNJMbcu28j3LhvG8x8caB4p3vL23Zo6Tz/KwbIaZk8ZoBCQbi1sgqCjn8BD8RLc33c8+ZMvtSfGRXPx8AwuHp7RfFttSnw01XUNfGF0HyLMKCytYt3uIxwqr6Gh8cQ1jEqr6og0mP3IWgDueX5L82O9EqLJ7pXA0IxELh6eQV1DI0+89ykAy9buZWBaAjd9Xv0D0j2FTRCEg674hO8vHandlyuf+oZGDpXXtPikX82S1TupqmtkwoAUrhmfdcLVQ4/YqJPOPywzkbjoCIqPHee/X9xGSWUdd10xws8/CZGup1E9EnKarh6S46OBE68emkRFRpDdK4Gpg9OYOTGbSYN6UdvgyEqJY09JFTkZPblkRCbDMhNPCIHW54+NiuTF2z/P6H5JLF5dwH2v7NB6utLtKAgk5My/MOekK4W8nPR2b+1seQWR3SuBRTeMZ+GyTSdNB97e+ZPionlmwee4bmI297/6CT/48wcnjcUQCWUKAumQlTdPC7n+E1+uIE4nJiqCX1w3lh9cNoynN+1n7qNrKfNOOx5Mwm29C+kcCgLp9jp6BdEeM+O2S4dy3/Xj2PBpKTMXr2HfkarOLPUkHX1jbxoDUVZd13zswmWbTrp9VqQldRaLdNA147PpmxzPvD/lc82D7/Dw1ydxXv8Uv7xWe4PbfjtrvGeW2iOtZqs9Wk10pLHtUAXJcVEaAyE+URCEAH83xYRaU08wmDo4jacX5PHNx9Zz3e/X8N1Lh57weGdNSTF5YCrfv2wodz+3hdioCOY+so5ePaL5xh/XnbA6HkDvxFiye8UzeVAaf//oIGU19Vw3MVshIKelIBA5Q0MyEnlmwee4fsm7/PqVHWQkxjIwLeGslh+tb2jkw/1lzaOp1+8p5dhxz/oP1XWN9E2OY/Kg1ObV8ZqWQe2XEt88H9OagmJe+vAgkQZPbyzki6P7cNmoTP//QCRkKQhEzkJ6z1heuO0C5jy6lvw9pXxWcZwbH17LsMxEntpQyHu7jnjerL2T/I3ul3RCU89bnxRx69KNXDGmL3MfXceGPUeorG0AIKd3D75yXj9Se8Sw+I0CMpNiqa5r5PpJ/X2a7C/SjK2HylmwdAOPf3MyeUN0ZSBtUxB0Q+HS1BMs32d8TCQr500j979eobSqjqGZiSTFRfNeQQmHyvfTsgUnwiAlIZrthyqIjjTmPLIOgJXr9zEssyfXTshmyuBUJg9KJSMxrsPThrQeof0vl4/gpy9t4/F39ygIpF0KApFOsHZ3CRXeCfOKKo7z46tGkZeTTm19I4fKaig8WuXt0PWMdP7bB/upbXCMzUrmlotymDwolbSesSedt6PThrQeoT3vgsG8s7OYN7YXse1QOSP6JPnxpyChSrePipylUw1Yi4mKYEBaAnk56Xw1tz/fv2wYMydm0eggKyWOwqPVJCdEtxkCcPa3vkZEGL/+6nkkxkWzcNkmqmrrz/r7le5HQSByljoyYK2jo5w7Q+/EWH5z/XkUFB3j3r99fNbnu37Ju83rI0j3oCCQoBKKI5c78qm9M0Y5n4nzh6Zzy4U5rFi/j+c/OODX15LQoyAQ6UKdNcr5TNxx2TAmDEjhR09/yN4S/46IltCiIBAJE9GREdw/azwRBrct36iJ86SZgkAkjPRPTeBnM8fyQWEZv3x5e6DLkSCh20dFwswVY/py49QBPPTmLqblpAW6HAkCCgIJG6HWCd1SZ9f+718aRf6eUn7w5w84JzWBmCg1DoQz/euLhKG46EguGJrOsZp6CoqONa+6prULwpOCQKQb8uU23ItHZBAdaZTX1HOgrEZrF4QxvwaBmV1uZtvNbKeZ3dXG48lm9jcz+8DMtpjZN/1Zj0hHhOKYho7Iy0nnD3NzMaCwtJpbntyotQvClN+CwMwigQeAK4BRwNfMbFSrw24FPnbOjQMuAn5lZjH+qklETpQ3JJ2MRM/0FmOykhUCYcqfVwSTgZ3OuV3OuVpgBTCj1TEOSDQzA3oCRwBNhiLSRdYUFFNSWUtUhPHerhK/TnXhK01h0fX8GQRZwL4W24XefS0tAkYCB4APgdudcxrlItIFWs57lJEYS6Nz3Lp0Y1CEgXQtfwaBtbHPtdr+IvA+0A84D1hkZifNk2tm88ws38zyi4qKOrtOkbDUct6jlIRoGh3MnjLA7/MeSfDxZxAUAv1bbGfj+eTf0jeBp53HTmA3MKL1iZxzDznncp1zub179/ZbwSJnI9Q6l1vOe9QzNorUHjHsK63u9HmP1NQT/PwZBOuBoWY2yNsBPAt4vtUxe4FLAcwsExgO7PJjTSLSBjPj4uEZvLG9iPoGtc6GG78FgXOuHlgI/APYCvzZObfFzOab2XzvYT8B8szsQ+BV4IfOOTVQigTA9JEZlFXXkf9paaBLkS7m1ykmnHMvAi+22re4xdcHgC/4swYR8c35Q9OJjjRe2/YZUwdrDqJwopHFIgJAYlw0UwensWrr4UCXIl1MQSAizS4ZkcGuokp2F1cGuhTpQgoCEWk2fWQmAK/qqiCsaBpqkQAI1ttM+6cmMCyzJ69t+4zvXDA40OVIF9EVgYic4JIRmazbfYTymrpAlyJdREEgIieYPjKD+kbH6u0axR8uFAQicoLxA3rRKyGa17Z9FuhSpIsoCETkBJERnlHGr2//TKOMw4SCQEROcsnIDI5W1bFp39FAlyJdQEEgIif5/LDeREWYBpeFCQWBiJwkKS6aKYNTeW2r+gnCgYJARNp0yYhMPvnsGHtLqgJdSrs0xXXnUBCISJumj8wAUPNQGFAQiEibzknrQU7vHrqNNAwoCESkXdNHZrJ2dwkVGmXcrSkIRKRdl47MpK7B8dYnWi+qO1MQiEi7JgxIITk+Wv0E3ZyCQETaFRUZwcXDe/PG9iIaGl2gyxE/OW0QmNmXzUyBIRKmLhmZyZHKWt7fp7WMuytf3uBnAZ+Y2c/NbKS/CxKR4HLhsN5ERhirNLis2zptEDjnbgTGAwXAH83sXTObZ2aJfq9ORAIuOT6aSQN7aZRxN+ZTk49zrhz4K7AC6AtcA2w0s9v8WJuIBInpIzPZfriCfUeCd5SxnDlf+giuMrNngNeAaGCyc+4KYBxwp5/rE5EgcKl3LWMNLuuefFmz+J+A+5xzb7bc6ZyrMrNv+acsEQkmg9J7MDi9h24j7aZ8aRr6MbCuacPM4s1sIIBz7lU/1SUiQebSkRms3XVEt5F2Q74EwV+AlssUNXj3iUg3sPLmaay8edppj7tkRCa1DY2UVWu6ie7GlyCIcs7VNm14v47xX0kiEoxyB/YiKS6K0qra0x8sIcWXICgys680bZjZDEATj4iEmejICC4a7lnC0jk1D7UWymsj+BIE84EfmdleM9sH/BC42b9liUiwWby6gAGp8dQ3Oo4dbwBgTUExi1cXBLgyOVu+DCgrcM5NBUYBo5xzec65nf4vTUSCydjsZJau3QvA0apa1hQUs3DZJsZmJwe4Mjlbvtw+ipl9CRgNxJkZAM65e/1Yl4gEmbycdB6YPYHZf1jL4fIaFjy5kQdvnEBeTnqgS5Oz5MuAssXA9cBtgOEZV3COn+sSkSCUl5NOao8YGhzUNzaS1iM20CVJJ/CljyDPOTcXKHXO/ScwDejv37JEJBitKSimrLqO3j1jqKxt4OoH3uZtLVoT8nwJghrv31Vm1g+oAwb5ryQRCUZNfQJDMnoyuHdP7p81ntp6x9xH1/Ln/H2BLk/Ogi9B8DczSwF+AWwE9gDL/ViTiAShzYVlLLphPMnx0QB8ZVw/lsydyIDUBP7lqc386uXtuq00RJ0yCLwL0rzqnDvqnPsrnr6BEc65u305uZldbmbbzWynmd3VzjEXmdn7ZrbFzFZ3+DsQkS4x/8KckzqGp4/M5JXvX8hXc7P53Ws7uWPl+xyvbwhQhXKmTnnXkHOu0cx+hadfAOfcceC4Lyc2s0jgAeAyoBBYb2bPO+c+bnFMCvAgcLlzbq+ZZZzRdyEiARMdGcHPZo5lQGoCv3x5BwfLanhoTi7JCdGBLk185EvT0MtmNtOa7hv13WRgp3Nul3daihXAjFbH3AA87ZzbC+Cc0xy3IiHIzFh4yVDun3Uem/Ye5drfv9Mt1i4I5dHCHeFLEHwfzyRzx82s3MwqzKzch+dlAS17kAq9+1oaBvQyszfMbIOZzW3rRN4V0fLNLL+oqMiHlxaRQJhxXhZPfHsyxcdq+eJv3uRP7+454XGNRA5OvowsTnTORTjnYpxzSd7tJB/O3dYVROuepChgIvAl4IvAf5jZsDZqeMg5l+ucy+3du7cPLy0igTJlcBpPL8ijZ2wUdz+3hcJSz5WBRiIHr9OOLDazz7e1v/VCNW0o5MTxBtnAgTaOKXbOVQKVZvYmnpXPdpyuLhEJXjm9e/Li7Rdw/ZJ3KSiqpLSqjvlPbGDxjRM1EjkI+dI09M8t/vwH8DfgHh+etx4YamaDzCwGmAU83+qY54ALzCzKzBKAKcBWH2sXkSCW3jOW//fdC0iIiaSqtoHymnp+/PwWFq8u4HB5zelPIF3mtFcEzrmrWm6bWX/g5z48r97MFgL/ACKBR51zW8xsvvfxxc65rWb2d2AznsVvHnbOfXQG34eIBKGNe0uprW+kb1IsZTX1mMFPX9rGz/++jfOH9mbmhCy+OLoPcdGRLF5dcFKz0ZqCYjYXljH/wpxOqaep49eXhXjCiU+TzrVSCJzry4HOuReBF1vtW9xq+xd4BquJSDfSciRycnw0t08fysJlm/j1V8exu7iSpzfu5/YV75MYG8WXxvZleJ9EFi7bREZiLMnx0c3PX3TD+EB/K92eL30Ev+P/OnkjgPOAD/xYk4h0A00jke9f9QngmbBu0Q3j2VxYxg++MJw7pg/jvV0lPLWxkOc/OEDV+gYyk2LZcbiCPklxzSHQXfsUgunqxJcrgvwWX9cDy51z7/ipHhHpJpqac5qCADxh0PTGHhFh5A1JJ29IOj+ZUc9LHx3irxsKOVx+nANlNdx2yZBuGwLBxpcgeAqocc41gGfEsJklOOdCf7SIiASFHrFRXDcxm34pcazdXUKjg4ff2s20nDSFQRfw5a6hV4H4FtvxwCr/lCMi4aqpT2BYRk9SE6KpqWvg5ic2sKZA01z7my9BEOecO9a04f06wX8liUg4ap7dNCGGwb17MrxPIg0NjazertkE/M2XIKg0swlNG2Y2Eaj2X0kiEo5azm4aGWH8YW4uMdGRvLbtM44drw9wdd2bL0HwPeAvZvaWmb0FrAQW+rUqEQl7/VMTeOCGCRQUHeP7K9+nsVFrHfiLL3MNrQdGALcAC4CRzrkN/i5MRORzQ9L50ZUjefnjw/zutZ2BLqfb8mXx+luBHs65j5xzHwI9zWyB/0sTEYFvnz+Ia8dncd+qHby85VCgy+mWfGkausk5d7RpwzlXCtzkt4pERFowM/7n2jGMzU7mjpXv88nhikCX1O34EgQRLRel8a48FuO/kkREThQXHcmSOROJj4lk3hMbKKuuC3RJXc6fi+T4EgT/AP5sZpea2SV4Fq5/yS/ViIi0o29yPL+/cSKFpVV8d/kmGtR53Gl8CYIf4hlUdgtwK56ZQuNP+QwRET+YNDCVe74ymtU7ivjly9sDXU634ctdQ43Ae8AuIBe4FK0ZICIBMnvKOZzXP4Xfv1FAybHjzfu1DOaZazcIzGyYmd1tZluBRXjXH3bOXeycW9RVBYqItPb9y4YRFWEUFFVSebxey2CepVNdEWzD8+n/Kufc+c653wENXVOWiEj7Pj+sN7/92nk4YOvBcm5durFbT1ntb6cKgpnAIeB1M/uDmV1K2wvSi4h0uSvH9COtRwwNDsZkJSsEzkK7QeCce8Y5dz2eUcVvAHcAmWb2ezP7QhfVJyLSpjUFxZRV1xETaby1s5g3tn8W6JJCli+dxZXOuaXOuS8D2cD7wF3+LkxEpD0tl8HM6d0T52DB0o2asvoM+XL7aDPn3BHn3BLn3CX+KkhE5HSap6yOjyYpPpq8nDSiIiLYsKc00KWFpA4FgYhIMGg5ZTXAHZcNo7ymjrjoyABWFboUBCIS8iYNTOX8IeksXl1AVa3WLugoBYGIdAt3XDaUkspannj300CXEnIUBCLSLUw8J5XPD+vNkjd3UdmFK5otXl1wUid1qI1yVhCISLdxx/ShHKms5fF393TZa47NTmbhsk3NM6KG4ihnBYGIdBvjB/TiouG9eejNXV22znFeTjq//qdx7DhcwQf7jrLgydAb5awgEJFu5XvTh3G0qo7H1+zx+2tV1dbzhzd3cedTm2l0UFPfiMMxMK2H31+7MykIRKRbOa9/CpeMyOChN3dRXuOfBWwqj9ezZHUBF/zsdf77xa1kJsUSaUZ6zxjKq+u5+oF32H+02i+v7Q8KAhHpdu6YPoyy6joee2dPp5732PF6HnxjJxf8/HX+96VtjOqXxD1XjeJgWQ1DMz2jnO+dMZqiiuPMWPQ2+45Uderr+4uCQES6nTHZyUwfmcnDb+06o2UtW98JVFFTxw+f2kzuf73Cz/++nTFZyfz1ljye+PYUauobm0c5A8yZNpCfXD2aipp6Zj30XkiEgYJARLql700fSnlNPX98Z3eHn9t0J1Bp1XH2l1Yz9X9eZWX+Pkb0SeSZBXk8/q3JTDynF3DyKGeAG6cO5Kn5eRw77gmDvSXBHQYKAhHpls7NSuYLozJ55O3dlFV17Kpg6qA0rp2QxY7DlRQeraa2oZH/uno0z956PuMH9PLpHGOyk1n6nSneMHiXT0sqz+Tb6BIKAhHptr43fRgVNfU88vYun5+z7VA5/7TkXR5+azcxkZ4lWG65MIcbpw7s8Oufm5XMspumUFXXwKyH3mNPcXCGgYJARLqtUf2SuOLcPjz6zh6OVtWe8tiq2nr+98WtfOm3b7O7uJL5F+bQ6CArJY4n1+494ymuR/dLZtl3plLjDYPdQRgGfg0CM7vczLab2U4za3cNAzObZGYNZnadP+sRkfBz+/ShHDtez8Nvtd9XsOrjw1z26zdZ8uYu/mliNj+9dgx/zt/HkIyeZPdKYNEN41m4bNMZh8GofkksnzeV2oZGrl/yLgVFx8702/ELvwWBmUUCDwBXAKOAr5nZqHaO+xnwD3/VIiKdZ+XN01h587RAl+GzEX2S+NKYvvzxnd3UNTSe8NiBo9Xc/EQ+3/lTPj1iI/nL/Gn8dOZYdhVXnnAnUF5OOotuGM/mwrKzqmP5TVM5VlPPzAfXUF37f0vAB3puIn9eEUwGdjrndjnnaoEVwIw2jrsN+CugdeZExC/6JsdRWdvAwbIaAOobGvn3Zz7kwl+8zuodRfzw8hG8cNsFTBqYCrR9J1BeTjrzL8w5qzqG90nknhmjKauuY8vBMqpq64NibiJ/BkEWsK/FdqF3XzMzywKuARb7sQ4RCXOXjMwgJjKCQ2U1HK2q5dJfrebJtXsZ3S+JV+64kFsuyiEmqmu6TL+a259fXDeWhkb4cH85Nz68lnHZyRwur+FAG6ORu2J206hOO9PJrI19rtX2b4AfOucazNo63Hsis3nAPIABAwZ0Vn0iEibyctL5n2vO5c6nNrP98DHM4HuXDuX26UM51XuPv1yX259fvLydw+XHOSctgQ2flvL69iIABqQmMGVQKlMGpzF1cGrzmIaMxFiS46ObryAW3TC+0+rxZxAUAv1bbGcDB1odkwus8P5DpANXmlm9c+7Zlgc55x4CHgLIzc1tHSYiIqd1XW5/7n3hY8pr6pl3wWC+d9mwgNWypqCYkmO1ZKXEUVZdz4OzJ9KrRzTv7TrC2l0lvLL1MH/ZUAhAVko8o/sl8c7OYnonxjaHQGfOburPIFgPDDWzQcB+YBZwQ8sDnHODmr42s8eAF1qHgIhIZ1hTUExVbQNZKXH8ZUMhFw7vHZCpops+0Q/J6ElyfDS3Tx/a/Ob+7fMH8e3zB9HY6Nh+uIK1u0p4b9cR1u05QqODw+XH+e4lQzq9br81ijnn6oGFeO4G2gr82Tm3xczmm9l8f72uiEhrLd98O+N20LOxubDstHckRUQYI/sm8Y3PDWLxnIn8btZ4IiOMzMTYsxrT0B5/XhHgnHsReLHVvjY7hp1z3/BnLSISGMFwq2nTm+/9qz4BTnzz7eqrgqY7j5pqaaqnvTrWFBRz24pNDG3jCqKzatfIYhHp9vx1O2hX8OUK4mwpCEREglhXhJiCQEQkzCkIRETCnIJARCTMKQhERMKcgkBEJMwpCEREwpyCQEQkzCkIRETCnIJARCTM+XWuIRGRjgqGuYnCja4IRETCnIJARCTMKQhERMKcgkBEJMwpCEREwpyCQEQkzCkIRETCnIJARCTMKQhERMKcgkBEJMxpigkRCVmajqJz6IpARCTMKQhERMKcgkBEJMypj0BEpBOEcn+FgkBEpB2h/ObeEQoCEZEQ4M9QUh+BiEiYUxCIiIQ5BYGISJhTH4GISAAEU0e0rghERMKcgkBEJMz5NQjM7HIz225mO83srjYen21mm71/1pjZOH/WIyIiJ/NbEJhZJPAAcAUwCviamY1qddhu4ELn3FjgJ8BD/qpHRETa5s/O4snATufcLgAzWwHMAD5uOsA5t6bF8e8B2X6sR0TCXDB10AYTfzYNZQH7WmwXeve159vAS209YGbzzCzfzPKLioo6sUQREfFnEFgb+1ybB5pdjCcIftjW4865h5xzuc653N69e3diiSIi4s+moUKgf4vtbOBA64PMbCzwMHCFc67Ej/WIiEgb/HlFsB4YamaDzCwGmAU83/IAMxsAPA3Mcc7t8GMtIiLSDr9dETjn6s1sIfAPIBJ41Dm3xczmex9fDNwNpAEPmhlAvXMu1181iYjIycy5Npvtg1Zubq7Lz88PdBkiIiHFzDa090FbI4tFRMKcgkBEJMwpCEREwpyCQEQkzCkIRETCnIJARCTMKQhERMKcgkBEJMwpCEREwpyCQEQkzCkIRETCnIJARCTMKQhERMKcgkBEJMwpCEREwpyCQEQkzIXcwjRmVgR82sGnpQPFfiinK6j2wAjV2kO1blDt/naOc653Ww+EXBCcCTPLD9UlMFV7YIRq7aFaN6j2QFLTkIhImFMQiIiEuXAJgocCXcBZUO2BEaq1h2rdoNoDJiz6CEREpH3hckUgIiLtUBCIiIS5bh8EZna5mW03s51mdleg62mPmfU3s9fNbKuZbTGz2737U83sFTP7xPt3r0DX2h4zizSzTWb2gnc7JGo3sxQze8rMtnl//tNCqPY7vP9fPjKz5WYWF6y1m9mjZvaZmX3UYl+7tZrZv3p/b7eb2RcDU3VzLW3V/gvv/5nNZvaMmaW0eCxoavdFtw4CM4sEHgCuAEYBXzOzUYGtql31wA+ccyOBqcCt3lrvAl51zg0FXvVuB6vbga0ttkOl9vuBvzvnRgDj8HwPQV+7mWUB3wVynXPnApHALIK39seAy1vta7NW7//9WcBo73Me9P4+B8pjnFz7K8C5zrmxwA7gXyEoaz+tbh0EwGRgp3Nul3OuFlgBzAhwTW1yzh10zm30fl2B580oC0+9j3sPexy4OiAFnoaZZQNfAh5usTvoazezJODzwCMAzrla59xRQqB2rygg3syigATgAEFau3PuTeBIq93t1ToDWOGcO+6c2w3sxPP7HBBt1e6ce9k5V+/dfA/I9n4dVLX7orsHQRawr8V2oXdfUDOzgcB4YC2Q6Zw7CJ6wADICWNqp/Ab4F6Cxxb5QqH0wUAT80dus9bCZ9SAEanfO7Qd+CewFDgJlzrmXCYHaW2iv1lD73f0W8JL361CrvdsHgbWxL6jvlzWznsBfge8558oDXY8vzOzLwGfOuQ2BruUMRAETgN8758YDlQRPU8opedvTZwCDgH5ADzO7MbBVdZqQ+d01s3/D07S7tGlXG4cFZe1NunsQFAL9W2xn47l0DkpmFo0nBJY655727j5sZn29j/cFPgtUfafwOeArZrYHT/PbJWb2JKFReyFQ6Jxb691+Ck8whELt04Hdzrki51wd8DSQR2jU3qS9WkPid9fMvg58GZjt/m9QVkjU3lJ3D4L1wFAzG2RmMXg6cJ4PcE1tMjPD00691Tn36xYPPQ983fv114Hnurq203HO/atzLts5NxDPz/g159yNhEbth4B9Zjbcu+tS4GNCoHY8TUJTzSzB+//nUjx9S6FQe5P2an0emGVmsWY2CBgKrAtAfe0ys8uBHwJfcc5VtXgo6Gs/iXOuW/8BrsTTo18A/Fug6zlFnefjuXzcDLzv/XMlkIbnbopPvH+nBrrW03wfFwEveL8OidqB84B878/+WaBXCNX+n8A24CPgCSA2WGsHluPpy6jD86n526eqFfg37+/tduCKIKx9J56+gKbf18XBWLsvfzTFhIhImOvuTUMiInIaCgIRkTCnIBARCXMKAhGRMKcgEBEJcwoCEZEwpyAQEQlzCgKRs2RmN5vZYu/X0Wb2hJk97p0yRCToKQhEzt5YYLN3SuuXgL3Oua87z/w/IkFPQSBy9sbgmav+DeAvzrl/C2w5Ih2jKSZEzpKZleKZhvhbzrm/BboekY6KCnQBIqHMzPoDx/BMmtY3wOWInBEFgcjZGQt8ANwEvGdm651zmwJck0iHqI9A5OyMAT50nmUWvwOsNLPkANck0iEKApGzMwb4EMA59wrwZ+DRgFYk0kHqLBYRCXO6IhARCXMKAhGRMKcgEBEJcwoCEZEwpyAQEQlzCgIRkTCnIBARCXP/Hz53eVaX/iSaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score, KFold\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "cv_scores = []\n",
    "cv_scores_std = []\n",
    "k_range = range(1, 135, 5)\n",
    "for i in k_range:\n",
    "    clf = KNeighborsClassifier(n_neighbors = i)\n",
    "    scores = cross_val_score(clf, iris_data.data, iris_data.target, scoring='accuracy', cv=KFold(n_splits=10, shuffle=True))\n",
    "    cv_scores.append(scores.mean())\n",
    "    cv_scores_std.append(scores.std())\n",
    "\n",
    "# Plot the relationship\n",
    "plt.errorbar(k_range, cv_scores, yerr=cv_scores_std, marker='x', label='Accuracy')\n",
    "plt.ylim([0.1, 1.1])\n",
    "plt.xlabel('$K$')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be seen that the accuracy first goes up when $K$ increases. It peeks around 15. Then, it keeps going down. Particularly, the performance (measured by the score mean) and its robustness/stableness (measured by the score std) drop substantially around K=85. One possible reason is that when K is bigger than 85, the model suffers from the underfitting issue severely. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Automated Parameter Tuning**: Use the [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV) method to accomplish automatic model selection. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2\n",
    "Check against the figure plotted above to see if the selected hyperparameter $K$ can lead to the highest misclassification accuracy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best K value:  21\n",
      "The accuracy: 0.9800\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "parameter_grid = {'n_neighbors': range(1, 135, 5)}\n",
    "knn_clf = KNeighborsClassifier()\n",
    "gs_knn = GridSearchCV(knn_clf, parameter_grid, scoring='accuracy', cv=KFold(n_splits=10, shuffle=True))\n",
    "gs_knn.fit(iris_data.data, iris_data.target)\n",
    "print('Best K value: ', gs_knn.best_params_['n_neighbors'])\n",
    "print('The accuracy: %.4f\\n' % gs_knn.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3\n",
    "It can be seen that GridSearchCV can help us to the automated hyperparameter tuning. Actually, it also store the intermediate results during the search procrss. The attribute 'cv_results_' of GridSearchCV contains much such informaiton. For example, this attribute contains the 'mean_test_score' and 'std_test_score' for the cross validation. Make use of this information to produce a plot similar to what we did in the manual way. Please check if the two plots comply with each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEICAYAAABS0fM3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAmO0lEQVR4nO3deXxU9b3/8dcnC0nYDGETEjShRRYJlDWKWluXoq0KFnvFWqytVanV29v+QPHaeq22t7TeXvtzqeh1rVWgIiD6c2kVrVdlB9kELDsJIPuekO37+2MmMctMMhNyMjM57+fjkYeZc74585lI5j3n+z3f7zHnHCIi4l9JsS5ARERiS0EgIuJzCgIREZ9TEIiI+JyCQETE51JiXUC0unTp4nJzc2NdhohIQlm2bNk+51zXUPsSLghyc3NZunRprMsQEUkoZrYt3D51DYmI+JyCQETE5xQEIiI+l3BjBCLSepWVlVFYWEhJSUmsS0lY6enp5OTkkJqaGvHPKAhEJG4UFhbSoUMHcnNzMbNYl5NwnHPs37+fwsJC8vLyIv45dQ2JSNwoKSmhc+fOCoEmMjM6d+4c9RmVgkBE4opC4NQ05fenIBCRhHbtEwu49okFsS4joSkIRETqmDNnDmbG+vXrY11Ki1AQiEjCmruiiBXbD7FoywHOmzqfuSuKmuW406dP5/zzz2fGjBnNcrxQKioqPDt2tBQEIpKQ5q4o4u7ZqymtqASg6FAxd89efcphcOzYMT766COefvrp6iCoqKhg0qRJ5OfnM2jQIB555BEAlixZwqhRoxg8eDAjR47k6NGjPPfcc9x+++3Vx7viiit4//33AWjfvj333nsvBQUFLFiwgPvvv58RI0YwcOBAbrnlFqruGLlx40YuueQSBg8ezNChQ9m0aRMTJkzg1VdfrT7u9ddfz7x5807ptVbR5aMiEpd+9dpaPt15JOz+FdsPVYdAleKyCu6ctYrpi7eH/JkBPTvyH1ee3eDzzp07l8suu4yzzjqLrKwsli9fzqJFi9iyZQsrVqwgJSWFAwcOUFpayrXXXsvMmTMZMWIER44cISMjo8FjHz9+nIEDB3L//fcH6hkwgHvvvReACRMm8Prrr3PllVdy/fXXM2XKFK6++mpKSkqorKzkRz/6EQ899BBjxozh8OHDfPzxxzz//PMNPl+kdEYgIgmpbgg0tj1S06dPZ/z48QCMHz+e6dOn88477zBx4kRSUgKfnbOystiwYQM9evRgxIgRAHTs2LF6fzjJycmMGzeu+vF7771HQUEB+fn5zJ8/n7Vr13L06FGKioq4+uqrgcAEsbZt23LhhReyceNG9uzZw/Tp0xk3blyjzxcpnRGISFxq7JP7eVPnU3SouN727MwMZt56bpOec//+/cyfP581a9ZgZlRUVGBmDBs2rN5lmc65kJdqpqSkUFn5RRjVvKY/PT2d5OTk6u233XYbS5cupVevXtx3332UlJRUdw+FMmHCBF588UVmzJjBM88806TXGIrOCEQkIU0e3ZeM1ORa2zJSk5k8um+Tjzlr1ixuuOEGtm3bxtatW9mxYwd5eXkMHTqUadOmUV5eDsCBAwfo168fO3fuZMmSJQAcPXqU8vJycnNz+eSTT6isrGTHjh0sXrw45HNVBUSXLl04duwYs2bNAgJnFjk5OcydOxeAkydPcuLECQBuvPFG/vjHPwJw9tkNB2U0FAQikpDGDsnmt9/Op01y4G0sOzOD3347n7FDspt8zOnTp1d3yVQZN24cO3fu5IwzzmDQoEEMHjyYl156iTZt2jBz5kzuuOMOBg8ezKWXXkpJSQnnnXceeXl55OfnM2nSJIYOHRryuTIzM7n55pvJz89n7Nix1V1MAC+88AIPP/wwgwYNYtSoUezevRuA7t27079/f37wgx80+TWGYg2dhsSj4cOHO92YRqR1WrduHf3794/qZ6omkzW1OyiRnDhxgvz8fJYvX85pp50Wtl2o36OZLXPODQ/V3rMzAjN7xsz2mNmaMPvNzB42s41mtsrMQsemiEgDZt56ri9C4J133qFfv37ccccdDYZAU3g5WPwc8Cjw5zD7Lwf6BL8KgMeD/40L8fQpI55qEZHYuOSSS9i+PfRlsafKszMC59wHwIEGmowB/uwCFgKZZtbDq3q8Xo9E6520vGh/5/p/lBgSrbs63jTl9xfLweJsYEeNx4XBbfWY2S1mttTMlu7du7dFivOa3pRE6ktPT2f//v0Kgyaquh9Benp6VD8Xy3kEodZKDfl/3zn3JPAkBAaLvSwq3lStpVJaUcl5U+czeXTfRq+KiLYrKZr2Xh67Ke2ldcnJyaGwsJDW8oEvFqruUBaNWAZBIdCrxuMcYGeMaolL4dZSAU7pEjmJjEKp5aWmpkZ1Zy1pHrHsGpoH3BC8eugc4LBzblcM62kyr1ZAfPDtDRSX1V6hsLisggff3tAsxxcRAW8vH50OLAD6mlmhmd1kZhPNbGKwyRvAZmAj8D/AbV7VEq1o3tibsgJiQ8d3zrF57zGmL94ecvo8wM4w2yVxaIxI4olnXUPOuesa2e+An3j1/E3VUHfMmK/05HBxGYUHiyk8eILCg8U89M5nIT+13zdvLZ3atSE7M4OcThmkB6fChzr+Xa+s4sON+zhZXsmizfvZc/QkAEkGlSFGRDpmpFJR6UhO0i39ROTUadG5OsJ1x0x6eSX3zFnN8dLIbiZxqLiM7z/zxRojXdqnkd0pg892H6G4rPbqiCfLK5m1rJDuHdM4p3dnzundmYLeWazacYh/n7OmVj1JBoeLy/j24x/zm7EDGZjdvBNLRMR/FAQ17DxUHLY7przSMWFEr+An/LbkdMqgV6e2fPPhDyg6VFKv/ekd03jku0MDZw4HAsctPFhcLwSqGLDw7otrrWb4pa7tMTPunLWK0opKsjMzmPSNs0hKMh54/VOuevRDbhyVx8+/cRbt0/S/UppGg+Li+3ePE6XlvLVmN68sL+TjTfvDtsvOzAi5LO7k0f24e/bqWp/aM1KTmXJ5f0bkZjEiN6tW+3BL5/bMzAi5pO3YIdnVN9mo+Yf6tbO68fu31/Psx1t4Y/Uu7rtqAKPPPj3kMUREGuKL1UfrDs7OWVbIgk37mfTySkb8+h1+/teVbD9wgp9e3Idffqt/VEvbRrsCYnMtnXta21R+c3U+r/x4FJltU5n4l+X86PmlPP3h5qiuYIp2YNyLq6NaCy8HgDW4LF5q9WcEoQZnf/7yShzQPi2Fbw3qwbihOYzIzSIpOPjauX1are6YxiZxhfvUHq4tENXxGzL0jE68fsf5PPvRVn7/9nreXb+nel/RoWKmzF5FSVkFVwzuWe9nX1+5k/teW1vrdxOufai2jc1paMpkOBFpea0+CEIN/jqgU9tUPp5yMRltkuv9TDRv7E0R7fEba5OSnMTNX+3NUx9u5vMjJ2vtKymrZMrs1UwJvmk3Jpr2xWUVTJm9ik93Ham+OiqnU1uyO2XwzqefazKcSIJo9UEQ7pr7QyfKQoZAIttTJwRquueb9dd4/80b6yJuH65tSVklz3+8lZPltQfBzaDucjFVk+GaIwh0tiHSfFp9EPTMzAg7ONuc4uGKi3CvNTszg5u/2rve9uc+3hpx+4bafnjX19l3rLR6bkXhwWJ+99b6kDUWHQrsK8jLYnhuVq2rnSJ9c9fSGyLNq9UHweTRfUNe1XMq9zWNV9G+1mjaN9TWzOjaIY2uHdIYckYnAP6ycFvI4EhNNv7ng808/v4mkpOMgdmncU7vLFyl488Lt9Ubr9h1uJi+p3eoDpiig8X87dPdlFXUPt0oLqtg6pvrFQRBuiRUotHqg6C5B2ebi1djDxD5a42mfbTHDhccv/12Pt84uzvLth1k0eYDLNqyn2c+3FLvjR0C3U6/e+uLdZXapCSRk5kRsi3A7iMl3PjsYq4ZlsMl/btXz+YWkYa1+iAA7wd/40m0rzXaK56a6+qoC/p05YI+XQEoLq2g/71vhT3W7NtGkdMpgy7t0khKsrBzMdqnpbBh91Fuf2kFHdNTuGJwT8YNzWHoGZm8+snOqMYUNAYhfuKLIGiK1h4YLSHS4Mhok0x2A+MbQ4PdTVXCnW38euxArhzckwWb9vPK8kJmLy/kpUXb6dq+DQdPlFEeXLipsTEFr8cgFDISbxQEEheiGa9o7Gzj/D5dOL9PFx4YO5A3Vu/iF3PWVIdAleKyCibPWslTH26ud/wNu4+GHINojiueNNAt8cgXM4sl/kU7Q3vskGyGnJFJQV4WH025KGS79mkp/MvwXpRVhF7fqazC0b1Der2vcGMQzbH8t+4xIfFIZwQSN7way2nostqnbxxRb3u4MQgH3DVrFT/5+pc5o3PbqOs4UVque0xIXNIZgbR60a7vFKp9ekoSF/TpwpxPivj6H95n8ssr2brveKPPXVnpaq1rFU77tBRKyiJb4lykuemMQFq95rys9vMjJUz7xyZeWrSd2SuKGPOVntxxUR9W7jhUawD4xlG5HC0pY/aKIgoPFleva9WtQxpPf7il1nLkyWYcPVnOZX/8gAfGDqy+mkqkpSgIWqFou1WiaZ+oV1M112W13Tum8x9Xns2PL/wST36wmb8s2sbs5UUkm1HhvrgqqWpJjgv6dGHSN/oy+uzTq5c0+XK3DvVCpmuHNH4xdw0Tnl7MVYN78osr+tOtQ3qz/g5EwvFNECTqG5jEp24d0/nFFQO49cIv8fX/ep9jJ8vrtTm9Yxov3FRQb3u4kHnzpxfw+PubePz9Tby3YQ93ju7LdwvO1C1JxXO+CQKJjdYewF07pHE8RAgA9VaCbUx6ajI/u/QsxnylJ798dQ2/fHUts5YXcUn/bpp3IJ5SEIicouZe2LB31/b85aYCXv1kJ7+Yu5qVOw5V79O8A/GCrhoSOUXNdde5msyMsUOy6ZCeWm9fYN5B6NVdRZpCQSByiqKdDBeN3YdLQm4vOlTCw+/+kx0HTpzyc4ioa0ikGbT0ZLg2KUn8998/47///hnn9M5i3NAcvpnfg3ZpKVrLSKKmIJCE1doHoqHh5byH53ZizvIiXlleyORZq7j31bWc3bMDqwqPaC0jiYqCQOKKH97co9HYZLg7Lu7D7Rd9meXbDzJrWREzFm+n7kpJzXmLUGmdFAQica6xbiczY9iZWQw7M4sZwXZ1aS0jaYgGi0VakXCXrDb3PbqldVEQiLQioS5lTTKYdOlZMapIEoGCQKQVqXsp62kZqVQ62Ly/8ZVSxb80RiASRqIOXNccU5hxyzlMeWU1j8zfyIAeHbk8v0eMq5N4pCAQiYGWChkz4/6xZ/PPPUf5Py+vJK9rO/qd3vGUjnntEwuAxA1KqU9dQ+IbM28915dvXmkpyUz73jDap6Vwy5+XcehEaaxLatC1TyyoDhtpGQoCER/o1jGdJyYMY/fhEm5/aQXlYe7jLP7kaRCY2WVmtsHMNprZlBD7TzOz18xspZmtNbMfeFmPiJ8NOaMTv756IB9u3MfUN1tu0Tp9wo9/no0RmFky8BhwKVAILDGzec65T2s0+wnwqXPuSjPrCmwwsxedc/F97ioSQiJ0O/3L8F58uvMIT324hbOzO3L1kJxYlyRxwMvB4pHARufcZgAzmwGMAWoGgQM6mJkB7YEDQOi7fIhIs7jnW/1Zv/sIU15ZzZe7doh1OadEA9fNw8uuoWxgR43HhcFtNT0K9Ad2AquBnzrn1Hkp4qHU5CQe++5QurRP43tPL2T59oMs2nKA86bOZ+6KoliXJzHgZRCEutFq3fWwRgOfAD2BrwCPmlm9a9vM7BYzW2pmS/fu3dvcdYr4Tuf2aVxX0IvDxeWUVQT+LKtWKlUY+I+XQVAI9KrxOIfAJ/+afgDMdgEbgS1Av7oHcs496Zwb7pwb3rVrV88KFvGT6Yt21NtWtVJpa6WB69C8HCNYAvQxszygCBgPfLdOm+3AxcD/mll3oC+w2cOaRBKSF33g4VYk1Uql/uPZGYFzrhy4HXgbWAf81Tm31swmmtnEYLMHgFFmthp4F7jLObfPq5pE5AtaqbRx0ZxBJPLZhqdLTDjn3gDeqLNtWo3vdwLf8LIGEQkt3N3PJo/uG8OqJBY0s1jEp+quVGoG/zl2oO5k5kNadE6kFYp0TKFqpdK9R0+yed9x+mef2oJ0kph0RiAidEwPfCZctPlAjCuRWFAQiAhpqclkZ2awaMv+WJciMaAgEBEACvKyWLzlAM7VnfcprZ2CQEQAGJmXxb5jpWzaq9ta+o2CQEQAKOjdGUDdQz6kIBARAHI7t6VbhzQWb9GAsd8oCEQECNzfeGReFos2a5zAbxQEIlKtoHdndh8pYccBrTfkJwoCEal2Tl4WAAs1TuC5eFqbSEEgItW+3K09We3aaGKZzygIRKSamTEyN4vFW3VG4CcKAhGpZWReFjsOFOu+BD6iIBCRWgp6B8YJNJ/APxQEIlJLv9M70jE9RfMJfERBICK1JCcZI3KzNGDsIwoCEamnoHcWm/cdZ8+RkliXIi2g0SAwsyvMTIEh4iMFeYF1hxZv1VmBH0TyBj8e+KeZ/d7M+ntdkIjE3tk9O9KuTbK6h3yi0SBwzn0PGAJsAp41swVmdouZdfC8OhGJiZTkJIblZunKIZ+IqMvHOXcEeAWYAfQArgaWm9kdHtYmIjFUkJfFZ58f48Dx0liXIh6LZIzgSjObA8wHUoGRzrnLgcHAJI/rE5EYKQiuO6TLSFu/SM4IvgM85Jwb5Jx70Dm3B8A5dwL4oafViUjMDMrJJC0lSd1DPpASQZv/AHZVPTCzDKC7c26rc+5dzyoTkZhqk5LE0DM66YzAByI5I3gZqKzxuCK4TURauYLeWXy66wiHi8tiXYp4KJIgSHHOVY8WBb9v411JIhIvCvI64xws1XyCVi2SINhrZldVPTCzMcA+70oSkXgx5IxM2iQnqXuolYtkjGAi8KKZPQoYsAO4wdOqRCQupKcmM7jXaSxUELRqjQaBc24TcI6ZtQfMOXfU+7JEJF4U5HXm8X9s4tjJctqnRfLZURJNRBPKzOxbwG3Az8zsXjO719uyRCRejMzLoqLSsXzbwViXIh6JZELZNOBa4A4CXUPfAc70uC4RiRPDzuxEcpJpPkErFskZwSjn3A3AQefcr4BzgV7eliUi8aJdWgoDs0/TAnStWCRBULUg+Qkz6wmUAXnelSQi8eacvCxWFh6ipKwi1qWIByIJgtfMLBN4EFgObAWme1iTiMSZgt5ZlFU4lm/XOEFr1GAQBG9I865z7pBz7hUCYwP9nHMRDRab2WVmtsHMNprZlDBtvmZmn5jZWjP7R9SvQEQ8N+zMLMxQ91Ar1eC1YM65SjP7A4FxAZxzJ4GTkRzYzJKBx4BLgUJgiZnNc859WqNNJvAn4DLn3HYz69akVyEinjotI5UBPTpqYlkrFUnX0N/MbJyZWZTHHglsdM5tDi5LMQMYU6fNd4HZzrntAFUrm4pIy5l567nMvPXcRtsV5HVm+faDVDrXAlVJS4okCH5OYJG5k2Z2xMyOmtmRCH4um8As5CqFwW01nQV0MrP3zWyZmYWcsRy8I9pSM1u6d+/eCJ5aRJrbyLwsTpZXcvxkeaxL8aVrn1jAtU8s8OTYkcwsbuotKUOdQdT9KJECDAMuBjKABWa20Dn3WZ0angSeBBg+fLg+jojEwMjgjWqOlJTTIT01xtVIc2o0CMzsq6G2O+c+aORHC6k93yAH2BmizT7n3HHguJl9QODOZ58hInElq10b+nbvwO4jxQQ+t0lrEcnCIZNrfJ9OoO9/GXBRIz+3BOhjZnlAETCewJhATa8Cj5pZCoGlrQuAhyKoSURiYGReFi8u2qZxglam0TEC59yVNb4uBQYCn0fwc+XA7cDbwDrgr865tWY20cwmBtusA94CVgGLgaecc2ua/nJExEtJBpUOlmw9yHlT5zN3RVGsS5Jm0JSlBAsJhEGjnHNvAG/U2TatzuMHCUxWE5E4NndFETOXfHH9R9GhYu6evRqAsUPqXgciiSSSReceMbOHg1+PAv8LrPS+NBGJJw++vYGS8spa24rLKvj9W+vD/szcFUWs2H6IRVsORHQGEW17aR6RnBEsrfF9OTDdOfeRR/WISJzaeag49PbDJZz723fJzswgp1MGOZ3akt0pg237j/PsR1sprQiER2NnEHNXFHH37NURt5fmE0kQzAJKnHMVEJgxbGZtnXMnvC1NROJJz8wMikKEQYf0FEZ9qQuFB0+wdNtBXlu1i4rK0IPJxWUV/NvMT7hz1qp6+6oCoG77B9/eoCDwWCRB8C5wCXAs+DgD+BswyquiRCT+TB7dl7tnr6a4xgqkGanJPDBmYK036vKKSnYfKeGC371Xb+JQlZsuqL+A8ePvbwrZNtyZiDSfSIIg3TlXFQI4546ZWVsPaxKROFT1Zn/nrFWUVlSSnZnB5NF9631aT0lOIqdT27BnENmZGdx1Wb962+d9sjNk+/TUJPYcKaFbx/RmeiVSVyRLTBw3s6FVD8xsGKCIFvGhsUOyGXJGJgV5WXw05aIGu2wmj+5LRmpyrW0ZqclMHt034vYpSUZpeSUX/+Ef/HnB1rBdTnJqIjkj+DfgZTOrmhXcg8CtK0VEwor0DKKx9oN7ZfLLuWu499W1vLKskN9cnc/A7NNa7HX4QSRrDS0xs35AXwLrB613zpV5XpmIJLyxQ7KZvng7QEQrnIZr/8JNI5m3cicPvL6Oqx79kBtH5XFW9/as2H6I0opKzps6v8GQkYZFstbQT4AXq2b8mlknM7vOOfcnz6sTEQHMjDFfyeZrfbvx4NvreeajLbX2R3KpadUcBQVHfZGMEdzsnDtU9cA5dxC42bOKRETCOC0jlV+Pzadr+7R6+4rLKvjVa2tZueMQ+4+dxNVYDyncHAVNWAuIZIwgyczMBX+rwTuPtfG2LBGR8PYdC32jxIMnyhjzWGC+a0ZqMtmdApPclmw5UOuyV2h8jkI0ZxCJfrYRSRC8DfzVzKYRuJ/AROBNT6sSEWlAuEtTu3VI4zdX51N48ASFB4spOlhM4aETHC+tCHGUwJnBDc8srjErOjAzek3RIaa+uT6iWc6tYUZ0JEFwF3AL8GMCg8UrCFw5JCISE+Emt/37N/tz6YDu9dqfN3V+2DkKh06UsrboMPuPlzb4nMVlFUx6eWW9iW+b9h6jvM5lrYk2IzqSq4YqzWwh0JvAZaNZwCteFyYiEk60l6aGC47ffju/+mdOlJYHzyCK+cGzS0Iep7zSkdelXa1tGz4/GrJtIs2IDhsEZnYWgZvJXAfsB2YCOOe+3jKliYiEF82lqZEER9s2KfTp3oE+3TuQ3cCs6GkThtXaFu5sA+ClRdsZP6IXSUmh7twbPxq6amg9gXsJX+mcO9859wgQuqNNRCTOeTUrOlTbtJQkendtx7/PWc010z5m3a4jzfMiPNJQEIwDdgPvmdn/mNnFhL4hvYhIqzJ2SDa//XY+bZIDb5HZmRm1upEaa/u7cYN45+cX8ofvDGbr/hNc8ciH/Ocb6zhRWt6iryNSYbuGnHNzgDlm1g4YC/wM6G5mjwNznHN/a5kSRURaXrRdT6HajhuWw8X9uzH1zfU8+cFm/t+qXdx31dkcP1keV5ebRjJYfBx4EXjRzLKA7wBTCCxFLSIiDchs24ap4wZxzbAc7pmzhpv/vLT63s8QH5ebRjKzuJpz7oBz7gnn3EVeFSQi0hoNz83i9X89n47pKdRdRLXqctNYiSoIRESk6VKTkzhaEnqcIJaXmyoIRERaUM/MjKi2twQFgYhICwp3A55wN+xpCQoCEZEWVPdy07ZtkimvdOR00hmBiIhv1JzctvieS8jplMGkl1dSHGZxPK8pCEREYqh9Wgq/v2YQW/ef4Pdvr49JDQoCEZEYG/WlLnz/3DN59qOtLNy8v8WfX0EgIhIH7rq8H2d2bsvkWSs5frJll6JQEIiIxIG2bVJ48JrBFB4sZuqbLdtFpCAQEYkTI/Oy+OF5ebywcBsfbdzXYs+rIBARiSOTvtGX3l3aceesVRwtKWuR51QQiIjEkYw2yTz4ncHsOlzMf76xrkWeU0EgIhJnhp3ZiZu/2pvpi3fwj8/2ev58CgIRkTj0s0vO4svd2nPXrFUcLva2i8jTIDCzy8xsg5ltNLMpDbQbYWYVZnaNl/WIiCSK9NRk/vCdwew9dpIHXv/U0+fyLAjMLBl4DLgcGABcZ2YDwrT7HfC2V7WIiCSiwb0y+fGFX2LWskKWbTvIoi0HOG/qfOauKGrW5/HyjGAksNE5t9k5VwrMAMaEaHcH8Aqwx8NaREQSUm7nthhQHrybTdUdzZozDLwMgmxgR43HhcFt1cwsG7gamOZhHSIiCeuhd/5JnRuaNfsdzbwMAguxre7r+SNwl3OuwSX3zOwWM1tqZkv37vV+BF1EJF6Eu3NZc97RzMsgKAR61XicA+ys02Y4MMPMtgLXAH8ys7F1D+Sce9I5N9w5N7xr164elSsiEn9a4o5mXgbBEqCPmeWZWRtgPDCvZgPnXJ5zLtc5lwvMAm5zzs31sCYRkYQS6o5mGanJzXpHs5RmO1IdzrlyM7udwNVAycAzzrm1ZjYxuF/jAiIijRg7JDC0euesVZRWVJKdmcHk0X2rtzcHz4IAwDn3BvBGnW0hA8A5d6OXtYiIJKqxQ7KZvng7ADNvPbfZj6+ZxSIiPqcgEBHxOQWBiIjPKQhERHxOQSAi4nMKAhERn1MQiIj4nIJARMTnFAQiIj6nIBAR8TkFgYiIzykIRER8TkEgIuJzCgIREZ9TEIiI+JyCQETE5zy9MY2ItD7R3hjFixupSPNSEIhIXFFwtDx1DYmI+JyCQETE5xQEIiI+pyAQEfE5BYGIiM8pCEREfE5BICLicwoCERGfUxCIiPicZhaLiG9o1nJoCgIRSVh6Y28eCgIRkTCiCZpEDiWNEYiI+JyCQETE59Q1JCISA/HUlaQzAhERn1MQiIj4nKdBYGaXmdkGM9toZlNC7L/ezFYFvz42s8Fe1iMiIvV5FgRmlgw8BlwODACuM7MBdZptAS50zg0CHgCe9KoeEREJzcszgpHARufcZudcKTADGFOzgXPuY+fcweDDhUCOh/WIiEgIXgZBNrCjxuPC4LZwbgLeDLXDzG4xs6VmtnTv3r3NWKKIiHh5+aiF2OZCNjT7OoEgOD/UfufckwS7jYYPHx7yGCIirZmXl5t6GQSFQK8aj3OAnXUbmdkg4Cngcufcfg/rERGRELzsGloC9DGzPDNrA4wH5tVsYGZnALOBCc65zzysRUREwvDsjMA5V25mtwNvA8nAM865tWY2Mbh/GnAv0Bn4k5kBlDvnhntVk4iI1GfOJVaX+/Dhw93SpUtjXYaISEIxs2XhPmhrZrGIiM8pCEREfE5BICLicwoCERGfUxCIiPicgkBExOcUBCIiPqcgEBHxOQWBiIjPKQhERHxOQSAi4nMKAhERn1MQiIj4nIJARMTnFAQiIj6nIBAR8bmEuzGNme0FtkX5Y12AfR6U0xJUe2wkau2JWjeodq+d6ZzrGmpHwgVBU5jZ0kS9BaZqj41ErT1R6wbVHkvqGhIR8TkFgYiIz/klCJ6MdQGnQLXHRqLWnqh1g2qPGV+MEYiISHh+OSMQEZEwFAQiIj7X6oPAzC4zsw1mttHMpsS6nnDMrJeZvWdm68xsrZn9NLg9y8z+bmb/DP63U6xrDcfMks1shZm9HnycELWbWaaZzTKz9cHf/7kJVPvPgv9e1pjZdDNLj9fazewZM9tjZmtqbAtbq5ndHfy73WBmo2NTdXUtoWp/MPhvZpWZzTGzzBr74qb2SLTqIDCzZOAx4HJgAHCdmQ2IbVVhlQP/xznXHzgH+Emw1inAu865PsC7wcfx6qfAuhqPE6X2/wu85ZzrBwwm8BrivnYzywb+FRjunBsIJAPjid/anwMuq7MtZK3Bf/vjgbODP/On4N9zrDxH/dr/Dgx0zg0CPgPuhrisvVGtOgiAkcBG59xm51wpMAMYE+OaQnLO7XLOLQ9+f5TAm1E2gXqfDzZ7HhgbkwIbYWY5wLeAp2psjvvazawj8FXgaQDnXKlz7hAJUHtQCpBhZilAW2AncVq7c+4D4ECdzeFqHQPMcM6ddM5tATYS+HuOiVC1O+f+5pwrDz5cCOQEv4+r2iPR2oMgG9hR43FhcFtcM7NcYAiwCOjunNsFgbAAusWwtIb8EbgTqKyxLRFq7w3sBZ4Ndms9ZWbtSIDanXNFwH8B24FdwGHn3N9IgNprCFdrov3t/hB4M/h9otXe6oPAQmyL6+tlzaw98Arwb865I7GuJxJmdgWwxzm3LNa1NEEKMBR43Dk3BDhO/HSlNCjYnz4GyAN6Au3M7HuxrarZJMzfrpndQ6Br98WqTSGaxWXtVVp7EBQCvWo8ziFw6hyXzCyVQAi86JybHdz8uZn1CO7vAeyJVX0NOA+4ysy2Euh+u8jM/kJi1F4IFDrnFgUfzyIQDIlQ+yXAFufcXudcGTAbGEVi1F4lXK0J8bdrZt8HrgCud19MykqI2mtq7UGwBOhjZnlm1obAAM68GNcUkpkZgX7qdc65/66xax7w/eD33wdebenaGuOcu9s5l+OcyyXwO57vnPseiVH7bmCHmfUNbroY+JQEqJ1Al9A5ZtY2+O/nYgJjS4lQe5Vwtc4DxptZmpnlAX2AxTGoLywzuwy4C7jKOXeixq64r70e51yr/gK+SWBEfxNwT6zraaDO8wmcPq4CPgl+fRPoTOBqin8G/5sV61obeR1fA14Pfp8QtQNfAZYGf/dzgU4JVPuvgPXAGuAFIC1eawemExjLKCPwqfmmhmoF7gn+3W4ALo/D2jcSGAuo+nudFo+1R/KlJSZERHyutXcNiYhIIxQEIiI+pyAQEfE5BYGIiM8pCEREfE5BICLicwoCERGfUxCInCIzu9XMpgW/TzWzF8zs+eCSISJxT0EgcuoGAauCS1q/CWx3zn3fBdb/EYl7CgKRU5dPYK3694GXnXP3xLYckehoiQmRU2RmBwksQ/xD59xrsa5HJFopsS5AJJGZWS/gGIFF03rEuByRJlEQiJyaQcBK4GZgoZktcc6tiHFNIlHRGIHIqckHVrvAbRZ/BMw0s9NiXJNIVBQEIqcmH1gN4Jz7O/BX4JmYViQSJQ0Wi4j4nM4IRER8TkEgIuJzCgIREZ9TEIiI+JyCQETE5xQEIiI+pyAQEfG5/w/LBnO9/PbXmwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Statistics\n",
    "cv_scores_means = gs_knn.cv_results_['mean_test_score']\n",
    "cv_scores_stds = gs_knn.cv_results_['std_test_score']\n",
    "\n",
    "# Plotting the relationships\n",
    "plt.errorbar(k_range, cv_scores_means, yerr=cv_scores_stds, marker='o', label='Accuracy')\n",
    "plt.ylim([0.1, 1.1])\n",
    "plt.xlabel('$K$')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Naive Bayes Classifier\n",
    "Naive Bayes methods are a set of supervised learning algorithms based on applying Bayes' theorem with the \"naive\" assumption of conditional independence between every pair of features given the value of the class variable. Bayes'theorem states the following relationship, given class variable $y$ and dependent feature vector $x_1$ through $x_n$,:\n",
    "\n",
    "$$P(y \\mid x_1, \\dots, x_n) = \\frac{P(y) P(x_1, \\dots x_n \\mid y)}\n",
    "                                 {P(x_1, \\dots, x_n)}$$\n",
    "\n",
    "Using the naive conditional independence assumption, we have\n",
    "\n",
    "$$\\begin{align}\\begin{aligned}P(y \\mid x_1, \\dots, x_n) \\propto P(y) \\prod_{i=1}^{n} P(x_i \\mid y)\\\\\\Downarrow\\\\\\hat{y} = \\arg\\max_y P(y) \\prod_{i=1}^{n} P(x_i \\mid y),\\end{aligned}\\end{align}$$\n",
    "\n",
    "Then, we can use Maximum A Posteriori (MAP) estimation to estimate $P(y)$ and $P(x_i \\mid y)$; the former is then the relative frequency of class $y$ in the training set.\n",
    "\n",
    "*References*:\n",
    "H. Zhang (2004). The optimality of Naive Bayes. Proc. FLAIRS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### Gaussian Naive Bayes\n",
    "\n",
    "[GaussianNB](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html#sklearn.naive_bayes.GaussianNB) implements the Gaussian Naive Bayes algorithm for classification on the data sets where features are continuous.   \n",
    "The likelihood of the features is assumed to be Gaussian:\n",
    "\n",
    "$$P(x_i \\mid y) = \\frac{1}{\\sqrt{2\\pi\\sigma^2_y}} \\exp\\left(-\\frac{(x_i - \\mu_y)^2}{2\\sigma^2_y}\\right)$$\n",
    "\n",
    "The parameters $\\sigma_y$ and $\\mu_y$  are estimated using maximum likelihood."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Demo**: In this demo, we show how to build a Gaussian Naive Bayes classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAcrElEQVR4nO3dfXBc1XkG8OeVkTFbPBIRLibY2iVTJiXgILDGCcFpmdoTwFPipoGJw9YlwURJGGfWzmSmziiu1+1oUpdJZZmkA2rr1I23fDTUTZw6Q8AhE/4oxjIRFuAYMKNVVAiW3UhxkDES+/aPu1derXa1u7qfZ+/zm9Fo793V3eOV/Nyz7z17jqgqiIjIXA1BN4CIiJxhkBMRGY5BTkRkOAY5EZHhGORERIa7IIgnvfTSSzWRSATx1ERExjpy5MgpVV1UvD+QIE8kEujr6wviqYmIjCUi2VL7WVohIjIcg5yIyHAMciIiwwVSIyei8JmYmMDw8DDeeeedoJsSeQsWLMCSJUvQ2NhY1eMZ5EQEABgeHsbChQuRSCQgIkE3J7JUFadPn8bw8DCuvPLKqn6GpRUiAgC88847aGlpYYgHTETQ0tJS0zsjBjkRTWGIh0OtvwcGOYVH8ZTKnGKZqCoMcgqHdBrYvPl8eKta2+l0kK0iH50+fRptbW1oa2vD4sWLccUVV0xtv/vuu1UfZ/fu3fj1r389tf35z38ex48fd9y+yclJzJs3D21tbbjmmmvQ1taGnTt3IpfLzfpzr7/+Oh555BHHzz8bXuyk4KkCo6NAT4+13d1thXhPD5BKWffzLX/4FP9eHP6eWlpa0N/fDwBIp9O4+OKL8bWvfa3m4+zevRs33HADFi9eDAD47ne/O+c2FVu4cOFUG9966y2sW7cOZ86cwdatW8v+jB3k69atc60dxdgjp+CJWOGdSlnh3dBwPsS7uxniYeTzO6g9e/ZgxYoVaGtrw3333YdcLofJyUmsX78ey5Ytw7XXXotdu3bh0UcfRX9/Pz7zmc9M9eRXrlyJ/v5+TE5Oorm5GVu2bMF1112HG2+8ESdPngQAvPrqq/jIRz6CFStWYOvWrWhubq7YpssuuwwPPfQQHnjgAQDAiRMn8PGPfxzXX389li9fjkOHDgEAtmzZgqeffhptbW3YtWtX2cc5oqq+fy1fvlyJZsjlVK1IsL5yuaBbFCkvv/xydQ/M5VRTKet3lEqV3nZo27Ztev/996uq6sDAgK5du1YnJiZUVfULX/iCZjIZffbZZ/XWW2+d+pnf/OY3qqp600036S9+8Yup/fb2xMSEAtADBw6oqurmzZv1m9/8pqqq3nLLLfrYY4+pquoDDzygTU1NM9o0MTFRcv/FF1+sp06d0rffflvPnj2rqqrHjh3TFStWqKrqk08+qWvXrp16fLnHFSv1+wDQpyUylaUVCge7R1do82b2yMPIfgcFWO+c7JKYR++gnnrqKRw+fBjt7e0AgLNnz2Lp0qW45ZZbcPz4caRSKaxZswaf+MQnKh7roosuwm233QYAWL58OZ555hkAwKFDh3DgwAEAwF133YVvfOMbVbdP8+9Kzp07h40bN+KFF17ABRdcgBMnTpR8fLWPqwVLKxQ8O8Ttckoud77MUvj2ncKjMMxtHp10VRX33HMP+vv70d/fj+PHj2Pr1q1oaWnB0aNHsXLlSuzatQtf/OIXKx5r/vz5U7fnzZuHyclJR2175ZVXEIvF0NLSgm9961tYunQpBgYG8Nxzz+HcuXMlf6bax9WCQU7BEwGam6f36OyaeXMze+RhVO4dlAcn3dWrV+Oxxx7DqVOnAFijW4aGhjAyMgJVxZ133ont27fj+eefB2BdkDxz5kxNz7FixQrs27cPAKoeYXLy5El8+ctfxle+8hUAwNjYGC6//HKICPbs2TPVUy9uT7nHOcHSCoVDOj191IMd5gzx8Cl+B1U4yghw/fe2bNkybNu2DatXr0Yul0NjYyMefPBBzJs3Dxs2bICqQkSwY8cOANZww3vvvRcXXXQRnnvuuaqeY9euXVi/fj127NiBNWvWoKmpqeTjzpw5M3URdf78+bj77ruRSqUAABs3bsQdd9yBhx9+GKtXr8aFF14IALj++uvx3nvv4brrrsOGDRvKPs4JceNsUKv29nblwhJE4XLs2DFcffXV1T04nbaGjNqhbYd7c7ORY//ffvttxGIxiAj27t2Lffv24fHHHw+0TaV+HyJyRFXbix/LHjkR1a7O3kEdPnwYmzZtQi6XwyWXXOLq2HM/MMiJaG6KQ9vQEAeAm2++eeqDPibixU4imhJEqZVmqvX3wCAnIgDWYganT59mmAdM8/ORL1iwoOqfYWmFiAAAS5YswfDwMEZGRoJuSuTZKwRVi0FORACAxsbGqlekoXBhaYWIyHAMciIiwzHIiYgMxyAnIjIcg5yIyHAMciIiwzHIiYgMxyAnIjIcg5yIyHB1EOQZAAlY/5REfpuIKDpcCXIR2S0iJ0XkRTeOV70MgA4AWQCa/94BhjkRRYlbPfJ/BXCrS8eqQSeA8aJ94/n9TrCXT0TmcGXSLFX9uYgk3DhWbYZq3F8Nu5dvnyDsXj4AJB0cl4jIG77VyEWkQ0T6RKTPvWkyW2vcXw2vevlERN7wLchVtVdV21W1fdGiRS4dtQtArGhfLL9/rrzo5RMRecfwUStJAL0A4gAk/70XjkogurRo277hpJdPROSdOlhYIgnXatfpNDB6NdA9AshZK8Q3A2i+AEg76eUTEXnHreGHDwP4HwAfFJFhEdngxnF9pQqMjgI9TwCb/wjQVivEewCMrgL0rqBbSERUklujVj7rxnECJQJ0d1u3e3qsAAeAVMraLxJY04iIZiNBrJjd3t6ufX19vj9vSarTQzqXA+bNm77NECeiEBCRI6raXrzf8IudDqXTwObNVpgDVmgvXz79MYX3ExGFUHSDfKom3mOFtR3i/f1AWxvw3ntWWcW+n2FORCFVB6NW5mhGTTxfFG9rA44cARoazt/f3MzyChGFVnR75MD0MLfZIV54fzrte9MoGjIZIJGw/uQSCWubqFbRDnJVq2xS6KtfnV5GYU/cJ9GbqCyTATo6gGzW+pPLZq1thjnVKrpBbod4T49VC8/lWBMPTDSnI+7sBMaLpvUZH7f2E9UiukEuYtW+C8eJd3db257UxKPX46xeNCcqGyozfU+5/UTlcBx58Tjy4m1XFE+NC1iTezmcF6ZuNKBgUpsCAiDn4LgZWCeDIVhz5XQhTK93ImGVU4rF48DgoN+tIRNwHHk5xaHtSU08mj3O6s1lOuJK73DCX67p6gJiRZN3xmLWfqJaMMh9EZWpcedaPqp1OuJqQjr8J89kEujttXrgItb33l5rP1EtGOS+8GIBjLBx0gOudjpi+0TxF6gc0uE+edrDDtevt7a/9z2rnMIQp7lgkPvCiwUwwsZpDzgJYBBWTXwQpUPcPlGUUxjS4T15ctghuY1B7gsPFsAIHa97wKVOFMUKQzq8J09vhx1ydFQURfcj+r5zcQGMUGpF6d6yWz3gSieE4pC2X+vwjVrxbtghFw6PKvbIySVe94BnOyGUe4dTqVwTjNYy/5Ry+6sX/gu85A0GObnE6/JRuRPFXoQppKvh3bDDcF/gJe8wyMlFXvaA7RNFS8G+i1w8vn+8G3YY3gu85C3WyMkwZwtun4apNeBk0ouhhl0o/Qni4C/wkrci3SPPDGSQ2JlAw/YGJHYmkBngFf5SwjPVKmvAs4vC6CgqJbJBnhnIoGN/B7JjWSgU2bEsOvZ3MMyLhGvMs/814PCcxKoVzgu85K3IBnnnwU6MT0zv3Y1PjKPzIHt3hcI11aq/NeBgT2LhHg9u3gmuvkU2yIfGSvfiyu2PqnBNtervh3yCO4mFe8IvX09wxbOzujZba7hPlLWKbJC3NpXuxZXbH1XejXmeC39rwMGdxMJ9LcC3E1w6PX2RF3sxGMdLL4b7RDkXkQ3yrlVdiDVO793FGmPoWsUr/IXCN9WqfzXg4E5i4R4P7uwEV2VPWBUYHZ2+Ype9otfoqMOeebhPlHOiqr5/LV++XMNg79G9Gu+Oq6RF491x3Xt0b9BNCqW9e1XjcVUR6/veiLxMe/eqxmKqVmpYX7GYH//+uJb+rxP3+omrEo9Pf03sr3i80k/uVdWYTv83xfL7S8jlVFOp6U+SSln7HREt/fqKw+N6D0Cflmh8pIOcqJJgTmI1Bp7Pyp3gnnlmr1onG8l/L25vXGs+QeVy05/IcYjPsR0hUS7II1taIapGMmnNE57L+TlfeLjHg5f6ZOoTT2SwcmWlunONJSO7nFLIlYXR/Z4Z04cLq6XS3esv9siJ6k1cK/dyq3lMXmFZxS6nFG87Uundg1vcfXeFMj1yfkSfiFxQTW+7hikERIDmZiCVArq7re3ubuu+5mYX1tb1a1rp2S6suvf8oo7fptSuvb1d+/r6fH9eIvJKAqXno4/DGl1ky6CmOeJVp4d28XboNcAqNRUTWCOvaiMiR1S1vdSzEBE5VG3ducbho8WhbVSIA359GplBTkQuCPcF2uD4c2GVQU7kkejNR8IJu2by5wTnSpCLyK0iclxEXhORLW4ck8hk4Zo1koLl/QnOcZCLyDwA3wFwG4APAfisiHzI6XGJTBauWSOp3rnRI18B4DVVfV1V3wXwCIC1LhyXyFjhmjWS6p0bQX4FgF8VbA/n900jIh0i0icifSMjIy48LVF4hWvWSKp3bgR5qfFAMwZOqmqvqraravuiRYtceFqi8ArfrJFUz9wI8mEASwu2lwB4w4XjEhmr1Hwkvb1+zdVCUePGR/QPA7hKRK4E8L8A1gG4y4XjEhktmWRwkz8c98hVdRLARgBPADgG4DFVfcnpcWeTGcggsTOBhu0NSOxMcMFkIoo0VybNUtUDAA64caxKMgMZdOzvmFo4OTuWRcf+DgBAchm7P0QUPcZ9srPzYOdUiNvGJ8bReZADdIkomowL8nKr3JfbH0YsDRGRm4wL8nKr3JfbHzZ2aSg7loVCp0pDDHMimivjgrxrVRdijdMH6MYaY+haZcYAXZaGqKLiNQICWDOAzGJckCeXJdF7ey/iTXEIBPGmOHpv7zXmQmc9lIbIQ+n09HUp7XUr0+kgW0UhZ+RSb8llSWOCu1hrUyuyYzNXUjGlNEQeUgVGR4GeHmu7u9sK8Z4ea8kz41bHIb8Y1yM3nemlIfKQvS5lKmWFd0PD+RC3160kKoFB7jPTS0NGMLnGXLjIsI0hThUYWVoJnMMFYU0uDYVeOm2VJ+zws2vMzc1m1Jnt9hbavJlhTrNij7xWhl6MisTY9cIas/07smvMo6Ph75kXtjeVAnK582WWwr85oiLskdfC0ItRkZnWoLAs0dNz/vdkSo1ZxHrnUNhe+9/T3Bz+9lNgRAM4y7e3t2tfX5/vz+uKwl6TLeRBkdiZKDlSJt4Ux+CmQf8b5DVV60KhLZcL7e+mJIelO6pfInJEVduL97O0UisDL0aVG6NeKtyNV67GbFJZovhvKcR/WxQODPJaGRgU5caoC6S+auWsMVNERTbI53Txz9Cg6FrVBSmxIp9C62tqgHI15lQq0BpzJgMkEla1J5GwtoncFMkaefHFP8D6UE5V47kNHd4m20uHmECQ25bzuTUeC1GNOZMBOjqA8YLpdWKx6pZ9y2SAzk5gaMhatLmriysORV25Gnkkg9zxxb8QBUW1InfBMyQSCSBb4lJEPA4MDpb/OScnAKpfvNhZwPHEVQZejOLUAMEYKvMnVW6/rbNzeogD1nZnHVXCyD2RDHLT5zSfC04NEIzWMn9S5fbb5noCoGiKZJBHtXeaXJbE4KZB5LblMLhpkCHug64uqyRSKBaz9s9mricAiqZIBjl7p+SXZNKqa8fjVgUuHq+uzj3XEwBFUyQvdtaTzEAGnQc7MTQ2hNamVnSt6uIJqU5w1AoVK3exk3OtGCwyc6hEVDLJ4KbqRLK0Ui+4/icRAQxyo3H9TyICGORGi+IwSiKaiUFusKgOoySi6RjkBuMwysoisTISRR6HH1LdcjQ5GlEIca4VihyO6qGoYJBT3eKoHooKBjnVLY7qoahgkFPd4qgeigpHQS4id4rISyKSE5EZBXgKXpRHbXBUD0WF07lWXgTw5wAecqEt5DInc7HUy4RNyWVJBjfVPUc9clU9pqrH3WoMuWuuozbsZcayWWsVu2zW2uaiwUThxBp5HZvrqA0uM0a1yGSstUkbGqzvPOH7r2JpRUSeArC4xF2dqvqDap9IRDoAdABAK5c58UVrU2vJBZcrjdrgMmNUreJFou13b4CZpThTVeyRq+pqVb22xFfVIZ4/Tq+qtqtq+6JFi+beYqraXEdtcJkxqhbfvYUDSyt1bK6jNjxfZqx4WogApokgd/DdWzg4mmtFRD4F4AEAiwCMAuhX1Vsq/RznWgk/z0atpNPA6CjQ3W0tYqkKbN4MNDdb95FREgmrnFIsHgcGB/1uTf3zZK4VVd2nqktU9UJVvayaECczJJPWf8RczvruSoirWiHe02OFtx3iPT3WfvbMjcNFosOBa3aSf0SsnjhghXdPj3U7lTrfQyej2Cf4evjMgck4jS35T9Uaq2bL5RjiRFXgNLYUDnY5pZBdZikhylMMEFWLQU7+KayJp1JWTzyVml4zL2BPMZAdy0KhU1MMMMyJpmOQk39ErNEphTXx7m5ru7l5RnmFC0MQVYcXO8lf6bTV87ZD2w7zEjVyLgxBVB32yMl/xaFd5kInF4Ygqg6DnEKLC0MQVYdBTqHFhSGIqsNx5ESmKbzGUGqb6hbHkRPVg3R6+lBNe0gn56mJNAY5kSk4Vw2VweGHRKbgXDVUBmvkRKbhXDWRxRo5UT2oca4aigYGOZEpapyrhqKDNXIiU5SbqwYoOVcNRQdr5ESm4TjyyGKNnKheVDlXDUUHg5yIyHAMciIiwzHIiYgMxyAnIjIcg5yIyHAMciIiwzHIiepR8edD+KnPusYgp1DLZIBEwpojKpGwtl09/kAGiZ0JNGxvQGJnApkBl58gCJyzPHIY5BRamQzQ0QFks1YWZbPWtlthnhnIoGN/B7JjWSgU2bEsOvZ3mB3mnLM8kvgRfQqtRMIK72LxODA46MLxdyaQHZv5BPGmOAY3ufAEQSkMbxvnLK8L5T6izyCn0GpoKN2BFLEm/nN8/O0NUMx8AoEgt82FJwgS5yyvS5xrhYzT2lrb/pqP31T6QOX2G4NzlkcOg5xCq6sLiMWm74vFrP2uHH9VF2KN058g1hhD1yqXniAInLM8kjgfOYVWMml97+wEhoasnnhX1/n9jo+/zDpQ58FODI0NobWpFV2ruqb2G4lzlkcSa+RE9YhzltclT2rkInK/iPxSRI6KyD4RaXZyPCJyCecsjxSnNfInAVyrqh8G8AqArztvEhER1cJRkKvqT1R1Mr/5LIAlzptERES1cHPUyj0AflzuThHpEJE+EekbGRlx8WmJiKKt4qgVEXkKwOISd3Wq6g/yj+kEMAmg7GebVbUXQC9gXeycU2uJiGiGikGuqqtnu19E7gbwpwBWaRBDYIiIIs7ROHIRuRXAXwH4Y1Udd6dJRERUC6c18m8DWAjgSRHpF5EHXWgTERHVwFGPXFX/wK2GEBHR3HCuFSIfeL1ABkUb51oh8pi9QMZ4/iqSvUAG4N68MRRt7JETeayz83yI28bHrf1EbmCQE3lsaKi2/US1YpATeczrBTKIGOREHvN6gQwiBjmRx5JJoLfXWjRaxPre28sLneQeBjmRD5JJYHDQWnltcDA8Ic5hkfWBww+JIorDIusHe+REEcVhkfWDQU4UURwWWT8Y5EQRxWGR9YNBThRRHBZZPxjkRBHFYZH1g6NWiCIsmWRw1wP2yImIDMcgJyIyHIOciMhwDHIiIsMxyImIDMcgJyIyHIOciMhwDHIiIsMxyImIDMcgJyIyHIOciMhwDHIiIsMxyImIDMcgJyIyHIOciMhwDHIiIsMxyIkqyAxkkNiZQMP2BiR2JpAZyATdJKJpuEIQ0SwyAxl07O/A+MQ4ACA7lkXH/g4AQHIZl9ahcGCPnGgWnQc7p0LcNj4xjs6DnQG1iGgmR0EuIn8rIkdFpF9EfiIi73erYURhMDQ2VNN+oiA47ZHfr6ofVtU2AD8C8NcutIkoNFqbWmvaTxQER0Guqr8t2Pw9AOqsOUTh0rWqC7HG2LR9scYYulZ1BdQiopkc18hFpEtEfgUgiVl65CLSISJ9ItI3MjLi9GmJfJFclkTv7b2IN8UhEMSb4ui9vZcXOilURHX2TrSIPAVgcYm7OlX1BwWP+zqABaq6rdKTtre3a19fX61tJSKKNBE5oqrtxfsrDj9U1dVVPse/A/hvABWDnIiI3ON01MpVBZufBPBLZ80hIqJaOf1A0N+JyAcB5ABkAXzJeZOIiKgWjoJcVT/tVkOIiGhu+MlOIiLDMciJiAzHICciMlzFceSePKnICKyLo6a4FMCpoBtRA7bXW2yvt9je8uKquqh4ZyBBbhoR6Ss1CD+s2F5vsb3eYntrx9IKEZHhGORERIZjkFenN+gG1Ijt9Rbb6y22t0askRMRGY49ciIiwzHIiYgMxyAvQUTuFJGXRCQnImWHFYnIrSJyXEReE5EtfraxqB3vE5EnReTV/PdLyjzuvfz6qv0i8sMA2jnr6yUiF4rIo/n7D4lIwu82FrWnUns/JyIjBa/pvUG0M9+W3SJyUkReLHO/iMiu/L/lqIjc4HcbS7SpUptvFpGxgtc3sKUkRWSpiDwtIsfy2ZAq8ZjgXmNV5VfRF4CrAXwQwM8AtJd5zDwAJwB8AMB8AC8A+FBA7f17AFvyt7cA2FHmcb8L8DWt+HoBuA/Ag/nb6wA8GvL2fg7At4NqY1Fb/gjADQBeLHP/GgA/BiAAPgrgkAFtvhnAj4JuZ74tlwO4IX97IYBXSvw9BPYas0degqoeU9XjFR62AsBrqvq6qr4L4BEAa71vXUlrAezJ394D4M8Casdsqnm9Cv8d3wewSkTExzYWCtPvtyJV/TmA/5vlIWsB/JtangXQLCKX+9O60qpoc2io6puq+nz+9hkAxwBcUfSwwF5jBvncXQHgVwXbw5j5i/XLZar6JmD9wQH4/TKPW5BfN/VZEfE77Kt5vaYeo6qTAMYAtPjSupmq/f1+Ov82+vsistSfps1JmP5ea3GjiLwgIj8WkWuCbgwA5Et+1wM4VHRXYK+x04UljFXtWqSzHaLEPs/Gcs7W3hoO06qqb4jIBwD8VEQGVPWEOy2sqJrXy9fXtIJq2rIfwMOqek5EvgTr3cSfeN6yuQnTa1ut52HNLfI7EVkD4L8AXFXhZzwlIhcDeBzAJlX9bfHdJX7El9c4skGu1a9FWs4wgMIe2BIAbzg8ZlmztVdE3hKRy1X1zfxbuZNljvFG/vvrIvIzWL0Kv4K8mtfLfsywiFwAoAnBvfWu2F5VPV2w+U8AdvjQrrny9e/VDYVBqaoHROQfReRSVQ1kQi0RaYQV4hlV/c8SDwnsNWZpZe4OA7hKRK4UkfmwLs75PhIk74cA7s7fvhvAjHcUInKJiFyYv30pgJsAvOxbC6t7vQr/HXcA+KnmryIFoGJ7i+qfn4RVNw2rHwL4y/zIio8CGLPLcWElIovtayQisgJWXp2e/ac8a4sA+BcAx1T1H8o8LLjXOOirwWH8AvApWGfXcwDeAvBEfv/7ARwoeNwaWFevT8AqyQTV3hYABwG8mv/+vvz+dgD/nL/9MQADsEZfDADYEEA7Z7xeAP4GwCfztxcA+A8ArwF4DsAHAv47qNTebwJ4Kf+aPg3gDwNs68MA3gQwkf/b3QBrDd0v5e8XAN/J/1sGUGY0VsjavLHg9X0WwMcCbOtKWGWSowD6819rwvIa8yP6RESGY2mFiMhwDHIiIsMxyImIDMcgJyIyHIOciMhwDHIiIsMxyImIDPf/pJ8vw4E4Mn4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Generate a synthetica 2D dataset\n",
    "X, y = make_classification(n_samples=50, n_features=2, n_informative=2,\n",
    "                           n_redundant=0, n_classes=3, n_clusters_per_class=1, \n",
    "                           weights=None, flip_y=0.01, class_sep=0.5, hypercube=True,\n",
    "                           shift=0.0, scale=1.0, shuffle=True, random_state=42)\n",
    "\n",
    "# Data split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.2, random_state=42)\n",
    "\n",
    "# Visualize the generated data\n",
    "colors = ['blue', 'yellow', 'green']\n",
    "for i, color in enumerate(colors):\n",
    "    plt.scatter(X_train[y_train == i, 0], X_train[y_train == i, 1], c=color)\n",
    "plt.scatter(X_test[:, 0], X_test[:,1], c='red', marker='x', label='Testing Data')\n",
    "plt.legend(loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing accuracy is: 0.8947\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Create and training a Gaussian Naive Bayes classifier model\n",
    "clf = GaussianNB()\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Use the model to predict testing data\n",
    "y_pred = clf.predict(X_test)\n",
    "accuracy = accuracy_score(y_pred, y_test)\n",
    "print('Testing accuracy is: %.4f\\n' % accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimated probability of classess: \n",
      " [0.383929 0.294643 0.321429]\n",
      "Estimated mean for each Gaussian distribution: \n",
      " [[5.018605 3.432558 1.460465 0.244186]\n",
      " [5.890909 2.766667 4.236364 1.306061]\n",
      " [6.591667 3.002778 5.566667 2.058333]]\n",
      "Estimated variance for each Gaussian distribution: \n",
      " [[0.126631 0.158010 0.033088 0.009443]\n",
      " [0.252342 0.089495 0.175647 0.033903]\n",
      " [0.305764 0.112492 0.276667 0.070764]]\n"
     ]
    }
   ],
   "source": [
    "# Explore the learned probability (model parameters)\n",
    "print('Estimated probability of classess: \\n', clf.class_prior_)\n",
    "print('Estimated mean for each Gaussian distribution: \\n', clf.theta_)\n",
    "print('Estimated variance for each Gaussian distribution: \\n', clf.sigma_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, for Class 0 and the first feature, we can have the following Gaussian disribution\n",
    "$$P(x_0 \\mid Class=0) = \\frac{1}{\\sqrt{2\\pi\\cdot0.2323}} \\exp\\left(-\\frac{(x_0 - 0.6090)^2}{2\\cdot0.2323}\\right)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'X' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Input \u001b[1;32mIn [15]\u001b[0m, in \u001b[0;36m<cell line: 3>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;66;03m# Use 10-fold cross validation to show a more robust prediction accuracy\u001b[39;00m\n\u001b[0;32m      2\u001b[0m clf \u001b[38;5;241m=\u001b[39m GaussianNB()\n\u001b[1;32m----> 3\u001b[0m scores \u001b[38;5;241m=\u001b[39m cross_val_score(clf, \u001b[43mX\u001b[49m, y, scoring\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124maccuracy\u001b[39m\u001b[38;5;124m'\u001b[39m, cv\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m10\u001b[39m)\n\u001b[0;32m      4\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mGaussian Naive Bayes accuracy range: [\u001b[39m\u001b[38;5;132;01m%.4f\u001b[39;00m\u001b[38;5;124m, \u001b[39m\u001b[38;5;132;01m%.4f\u001b[39;00m\u001b[38;5;124m]; mean: \u001b[39m\u001b[38;5;132;01m%.4f\u001b[39;00m\u001b[38;5;124m; std: \u001b[39m\u001b[38;5;132;01m%.4f\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m'\u001b[39m \u001b[38;5;241m%\u001b[39m (scores\u001b[38;5;241m.\u001b[39mmin(), scores\u001b[38;5;241m.\u001b[39mmax(), scores\u001b[38;5;241m.\u001b[39mmean(), scores\u001b[38;5;241m.\u001b[39mstd()))\n",
      "\u001b[1;31mNameError\u001b[0m: name 'X' is not defined"
     ]
    }
   ],
   "source": [
    "# Use 10-fold cross validation to show a more robust prediction accuracy\n",
    "clf = GaussianNB()\n",
    "scores = cross_val_score(clf, , y, scoring='accuracy', cv=10)\n",
    "print('Gaussian Naive Bayes accuracy range: [%.4f, %.4f]; mean: %.4f; std: %.4f\\n' % (scores.min(), scores.max(), scores.mean(), scores.std()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unable to resolve this error, I have googled, considered the solution uploaded, but I am unable to find out the reason. \n",
    "\n",
    "## A humble request to kindly do the needful y considering this error while evaluating this workshop assignment. Thank you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 4\n",
    "Given the training data generated as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Means estaimated manually: \n",
      " [[-2.     -1.3333]\n",
      " [ 2.      1.3333]]\n",
      "Variances estaimated manually: \n",
      " [[0.6667 0.2222]\n",
      " [0.6667 0.2222]]\n"
     ]
    }
   ],
   "source": [
    "X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])\n",
    "y = np.array([1, 1, 1, 2, 2, 2])\n",
    "\n",
    "# Firstly, let's do the parameter estimation manually without using the model\n",
    "X_0_C_1=X[y==1][:,0]\n",
    "X_1_C_1=X[y==1][:,1]\n",
    "X_0_C_2=X[y==2][:,0]\n",
    "X_1_C_2=X[y==2][:,1]\n",
    "\n",
    "manual_means = np.array([[X_0_C_1.mean(), X_1_C_1.mean()], [X_0_C_2.mean(), X_1_C_2.mean()]])\n",
    "np.set_printoptions(precision=4)\n",
    "print('Means estaimated manually: \\n', manual_means)\n",
    "manual_vars = np.array([[X_0_C_1.var(), X_1_C_1.var()], [X_0_C_2.var(), X_1_C_2.var()]])\n",
    "print('Variances estaimated manually: \\n', manual_vars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Train a GaussianNB model and print out the learned model parameters (parameters of probability distributions). And check if the learned parameters comply with the manually estimated ones as shown above. Predict the label of a data [-0.8,-1]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimated means: \n",
      " [[-2.     -1.3333]\n",
      " [ 2.      1.3333]]\n",
      "Estimated variance: \n",
      " [[0.6667 0.2222]\n",
      " [0.6667 0.2222]]\n",
      "Class label is: 1.0000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Create and training a Gaussian Naive Bayes classifier model\n",
    "clf = GaussianNB()\n",
    "clf.fit(X, y)\n",
    "\n",
    "print('Estimated means: \\n', clf.theta_)\n",
    "print('Estimated variance: \\n', clf.sigma_)\n",
    "\n",
    "# Prediction\n",
    "X_new = [[-0.8,-1]]\n",
    "y_pred = clf.predict(X_new)\n",
    "print('Class label is: %.4f\\n' % y_pred[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### Multinomial Naive Bayes\n",
    "The [MultinomialNB](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html#sklearn.naive_bayes.MultinomialNB) classification model is suitable for classification with discrete features. To let the model handle to categorical data, we often need to transform the categorical values to numberic ones, through [encoding](https://scikit-learn.org/stable/modules/preprocessing.html#encoding-categorical-features) methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Outlook Humidity    Wind Play\n",
      "0     Sunny     High    Weak   No\n",
      "1     Sunny     High  Strong   No\n",
      "2  Overcast     High    Weak  Yes\n",
      "3      Rain     High    Weak  Yes\n",
      "4      Rain   Normal    Weak  Yes\n",
      "\n",
      "Data shape:  (14, 4)\n"
     ]
    }
   ],
   "source": [
    "# Load the weather data\n",
    "weather_data = pd.read_csv('weather.csv')\n",
    "print(weather_data.head())\n",
    "print('\\nData shape: ', weather_data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data pre-processing and preparation\n",
    "# Firstly, we need to encode categorical values\n",
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "enc = OrdinalEncoder()\n",
    "coded_data = enc.fit_transform(weather_data)\n",
    "\n",
    "X = coded_data[:, 0:-1]\n",
    "y = coded_data[:, -1]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB()"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creat and train a model\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "clf_mnb = MultinomialNB()\n",
    "clf_mnb.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy is: 0.7500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the model\n",
    "y_pred = clf_mnb.predict(X_test)\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print('Accuracy is: %.4f\\n' % accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimated probability of classess: \n",
      " [0.4 0.6]\n",
      "Estimated class-conditional probabilities for each feature: \n",
      " [[0.6364 0.1818 0.1818]\n",
      " [0.4118 0.2941 0.2941]]\n"
     ]
    }
   ],
   "source": [
    "# Explore the learned model parameters (probabilities)\n",
    "# Note that the probabilities are in the logorithmic form. Why? The log-sum-exp trick for underflow of probability products\n",
    "print('Estimated probability of classess: \\n', np.e**clf_mnb.class_log_prior_)\n",
    "print('Estimated class-conditional probabilities for each feature: \\n', np.e**clf_mnb.feature_log_prob_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 5\n",
    "Given the training data generated as follows. The number of data instances (6) is small while the demensionality of the data is relatively highly (100)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.random.randint(5, size=(6, 100))\n",
    "y = np.array([1, 2, 3, 4, 5, 6])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train a MultinomialNB model, and predict the label of a data X_new = [[1,2,1,0,2,3,0,3,2,1,1,3,3,0,4,2,2,0,0,2,2,3,4,4,4,4,0,3,3,\n",
    "          1,1,1,2,3,1,3,0,2,2,0,4,2,4,3,2,0,1,1,1,2,3,0,0,3,4,3,3,4,\n",
    "          2,1,0,0,0,0,4,1,2,0,0,4,4,0,4,1,3,1,1,1,3,1,1,1,4,3,1,1,3,\n",
    "          2,0,0,0,3,4,1,1,4,3,2,3,4]]:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# [Your code here ...]\n",
    "\n",
    "# Training a MultinomialNB model\n",
    "clf_mnb = MultinomialNB()\n",
    "clf_mnb.fit(X, y)\n",
    "\n",
    "#print('Estimated probability of classess: \\n', np.e**clf_mnb.class_log_prior_)\n",
    "#print('Estimated class-conditional probabilities for each feature: \\n', np.e**clf_mnb.feature_log_prob_)\n",
    "\n",
    "# Predict the class of the new data instance\n",
    "#X_new = np.random.randint(5, size=(1, 100))\n",
    "X_new = [[1,2,1,0,2,3,0,3,2,1,1,3,3,0,4,2,2,0,0,2,2,3,4,4,4,4,0,3,3,\n",
    "          1,1,1,2,3,1,3,0,2,2,0,4,2,4,3,2,0,1,1,1,2,3,0,0,3,4,3,3,4,\n",
    "          2,1,0,0,0,0,4,1,2,0,0,4,4,0,4,1,3,1,1,1,3,1,1,1,4,3,1,1,3,\n",
    "          2,0,0,0,3,4,1,1,4,3,2,3,4]]\n",
    "clf_mnb.predict(X_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our lecture, we discussed that if there is no occurence of some feature values, zero probabilities will appear. To overcome this issue, Laplace correction (smoothing) is proposed, as shown in the follow formula. In the [MultinomialNB](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html#sklearn.naive_bayes.MultinomialNB) implementation, the parameter 'alpha' controls the way we apply smoothing. The default value is 'alpha=1.0'. Please create and train a model with no Laplace smoothing for the above data set. Compare the leaned model parameters (probabilities) with the case 'alpha=1', by checking if there are zero probabilities (note that due to the accuracy issue, zero might be represented as a signficantly small number by the computer).\n",
    "$$p(x_{yi}|y)=\\frac{N_{yi}+\\alpha}{N_y+{\\alpha}n}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimated probability of classess: \n",
      " [0.1667 0.1667 0.1667 0.1667 0.1667 0.1667]\n",
      "Estimated class-conditional probabilities for each feature: \n",
      " [[4.6729e-03 9.3458e-03 4.6729e-13 9.3458e-03 4.6729e-03 9.3458e-03\n",
      "  9.3458e-03 1.4019e-02 4.6729e-03 4.6729e-03 1.4019e-02 1.4019e-02\n",
      "  1.8692e-02 1.4019e-02 4.6729e-03 1.4019e-02 4.6729e-13 1.8692e-02\n",
      "  1.4019e-02 4.6729e-03 4.6729e-03 4.6729e-03 9.3458e-03 4.6729e-03\n",
      "  4.6729e-03 4.6729e-03 4.6729e-03 1.8692e-02 4.6729e-03 4.6729e-13\n",
      "  4.6729e-13 4.6729e-13 4.6729e-13 1.8692e-02 1.4019e-02 9.3458e-03\n",
      "  9.3458e-03 1.4019e-02 9.3458e-03 1.8692e-02 1.8692e-02 4.6729e-03\n",
      "  1.8692e-02 1.8692e-02 1.8692e-02 1.8692e-02 1.4019e-02 1.4019e-02\n",
      "  4.6729e-13 1.4019e-02 1.8692e-02 1.4019e-02 9.3458e-03 9.3458e-03\n",
      "  1.8692e-02 4.6729e-03 1.4019e-02 9.3458e-03 9.3458e-03 4.6729e-03\n",
      "  1.8692e-02 1.4019e-02 4.6729e-03 1.8692e-02 1.8692e-02 4.6729e-03\n",
      "  1.8692e-02 1.4019e-02 9.3458e-03 4.6729e-13 4.6729e-13 9.3458e-03\n",
      "  4.6729e-03 1.4019e-02 1.8692e-02 1.8692e-02 9.3458e-03 1.8692e-02\n",
      "  1.8692e-02 9.3458e-03 4.6729e-03 4.6729e-13 4.6729e-03 1.8692e-02\n",
      "  4.6729e-03 1.8692e-02 1.8692e-02 1.4019e-02 1.8692e-02 9.3458e-03\n",
      "  4.6729e-13 4.6729e-13 9.3458e-03 1.4019e-02 1.4019e-02 4.6729e-13\n",
      "  4.6729e-03 4.6729e-13 9.3458e-03 4.6729e-13]\n",
      " [1.9048e-02 1.9048e-02 9.5238e-03 9.5238e-03 1.4286e-02 1.4286e-02\n",
      "  9.5238e-03 1.9048e-02 4.7619e-13 9.5238e-03 1.4286e-02 1.4286e-02\n",
      "  1.4286e-02 4.7619e-03 4.7619e-03 4.7619e-03 4.7619e-13 1.4286e-02\n",
      "  4.7619e-03 1.4286e-02 4.7619e-03 1.9048e-02 1.4286e-02 4.7619e-13\n",
      "  1.9048e-02 9.5238e-03 1.9048e-02 1.9048e-02 4.7619e-13 4.7619e-13\n",
      "  1.4286e-02 9.5238e-03 9.5238e-03 9.5238e-03 1.9048e-02 4.7619e-03\n",
      "  1.9048e-02 1.4286e-02 1.9048e-02 1.4286e-02 1.9048e-02 4.7619e-13\n",
      "  4.7619e-03 9.5238e-03 1.9048e-02 4.7619e-13 9.5238e-03 9.5238e-03\n",
      "  1.4286e-02 4.7619e-03 1.4286e-02 9.5238e-03 1.9048e-02 4.7619e-13\n",
      "  9.5238e-03 4.7619e-03 4.7619e-03 1.4286e-02 1.9048e-02 1.9048e-02\n",
      "  1.4286e-02 4.7619e-13 1.9048e-02 1.4286e-02 1.4286e-02 1.4286e-02\n",
      "  4.7619e-13 4.7619e-13 1.9048e-02 4.7619e-13 4.7619e-13 1.9048e-02\n",
      "  4.7619e-03 9.5238e-03 1.4286e-02 1.4286e-02 4.7619e-13 4.7619e-13\n",
      "  1.4286e-02 9.5238e-03 4.7619e-13 4.7619e-13 1.4286e-02 1.9048e-02\n",
      "  4.7619e-03 4.7619e-13 4.7619e-13 9.5238e-03 4.7619e-03 9.5238e-03\n",
      "  1.9048e-02 4.7619e-03 9.5238e-03 1.9048e-02 4.7619e-13 9.5238e-03\n",
      "  1.9048e-02 4.7619e-03 1.4286e-02 4.7619e-13]\n",
      " [1.4218e-02 1.4218e-02 1.8957e-02 4.7393e-03 9.4787e-03 9.4787e-03\n",
      "  9.4787e-03 1.8957e-02 4.7393e-13 4.7393e-03 4.7393e-13 4.7393e-13\n",
      "  9.4787e-03 9.4787e-03 9.4787e-03 1.4218e-02 1.8957e-02 9.4787e-03\n",
      "  1.8957e-02 9.4787e-03 9.4787e-03 4.7393e-03 4.7393e-13 4.7393e-13\n",
      "  1.8957e-02 4.7393e-13 9.4787e-03 1.8957e-02 1.4218e-02 1.8957e-02\n",
      "  1.8957e-02 1.8957e-02 4.7393e-13 1.8957e-02 4.7393e-13 4.7393e-13\n",
      "  9.4787e-03 4.7393e-03 9.4787e-03 1.4218e-02 9.4787e-03 9.4787e-03\n",
      "  9.4787e-03 1.4218e-02 9.4787e-03 4.7393e-03 4.7393e-03 1.4218e-02\n",
      "  4.7393e-13 4.7393e-13 4.7393e-03 4.7393e-13 9.4787e-03 1.4218e-02\n",
      "  9.4787e-03 1.8957e-02 1.4218e-02 9.4787e-03 1.8957e-02 1.4218e-02\n",
      "  1.4218e-02 1.8957e-02 1.8957e-02 1.4218e-02 4.7393e-03 9.4787e-03\n",
      "  1.8957e-02 1.4218e-02 9.4787e-03 4.7393e-03 4.7393e-03 4.7393e-03\n",
      "  1.8957e-02 9.4787e-03 1.8957e-02 1.4218e-02 4.7393e-03 1.4218e-02\n",
      "  4.7393e-13 4.7393e-13 4.7393e-13 4.7393e-13 9.4787e-03 1.4218e-02\n",
      "  4.7393e-03 1.4218e-02 4.7393e-03 4.7393e-03 4.7393e-03 1.8957e-02\n",
      "  1.4218e-02 1.8957e-02 1.8957e-02 1.8957e-02 4.7393e-13 4.7393e-03\n",
      "  1.8957e-02 1.8957e-02 4.7393e-13 9.4787e-03]\n",
      " [2.0725e-02 5.1813e-13 2.0725e-02 1.5544e-02 5.1813e-13 5.1813e-13\n",
      "  1.5544e-02 5.1813e-03 5.1813e-13 5.1813e-13 1.5544e-02 1.5544e-02\n",
      "  1.5544e-02 1.5544e-02 1.5544e-02 1.5544e-02 1.0363e-02 2.0725e-02\n",
      "  5.1813e-13 5.1813e-13 2.0725e-02 5.1813e-03 5.1813e-13 5.1813e-03\n",
      "  2.0725e-02 1.5544e-02 2.0725e-02 5.1813e-13 5.1813e-03 5.1813e-03\n",
      "  1.0363e-02 1.5544e-02 5.1813e-03 5.1813e-03 2.0725e-02 1.0363e-02\n",
      "  5.1813e-13 1.5544e-02 5.1813e-13 1.5544e-02 1.0363e-02 1.5544e-02\n",
      "  5.1813e-03 1.0363e-02 1.0363e-02 5.1813e-03 2.0725e-02 5.1813e-13\n",
      "  5.1813e-03 1.5544e-02 1.0363e-02 5.1813e-13 5.1813e-13 1.0363e-02\n",
      "  1.5544e-02 5.1813e-03 5.1813e-03 5.1813e-03 2.0725e-02 2.0725e-02\n",
      "  1.5544e-02 5.1813e-13 5.1813e-13 1.0363e-02 1.0363e-02 5.1813e-03\n",
      "  5.1813e-13 2.0725e-02 1.5544e-02 5.1813e-13 1.0363e-02 2.0725e-02\n",
      "  1.5544e-02 5.1813e-03 1.0363e-02 5.1813e-13 2.0725e-02 5.1813e-13\n",
      "  1.5544e-02 2.0725e-02 1.0363e-02 5.1813e-03 1.5544e-02 1.0363e-02\n",
      "  5.1813e-13 1.5544e-02 5.1813e-03 2.0725e-02 2.0725e-02 1.0363e-02\n",
      "  2.0725e-02 1.5544e-02 5.1813e-13 5.1813e-13 5.1813e-13 1.0363e-02\n",
      "  2.0725e-02 5.1813e-13 1.5544e-02 1.5544e-02]\n",
      " [1.8349e-02 9.1743e-03 1.8349e-02 4.5872e-13 4.5872e-03 1.8349e-02\n",
      "  1.3761e-02 9.1743e-03 9.1743e-03 1.8349e-02 4.5872e-13 1.3761e-02\n",
      "  9.1743e-03 9.1743e-03 1.8349e-02 1.8349e-02 9.1743e-03 1.3761e-02\n",
      "  1.3761e-02 4.5872e-03 1.8349e-02 1.8349e-02 1.8349e-02 9.1743e-03\n",
      "  1.3761e-02 1.3761e-02 4.5872e-03 9.1743e-03 1.8349e-02 1.8349e-02\n",
      "  9.1743e-03 4.5872e-13 4.5872e-13 4.5872e-03 1.3761e-02 1.8349e-02\n",
      "  4.5872e-13 4.5872e-13 4.5872e-03 1.8349e-02 4.5872e-13 1.3761e-02\n",
      "  4.5872e-03 9.1743e-03 9.1743e-03 4.5872e-03 4.5872e-03 4.5872e-03\n",
      "  1.8349e-02 4.5872e-03 1.8349e-02 1.8349e-02 1.3761e-02 1.8349e-02\n",
      "  4.5872e-13 4.5872e-13 4.5872e-13 1.3761e-02 4.5872e-03 1.3761e-02\n",
      "  9.1743e-03 4.5872e-13 1.8349e-02 4.5872e-03 4.5872e-03 9.1743e-03\n",
      "  1.3761e-02 1.3761e-02 4.5872e-13 1.3761e-02 1.8349e-02 1.3761e-02\n",
      "  9.1743e-03 9.1743e-03 1.3761e-02 1.3761e-02 4.5872e-13 4.5872e-13\n",
      "  4.5872e-03 1.8349e-02 9.1743e-03 1.8349e-02 1.3761e-02 4.5872e-13\n",
      "  4.5872e-03 1.3761e-02 1.8349e-02 1.8349e-02 1.3761e-02 4.5872e-03\n",
      "  1.8349e-02 4.5872e-03 9.1743e-03 4.5872e-13 9.1743e-03 4.5872e-03\n",
      "  1.3761e-02 1.3761e-02 4.5872e-03 4.5872e-03]\n",
      " [1.4019e-02 4.6729e-03 9.3458e-03 4.6729e-03 1.4019e-02 9.3458e-03\n",
      "  1.8692e-02 4.6729e-13 1.8692e-02 1.8692e-02 4.6729e-13 4.6729e-03\n",
      "  4.6729e-13 1.8692e-02 4.6729e-13 4.6729e-03 4.6729e-13 4.6729e-03\n",
      "  1.4019e-02 1.8692e-02 1.8692e-02 4.6729e-13 9.3458e-03 9.3458e-03\n",
      "  4.6729e-13 4.6729e-13 4.6729e-13 1.4019e-02 1.4019e-02 1.8692e-02\n",
      "  9.3458e-03 1.8692e-02 1.8692e-02 1.4019e-02 1.4019e-02 4.6729e-03\n",
      "  9.3458e-03 9.3458e-03 4.6729e-13 1.8692e-02 4.6729e-03 4.6729e-03\n",
      "  9.3458e-03 1.4019e-02 4.6729e-13 1.8692e-02 1.4019e-02 1.8692e-02\n",
      "  1.4019e-02 1.4019e-02 1.4019e-02 1.4019e-02 9.3458e-03 4.6729e-13\n",
      "  4.6729e-13 1.8692e-02 1.8692e-02 9.3458e-03 4.6729e-13 1.4019e-02\n",
      "  1.8692e-02 1.8692e-02 4.6729e-13 1.4019e-02 4.6729e-03 4.6729e-13\n",
      "  4.6729e-13 4.6729e-13 1.4019e-02 1.4019e-02 1.8692e-02 1.4019e-02\n",
      "  1.8692e-02 4.6729e-03 9.3458e-03 1.8692e-02 4.6729e-13 4.6729e-13\n",
      "  1.8692e-02 1.8692e-02 4.6729e-13 1.8692e-02 4.6729e-13 1.4019e-02\n",
      "  1.8692e-02 4.6729e-03 4.6729e-13 1.4019e-02 1.8692e-02 1.8692e-02\n",
      "  4.6729e-03 9.3458e-03 9.3458e-03 4.6729e-03 9.3458e-03 4.6729e-03\n",
      "  1.4019e-02 1.8692e-02 9.3458e-03 1.4019e-02]]\n"
     ]
    }
   ],
   "source": [
    "# [Your code here ...]\n",
    "\n",
    "# Create and train a MultinomialNB model with no Laplace smoothing\n",
    "clf_mnb = MultinomialNB(alpha=0)\n",
    "clf_mnb.fit(X, y)\n",
    "\n",
    "print('Estimated probability of classess: \\n', np.e**clf_mnb.class_log_prior_)\n",
    "print('Estimated class-conditional probabilities for each feature: \\n', np.e**clf_mnb.feature_log_prob_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### Comparasion on Iris data\n",
    "## Task 6\n",
    "Compare the prediction accuaracy between KNN clasifier (use the optimal K you've identied) and Gaussian Naive Bayes. Use 10-cross validation to report the accuracy mean and standard deviation (Note this is to ensure the comparison is based on robust performace). Which classifidation mdoel is more accurate on Iris data set? Use t-test to show if the difference is statistically significant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gaussian Naive Bayes accuracy range: [0.8667, 1.0000]; mean: 0.9533; std: 0.0427\n",
      "\n",
      "KNN Classifier accuracy range: [0.9333, 1.0000]; mean: 0.9800; std: 0.0306\n",
      "\n",
      "t, p: -1.5240, 0.1449\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# [Your code here ...]\n",
    "\n",
    "clf_gnb = GaussianNB()\n",
    "clf_knn = KNeighborsClassifier(n_neighbors=13)\n",
    "\n",
    "scores = cross_val_score(clf_gnb, iris_data['data'], iris_data['target'], scoring='accuracy', cv=10)\n",
    "print('Gaussian Naive Bayes accuracy range: [%.4f, %.4f]; mean: %.4f; std: %.4f\\n'\n",
    "      % (scores.min(), scores.max(), scores.mean(), scores.std()))\n",
    "scores_gnb = scores\n",
    "\n",
    "scores = cross_val_score(clf_knn, iris_data['data'], iris_data['target'], scoring='accuracy', cv=10)\n",
    "print('KNN Classifier accuracy range: [%.4f, %.4f]; mean: %.4f; std: %.4f\\n'\n",
    "      % (scores.min(), scores.max(), scores.mean(), scores.std()))\n",
    "scores_knn = scores\n",
    "\n",
    "# This is to show t-test on their performances.\n",
    "from scipy.stats import ttest_ind\n",
    "t, p = ttest_ind(scores_gnb, scores_knn)\n",
    "print ('t, p: %.4f, %.4f\\n' % (t, p))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "### The averaged accuracy of the K nearest neighbors classifier is higher than that of the Naive Bayes classifier. But in terms of t-test, the two performances of the two classifiers are not statistically significantly different if we use the significance level 0.05, as the p value 0.1449 > 0.05."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}