Lab_5_Decision_Trees.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Lab 5 - Decision Trees.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "NPpBYbohZahG",
        "FXNP_SjRZah8",
        "sTU_40txZaiA",
        "4mvofbuTZaiD",
        "3QxYp5PzZaiM",
        "sr7KLqdWZaiV",
        "cdl1AO03ZajB"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "latex_envs": {
      "LaTeX_envs_menu_present": true,
      "autoclose": false,
      "autocomplete": true,
      "bibliofile": "biblio.bib",
      "cite_by": "apalike",
      "current_citInitial": 1,
      "eqLabelWithNumbers": true,
      "eqNumInitial": 1,
      "hotkeys": {
        "equation": "Ctrl-E",
        "itemize": "Ctrl-I"
      },
      "labels_anchors": false,
      "latex_user_defs": false,
      "report_style_numbering": false,
      "user_envs_cfg": false
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SflHIEqKZahE"
      },
      "source": [
        "<h1 align=\"center\">Zewail City of Science and Technology</h1>\n",
        "<h2 align=\"center\">CIE 417 (Fall 2020)</h2>\n",
        "<h2 align=\"center\">Lab 5: Decision Trees </h2>\n",
        "<h3 align=\"center\">Eng. Ahmed Wael</h3>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NPpBYbohZahG"
      },
      "source": [
        "## Import the needed libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RIQmcJWKZahG"
      },
      "source": [
        "import numpy as np\n",
        "from collections import Counter\n",
        "from scipy import stats\n",
        "from math import log2, sqrt\n",
        "import pandas as pd\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from scipy import stats\n",
        "import seaborn as sns\n",
        "from sklearn import metrics\n",
        "from sklearn import tree\n",
        "from sklearn import preprocessing\n",
        "import matplotlib.pyplot as plt\n",
        "import six\n",
        "\n",
        "from sklearn.tree import export_graphviz\n",
        "from six import StringIO  \n",
        "from IPython.display import Image  \n",
        "import pydotplus\n",
        "from tqdm.notebook import tqdm_notebook as tqdm\n",
        "from sklearn.model_selection import GridSearchCV"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MJaOK2MDZahN"
      },
      "source": [
        "# Exploring the dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sw7HRQwoZahN"
      },
      "source": [
        "## We are going to use a very famous dataset called [Adult Dataset](https://archive.ics.uci.edu/ml/datasets/adult)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UC0xkm7BZahO"
      },
      "source": [
        "The objective of this dataset is to predict whether income exceeds 50,000 dollars per year based on census data. \n",
        "So, we will use the decision tree algorithm to predict if an individual earns more than $50K annually and is therefore eligible for the new credit card.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dObphtdMZahO"
      },
      "source": [
        "### Dataset attributes \n",
        "\n",
        "1. age : continuous variable\n",
        "2. state_sample\t: continuous variable\n",
        "3. education_num : continuous variable\n",
        "4. capital_gain : continuous variable\n",
        "5. capital_loss : continuous variable\n",
        "6. hours_per_week : continuous variable\n",
        "7. workclass : categorical variable\n",
        "8. education : categorical variable\n",
        "9. marital_status : categorical variable\n",
        "10. occupation : categorical variable\n",
        "11. relationship\t: categorical variable\n",
        "12. c1\t: categorical variable\n",
        "13. c2\t: categorical variable\n",
        "14. c3 : categorical variable"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZK8ufVkbZahP"
      },
      "source": [
        "### Load the dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PPf3gf--ZahR",
        "outputId": "4f8e3e42-2a43-4feb-8c02-89008258fedf"
      },
      "source": [
        "data = pd.read_csv(\"adult_data.csv\")\n",
        "data.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>workclass</th>\n",
              "      <th>state_sample</th>\n",
              "      <th>education</th>\n",
              "      <th>education_num</th>\n",
              "      <th>marital_status</th>\n",
              "      <th>occupation</th>\n",
              "      <th>relationship</th>\n",
              "      <th>c1</th>\n",
              "      <th>c2</th>\n",
              "      <th>capital_gain</th>\n",
              "      <th>capital_loss</th>\n",
              "      <th>hours_per_week</th>\n",
              "      <th>c3</th>\n",
              "      <th>eligible</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>39</td>\n",
              "      <td>State-gov</td>\n",
              "      <td>77516</td>\n",
              "      <td>Bachelors</td>\n",
              "      <td>13</td>\n",
              "      <td>Never-married</td>\n",
              "      <td>Adm-clerical</td>\n",
              "      <td>Not-in-family</td>\n",
              "      <td>c14</td>\n",
              "      <td>c21</td>\n",
              "      <td>2174</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>c337</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>50</td>\n",
              "      <td>Self-emp-not-inc</td>\n",
              "      <td>83311</td>\n",
              "      <td>Bachelors</td>\n",
              "      <td>13</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Exec-managerial</td>\n",
              "      <td>Husband</td>\n",
              "      <td>c14</td>\n",
              "      <td>c21</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>13</td>\n",
              "      <td>c337</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>38</td>\n",
              "      <td>Private</td>\n",
              "      <td>215646</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Divorced</td>\n",
              "      <td>Handlers-cleaners</td>\n",
              "      <td>Not-in-family</td>\n",
              "      <td>c14</td>\n",
              "      <td>c21</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>c337</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>53</td>\n",
              "      <td>Private</td>\n",
              "      <td>234721</td>\n",
              "      <td>11th</td>\n",
              "      <td>7</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Handlers-cleaners</td>\n",
              "      <td>Husband</td>\n",
              "      <td>c12</td>\n",
              "      <td>c21</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>c337</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>28</td>\n",
              "      <td>Private</td>\n",
              "      <td>338409</td>\n",
              "      <td>Bachelors</td>\n",
              "      <td>13</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Prof-specialty</td>\n",
              "      <td>Wife</td>\n",
              "      <td>c12</td>\n",
              "      <td>c20</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>c34</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   age          workclass  state_sample   education  education_num  \\\n",
              "0   39          State-gov         77516   Bachelors             13   \n",
              "1   50   Self-emp-not-inc         83311   Bachelors             13   \n",
              "2   38            Private        215646     HS-grad              9   \n",
              "3   53            Private        234721        11th              7   \n",
              "4   28            Private        338409   Bachelors             13   \n",
              "\n",
              "        marital_status          occupation    relationship   c1   c2  \\\n",
              "0        Never-married        Adm-clerical   Not-in-family  c14  c21   \n",
              "1   Married-civ-spouse     Exec-managerial         Husband  c14  c21   \n",
              "2             Divorced   Handlers-cleaners   Not-in-family  c14  c21   \n",
              "3   Married-civ-spouse   Handlers-cleaners         Husband  c12  c21   \n",
              "4   Married-civ-spouse      Prof-specialty            Wife  c12  c20   \n",
              "\n",
              "   capital_gain  capital_loss  hours_per_week    c3  eligible  \n",
              "0          2174             0              40  c337         0  \n",
              "1             0             0              13  c337         0  \n",
              "2             0             0              40  c337         0  \n",
              "3             0             0              40  c337         0  \n",
              "4             0             0              40   c34         0  "
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OwjQong-ZahV"
      },
      "source": [
        "### Describe the dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nQ58WaTZZahW",
        "outputId": "b1f9c0d2-ca5d-41f4-a8f5-6a5c48b13926"
      },
      "source": [
        "data.describe()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>state_sample</th>\n",
              "      <th>education_num</th>\n",
              "      <th>capital_gain</th>\n",
              "      <th>capital_loss</th>\n",
              "      <th>hours_per_week</th>\n",
              "      <th>eligible</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>30162.000000</td>\n",
              "      <td>3.016200e+04</td>\n",
              "      <td>30162.000000</td>\n",
              "      <td>30162.000000</td>\n",
              "      <td>30162.000000</td>\n",
              "      <td>30162.000000</td>\n",
              "      <td>30162.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>38.437902</td>\n",
              "      <td>1.897938e+05</td>\n",
              "      <td>10.121312</td>\n",
              "      <td>1092.007858</td>\n",
              "      <td>88.372489</td>\n",
              "      <td>40.931238</td>\n",
              "      <td>0.248922</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>13.134665</td>\n",
              "      <td>1.056530e+05</td>\n",
              "      <td>2.549995</td>\n",
              "      <td>7406.346497</td>\n",
              "      <td>404.298370</td>\n",
              "      <td>11.979984</td>\n",
              "      <td>0.432396</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>17.000000</td>\n",
              "      <td>1.376900e+04</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>28.000000</td>\n",
              "      <td>1.176272e+05</td>\n",
              "      <td>9.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>40.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>37.000000</td>\n",
              "      <td>1.784250e+05</td>\n",
              "      <td>10.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>40.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>47.000000</td>\n",
              "      <td>2.376285e+05</td>\n",
              "      <td>13.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>45.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>90.000000</td>\n",
              "      <td>1.484705e+06</td>\n",
              "      <td>16.000000</td>\n",
              "      <td>99999.000000</td>\n",
              "      <td>4356.000000</td>\n",
              "      <td>99.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                age  state_sample  education_num  capital_gain  capital_loss  \\\n",
              "count  30162.000000  3.016200e+04   30162.000000  30162.000000  30162.000000   \n",
              "mean      38.437902  1.897938e+05      10.121312   1092.007858     88.372489   \n",
              "std       13.134665  1.056530e+05       2.549995   7406.346497    404.298370   \n",
              "min       17.000000  1.376900e+04       1.000000      0.000000      0.000000   \n",
              "25%       28.000000  1.176272e+05       9.000000      0.000000      0.000000   \n",
              "50%       37.000000  1.784250e+05      10.000000      0.000000      0.000000   \n",
              "75%       47.000000  2.376285e+05      13.000000      0.000000      0.000000   \n",
              "max       90.000000  1.484705e+06      16.000000  99999.000000   4356.000000   \n",
              "\n",
              "       hours_per_week      eligible  \n",
              "count    30162.000000  30162.000000  \n",
              "mean        40.931238      0.248922  \n",
              "std         11.979984      0.432396  \n",
              "min          1.000000      0.000000  \n",
              "25%         40.000000      0.000000  \n",
              "50%         40.000000      0.000000  \n",
              "75%         45.000000      0.000000  \n",
              "max         99.000000      1.000000  "
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xW6Qd4hLZahb",
        "outputId": "b1225e0f-ab29-42fc-d040-cfb2735d40d8"
      },
      "source": [
        "print(f'dataset shape is {data.shape}')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "dataset shape is (30162, 15)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XEnBDMJqZahi"
      },
      "source": [
        "### Which ones are categorical and which are numerical?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4PVquIb5Zahi",
        "outputId": "8e1898dd-bd14-476f-fc72-a2abe73043a8"
      },
      "source": [
        "data.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 30162 entries, 0 to 30161\n",
            "Data columns (total 15 columns):\n",
            " #   Column          Non-Null Count  Dtype \n",
            "---  ------          --------------  ----- \n",
            " 0   age             30162 non-null  int64 \n",
            " 1   workclass       30162 non-null  object\n",
            " 2   state_sample    30162 non-null  int64 \n",
            " 3   education       30162 non-null  object\n",
            " 4   education_num   30162 non-null  int64 \n",
            " 5   marital_status  30162 non-null  object\n",
            " 6   occupation      30162 non-null  object\n",
            " 7   relationship    30162 non-null  object\n",
            " 8   c1              30162 non-null  object\n",
            " 9   c2              30162 non-null  object\n",
            " 10  capital_gain    30162 non-null  int64 \n",
            " 11  capital_loss    30162 non-null  int64 \n",
            " 12  hours_per_week  30162 non-null  int64 \n",
            " 13  c3              30162 non-null  object\n",
            " 14  eligible        30162 non-null  int64 \n",
            "dtypes: int64(7), object(8)\n",
            "memory usage: 3.5+ MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8fq7BGanZahl"
      },
      "source": [
        "## Data Exploration"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "30h0z3D_Zahm",
        "scrolled": true,
        "outputId": "9dc3cc23-0baa-4fe5-b295-0694d6278a9e"
      },
      "source": [
        "## Is the dataset balanced?\n",
        "\n",
        "sns.countplot(x='eligible',data=data)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<AxesSubplot:xlabel='eligible', ylabel='count'>"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEGCAYAAABPdROvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAQK0lEQVR4nO3df+xddX3H8eeLFhlTYUALYy2zRBsj4IZr06HuDzaS0S3ZYAqzbo5uw9QQXGayLIFlGUbTRDcdEyZk3UBatgmdP0ZnZEqq0xgJ+K0hg5YxG2HQUWkVgmAirvjeH9/z1dsv91sv38/33tvr9/lITu657/P5nPs5zbd55XPOueemqpAkab6OGfcAJEmTzSCRJDUxSCRJTQwSSVITg0SS1GTpuAcwasuWLatVq1aNexiSNFF27dr1zapa3m/boguSVatWMTU1Ne5hSNJESfI/c23z1JYkqYlBIklqYpBIkpoYJJKkJgaJJKmJQSJJamKQSJKaGCSSpCYGiSSpyaL7ZvtCWPOn28Y9BB2Fdv3VZeMegjQWzkgkSU0MEklSE4NEktTEIJEkNTFIJElNDBJJUhODRJLUxCCRJDUxSCRJTQwSSVITg0SS1MQgkSQ1MUgkSU0MEklSE4NEktTEIJEkNTFIJElNDBJJUhODRJLUxCCRJDUZWpAkOSPJ55M8mGR3kj/u6icnuSvJ17rXk3r6XJ1kb5KHklzYU1+T5P5u23VJ0tWPS3J7V78nyaphHY8kqb9hzkgOAX9SVa8BzgOuTHIWcBWws6pWAzu793TbNgBnA+uBG5Is6fZ1I7AJWN0t67v65cBTVfUq4Frg/UM8HklSH0MLkqraX1Vf7dafAR4EVgAXAVu7ZluBi7v1i4Dbquq5qnoY2AusS3I6cEJV3V1VBWyb1WdmXx8DLpiZrUiSRmMk10i6U06vA+4BTquq/TAdNsCpXbMVwGM93fZ1tRXd+uz6YX2q6hDwNHBKn8/flGQqydTBgwcX6KgkSTCCIEnyMuDjwLuq6ttHatqnVkeoH6nP4YWqLVW1tqrWLl++/EcNWZL0Igw1SJIcy3SI/FNVfaIrP9GdrqJ7PdDV9wFn9HRfCTze1Vf2qR/WJ8lS4ETgyYU/EknSXIZ511aAm4AHq+qvezbtADZ26xuBO3rqG7o7sc5k+qL6vd3pr2eSnNft87JZfWb2dQnwue46iiRpRJYOcd9vBH4PuD/JfV3tz4D3AduTXA48ClwKUFW7k2wH9jB9x9eVVfV81+8K4BbgeODOboHpoLo1yV6mZyIbhng8kqQ+hhYkVfUl+l/DALhgjj6bgc196lPAOX3q36ULIknSePjNdklSE4NEktTEIJEkNTFIJElNDBJJUhODRJLUxCCRJDUxSCRJTQwSSVITg0SS1MQgkSQ1MUgkSU0MEklSE4NEktTEIJEkNTFIJElNDBJJUhODRJLUxCCRJDUxSCRJTQwSSVITg0SS1MQgkSQ1MUgkSU0MEklSE4NEktTEIJEkNTFIJElNDBJJUhODRJLUxCCRJDUxSCRJTQwSSVITg0SS1MQgkSQ1MUgkSU0MEklSk6EFSZKbkxxI8kBP7d1J/jfJfd3y6z3brk6yN8lDSS7sqa9Jcn+37bok6erHJbm9q9+TZNWwjkWSNLdhzkhuAdb3qV9bVed2y6cBkpwFbADO7vrckGRJ1/5GYBOwultm9nk58FRVvQq4Fnj/sA5EkjS3oQVJVX0ReHLA5hcBt1XVc1X1MLAXWJfkdOCEqrq7qgrYBlzc02drt/4x4IKZ2YokaXTGcY3knUn+szv1dVJXWwE81tNmX1db0a3Prh/Wp6oOAU8Dp/T7wCSbkkwlmTp48ODCHYkkaeRBciPwSuBcYD/wwa7ebyZRR6gfqc8Li1VbqmptVa1dvnz5ixqwJOnIRhokVfVEVT1fVd8H/h5Y123aB5zR03Ql8HhXX9mnflifJEuBExn8VJokaYGMNEi6ax4zfguYuaNrB7ChuxPrTKYvqt9bVfuBZ5Kc113/uAy4o6fPxm79EuBz3XUUSdIILR3WjpN8FDgfWJZkH3ANcH6Sc5k+BfUI8A6AqtqdZDuwBzgEXFlVz3e7uoLpO8COB+7sFoCbgFuT7GV6JrJhWMciSZrb0IKkqt7ap3zTEdpvBjb3qU8B5/Spfxe4tGWMkqR2frNdktTEIJEkNTFIJElNDBJJUhODRJLUxCCRJDUxSCRJTQwSSVITg0SS1MQgkSQ1MUgkSU0GCpIkOwepSZIWnyM+tDHJTwA/yfQTfE/ihz8mdQLwM0MemyRpAvyop/++A3gX06Gxix8GybeBDw9vWJKkSXHEIKmqDwEfSvJHVXX9iMYkSZogA/0eSVVdn+QNwKrePlW1bUjjkiRNiIGCJMmtwCuB+4CZXy4swCCRpEVu0F9IXAuc5W+iS5JmG/R7JA8APz3MgUiSJtOgM5JlwJ4k9wLPzRSr6jeHMipJ0sQYNEjePcxBSJIm16B3bX1h2AORJE2mQe/aeobpu7QAXgIcC3ynqk4Y1sAkSZNh0BnJy3vfJ7kYWDeMAUmSJsu8nv5bVf8K/MrCDkWSNIkGPbX1pp63xzD9vRK/UyJJGviurd/oWT8EPAJctOCjkSRNnEGvkfzBsAciSZpMg/6w1cokn0xyIMkTST6eZOWwBydJOvoNerH9I8AOpn+XZAXwb11NkrTIDRoky6vqI1V1qFtuAZYPcVySpAkxaJB8M8nbkizplrcB3xrmwCRJk2HQIPlD4LeBbwD7gUsAL8BLkga+/fe9wMaqegogycnAB5gOGEnSIjbojOTnZkIEoKqeBF43nCFJkibJoEFyTJKTZt50M5JBZzOSpB9jgwbJB4EvJ3lvkvcAXwb+8kgdktzcfe/kgZ7ayUnuSvK17rU3nK5OsjfJQ0ku7KmvSXJ/t+26JOnqxyW5vavfk2TVizhuSdICGShIqmob8GbgCeAg8KaquvVHdLsFWD+rdhWws6pWAzu79yQ5C9gAnN31uSHJkq7PjcAmYHW3zOzzcuCpqnoVcC3w/kGORZK0sAZ++m9V7amqv62q66tqzwDtvwg8Oat8EbC1W98KXNxTv62qnquqh4G9wLokpwMnVNXdVVXAtll9Zvb1MeCCmdmKJGl05vUY+QanVdV+gO711K6+Anisp92+rraiW59dP6xPVR0CngZO6fehSTYlmUoydfDgwQU6FEkSjD5I5tJvJlFHqB+pzwuLVVuqam1VrV2+3C/kS9JCGnWQPNGdrqJ7PdDV9wFn9LRbCTze1Vf2qR/WJ8lS4EReeCpNkjRkow6SHcDGbn0jcEdPfUN3J9aZTF9Uv7c7/fVMkvO66x+Xzeozs69LgM9111EkSSM0tO+CJPkocD6wLMk+4BrgfcD2JJcDjwKXAlTV7iTbgT1M/3DWlVX1fLerK5i+A+x44M5uAbgJuDXJXqZnIhuGdSySpLkNLUiq6q1zbLpgjvabgc196lPAOX3q36ULIknS+BwtF9slSRPKIJEkNTFIJElNDBJJUhODRJLUxCCRJDUxSCRJTQwSSVITg0SS1MQgkSQ1MUgkSU0MEklSE4NEktTEIJEkNTFIJElNDBJJUhODRJLUZGi/kChp9B59z2vHPQQdhX72L+4f6v6dkUiSmhgkkqQmBokkqYlBIklqYpBIkpoYJJKkJgaJJKmJQSJJamKQSJKaGCSSpCYGiSSpiUEiSWpikEiSmhgkkqQmBokkqYlBIklqYpBIkpoYJJKkJgaJJKnJWIIkySNJ7k9yX5KprnZykruSfK17Pamn/dVJ9iZ5KMmFPfU13X72JrkuScZxPJK0mI1zRvLLVXVuVa3t3l8F7Kyq1cDO7j1JzgI2AGcD64Ebkizp+twIbAJWd8v6EY5fksTRdWrrImBrt74VuLinfltVPVdVDwN7gXVJTgdOqKq7q6qAbT19JEkjMq4gKeCzSXYl2dTVTquq/QDd66ldfQXwWE/ffV1tRbc+u/4CSTYlmUoydfDgwQU8DEnS0jF97hur6vEkpwJ3JfmvI7Ttd92jjlB/YbFqC7AFYO3atX3bSJLmZywzkqp6vHs9AHwSWAc80Z2uons90DXfB5zR030l8HhXX9mnLkkaoZEHSZKXJnn5zDrwq8ADwA5gY9dsI3BHt74D2JDkuCRnMn1R/d7u9NczSc7r7ta6rKePJGlExnFq6zTgk92dukuBf66qf0/yFWB7ksuBR4FLAapqd5LtwB7gEHBlVT3f7esK4BbgeODObpEkjdDIg6Sqvg78fJ/6t4AL5uizGdjcpz4FnLPQY5QkDe5ouv1XkjSBDBJJUhODRJLUxCCRJDUxSCRJTQwSSVITg0SS1MQgkSQ1MUgkSU0MEklSE4NEktTEIJEkNTFIJElNDBJJUhODRJLUxCCRJDUxSCRJTQwSSVITg0SS1MQgkSQ1MUgkSU0MEklSE4NEktTEIJEkNTFIJElNDBJJUhODRJLUxCCRJDUxSCRJTQwSSVITg0SS1MQgkSQ1MUgkSU0MEklSE4NEktTEIJEkNTFIJElNJj5IkqxP8lCSvUmuGvd4JGmxmeggSbIE+DDwa8BZwFuTnDXeUUnS4jLRQQKsA/ZW1der6nvAbcBFYx6TJC0qS8c9gEYrgMd63u8DfnF2oySbgE3d22eTPDSCsS0Wy4BvjnsQR4N8YOO4h6DD+bc545osxF5eMdeGSQ+Sfv869YJC1RZgy/CHs/gkmaqqteMehzSbf5ujM+mntvYBZ/S8Xwk8PqaxSNKiNOlB8hVgdZIzk7wE2ADsGPOYJGlRmehTW1V1KMk7gc8AS4Cbq2r3mIe12HjKUEcr/zZHJFUvuKQgSdLAJv3UliRpzAwSSVITg0Tz4qNpdLRKcnOSA0keGPdYFguDRC+aj6bRUe4WYP24B7GYGCSaDx9No6NWVX0ReHLc41hMDBLNR79H06wY01gkjZlBovkY6NE0khYHg0Tz4aNpJP2AQaL58NE0kn7AINGLVlWHgJlH0zwIbPfRNDpaJPkocDfw6iT7klw+7jH9uPMRKZKkJs5IJElNDBJJUhODRJLUxCCRJDUxSCRJTQwSaYiSPJJkWbf+5QHa/8PMAzCTPDtHm1uSXLKwI5Xmb6J/aleaJFX1hgHavH0UY5EWkjMSaYEkeVuSe5Pcl+Tvusft925/tns9JskNSXYn+VSST8/MMJL8R5K1PX0+mOSrSXYmWd7nM9ck+UKSXUk+k+T0YR+nNJtBIi2AJK8B3gK8sarOBZ4HfneO5m8CVgGvBd4OvH6Odi8FvlpVvwB8Abhm1mceC1wPXFJVa4Cbgc1NByLNg6e2pIVxAbAG+EoSgOOBA3O0/SXgX6rq+8A3knx+jnbfB27v1v8R+MSs7a8GzgHu6j5zCbB/vgcgzZdBIi2MAFur6urDisnvz9F2PmY/zyjA7qqaa0YjjYSntqSFsRO4JMmpAElOTvKKOdp+CXhzd63kNOD8OdodA8zcnfU7Xb9eDwHLk7y++8xjk5zdcAzSvDgjkRZAVe1J8ufAZ5McA/wfcOUczT/O9KmwB4D/Bu4Bnu7T7jvA2Ul2ddvfMuszv9ddpL8uyYlM/3/+G8AnMWukfPqvNAZJXlZVzyY5BbiX6Yv03xj3uKT5cEYijcenkvwU8BLgvYaIJpkzEklSEy+2S5KaGCSSpCYGiSSpiUEiSWpikEiSmvw/riPu6U2f/PAAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kxRk3vB3Zahr"
      },
      "source": [
        "\n",
        "categorical = ['workclass', 'education', 'marital_status', 'occupation', \n",
        "                   'relationship', 'c1', 'c2', 'c3']\n",
        "numerical = ['age', 'education_num','capital_gain', 'capital_loss',\n",
        "                'hours_per_week']\n",
        " \n",
        "X = pd.concat([data[categorical], data[numerical]], axis=1) ## Just sort them, no need for encoding\n",
        "y = data['eligible']\n",
        "X_train , X_test , y_train , y_test = train_test_split(X,y,test_size = 0.2 , random_state = 10)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kRnr-2m7Zahu"
      },
      "source": [
        "# What is Entropy ?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BZrjq9miZahu"
      },
      "source": [
        "<img src=\"https://i.ibb.co/x88jGKC/Entropy-Versus-Probability.png\">\n",
        "<br/> \n",
        "<br/> \n",
        "source: http://matlabdatamining.blogspot.com/2006/11/introduction-to-entropy.html"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7oNi2KuJZahv"
      },
      "source": [
        "$$Entropy\\ H(\\pi) = -\\sum \\pi log_2(\\pi)$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7e_A6GBe11nA"
      },
      "source": [
        "## Entropy Implementation "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KtJ5JBp4Zahw"
      },
      "source": [
        "def entropy(class_y):\n",
        "    \"\"\" \n",
        "    Input: \n",
        "        - class_y: list of class labels (0's and 1's)\n",
        "    \n",
        "    Output: \n",
        "        - The entropy\n",
        "        \n",
        "    Compute the entropy for a list of classes\n",
        "    \"\"\"\n",
        "    \n",
        "    if len(class_y) <=1: ## Handling if there is only 1 or 0 labels    (labels are the possible values for the Random Variable) \n",
        "        return 0\n",
        "    \n",
        "    total_count = np.bincount(class_y) # count (now total_count[0] has the count of 0's and total_count[1] has the count of 1's)\n",
        "    probabilities = total_count[np.nonzero(total_count)] / len(class_y) # Find the probabilities\n",
        "    if len(probabilities) <= 1 : ## Handling if the length of the probabilities is less than or equal to 1   (will this ever happen?!!)\n",
        "        return 0\n",
        "\n",
        "    return - np.sum(probabilities * np.log(probabilities)) / np.log(len(probabilities) ) ## Entropy equation"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NeTa1OXMOb-N"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AKRKcUO2Zahz"
      },
      "source": [
        "# Information Gain"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "znSgzwaHZahz"
      },
      "source": [
        "$$Conditional\\ Entropy\\ H(Y|X)= -\\sum p(Y=y, X=x )\\log\\frac{p(Y=y, X=x )}{p(X=x)}$$\n",
        "\n",
        "$$Information\\ Gain\\ IG(Y|X)= H(Y) - H(Y|X)$$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-GyKZeFQZahz"
      },
      "source": [
        "def information_gain(previous_y, current_y):\n",
        "    \"\"\"\n",
        "    Inputs:\n",
        "        - previous_y : the distribution of original labels (0's and 1's) ---->  [0,0,0,1,1,1]\n",
        "        - current_y  : the distribution of labels after splitting based on a particular    \n",
        "                     split attribute and split value   ----->  [[0,0], [1,1,1,0]] for example\n",
        "    \n",
        "    Outputs:\n",
        "        - info_gain : The information gain after partitioning\n",
        "        \n",
        "    Compute and return the information gain from partitioning the previous_y labels into the current_y labels.\n",
        "    \n",
        "    \"\"\" \n",
        "    conditional_entropy = 0 ## IG = H(Y) - CE(Y|X)\n",
        "    for y in current_y:\n",
        "        conditional_entropy += (entropy(y)*len(y)/len(previous_y))       #???\n",
        "\n",
        "    info_gain = entropy(previous_y) - conditional_entropy\n",
        "    return info_gain\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zErFf3e4Zah3"
      },
      "source": [
        "### Test the information gain and the entropy implementations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IMKabbTsZah3",
        "outputId": "d3d6c790-4583-4bd9-9229-dcce173a0adc"
      },
      "source": [
        "# TEST CASE\n",
        "test_class_y = [0,0,0,1,1,1,1,1]\n",
        "print(entropy(test_class_y))\n",
        " \n",
        "previous_y = [0,0,0,1,1,1]\n",
        "current_y = [[0,0], [1,1,1,0]]\n",
        "print(information_gain(previous_y, current_y))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0.954434002924965\n",
            "0.4591479170272448\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4DlBNkPkZah7"
      },
      "source": [
        "# The Decision Tree ID3 algorithm\n",
        "1. Find best feature\n",
        "2. Find best split for the best feature\n",
        "3. Parition classes based on steps 1 and 2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FXNP_SjRZah8"
      },
      "source": [
        "## Step 3 : partition_classes\n",
        "\n",
        "- One of the basic operations is to split a tree on one attribute - step 1 - with a specific value for that attribute - step 2.\n",
        "\n",
        "\n",
        "- We need to also take into consideration if the split attribute is either a numerical or categorical.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wf1892vLZah9"
      },
      "source": [
        "def partition_classes(X, y, split_attribute, split_val):\n",
        "    \"\"\"\n",
        "    Inputs:\n",
        "    - X               : (N,D) list containing all data attributes\n",
        "    - y               : a list of labels\n",
        "    - split_attribute : column index of the attribute to split on\n",
        "    - split_val       : either a numerical or categorical value to divide the split_attribute\n",
        "    \n",
        "    Outputs:\n",
        "    - X_left          : X left after partitioning\n",
        "    - X_right         : X right after partitioning\n",
        "    - y_left          : y left after partitioning\n",
        "    - y_right         : y right after partitioning\n",
        "    \n",
        "    Partition the data(X) and labels(y) based on the split value    \n",
        "    \"\"\"\n",
        "    \"\"\"\n",
        "    Example:\n",
        "    \n",
        "    X = [[3, 'aa', 10],                 y = [1,\n",
        "         [1, 'bb', 22],                      1,\n",
        "         [2, 'cc', 28],                      0,\n",
        "         [5, 'bb', 32],                      0,\n",
        "         [4, 'cc', 32]]                      1]\n",
        "    \n",
        "    Here, columns 0 and 2 represent numeric attributes, while column 1 is a categorical attribute.\n",
        "    \n",
        "    Consider the case where we call the function with split_attribute = 0 (the index of attribute) and split_val = 3 (the value of attribute).\n",
        "    Then we divide X into two lists - X_left, where column 0 is <= 3 and X_right, where column 0 is > 3.\n",
        "    \n",
        "    X_left = [[3, 'aa', 10],                 y_left = [1,\n",
        "              [1, 'bb', 22],                           1,\n",
        "              [2, 'cc', 28]]                           0]\n",
        "              \n",
        "    X_right = [[5, 'bb', 32],                y_right = [0,\n",
        "               [4, 'cc', 32]]                           1]\n",
        "\n",
        "    Consider another case where we call the function with split_attribute = 1 and split_val = 'bb'\n",
        "    Then we divide X into two lists, one where column 1 is 'bb', and the other where it is not 'bb'.\n",
        "        \n",
        "    X_left = [[1, 'bb', 22],                 y_left = [1,\n",
        "              [5, 'bb', 32]]                           0]\n",
        "              \n",
        "    X_right = [[3, 'aa', 10],                y_right = [1,\n",
        "               [2, 'cc', 28],                           0,\n",
        "               [4, 'cc', 32]]                           1]\n",
        "               \n",
        "    \"\"\"\n",
        "    \n",
        "    X = np.array(X)\n",
        "    column_split = X[:,split_attribute]\n",
        "    X_left=[]\n",
        "    y_right = []\n",
        "    X_right = []\n",
        "    y_left = []\n",
        "    \n",
        "    counter=0 ## Counter for appending\n",
        "\n",
        "    if isinstance(split_val,str) == False: ## Numerical attribute\n",
        "        for i in column_split:\n",
        "            if i <= split_val:\n",
        "                X_left.append(X[counter])\n",
        "                y_left.append(y[counter])\n",
        "            else:\n",
        "                X_right.append(X[counter])\n",
        "                y_right.append(y[counter])\n",
        "            counter+=1\n",
        "            \n",
        "    else: ## Categorical attribute\n",
        "        for i in column_split:\n",
        "            if i == split_val: ## == and NOT <=\n",
        "                X_left.append(X[counter])\n",
        "                y_left.append(y[counter])\n",
        "            else:\n",
        "                X_right.append(X[counter])\n",
        "                y_right.append(y[counter])\n",
        "            counter+=1\n",
        "\n",
        "    return X_left, X_right, y_left, y_right "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sTU_40txZaiA"
      },
      "source": [
        "## Step 2 : Find Best Split\n",
        "\n",
        "- In this step, we find the *order* of features splitting. \n",
        "\n",
        "\n",
        "- To do so, we calculate the information gain of each value in the attribute list. \n",
        "\n",
        "\n",
        "- Then, we find the optimal split value. This corresponds to the one with the highest information gain\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d6BTbdh7ZaiB"
      },
      "source": [
        "def find_best_split(X, y, split_attribute):\n",
        "    \n",
        "    \"\"\"\n",
        "    Inputs:\n",
        "        - X               : (N,D) list containing all data attributes\n",
        "        - y               : a list array of labels\n",
        "        - split_attribute : Column of X on which to split\n",
        "    \n",
        "    Outputs:\n",
        "        - best_split_val  : optimal split value for a given attribute\n",
        "        - best_info_gain  : the corresponding information gain\n",
        "    \n",
        "    Compute and return the optimal split value for a given attribute, along with the corresponding information gain\n",
        "\n",
        "    \"\"\"\n",
        "    \n",
        "    \"\"\"\n",
        "    \n",
        "      Example:\n",
        "    \n",
        "        X = [[3, 'aa', 10],                 y = [1,\n",
        "             [1, 'bb', 22],                      1,\n",
        "             [2, 'cc', 28],                      0,\n",
        "             [5, 'bb', 32],                      0,\n",
        "             [4, 'cc', 32]]                      1]\n",
        "    \n",
        "        split_attribute = 0\n",
        "        \n",
        "        Starting entropy: 0.971\n",
        "        \n",
        "        Calculate information gain at splits:\n",
        "           split_val = 1  -->  info_gain = 0.17\n",
        "           split_val = 2  -->  info_gain = 0.01997\n",
        "           split_val = 3  -->  info_gain = 0.01997\n",
        "           split_val = 4  -->  info_gain = 0.32\n",
        "           split_val = 5  -->  info_gain = 0\n",
        "        \n",
        "       best_split_val = 4; info_gain = .32; \n",
        "    \"\"\"\n",
        "    \n",
        "    \n",
        "    best_info_gain = 0 # Initialize the best information gain\n",
        "    X = np.array(X)\n",
        "    column_split = X[:,split_attribute]\n",
        "\n",
        "    column_split = np.unique(column_split) ## Get the unique values only !\n",
        "    best_split_val = column_split[0] ## Set the best split as the initial one ( in case it is the only one)\n",
        "\n",
        "    for split_val in column_split:\n",
        "        current_X_left, current_X_right, current_y_left, current_y_right = partition_classes(X, y, split_attribute, split_val) ## Call the partition classes function\n",
        "        current_y = []\n",
        "        current_y.append(current_y_left)\n",
        "        current_y.append(current_y_right)\n",
        "\n",
        "        current_info_gain = information_gain(y,current_y) ## Calculate the information gain\n",
        "        if current_info_gain > best_info_gain: ## check if it is better than before\n",
        "            best_info_gain = current_info_gain\n",
        "            best_split_val = split_val\n",
        "\n",
        "    return best_split_val, best_info_gain"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4mvofbuTZaiD"
      },
      "source": [
        "## Step 1 : Find Best Feature\n",
        "\n",
        "- We use steps 2 and 3 to find the best feature"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wWrtRx_fZaiE"
      },
      "source": [
        "def find_best_feature(X, y):\n",
        "    \"\"\"\n",
        "    Inputs:\n",
        "        - X              : (N,D) list containing all data attributes\n",
        "        - y              : a list of labels\n",
        "    \n",
        "    Outputs:\n",
        "        - best_feature   : Best Feature to split on\n",
        "        - best_split_val : Best Split Value for this feature\n",
        "        \n",
        "    Compute and return the optimal attribute to split on and optimal splitting value\n",
        "    \"\"\"\n",
        "    \n",
        "    \"\"\"\n",
        "        \n",
        "    Example:\n",
        "    \n",
        "        X = [[3, 'aa', 10],                 y = [1,\n",
        "             [1, 'bb', 22],                      1,\n",
        "             [2, 'cc', 28],                      0,\n",
        "             [5, 'bb', 32],                      0,\n",
        "             [4, 'cc', 32]]                      1]\n",
        "    \n",
        "        split_attribute = 0\n",
        "        \n",
        "        Starting entropy: 0.971\n",
        "        \n",
        "        Calculate information gain at splits:\n",
        "           feature 0:  -->  info_gain = 0.32\n",
        "           feature 1:  -->  info_gain = 0.17\n",
        "           feature 2:  -->  info_gain = 0.4199\n",
        "        \n",
        "       best_split_feature: 2 best_split_val: 22\n",
        "    \"\"\"\n",
        "    \n",
        "    \n",
        "    best_info_gain = 0\n",
        "    best_feature = 0\n",
        "    best_split_val = 0\n",
        "    for feature_index in range(len(X[0])): ## Loop over the features , find the best split \n",
        "        current_best_split_val, current_best_info_gain =  find_best_split(X, y,  feature_index)\n",
        "        if current_best_info_gain > best_info_gain:\n",
        "            best_info_gain = current_best_info_gain\n",
        "            best_feature = feature_index\n",
        "            best_split_val = current_best_split_val\n",
        "    return best_feature, best_split_val"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jFnM5rQWZaiG"
      },
      "source": [
        "# Decision Tree ID3 Full Algorithm : Putting Everything Together"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uu1wESqIZaiG"
      },
      "source": [
        "- We will use a recursive algorithm, so we need to have a base case - a stopping condition for splitting- . It can be one of the following:\n",
        "\n",
        "    - If all data points in that node have the same label\n",
        "\n",
        "    - If the current node is at the maximum depth. In this case, you may assign the mode of the labels as the class label\n",
        "\n",
        "\n",
        "- We are going to use a class, so you might need to revise the OOP part in the first lab.\n",
        "\n",
        "\n",
        "- We are going to use a dictionary to store the tree information. It will contain the class labels, feature and value to split, and reference to the right and left children.\n",
        "\n",
        "- For example:\n",
        "```\n",
        " node = {\n",
        "            'isLeaf': False,\n",
        "            'split_attribute': 1,\n",
        "            'split_value': 3,\n",
        "            'is_categorical': True,\n",
        "            'leftTree': leftTree,  --> This is a instance of the class itself\n",
        "            'rightTree': rightTree --> This is a instance of the class itself\n",
        "        };\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uAazOKWZZaiH"
      },
      "source": [
        "<img src=\"https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1545934190/1_r5ikdb.png\">\n",
        "<br/> \n",
        "<br/> \n",
        "source: https://www.datacamp.com/community/tutorials/decision-tree-classification-python"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gyzBMR2cZaiH"
      },
      "source": [
        "class MyDecisionTree(object):\n",
        "    def __init__(self, max_depth=None):\n",
        "        \"\"\"        \n",
        "        Args:\n",
        "        \n",
        "        max_depth: maximum depth of the tree including the root node.\n",
        "        \"\"\"\n",
        "        self.tree = {}\n",
        "        self.residual_tree = {} ## For prediction\n",
        "        self.max_depth = max_depth\n",
        "\n",
        "        \n",
        "    def fit(self, X, y, depth):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "        X     : N*D matrix corresponding to the data points\n",
        "        Y     : N*1 array corresponding to the labels of the data points\n",
        "        depth : depth of node of the tree\n",
        "        \n",
        "        Output:\n",
        "        node : A dictionary that contains this node information\n",
        "        \"\"\"\n",
        "\n",
        "        # base cases\n",
        "        unique_labels = np.unique(y) \n",
        "        if (len(unique_labels)==1) or (depth == max_depth): ## IF we reach the maximum depth or the unique labels = 1\n",
        "            unique_labels, counts_unique_labels = np.unique(y, return_counts=True)\n",
        "            index = counts_unique_labels.argmax()\n",
        "            classification = unique_labels[index]    \n",
        "            return classification      #majority voting?\n",
        "    \n",
        "        best_feat, best_split  = find_best_feature(X, y) ## Find best feature (Step 1)\n",
        "        best_split, information_gain = find_best_split(X, y, best_feat) ## Split on this feature (Step 2)\n",
        "        X_left, X_right, y_left, y_right = partition_classes(X, y, best_feat, best_split) ## Partition on the best feature and split (Step 3)\n",
        "        \n",
        "        if isinstance(best_split,str):\n",
        "            question = \"{} == {}\".format(best_feat, best_split) ## Represnt the sub-tree as a question and an answer\n",
        "        else:\n",
        "            question = \"{} <= {}\".format(best_feat, best_split) ## Represnt the sub-tree as a question and an answer\n",
        "        node = {question: []}\n",
        "        \n",
        "        # find answers (recursion)\n",
        "        depth+=1 \n",
        "        yes_answer = self.fit(X_left,y_left, depth)  # RECURSION on the left sub tree\n",
        "        no_answer = self.fit(X_right, y_right, depth) ## RECURSION on the right sub tree\n",
        "        \n",
        "        if yes_answer == no_answer: ## Both trees are the same\n",
        "            node = yes_answer\n",
        "        else:\n",
        "            node[question].append(yes_answer) ## Append the questions to the answers\n",
        "            node[question].append(no_answer)\n",
        "        self.tree = node ## The tree is equal to the node\n",
        "        return node\n",
        "        \n",
        "    def predict(self, record,flag=1):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "        \n",
        "        record: D*1, a single data point that should be classified\n",
        "        \n",
        "        Output:\n",
        "        prediction: True if the predicted class label is 1, False otherwise      \n",
        "        \n",
        "        classify a sample in test data set using self.tree and return the predicted label\n",
        "        \"\"\"\n",
        "        if flag == 1: ## First time\n",
        "            self.residual_tree = self.tree\n",
        "        question = list(self.residual_tree.keys())[0]\n",
        "        feature, comparison, value = question.split() ## Split the question to get the feature and its value\n",
        "\n",
        "        \n",
        "        ## Check if str or int\n",
        "        if comparison == \"==\": # String\n",
        "            if record[int(feature)] == value:\n",
        "                answer = self.residual_tree[question][0]  ## Left sub tree\n",
        "            else:\n",
        "                answer = self.residual_tree[question][1] ## Right sub-tree\n",
        "        elif comparison == \"<=\": #Integer\n",
        "            if record[int(feature)] <= float(value):\n",
        "                answer = self.residual_tree[question][0] ## Right Sub-tree\n",
        "            else:\n",
        "                answer = self.residual_tree[question][1] ## Left sub-tree        try: ## Integer\n",
        "                \n",
        "        # base case\n",
        "        if not isinstance(answer, dict):  ## If we have the answer\n",
        "            return answer\n",
        "    \n",
        "        # recursion\n",
        "        else:\n",
        "            self.residual_tree = answer ## The residual tree is the answer !\n",
        "            return self.predict(record,0) ## have flag = 0 so the residual tree is our sub-problem"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3QxYp5PzZaiM"
      },
      "source": [
        "### Evaluate the ID3 implementation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZIw7gxOXZaiM"
      },
      "source": [
        "def DecisionTreeEvalution(id3,X,y, verbose=False):\n",
        "\n",
        "    y_predicted = []\n",
        "    for record in X: \n",
        "        y_predicted.append(id3.predict(record))\n",
        "    # Comparing predicted and true labels\n",
        "    results = [prediction == truth for prediction, truth in zip(y_predicted, y)]\n",
        "    # Accuracy\n",
        "    accuracy = float(results.count(True)) / float(len(results))\n",
        "    if verbose:\n",
        "        print(\"accuracy: %.4f\" % accuracy)\n",
        "    return accuracy"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l_bYeL5-ZaiP",
        "outputId": "f2e5dad2-5e7b-46f3-f987-247218327c86"
      },
      "source": [
        "#### Convert the dataframes to numpy arrays as our implementation works on NumPy arrays\n",
        "X_train , X_test , y_train , y_test = X_train.to_numpy() , X_test.to_numpy() , y_train.to_numpy() , y_test.to_numpy()\n",
        "max_depth = 7\n",
        "inital_depth = 0\n",
        "id3_dt = MyDecisionTree(max_depth)\n",
        "# Building a tree\n",
        "print(\"fitting the decision tree\")\n",
        "id3_dt.fit(X_train, y_train, inital_depth)\n",
        "\n",
        "# Evaluating the decision tree\n",
        "DecisionTreeEvalution(id3_dt,X_test,y_test, True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "fitting the decision tree\n",
            "accuracy: 0.7956\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "0.7956240676280457"
            ]
          },
          "execution_count": 31,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sr7KLqdWZaiV"
      },
      "source": [
        "## What is the final tree"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "739SulVKZaiW",
        "outputId": "9372a06f-0d0c-4a8a-adf0-0cfca87a143f"
      },
      "source": [
        "id3_dt.tree"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'2 ==  Married-civ-spouse': [{'9 <= 12': [{'10 <= 5013': [{'9 <= 8': [{'8 <= 36': [{'3 ==  Tech-support': [{'8 <= 30': [0,\n",
              "              1]},\n",
              "            {'7 == c323': [1, 0]}]},\n",
              "          0]},\n",
              "        {'8 <= 33': [{'8 <= 24': [0, {'11 <= 1740': [0, 1]}]},\n",
              "          {'11 <= 1740': [0, {'11 <= 1977': [1, 0]}]}]}]},\n",
              "      {'8 <= 61': [{'1 ==  Preschool': [0, 1]},\n",
              "        {'10 <= 10566': [{'10 <= 9386': [{'0 ==  Local-gov': [0, 1]}, 0]},\n",
              "          1]}]}]},\n",
              "    {'10 <= 5013': [{'11 <= 1740': [{'12 <= 30': [{'4 ==  Wife': [1, 0]},\n",
              "          {'3 ==  Exec-managerial': [{'11 <= 0': [1, 0]}, 1]}]},\n",
              "        {'11 <= 1977': [{'8 <= 28': [{'4 ==  Husband': [1, 0]}, 1]},\n",
              "          {'11 <= 2051': [0, {'0 ==  Local-gov': [0, 1]}]}]}]},\n",
              "      {'8 <= 62': [1,\n",
              "        {'12 <= 45': [1, {'0 ==  Self-emp-not-inc': [0, 1]}]}]}]}]},\n",
              "  {'10 <= 6849': [{'9 <= 12': [{'8 <= 28': [{'0 ==  Private': [{'12 <= 49': [0,\n",
              "            {'7 == c330': [1, 0]}]},\n",
              "          0]},\n",
              "        {'12 <= 41': [0, {'11 <= 2206': [0, {'8 <= 34': [0, 1]}]}]}]},\n",
              "      {'8 <= 27': [{'11 <= 1974': [{'12 <= 39': [0,\n",
              "            {'2 ==  Married-AF-spouse': [1, 0]}]},\n",
              "          {'3 ==  Prof-specialty': [0, 1]}]},\n",
              "        {'12 <= 44': [{'9 <= 14': [{'11 <= 2206': [0, 1]}, 0]},\n",
              "          {'11 <= 2339': [{'9 <= 14': [0, 1]}, 1]}]}]}]},\n",
              "    {'9 <= 10': [{'10 <= 7978': [{'0 ==  Local-gov': [1, 0]},\n",
              "        {'8 <= 19': [0,\n",
              "          {'12 <= 56': [1, {'0 ==  Self-emp-not-inc': [0, 1]}]}]}]},\n",
              "      1]}]}]}"
            ]
          },
          "execution_count": 18,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IaDoqFwLZaii"
      },
      "source": [
        "# Compare with Sklearn Decision Tree"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DLtCd6LKZaij"
      },
      "source": [
        "Sklearn decision tree implementation can't handle categorical data \n",
        "\n",
        "\n",
        "https://github.com/scikit-learn/scikit-learn/issues/5442\n",
        "\n",
        "https://stackoverflow.com/questions/38108832/passing-categorical-data-to-sklearn-decision-tree"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JnBN_WMoZaij",
        "scrolled": true
      },
      "source": [
        "X_train , X_test , y_train , y_test = train_test_split(X,y,test_size = 0.2 , random_state = 10)\n",
        "\n",
        "X_train = pd.get_dummies(X_train,drop_first=True)\n",
        "X_test = pd.get_dummies(X_test,drop_first=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1w8ab6GIZaim",
        "outputId": "9b16a5bc-4731-4621-aa49-26511a397721"
      },
      "source": [
        "clf = tree.DecisionTreeClassifier( criterion='entropy', max_depth = max_depth)\n",
        "clf.fit(X_train, y_train)\n",
        "y_pred = clf.predict(X_test)\n",
        "print(\"Accuracy:\",metrics.accuracy_score(y_test, y_pred))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 0.8534725675451682\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xz1zQu3kZaip"
      },
      "source": [
        "# Visualize the tree"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "code_folding": [],
        "id": "DStI5xl8Zaiq",
        "scrolled": true,
        "outputId": "2bf087c2-0142-4946-ec9c-337584d2b873"
      },
      "source": [
        "feature_names = list(X_train.columns)\n",
        "\n",
        "dot_data = StringIO()\n",
        "export_graphviz(clf, out_file=dot_data,  \n",
        "                filled=True, rounded=True,\n",
        "                special_characters=True,feature_names = feature_names,class_names=['0','1'])\n",
        "graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  \n",
        "graph.write_png('credit_tree.png')\n",
        "Image(graph.create_png())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "ename": "NameError",
          "evalue": "name 'pydotplus' is not defined",
          "output_type": "error",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-34-dde64d2182e7>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      5\u001b[0m                 \u001b[0mfilled\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mrounded\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m                 special_characters=True,feature_names = feature_names,class_names=['0','1'])\n\u001b[1;32m----> 7\u001b[1;33m \u001b[0mgraph\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpydotplus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgraph_from_dot_data\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdot_data\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgetvalue\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      8\u001b[0m \u001b[0mgraph\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwrite_png\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'credit_tree.png'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[0mImage\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mgraph\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcreate_png\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;31mNameError\u001b[0m: name 'pydotplus' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wNIkus7hZai2"
      },
      "source": [
        "# Plot the features importance"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k9GxtfYVZai2",
        "scrolled": true,
        "outputId": "102a49d4-fca9-403e-960b-6e195cd3b255"
      },
      "source": [
        "feature_names = list(X_train.columns)\n",
        "\n",
        "importances = clf.feature_importances_ ## most important line in this cell\n",
        "\n",
        "indices = np.argsort(importances)[::-1]\n",
        "for f in range(X_train.shape[1]):\n",
        "    print(f'{f+1} feature {feature_names[indices[f]]} {importances[indices[f]]}')\n",
        "\n",
        "\n",
        "plt.figure(figsize=(20, 20))\n",
        "plt.title(\"Feature importances\")\n",
        "plt.bar(range(X_train.shape[1]), importances[indices],\n",
        "        color=\"r\", align=\"center\")\n",
        "plt.xticks(range(X_train.shape[1]), indices)\n",
        "plt.xlim([-1, X_train.shape[1]])\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "1 feature marital_status_ Married-civ-spouse 0.42598171378711264\n",
            "2 feature education_num 0.20347421738241356\n",
            "3 feature capital_gain 0.19763928810771172\n",
            "4 feature age 0.06635900558173644\n",
            "5 feature capital_loss 0.05179449844577836\n",
            "6 feature hours_per_week 0.03032587545929722\n",
            "7 feature occupation_ Exec-managerial 0.006505086074851692\n",
            "8 feature relationship_ Wife 0.0035377261581412077\n",
            "9 feature occupation_ Prof-specialty 0.002989181157133509\n",
            "10 feature workclass_ Private 0.002001788168732481\n",
            "11 feature occupation_ Transport-moving 0.0014692789344171583\n",
            "12 feature occupation_ Machine-op-inspct 0.0014266221811375876\n",
            "13 feature education_ Preschool 0.0011482603771392152\n",
            "14 feature occupation_ Other-service 0.0008713530897137304\n",
            "15 feature workclass_ Self-emp-not-inc 0.0007441716952441054\n",
            "16 feature c3_c330 0.0007343534616974588\n",
            "17 feature marital_status_ Married-AF-spouse 0.0006246656367571916\n",
            "18 feature c3_c336 0.000549269849029299\n",
            "19 feature workclass_ Local-gov 0.0005483642526015628\n",
            "20 feature occupation_ Tech-support 0.0005300222369840512\n",
            "21 feature occupation_ Sales 0.0004138827656782659\n",
            "22 feature c3_c338 0.00033137519669127657\n",
            "23 feature education_ 9th 0.0\n",
            "24 feature marital_status_ Widowed 0.0\n",
            "25 feature occupation_ Protective-serv 0.0\n",
            "26 feature workclass_ Self-emp-inc 0.0\n",
            "27 feature occupation_ Priv-house-serv 0.0\n",
            "28 feature workclass_ State-gov 0.0\n",
            "29 feature workclass_ Without-pay 0.0\n",
            "30 feature occupation_ Handlers-cleaners 0.0\n",
            "31 feature occupation_ Farming-fishing 0.0\n",
            "32 feature education_ 11th 0.0\n",
            "33 feature occupation_ Craft-repair 0.0\n",
            "34 feature occupation_ Armed-Forces 0.0\n",
            "35 feature marital_status_ Separated 0.0\n",
            "36 feature education_ Assoc-acdm 0.0\n",
            "37 feature marital_status_ Never-married 0.0\n",
            "38 feature marital_status_ Married-spouse-absent 0.0\n",
            "39 feature education_ 12th 0.0\n",
            "40 feature education_ 1st-4th 0.0\n",
            "41 feature education_ Some-college 0.0\n",
            "42 feature education_ Prof-school 0.0\n",
            "43 feature education_ 5th-6th 0.0\n",
            "44 feature education_ Masters 0.0\n",
            "45 feature education_ HS-grad 0.0\n",
            "46 feature education_ 7th-8th 0.0\n",
            "47 feature education_ Bachelors 0.0\n",
            "48 feature education_ Assoc-voc 0.0\n",
            "49 feature education_ Doctorate 0.0\n",
            "50 feature c3_c39 0.0\n",
            "51 feature relationship_ Not-in-family 0.0\n",
            "52 feature c3_c322 0.0\n",
            "53 feature c3_c324 0.0\n",
            "54 feature c3_c325 0.0\n",
            "55 feature c3_c326 0.0\n",
            "56 feature c3_c327 0.0\n",
            "57 feature c3_c328 0.0\n",
            "58 feature c3_c329 0.0\n",
            "59 feature c3_c33 0.0\n",
            "60 feature c3_c331 0.0\n",
            "61 feature c3_c332 0.0\n",
            "62 feature c3_c333 0.0\n",
            "63 feature c3_c334 0.0\n",
            "64 feature c3_c335 0.0\n",
            "65 feature c3_c337 0.0\n",
            "66 feature c3_c339 0.0\n",
            "67 feature c3_c34 0.0\n",
            "68 feature c3_c340 0.0\n",
            "69 feature c3_c35 0.0\n",
            "70 feature c3_c36 0.0\n",
            "71 feature c3_c37 0.0\n",
            "72 feature c3_c323 0.0\n",
            "73 feature c3_c321 0.0\n",
            "74 feature relationship_ Other-relative 0.0\n",
            "75 feature c3_c320 0.0\n",
            "76 feature c3_c38 0.0\n",
            "77 feature relationship_ Unmarried 0.0\n",
            "78 feature c1_c11 0.0\n",
            "79 feature c1_c12 0.0\n",
            "80 feature c1_c13 0.0\n",
            "81 feature c1_c14 0.0\n",
            "82 feature c2_c21 0.0\n",
            "83 feature c3_c31 0.0\n",
            "84 feature c3_c310 0.0\n",
            "85 feature c3_c311 0.0\n",
            "86 feature c3_c312 0.0\n",
            "87 feature c3_c313 0.0\n",
            "88 feature c3_c314 0.0\n",
            "89 feature c3_c315 0.0\n",
            "90 feature c3_c316 0.0\n",
            "91 feature c3_c317 0.0\n",
            "92 feature c3_c318 0.0\n",
            "93 feature c3_c319 0.0\n",
            "94 feature c3_c32 0.0\n",
            "95 feature relationship_ Own-child 0.0\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABIcAAARuCAYAAABJOlfRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABF30lEQVR4nO3dffys510X+M+3CeEhtFRIWkrSkgKBGt1Su4cASylUhG2qEhArrUCXBzfblYpVqhZ1pS6riIB2cauxW7IqCFWBdqMU0oIKrqWYEyhpUho8hGBOU8jpA8VSbBt67R8zJ50zv/u+Z37nIeck3/f79ZrX+c3M977muq/rfpjf59wzvxpjBAAAAICeHnW+OwAAAADA+SMcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAABrVfVXq+pV57sfAAAPpRpjnO8+AACPAFV1T5LHJ/m9jYc/c4xx3xm2+WfGGD91Zr17+KmqlyX5jDHG157vvgAAj2yuHAIAzqY/Psb4+I3baQdDZ0NVXXw+X/90PVz7DQA8PAmHAIBzqqo+oaq+v6reUVVvr6r/o6ouWj/36VX1b6vqXVX1zqr651X12PVzP5DkSUn+dVW9r6r+clV9cVUd32r/nqr6I+ufX1ZVP1JVP1hVv53k65def6KvL6uqH1z/fFVVjar6hqq6t6reU1UvrKrPqarbq+q3qur/2lj266vqP1bVP6iq91bV26rqSzae/5Squrmq3l1Vx6rqf9563c1+vzDJX03y1et1/6V13TdU1S9X1X+tqrur6n/ZaOOLq+p4VX1rVd2/Xt9v2Hj+Y6vqe6vq19f9+/+q6mPXz31eVb1xvU6/VFVfvLVed69f89eq6msOtQEAABc8/ysFAJxr/zTJbyb5jCSXJvk3Se5N8o+TVJLvTPKzSR6T5EeTvCzJi8cYX1dVX5iNj5VthhYLrk/y3CQvSPLRSX544fX38blJrk7yzCQ3J/nJJH8kyUcl+cWq+ldjjJ/ZqP2RJJcl+RNJfqyqnjzGePe6H3cm+ZQkT0nyhqq6e4zx0zP9viwHP1Z2f5I/luTudX9+oqpuHWP8wvr5T07yCUmuSPKlSX6kql47xnhPku9J8geS/A9JfmPd1w9X1RVJfjzJ163X7UuS/GhVPSXJ+5N8X5LPGWPcVVVPSPKJe44bAPAw4cohAOBseu366pPfqqrXVtXjk1yXVdjzO2OM+5P8/STPS5IxxrExxhvGGB8YY5xI8veSfNEZ9uHnxhivHWN8OKvAafb19/QdY4z/NsZ4fZLfSfLDY4z7xxhvT/Ifkvyhjdr7k7x8jPGhMca/SHJXkj9aVU9M8owkf2Xd1puTvCqrQOZAv8cYvzvVkTHGj48xfnWs/EyS1yf5wo2SDyX539ev/7ok70vyWVX1qCTfmOTPjzHePsb4vTHGG8cYH0jytUleN8Z43fq135DkaJLnrNv8cJI/WFUfO8Z4xxjjzkOMHQDwMODKIQDgbPqKzS+Prqprs7rC5h1VdfLhR2V15U6q6nFZXZnyhUkevX7uPWfYh3s3fv7Updff029u/Py7E/c/fuP+28epf+3j17O6UuhTkrx7jPFft547MtPvSVV1XZJvT/KZWa3HxyV5y0bJu8YYD2zcf/+6f5cl+ZgkvzrR7KcmeW5V/fGNxz4qyb8bY/xOVX11kpck+f6q+o9JvnWM8bZdfQUAHj5cOQQAnEv3JvlAksvGGI9d3x4zxvgD6+e/M8lI8tQxxmOyuoqlNpbf/rOqv5NVIJIkWX930OVbNZvL7Hr9s+2K2kihsvrOpPvWt0+sqkdvPff2mX4fuF9VH53Vx+6+J8njxxiPTfK6nDpec96Z5L8l+fSJ5+5N8gMb4/PYMcalY4y/kyRjjFvGGF+a5AlJ3pbk/97j9QCAhxHhEABwzowx3pHVR5++t6oeU1WPWn8J9cmPjj06q48+/db6u2/+0lYTv5nk0zbu/0qSj6mqP1pVH5Xkr2f1/Tyn+/pn2+OSfEtVfVRVPTfJ78/qI1v3Jnljku+sqo+pqqcm+aYk/3yhrd9MctX6I2FJcklW63oiyQPrq4i+bJ9OrT9id1OSv7f+YuyLqurz14HTDyb541X1P64f/5j1l1tfWVWPr6ovr6pLswrZ3pfk9w45JgDABU44BACcay/IKth4a1YfGfuRrK5CSZK/meTpSd6b1Zci/9jWst+Z5K+vv8PoJWOM9yb5s1l9X8/bs7qS6HiWLb3+2fbzWX159TuT/K0kf3KM8a71c89PclVWVxG9Jsm3r7/fZ86/Wv/7rqr6hfVH0r4lyb/Maj3+dFZfkL2vl2T1EbRbk7w7yXcledQ6uLo+q7+OdiKrK4n+UlbvEx+V5FvXfX53Vt8H9WcP8ZoAwMNAnfqxeAAATkdVfX1Wf1ntGee7LwAAh+HKIQAAAIDGhEMAAAAAjflYGQAAAEBjrhwCAAAAaEw4BAAAANDYxee7A1Muu+yycdVVV53vbgAAAAA8Ytx2223vHGNcvv34BRkOXXXVVTl69Oj57gYAAADAI0ZV/frU4z5WBgAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANDYxee7A4dWtbtmjHPfDwAAAIBHAFcOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0Nhe4VBVPbuq7qqqY1X10oW6z6mq36uqP3nYZQEAAAB46O0Mh6rqoiSvSHJdkmuSPL+qrpmp+64ktxx2WQAAAADOj32uHLo2ybExxt1jjA8meXWS6yfq/lySH01y/2ksCwAAAMB5sE84dEWSezfuH18/9qCquiLJVya58bDLAgAAAHD+7BMO1cRjY+v+y5P8lTHG753GsqvCqhuq6mhVHT1x4sQe3QIAAADgTF28R83xJE/cuH9lkvu2ao4keXVVJcllSZ5TVQ/suWySZIzxyiSvTJIjR45MBkgAAAAAnF37hEO3Jrm6qp6c5O1JnpfkT28WjDGefPLnqvonSf7NGOO1VXXxrmUBAAAAOH92hkNjjAeq6kVZ/RWyi5LcNMa4s6peuH5++3uGdi57droOAAAAwJmqMS68T3AdOXJkHD16dPrJmvoaoy0X4DoBAAAAnE9VddsY48j24/t8ITUAAAAAj1DCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGN7hUNV9eyququqjlXVSyeev76qbq+qN1fV0ap6xsZz91TVW04+dzY7DwAAAMCZuXhXQVVdlOQVSb40yfEkt1bVzWOMt26U/XSSm8cYo6qemuRfJnnKxvPPGmO88yz2GwAAAICzYJ8rh65NcmyMcfcY44NJXp3k+s2CMcb7xhhjfffSJCMAAAAAXPD2CYeuSHLvxv3j68dOUVVfWVVvS/LjSb5x46mR5PVVdVtV3TD3IlV1w/ojaUdPnDixX+8BAAAAOCP7hEM18diBK4PGGK8ZYzwlyVck+Y6Np75gjPH0JNcl+eaqeubUi4wxXjnGODLGOHL55Zfv0S0AAAAAztQ+4dDxJE/cuH9lkvvmiscYP5vk06vqsvX9+9b/3p/kNVl9TA0AAACAC8A+4dCtSa6uqidX1SVJnpfk5s2CqvqMqqr1z09PckmSd1XVpVX16PXjlyb5siR3nM0VAAAAAOD07fxrZWOMB6rqRUluSXJRkpvGGHdW1QvXz9+Y5KuSvKCqPpTkd5N89fovlz0+yWvWudHFSX5ojPGT52hdAAAAADik+sgfGbtwHDlyZBw9enT6yZr6CqQtF+A6AQAAAJxPVXXbGOPI9uP7fKwMAAAAgEco4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0Nhe4VBVPbuq7qqqY1X10onnr6+q26vqzVV1tKqese+yAAAAAJw/O8OhqrooySuSXJfkmiTPr6prtsp+OslnjzGeluQbk7zqEMsCAAAAcJ7sc+XQtUmOjTHuHmN8MMmrk1y/WTDGeN8YY6zvXppk7LssAAAAAOfPPuHQFUnu3bh/fP3YKarqK6vqbUl+PKurh/ZeFgAAAIDzY59wqCYeGwceGOM1Y4ynJPmKJN9xmGWTpKpuWH9f0dETJ07s0S0AAAAAztQ+4dDxJE/cuH9lkvvmiscYP5vk06vqssMsO8Z45RjjyBjjyOWXX75HtwAAAAA4U/uEQ7cmubqqnlxVlyR5XpKbNwuq6jOqqtY/Pz3JJUnetc+yAAAAAJw/F+8qGGM8UFUvSnJLkouS3DTGuLOqXrh+/sYkX5XkBVX1oSS/m+Sr119QPbnsOVoXAAAAAA6pPvJHxi4cR44cGUePHp1+sqa+xmjLBbhOAAAAAOdTVd02xjiy/fg+HysDAAAA4BFKOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhsr3Coqp5dVXdV1bGqeunE819TVbevb2+sqs/eeO6eqnpLVb25qo6ezc4DAAAAcGYu3lVQVRcleUWSL01yPMmtVXXzGOOtG2W/luSLxhjvqarrkrwyyeduPP+sMcY7z2K/AQAAADgL9rly6Nokx8YYd48xPpjk1Umu3ywYY7xxjPGe9d03Jbny7HYTAAAAgHNhn3DoiiT3btw/vn5szjcl+YmN+yPJ66vqtqq6YW6hqrqhqo5W1dETJ07s0S0AAAAAztTOj5UlqYnHxmRh1bOyCoeesfHwF4wx7quqxyV5Q1W9bYzxswcaHOOVWX0cLUeOHJlsHwAAAICza58rh44neeLG/SuT3LddVFVPTfKqJNePMd518vExxn3rf+9P8pqsPqYGAAAAwAVgn3Do1iRXV9WTq+qSJM9LcvNmQVU9KcmPJfm6McavbDx+aVU9+uTPSb4syR1nq/MAAAAAnJmdHysbYzxQVS9KckuSi5LcNMa4s6peuH7+xiR/I8knJfmHVZUkD4wxjiR5fJLXrB+7OMkPjTF+8pysCQAAAACHVmNceF/vc+TIkXH06NHpJ2vqK5C2XIDrBAAAAHA+VdVt64t5TrHPx8oAAAAAeIQSDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaOzi892Bc6pqd80Y574fAAAAABcoVw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoLG9wqGqenZV3VVVx6rqpRPPf01V3b6+vbGqPnvfZQEAAAA4f3aGQ1V1UZJXJLkuyTVJnl9V12yV/VqSLxpjPDXJdyR55SGWBQAAAOA82efKoWuTHBtj3D3G+GCSVye5frNgjPHGMcZ71nfflOTKfZcFAAAA4PzZJxy6Ism9G/ePrx+b801JfuI0lwUAAADgIXTxHjU18diYLKx6Vlbh0DNOY9kbktyQJE960pP26BYAAAAAZ2qfK4eOJ3nixv0rk9y3XVRVT03yqiTXjzHedZhlk2SM8coxxpExxpHLL798n74DAAAAcIb2CYduTXJ1VT25qi5J8rwkN28WVNWTkvxYkq8bY/zKYZYFAAAA4PzZ+bGyMcYDVfWiJLckuSjJTWOMO6vqhevnb0zyN5J8UpJ/WFVJ8sD6KqDJZc/RugAAAABwSDXG5FcAnVdHjhwZR48enX6ypr7GaMvJdTpMLQAAAMAjWFXdNsY4sv34Pl9I3YcwCQAAAGhmn+8cAgAAAOARSjgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADS2VzhUVc+uqruq6lhVvXTi+adU1c9V1Qeq6iVbz91TVW+pqjdX1dGz1XEAAAAAztzFuwqq6qIkr0jypUmOJ7m1qm4eY7x1o+zdSb4lyVfMNPOsMcY7z7CvAAAAAJxl+1w5dG2SY2OMu8cYH0zy6iTXbxaMMe4fY9ya5EPnoI8AAAAAnCP7hENXJLl34/7x9WP7GkleX1W3VdUNc0VVdUNVHa2qoydOnDhE8wAAAACcrn3CoZp4bBziNb5gjPH0JNcl+eaqeuZU0RjjlWOMI2OMI5dffvkhmgcAAADgdO0TDh1P8sSN+1cmuW/fFxhj3Lf+9/4kr8nqY2oAAAAAXAD2CYduTXJ1VT25qi5J8rwkN+/TeFVdWlWPPvlzki9LcsfpdhYAAACAs2vnXysbYzxQVS9KckuSi5LcNMa4s6peuH7+xqr65CRHkzwmyYer6sVJrklyWZLXVNXJ1/qhMcZPnpM1AQAAAODQdoZDSTLGeF2S1209duPGz7+R1cfNtv12ks8+kw4CAAAAcO7s87EyAAAAAB6hhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGN7hUNV9eyququqjlXVSyeef0pV/VxVfaCqXnKYZQEAAAA4f3aGQ1V1UZJXJLkuyTVJnl9V12yVvTvJtyT5ntNYFgAAAIDzZJ8rh65NcmyMcfcY44NJXp3k+s2CMcb9Y4xbk3zosMsCAAAAcP7sEw5dkeTejfvH14/t40yWBQAAAOAc2yccqonHxp7t771sVd1QVUer6uiJEyf2bB4AAACAM7FPOHQ8yRM37l+Z5L4929972THGK8cYR8YYRy6//PI9mwcAAADgTOwTDt2a5OqqenJVXZLkeUlu3rP9M1kWAAAAgHPs4l0FY4wHqupFSW5JclGSm8YYd1bVC9fP31hVn5zkaJLHJPlwVb04yTVjjN+eWvYcrQsAAAAAh1Rj7Pv1QQ+dI0eOjKNHj04/WVNfY7Tl5DodpvZ06gEAAAAeJqrqtjHGke3H9/lYGQAAAACPUMIhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADR28fnuwMNW1X51Y5zbfgAAAACcAVcOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABo7OLz3YE2qnbXjHHu+wEAAACwYa8rh6rq2VV1V1Udq6qXTjxfVfV96+dvr6qnbzx3T1W9pareXFVHz2bnAQAAADgzO68cqqqLkrwiyZcmOZ7k1qq6eYzx1o2y65Jcvb59bpJ/tP73pGeNMd551noNAAAAwFmxz5VD1yY5Nsa4e4zxwSSvTnL9Vs31Sf7ZWHlTksdW1RPOcl8BAAAAOMv2CYeuSHLvxv3j68f2rRlJXl9Vt1XVDXMvUlU3VNXRqjp64sSJPboFAAAAwJnaJxya+ibl7W9OXqr5gjHG07P66Nk3V9Uzp15kjPHKMcaRMcaRyy+/fI9uAQAAAHCm9gmHjid54sb9K5Pct2/NGOPkv/cneU1WH1MDAAAA4AKwTzh0a5Krq+rJVXVJkucluXmr5uYkL1j/1bLPS/LeMcY7qurSqnp0klTVpUm+LMkdZ7H/AAAAAJyBnX+tbIzxQFW9KMktSS5KctMY486qeuH6+RuTvC7Jc5IcS/L+JN+wXvzxSV5TVSdf64fGGD951tcCAAAAgNNSY2x/fdD5d+TIkXH06NHpJ2vq6422nFynw9Sei7bPpC8AAAAAZ1FV3TbGOLL9+D4fKwMAAADgEUo4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANCYcAgAAACgMeEQAAAAQGPCIQAAAIDGhEMAAAAAjQmHAAAAABoTDgEAAAA0JhwCAAAAaEw4BAAAANDYxee7A8yo2l0zxrnvBwAAAPCI5sohAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMaEQwAAAACNCYcAAAAAGhMOAQAAADQmHAIAAABoTDgEAAAA0JhwCAAAAKAx4RAAAABAY8IhAAAAgMYuPt8d4Cyo2q9ujHPbDwAAAOBhx5VDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGhMOAQAAADQmHAIAAAAoDF/yr6jqt01/uw9AAAAtODKIQAAAIDGXDnEbq40AgAAgEcsVw4BAAAANObKIc4uVxkBAADAw8peVw5V1bOr6q6qOlZVL514vqrq+9bP315VT993WZqr2n07TO0+4RQAAADwoJ3hUFVdlOQVSa5Lck2S51fVNVtl1yW5en27Ick/OsSycG4IkgAAAGCnfT5Wdm2SY2OMu5Okql6d5Pokb92ouT7JPxtjjCRvqqrHVtUTkly1x7JwYTjMR+L2DZYOU7/5cbuz3ZfTbRsAAIBHvH0+VnZFkns37h9fP7ZPzT7LAheqw36U77BXa7m6CwAA4Lzb58qhqd/Oti8rmKvZZ9lVA1U3ZPWRtCR5X1XdtUffkuSyJO/camz/+sPUnuv6Ln3psp4XUl+6rOduh6l/uLatLw992/ry0LetLw992/ry0LetLw992/ry0LetLw992/ry0Ld9ofXlUycfHWMs3pJ8fpJbNu5/W5Jv26r5x0mev3H/riRP2GfZM70lOXqu6s9l2/rSez0vpL50Wc8LqS9d1vNC6kuX9byQ+tJlPS+kvnRZzwupL13W80LqS5f1vJD60mU9L6S+dFnPC6kvXdZz6bbPx8puTXJ1VT25qi5J8rwkN2/V3JzkBeu/WvZ5Sd47xnjHnssCAAAAcJ7s/FjZGOOBqnpRkluSXJTkpjHGnVX1wvXzNyZ5XZLnJDmW5P1JvmFp2XOyJgAAAAAc2j7fOZQxxuuyCoA2H7tx4+eR5Jv3XfYse+U5rD+XbR+2vktfuqznYesfrm0ftr5LX7qs52HrH65tH7a+S1+6rOdh6x+ubR+2vktfuqznYesfrm0ftr5LX7qs52HrH65tH7a+S1+6rOesWn9GDQAAAICG9vnOIQAAAAAeqc7Gt1o/VLckT0zy75L8cpI7k/z59eP/Ismb17d7krx5Ytmbktyf5I7TfZ0dyzw7q7/SdizJSxfqPibJf0ryS+u2/+ae635Rkl9M8m8O016SlyQZSS5bai/JZyf5uSRvSfKvkzxmofYTk7whyX9e//v7Zvr82CQ/kuRt67H8/EOu24N9n1vPublf2Fa+I8nt6/rXJ/mUpXFM8rQkb1rXH01y7UT//8J6mTuS/HCSj9l6/rM2+vjmJL+d5MVzbc/1ff3cn8tqO7szyd/d9fpz87/x/D3rOX9zZr7lfmL+J8dwof6719vA7Ulek+Sxu8ZuZj0n9+GZ2rn5fO76/oeTHNnR79naXXO7zz6ahX1ua/k/vx6fOyfaPjAmWdg/5/oyNY4L/X5ZkrdvrPNzFvqyNN5z7R/YvhZqJ7ethfrJsZnp+2GPLZPzuVD/tEzv/1N9mVvPqdrZ/XOhLwfmdK527tgy1ZeF2rl+zK3nXP3k9jUzLgfWcUf93HxO1U7O5Y6+z22LS/v7vufzyXXNxDl5YT3nzltzYz5Xf2BOF2rnzs+Lx9ntcVlo/8C4LLWd6XPLXNtzx4u5+rlxP3BOzPIxfap+bo6mag91fl6Yo6naA+O9o+3Z9/Iz9U/L9DF0qnbpPe6BfW7HmE/Vz435VO3ccW5uW1nqy2NzcJ+eHJepY8Xcdr7Q9tx2PrWeh32vuNTve7L1XnVuXGZql96LTNUv7RdT9XP7xYHapeP5TNsvy/x+NNl+po9dU20v7XNT9ZNzNFO7tM89Nge3rcNu53P73FTt0nxO1c/N54HaHfM51fbLMj+fk+1nfh/d3od25iH73A69wPm8JXlCkqevf350kl9Jcs1Wzfcm+RsTyz4zydOzXzi083W26i9K8qtJPi3JJVn9QjJZn6SSfPz6549K8vNJPm+PPv3FJD+UgwHKbHtZvSm9Jcmv5+CbyVPay+ovy33R+udvTPIdC7V/N+sALMlLk3zXTJ//aZI/s/75kmzsjLvWbbvv+4zb5tzPzWFOPTh9S5Ibl8Yxq4PCdevHn5Pk32+95hVJfi3Jx67v/8skX79jW/mNJJ861/ZC35+V5KeSfPT6ucctvf7S/G/055655xbmf3IMF+q/LMnF65+/6+T2Mtf3qfWc24cXaufm8/dn9Qbs3+fgG4Ttfs/W7prbffbRLOxzG8v+wazecH1cVt8R91NJrt54fmpMZvfPhb5MbVtztS9L8pKJvk71ZWm859o/sH0t1M5tW3P1k2Mz1ffTOLZMzudC/dz+PzWOc+s5VTu7fy705cCcztXOHVvmxnCmdq4fc+s5Vz+5fc2My4F13FE/N59TtbPniYW+H9gWs7C/T43jwrFrcl0zcU6eW8+5Y9vcmC/UT87pTO3iuWW7fte4TLQ/uw1M1E6eW5b6MnW8WGj/wLhn/pw4d9yaqz8wRwu1hz0/Tx2f52onx3uufm4MF9o/sN8t1M7tz5P73MKYz9VPjflc7eI+MbGtLJ3Tp/bppePR9rFidjufantqjhbW87DvFZf6fU8OHvfm5miqdum9yFT97BzN1M/9bnGgdum4NdP2yzJ/7pqqn3tfPNmXuePWTNtz71umapd+r5zabg+7nc+d/6dql+Zzqn5uPif3iYX5nGp7aT6n6pf20clsYGo+D3N7WH2sbIzxjjHGL6x//q9ZpWpXnHy+qirJn8rqfwq2l/3ZJO8+G68z4dokx8YYd48xPpjk1Umun2l7jDHet777UevbWOpPVV2Z5I8medUh2/v7Sf7ydvsz7X1Wkp9d//yGJF+1UHt9Vhtw1v9+xUSfH5PVm+jvX/fzg2OM3zrEup3S913jtj33c3M4xvjtjde4dI/2R5LHrB//hCT3ba9DVifDj62qi7M6OU7VnPQlSX51jPHrc20vbH//a5K/M8b4wPq5+3e8/uT8H8bU/MyN4UL968cYD6zvvinJlRvLT/V9cj1n9uG52sn5HGP88hjjrj3Xc7J2webcPmhh25rc57b8/iRvGmO8fz2GP5PkKzfanhqT2f1zoS8HxvGwx6qpviyN4cIcHdi+Fmont62Fvk+OzdL5Yd9jS2bmc6F+bv+fGse59Zyqnd0/D3Nu21F74NiyMIZTtXPH57n1nKuf3L6W5nNmXafq5+Zzqnb2PLEwjlPb4tL+fpjz+QEL5+R9jkMPHtv2PC5u1i8d/7drZ7fdqfr1/V3nucnj8h61c+fbxbaX3odu1c+N+9Q5cek914H6hTmaqj3U+Xlhjg7zPmixfmYMp+rn9rup2rnxntvn5sZ8sn5mzOdqd+0TyanbymRfFvbpyXGZOVZMbue73sNvzdHceh7qveJcvxfs/F3kwYYP+X5uzznarN/n2LXpjN+fL9jn2HWKHcetTYeZo7nfK+e2rUNt51NzulA7OZ8L9Qfmc8c+cWA+9/09eI/6uX109tx/iPmc9LAKhzZV1VVJ/lBW/yN80hcm+c0xxn8+x6+z7Yok927cP56FMKmqLqqqN2d1afobxhhLbSfJy7Pa6D68b3tV9eVJ3j7G+KU927sjyZevf35uVinoXO3jxxjvSFZvfLO6ymDbpyU5keT/qapfrKpXVdWl+/Rlru87xm127rfnsKr+VlXdm+Rrsvpfj6X2X5zku9f135Pk2zbbHmO8ff34f0nyjiTvHWO8fmI9T3pePrKzLrY90ffPTPKFVfXzVfUzVfU5c6+/Y/5PWYUkr6+q26rqhonnX56JbW9uDOfqN3xjkp9IFsfuwHou9H+29pD72a5+72Nzbk8x05e5fW7THUmeWVWfVFUfl4983GfJ4v4505fJcVwYwxdV1e1VdVNV/b4d/Zk11/7U9rXHfD64bS3U73Ps2rbvsWXnfG7Vvzg79v8Zp6znlIX9c64vycKcbtYe4tgyeyzf0Y+TJtdzz3PynMNst/vsnye9OHvM5Vbfp7bFyf39NM7nU+s6d07eZz1nj20z5uqn5vSU2j223Qfr99wWt/uytA1s1u5zHppaz6X3oZv1B8Z94Zw4edw6zPuPPWv3OT8fmKMdbR8Y7z36csoYLtS/OFv73ULt3HY+d46dO1cc5py8T+3c8XxzW5nry9w+fWBc1vUvz8Fjxdx2vus9/OYcza7nId8rzvU7mX6vOjcuu97XbttVvz1Hk/Uz63qgdsdxa64vc8etqfq5OV1az6nj1lT9izM9R1O1c/vc3LZ12O18yj61m/M5Wz8xn5O1C/O51Jep+Zyrn5vPl2f+95Yzy0PGaVxudL5vST4+yW1J/sTW4/8oybcuLHdV9vhY2a7Xmah7bpJXbdz/uiT/YI/2H5vVdxH8wYWaP5bkH65//uJMXDo20d5Ts3rz+Qnrx+/JRy7nn2wvyVOyulzwtiTfnuRdC7W/tfW675noy5EkDyT53PX9/zNbl6tPtZ/V//JM9n1p3ObmfmkOszqoHfjOp832k3xfkq9aP/6nkvzUVu3vS/Jvk1ye1dUJr03ytTPzc0mSd2Z1AMwebZ/S96wOtN+X1Udmrs3q0ump13/BrjHceI2Tn6N9XFYfh3zmYba9zTHcVZ/kr2X1Wd9aGruZ9Ty5zFU59eMcs7U7tpd/n49c+r2r3w/WLux7p8ztPvt8Jva5mWW+KckvZPU/MDcm+ftbz2+Pyc79c6Ivi+O4Vfv4rC55f1SSv5Xkprm+7DuGU3M0t4/OzOcp29bCes6OzULf9zq27JrPifrZ/X+hLwfWc652bvxm+rI0pw/WZsfxebMvu2qn+rFrPhfqD2xf2+OytI4z9bPzOVG7eCyfGfPJbTET+/vcOGb+HH1gXTNzTl5az3Xd5LFtasx31E9tu7PHzUzv+w/W77l9bZ9zl7bz7dpdx8S59Zw7Xmy3P/Wea+6cOLetLL7/yKnnuV21e52fp+Zood+T471HX04Zw4X2D+x3C7VL+/PUPrd0rpg9J+fgR1yWaueOc9vbytz8z+3TU+Myd6yY3M7n2l6Yo13vU3a+V5zq98byB96rLozL0vvaU+Znj/qp49Zs/cS6TvV76Rw6Vb903Jqqn5vTpfU8cNyaaXtyjmZqJ/e5zG+3h9rOZ45zu2q3j3P7/K568jg3Vfvdc/O5sJ5zx8W5+qn53PV7y2Iesut2Wgudz1tWB/tbkvzFrccvTvKbSa5cWPaq7BkOzb3OTO3nJ7lla0P6tj1f59uz/Bn478zqSqR7svr88fuT/OCO9v63rP63/J717YGs/hflk/dpL6uU8j/N1Wb1pVhPWNc+IcldE/345CT3bNz/wiQ/vse6/ehc3+fGbW7ud81hVp/lnvul6tuz+nKx9+YjB5FK8ttbdc9N8v0b919wcoedaPP6JK/fuD/b9lTfk/xkki/euP+rWZ2Qt1//3+0zhhP9e9nmtrjntvLgGC7VJ/mfsvpiuo/bNXYz63n51D68VLu0n+XUk8niema/cOiUuT3sPp/1PrfHsn87yZ/demx7THbunxPb+c5xnOn39mufcv+QYzjV/uQ+mlP3/wPb1sJ6zo7NVN9z+seWU+Zzqj7L+/9UXybXc27M58Zvj74/2N52bZL/LgvHlq1ld9XOncvn1nO231Pb145xmRrfpfrt+TyldmkuF+Z/n/Po387qi17P5Hx+VVZvLvc5Jx84DmXm2DY15nP1C3M6e9zM9Lb7YP2u7WuP9rfncPv8vHhMnFnP2fehO/py8j3X3DlxcluZq5+ao6XaqfnZ1fbmHO1Z++B47+jLgTFcGJcD+92efZk932Z9jp0b87n6XfvFdu3UmC9si3PzP7lPz4zL3Pv5ye18ru1d2/nUmGzvzwt9WTyObrT1suw4n2/X7jM/2/VLczTX/va6TtTO/n62Z9tXTbW9NS77vJ/bXM99fn8+2fbOOZrp94P73Ny2NTefS9vi9pwu1U7N5662N+dzpvan5+Zzz7YfnM+FcZmaz5dn/vetnfO56/aw+lhZVVVWn8X75THG39t6+o8kedsY4/g5fp0ptya5uqqeXFWXZHU56M0zbV9eVY9d//yxJ/s91/AY49vGGFeOMa5at/tvxxhfu6O9XxxjPG6McdV6ueNZfSHmb8y1V1WPW7fxqCR/Pasv35p77Zuz2smy/vf/nej3byS5t6o+a/3QlyR56x7r9lVTfU/yewvjdmDu5+awqq7e6MKXn2xjYV7uS/JF6/o/nNW36G/6L0k+r6o+bv2aX5LVd0pMeX5OvQR9su2F7e+167pU1Wdm9b9Ld0y8/o/Nzf9mZ9aXQz765M9ZfWHbHSefX9hWJsdwof7ZSf5Kki8fY7x/j7GbWs93zozpZO1h9rNd+9ietuf2QXN9mdrnZpY/WfekrK7e2PXxjtn9c2FcXpuD45iZfj9h47W+MhvbzGEsjMuB7WuhdnLbWljPnceuLYc5tkzO58L+vOvY8qCFfWiqdnL/3NH3A3M6VTvGeMs+x5ZdtQv9mJvPw56Tp8blUNvtvvvn2uxcLvR9cluc2N//2Wmczw+s69w5eY/1nD22zTilfse2u107u+1u1++5LW63v7QNbK/na7N8Hpoal6X3odt9mRr3uXPi3HHrMO8/JmsPe36emaO52rnxXur31BjO1U/td3N9md3OZ86xS+fRvc/JU7V7HM+3t63Jviy8zz4wLgvvc16bie18x3v4qfPi1Hoe6r3iVL/Xbc69Vz0wLrve126bq184F83VT71vmaq9deG8ONf25H60sK6vzcE5ff/CuEzN51zbB+Zood+T+9zCtnXY7fyAhfPc5Hwu1B+Yz5naX1g4P8+1PTmfC+v52hycz7+w8HvLmechS8nRhXZL8oysPtd4ew7+idZ/kuSFC8v+cFafP/7QevK+6XReZ2GZ52T1V0h+NclfW6h7alZ/du729Qax9zeJZ/rSsZ3tZf7b8h9sL6v/nfyV9e3v5OBlrpu1n5RVWvqf1/9+4kx/n5bVnzq8PauNe/JP3s+t22bfl9Zzau7n5jCrK5PuWD/+r7P6UtPZcVy3c1tWl0n+fJL/fqKPfzOrk94dSX4g62+U36r5uKwuG/+ErT4eaHuh75dk9b8rd2R1+e4f3uf1F+b/09av/UtZ/XnEpe12c/4nx3Ch/lhW38l1cl1uXBq7hfU8sA8v1M7N51eul/1AVsn6LQv9Xqxdmtt99tHs2Oc2lv8PWZ0gfinJl+w6rmVh/1zoy4FxXKj9gaz+NOntWZ3Mn7DQl9kxXGj/wPa1UDu5bS3UT47NVN9P49gyOZ8L9XP7/9Q4zq3nVO3s/rnQlwNzOlc7d2yZG8OZ2rl+zK3nXP3k9jUzLpPb7UL93HxO1c6eJxb6Prctzu7vO47nX5yPHLvm9tGnZeucPLeeC+etpX16qn5uTqdql7bdXcfZU8Zlpv25cZmqnTy3LPUlM+9DZ9qf276mzolLx/Sp+rn9Yqr2sOfnufdQU7VL+9zk+5aFMZxqf+4YOlW7tJ0f2Od2jPlU/dyYT9UujfnUtrLUl6fl4D69+L41px4rlrbzA20vnBen1vOw7xXn5nPyverUuCzUzs3PXP3ccWuufup9y8732Dn1vDjX9txxa65+6v3cbF9m5nOu7QNztFC7tM89LQe328Nu53NzOlW7tM9N1c8d5w7U7jgPTbW9dFycqp/dR7f3oaVj6GFuJ09GAAAAADT0sPpYGQAAAABnl3AIAAAAoDHhEAAAAEBjwiEAAACAxoRDAAAAAI0JhwAAAAAaEw4BAAAANCYcAgAAAGjs/wfrA9gs/MR/JwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 1440x1440 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8lQSQNviZai5"
      },
      "source": [
        "# What the most important hyper parameters in DT ?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CHJ4I4XGZai6"
      },
      "source": [
        "- ``criterion`` : optional (default=”gini”) or Choose attribute selection measure: This parameter allows us to use the different-different attribute selection measure. Supported criteria are “gini” for the Gini index and “entropy” for the information gain.\n",
        "\n",
        "\n",
        "- ``splitter`` : string, optional (default=”best”) or Split Strategy: This parameter allows us to choose the split strategy. Supported strategies are “best” to choose the best split and “random” to choose the best random split.\n",
        "\n",
        "\n",
        "- ``max_depth`` : int or None, optional (default=None) or Maximum Depth of a Tree: The maximum depth of the tree. If None, then nodes are expanded until all the leaves contain less than min_samples_split samples. The higher value of maximum depth causes overfitting, and a lower value causes underfitting \n",
        "\n",
        "\n",
        "- ``min_samples_split`` : int or float, default=2 . The minimum number of samples required to split an internal node\n",
        "\n",
        "\n",
        "- ``class_weight``: dict, list of dict or “balanced”, default=None"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BCmaaM2tZai7"
      },
      "source": [
        "## Hyperparameter tuning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "referenced_widgets": [
            "81032e5d9c4c4b70847c51484a88a791"
          ]
        },
        "id": "qgC8UcFiZai8",
        "outputId": "4203b8de-7d22-470a-b612-453e1d0a01d1"
      },
      "source": [
        "clf = tree.DecisionTreeClassifier( criterion='entropy', max_depth = max_depth)\n",
        "\n",
        "print(\"Start fitting the data\")\n",
        "\n",
        "hp = {\"criterion\":['gini','entropy'], 'max_depth':np.arange(3,9)}\n",
        "for cv in tqdm(range(3,6)):\n",
        "    grid = GridSearchCV(clf, param_grid  = hp  , cv = cv)\n",
        "    grid.fit(X_train,y_train)\n",
        "    print(f\"score for {cv} fold CV is : {grid.score(X_test,y_test)}\")\n",
        "    print(f\"Best result {grid.best_params_}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Start fitting the data\n"
          ]
        },
        {
          "ename": "NameError",
          "evalue": "name 'tqdm' is not defined",
          "output_type": "error",
          "traceback": [
            "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[1;32m<ipython-input-36-5ea27a45c18d>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[0mhp\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m{\u001b[0m\u001b[1;34m\"criterion\"\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'gini'\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;34m'entropy'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'max_depth'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0marange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m9\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 6\u001b[1;33m \u001b[1;32mfor\u001b[0m \u001b[0mcv\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mtqdm\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m6\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      7\u001b[0m     \u001b[0mgrid\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mGridSearchCV\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mclf\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mparam_grid\u001b[0m  \u001b[1;33m=\u001b[0m \u001b[0mhp\u001b[0m  \u001b[1;33m,\u001b[0m \u001b[0mcv\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcv\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      8\u001b[0m     \u001b[0mgrid\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my_train\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
            "\u001b[1;31mNameError\u001b[0m: name 'tqdm' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4kHW8ps2ZajA"
      },
      "source": [
        "# Avoiding Overfitting :\n",
        "\n",
        "1. Acquire more training data\n",
        "\n",
        "\n",
        "2. Remove irrelevant attributes\n",
        "\n",
        "\n",
        "3. **Grow full tree, then post-prune** \n",
        "\n",
        "\n",
        "4. Ensemble learning. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cdl1AO03ZajB"
      },
      "source": [
        "## Pruning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r-Emt5vWZajC"
      },
      "source": [
        "We can perform the pruning either before building the fully grown tree - pre-pruning - or after building it - post-pruning-\n",
        "\n",
        "- We have already performed pre-pruning by specifiying the maximum depth in our implementation, or by controlling the different hyperparameters in sklearn implementation such as the min_samples_split and max_leaf_nodes\n",
        "\n",
        "\n",
        "- However, this approach is not preferred or shouldn't be used alone, because a seemingly worthless split early on in the tree might be followed by a very good split.\n",
        "\n",
        "\n",
        "- Therefore, it should be combined with post-pruning, also known as cost-complexity pruning and as weakest link pruning."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "12RmMUP8ZajD"
      },
      "source": [
        "We can perform post-purning to prune the fully grown tree in a *bottom-up manner*.\n",
        "\n",
        "- The idea is basically about, starting at the leaves, each node is replaced with its most popular class.\n",
        "\n",
        "\n",
        "- If the prediction accuracy is not affected then the change is kept. \n",
        "\n",
        "\n",
        "\n",
        "We can also explain it as follows :\n",
        "\n",
        "- Rather than considering every possible subtree, we consider a sequence of trees indexed by a nonnegative tuning parameter α.\n",
        "\n",
        "\n",
        "- For each value of α there corresponds a subtree $$T ⊂ T_0$$ such that Cost $$T = Error(T) + α |T|$$ is small as possible\n",
        "\n",
        "\n",
        "- $$|T|$$ indicates the number of terminal nodes of the tree T\n",
        "\n",
        "\n",
        "- The tuning parameter α controls a trade-off between the subtree’s complexity and its fit to the training data. When α = 0, then the subtree T will simply equal $$T_0$$\n",
        "\n",
        "\n",
        "- as we increase α from zero, branches get pruned from the tree"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CgZ8Oz8xZajE",
        "outputId": "9e2106d1-65a9-4d45-91fc-cbea389b5328"
      },
      "source": [
        "X_train , X_test , y_train , y_test = train_test_split(X,y,test_size = 0.2 , random_state = 10)\n",
        "\n",
        "X_train = pd.get_dummies(X_train,drop_first=True)\n",
        "X_test = pd.get_dummies(X_test,drop_first=True)\n",
        "clf = tree.DecisionTreeClassifier( criterion='entropy', max_depth = max_depth)\n",
        "path = clf.cost_complexity_pruning_path(X_train, y_train) ## Pruning !\n",
        "\n",
        "ccp_alphas, impurities = path.ccp_alphas, path.impurities ## Alpha we have been talking about\n",
        "fig, ax = plt.subplots(figsize=(10,10))\n",
        "ax.plot(ccp_alphas[:-1], impurities[:-1], marker='o', drawstyle=\"steps-post\")\n",
        "ax.set_xlabel(\"effective alpha\")\n",
        "ax.set_ylabel(\"total impurity of leaves\")\n",
        "ax.set_title(\"Total Impurity vs effective alpha for training set\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Text(0.5, 1.0, 'Total Impurity vs effective alpha for training set')"
            ]
          },
          "execution_count": 111,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAJcCAYAAAC8DwN/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA14ElEQVR4nO3de5hddX33/feXYdARiaMFaTKAINK0sQqRg1KtNx6DPlYijwc81LbPfQtq8XD7mGiqV8XTjd55tGrVG6mH2ioHjWGkSonnQ1XkYCCBYFoEhEyiBHQM4GjC5Pv8sdaEnWHPzJ7MrL3XzH6/rmuuzPqtw/7uvXbCh99vrd+KzESSJEn1sF+nC5AkSdL9DGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM2kWRURGxGM6XcdUIuLPI2Jzp+uYjoh4T0TcGRG/KJdfEBG3R8Q9EbF0Fl+nI5/NdL47s/k9i4gnR8R/lZ/j8tk45kxFxMsj4muzva00V4TznKkbRMQ9DYsPAX4PjJbLZ2Xm55vscwrwucw8bBqvk8AxmXlTk3XfKY/3ydYrb4+IuBX4H5n5jU7X0kxEHA78J/CozLyjbPsZ8KbM/PIMjz3hOWun6dQxmzVHxDeBSzPzwzM9Vnm8fwa2ZObbZ+N4c01EHAncAvRm5n0dLkdz1P6dLkBqh8x86NjvdQ8iVYuI/efgfzQeBdw1Fswa2m7oUD3zyT5/jvvyXZqj3z+prRzWVFeLiAdFxIciYmv586Gy7UDg34FF5XDPPRGxKCJOiogfRcRwRGyLiI9GxAH78LqnRMSWiFgZEXeUx1oeEc+NiP+MiF9FxN81bH9ORKyJiIsj4u6I+ElEHNuwfq9hroj454h4z7jXeks5JPiZsbZy/b8CRwD/Vr7PlRHx1Yh43biaNzQb9oqIyyPi7HFt10XE6VH4h/I9/qY8xp9O8Jk8LCI+VX4WQ+UwZk9EPBP4esO5uLDsCe0Brit70CjPz5ciYntE3BIRr284dk9E/F1E/Kz8/K6JiMMj4nvlJteVx37JuM/mrRGxZlydH46Ij0xW8wTvr+XvTnn+zouIr5f1fjciHjVus2dGMRz564j4WEREue/REfGtiLgrimHgz0dE/wSv8zPg0dx/7h9Ufo6Xlt/BmyLiVQ3bj30PPxcRO4C/Hne8M4GXAyvL4/1b2X5r+f3bANwbEfuXn+3Y+dgUES9oOM5fR8R/NCxnRLx6gvc7nW17IuID5edyS0ScXW7ftKOirHmorHFzRDyjbN+vof67IuILEfGIcrex79Rw+Rmc3OzY0qQy0x9/uuoHuBV4Zvn7u4ArgEcChwA/BN5drjuFYnimcd/jgSdR9DofCdwIvLFhfQKPmeB1v0PRYzd27PuAvwd6gVcB24ELgIOAxwK/Ax5dbn8OsAt4Ybn9m7l/6OQBrwv8M/Ceca/1fuBBQN/499b4mZTLLwZ+3LB8LHAXcECT9/VK4AcNy0uA4fK1lgHXAP1AAH8CLJzg8xkEPgEcWJ6PKymGnCc6F3veM8X/aF5Tfp4HUASOm4Fl5foVwEZgcVnHscAfTPDZ7Xktil6l3wILyuUeYBvwpKlqbvL+Wv7ulOfvbuCp5ef4YeA/xm37lfJzPYLiu3Nque4xwLPK/Q6hCAsfauXvQ7n8XeDjwIOB48pjP2Pc93B5+Zn3NTneP1N+98a9xrXA4WP7AC8CFpXHeQlw79h3gyL0tfp+p7Ptq4FNwGHAw4FvlNvv3+R9LAZuBxaVy0cCR5e/v5Hi343Dys/5E8CFDds1PaY//rT6Y8+Zut3LgXdl5h2ZuR14J/CXE22cmddk5hWZeV9m3krxj/J/28fX3gW8NzN3ARcBBwMfzsy7M/MGiqGmxzdsf01mrim3/yDFfzyf1OJr7QbekZm/z8yRFrb/MnBMRBxTLv8lcHFm7myy7SXAcQ09Oy8H1mbm78v3eBDwxxTXuN6YmdvGHyAiDgWeQxFW7s1i+PIfgDNafH8nAodk5rsyc2dm3gz8U8P+/wN4e2ZuzsJ1mXnXVAfNzJ8DP6EIIwBPB36bmVdMt+Z9+O58NTO/V36ObwNOjuLauzHvy8zhzLwN+DZFkCIzb8rMr5fnejvFd6Wl72h5/KcAb8nM32XmtcAn2fvvxI8yczAzd7f4XRrzkcy8fWyfzPxiZm4tj3Mx8F/ASZPs3/T9TnPbF1P8HduSmb8G3jfJMUYpgteSiOjNzFsz82flurOAt5XH+T1FaH3hRD1w0nQZztTtFgE/b1j+ednWVET8UUR8JSJ+UQ7r/C+KULUv7srMsZsSxv4j98uG9SPAQxuWbx/7JTN3A1smq3Wc7Zn5u1YLK/+D8wXgFRGxH/BS4F8n2PZu4KvcH0rOAD5frvsW8FHgY8AvI+L8iFjQ5DCPougR3FYO+w1ThJdHtljyoyiGPYcb9v874NBy/eHAzybaeQoXULx/gJeVy9OueR++O43n+x7gV+x9vn/R8PtvKb8rEfHIiLioHI7bAXxuitdptAj4VXlOx/wcGGhW1zTttV9EvDIirm347P50ijqbvt9pbrtoXB0TvpcsbrZ4I0XwuqP8TMc+/0cBlzTUfiNFmDu0yaGkaTOcqdttpfiHdswRZRsUQxPj/R/gpxR3yi2gCABRaYX329NrUgamw7i/1t9S3IU65g/H7TvVbdnN1n+WohfsGRS9RT+aZP8LgZeW19f0UfRWFAfO/EhmHk8xVPtHFEOM491OcQftwZnZX/4syMzHTlF34/63NOzbn5kHZeZzG9Yf3eKxxvsicEpEHAa8gPvD2XRrnu53p/F8PxR4BPef78mcS3E+H1++ziumeJ1GW4FHRMRBDW1HAEMNy/vyXdqrvexl/SfgbIrh5X7g+mnUua+2Ufy9GXP4RBsCZOYFmfkUin8jkuLSACjO/XPGfd8enJlDTP35SFMynKnbXQi8PSIOiYiDKa5Z+ly57pfAH0TEwxq2PwjYAdwTEX8MvKaNtR4fxUX2+1P8H/3vKa57geJ6npeVFzyfyvSHWn9JcZ3WHmUY2w18gAl6zRpcRvEfsHdRDH/uBoiIEyPiiRHRS3FN0e+4fwqTxtfaBnwN+EBELCgvuD46Ilp9H1cCO8oLuPvKz+FPI+LEcv0ngXdHxDFReHxE/MFE731cbdsprhf8DEUAvHEfa57ud+e5EfGUKG4aeDfFNYCt9FodBNxDcUH6AM3DcFPl8X8InBsRD46IxwP/nbIntEWTfp6lAylCzHaAiPgbip6zqn0BeENEDERxk8RbJtowIhZHxNMj4kEU39sR7v/unge8d2wov/z347Ry3XaKvzdTfQbShAxn6nbvAa4GNlBcMP6Tso3M/ClFeLu5HL5YRHEh/ssoLtb+J+DiNtb6ZYoLp39NcQ3Q6eX1ZwBvAP6C4kL8l1NcqD4d51KE1OGIeHND+78Aj+P+wNpUOQy6Fngm9/csASyg+Jx+TTE8dhfw/01wmFdSXMy/qdx+DbCwleLL4eG/oLi26BbgTopANhasP0jxH+avUQSkT1H08EExbPXZ8r2/eIKXuKDJe5tuzdP97lwAvINiOPN4ivPaincCTwB+QzHcvLbF/ca8lOKi9q0U1xO+IzO/Po39P0VxndZwRAw22yAzN1GE/h9RhLnHAT+YZp374p8ovgMbgPUU/1NxH03+h4HierP3UXyXfkExXD12B/WHgUuBr0XE3RT/k/REgMz8LfBe4AflZ9DqdaHSHk5CK80BEXEOxZ18r2jz674SOLMc2lGbRJdP5NouEfEc4LzMHD9NidRR9pxJaioiHgK8Fji/07VIs6Ec8n5uFPOsDVD0TF7S6bqk8Qxnkh4gIpZRXDvzSx44lCfNVUEx7PtrimHNGymuM5VqxWFNSZKkGrHnTJIkqUbm1WzGBx98cB555JGdLkOSJGlK11xzzZ2Zecj49nkVzo488kiuvvrqTpchSZI0pYj4ebN2hzUlSZJqxHAmSZJUI4YzSZKkGjGcSZIk1YjhTJIkqUYMZ5IkSTViOJMkSaoRw5kkSVKNGM4kSZJqxHAmSZJUI4YzSZKkGjGcSZIk1YjhTJIkqUYMZ5IkSTViOJMkSaoRw5kkSVKNGM4kSZJqxHAmSZJUI4YzSZKkGjGcSZIk1cj+nS5AkiSpDgbXD7F63Wa2Do+wqL+PFcsWs3zpQNvrMJxJkqSuN7h+iFVrNzKyaxSAoeERVq3dCND2gOawpiRJ6nqr123eE8zGjOwaZfW6zW2vxXAmSZK63tbhkWm1V8lwJkmSut6i/r5ptVfJcCZJkrreimWL6evt2autr7eHFcsWt70WbwiQJEldb+yi/5VrNrBzdDcD3q0pSZLUWcuXDnDhlbcBcPFZJ3esDoc1JUmSasRwJkmSVCOVhrOIODUiNkfETRHx1gm2OSUiro2IGyLiuw3tt0bExnLd1VXWKUmSVBeVXXMWET3Ax4BnAVuAqyLi0szc1LBNP/Bx4NTMvC0iHjnuME/LzDurqlGSJKluquw5Owm4KTNvzsydwEXAaeO2eRmwNjNvA8jMOyqsR5IkqfaqDGcDwO0Ny1vKtkZ/BDw8Ir4TEddExCsb1iXwtbL9zIleJCLOjIirI+Lq7du3z1rxkiRJnVDlVBrRpC2bvP7xwDOAPuBHEXFFZv4n8OTM3FoOdX49In6amd97wAEzzwfOBzjhhBPGH1+SJGlOqbLnbAtweMPyYcDWJttcnpn3lteWfQ84FiAzt5Z/3gFcQjFMKkmSNK9VGc6uAo6JiKMi4gDgDODScdt8GfjziNg/Ih4CPBG4MSIOjIiDACLiQODZwPUV1ipJklQLlQ1rZuZ9EXE2sA7oAT6dmTdExKvL9edl5o0RcTmwAdgNfDIzr4+IRwOXRMRYjRdk5uVV1SpJklQXlT6+KTMvAy4b13beuOXVwOpxbTdTDm9KkiR1E58QIEmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlG9u90AZIkqTWD64dYvW4zW4dHWNTfx4pli1m+dKDTZWmWGc4kSZoDBtcPsWrtRkZ2jQIwNDzCqrUbAQxo84zhTJKkOWD1us17gtmYkV2jrFyzgQuvvK1DVc0/m7btYMnCBR2twWvOJEmaA7YOjzRt3zm6u82VzG9LFi7gtOM62xNpz5kkSXPAov4+hpoEtIH+Pi4+6+QOVKSq2HMmSdIcsGLZYvp6e/Zq6+vtYcWyxR2qSFWx50ySpDlg7KL/lWs2sHN0NwPerTlvGc4kSZojli8d2HPxv0OZ85fDmpIkSTViOJMkSaoRw5kkSVKNGM4kSZJqxHAmSZJUI4YzSZKkGjGcSZIk1YjhTJIkqUYMZ5IkSTViOJMkSaoRw5kkSVKNGM4kSZJqxHAmSZJUI4YzSZKkGjGcSZIk1YjhTJIkqUYMZ5IkSTViOJMkSaqRSsNZRJwaEZsj4qaIeOsE25wSEddGxA0R8d3p7CtJkjTf7F/VgSOiB/gY8CxgC3BVRFyamZsatukHPg6cmpm3RcQjW91XkiRpPqqy5+wk4KbMvDkzdwIXAaeN2+ZlwNrMvA0gM++Yxr6SJEnzTpXhbAC4vWF5S9nW6I+Ah0fEdyLimoh45TT2BSAizoyIqyPi6u3bt89S6ZIkSZ1R2bAmEE3assnrHw88A+gDfhQRV7S4b9GYeT5wPsAJJ5zQdBtJkqS5ospwtgU4vGH5MGBrk23uzMx7gXsj4nvAsS3uK0mSNO9UOax5FXBMRBwVEQcAZwCXjtvmy8CfR8T+EfEQ4InAjS3uK0mSNO9U1nOWmfdFxNnAOqAH+HRm3hARry7Xn5eZN0bE5cAGYDfwycy8HqDZvlXVKkmSVBdVDmuSmZcBl41rO2/c8mpgdSv7SpIkzXc+IUCSJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNWI4UySJKlGDGeSJEk1sn+nC5Akzczg+iFWr9vM1uERFvX3sWLZYpYvHeh0WZL2keFMkuawwfVDrFq7kZFdowAMDY+wau1GAAOaNEcZziRpDlu9bvOeYDZmZNcoK9ds4MIrb+tQVarSpm07WLJwQafLUIW85kyS5rCtwyNN23eO7m5zJWqXJQsXcNpx9orOZ/acSdIctqi/j6EmAW2gv4+Lzzq5AxVJmil7ziRpDluxbDF9vT17tfX19rBi2eIOVSRppqYMZxFxdEQ8qPz9lIh4fUT0V16ZJGlKy5cOcO7pj+OAnuKf84H+Ps49/XHeDCDNYa0Ma34JOCEiHgN8CrgUuAB4bpWFSZJas3zpwJ6L/x3KlOa+VoY1d2fmfcALgA9l5v8EFlZbliRJUndqJZztioiXAn8FfKVs662uJEmSpO7VSjj7G+Bk4L2ZeUtEHAV8rtqyJEmSutOU15xl5qaIeAtwRLl8C/C+qguTJEnqRq3crfkXwLXA5eXycRFxacV1SZIkdaVWhjXPAU4ChgEy81rgqMoqkiRJ6mKthLP7MvM349qyimIkSZK6XSvznF0fES8DeiLiGOD1wA+rLUuSJKk7tdJz9jrgscDvKSaf/Q3wxgprkiRJ6lqt9Jwtzsy3AW+ruhhJkqRu10rP2Qcj4qcR8e6IeGzlFUmSJHWxKcNZZj4NOAXYDpwfERsj4u1VFyZJktSNWuk5IzN/kZkfAV5NMefZ31dZlCRJUrdqZRLaP4mIcyLieuCjFHdqHlZ5ZZIkSV2olRsCPgNcCDw7M7dWXI8kSVJXa+XZmk9qRyGSJElqIZyVE8+eCywBHjzWnpmPrrAuSZKkrtTKDQGfAf4PcB/wNOBfgH+tsihJkqRu1Uo468vMbwKRmT/PzHOAp1dbliRJUndq5YaA30XEfsB/RcTZwBDwyGrLkiRJ6k6t9Jy9EXgIxQPPjwdeAfxVhTVJkiR1rVbu1rwKICIyM/+m+pIkSZK6VyuT0J4cEZuAG8vlYyPi45VXJkmS1IVaGdb8ELAMuAsgM68DnlphTZIkSV2r1Wdr3j6uabSCWiRJkrpeK3dr3h4RfwZkRBxAcWPAjdWWJUmS1J1a6Tl7NfC3wACwBTiuXJYkSdIsa+VuzTuBl7ehFkmSpK43YTiLiH8EcqL1mfn6SiqSJEnqYpP1nF3dtiokSZIETBLOMvOz7SxEkiRJLU6lIUmSpPYwnEmSJNXIhOEsIt5f/vmi9pUjSZLU3SbrOXtuRPQCq9pVjCRJUreb7G7Ny4E7gQMjYgcQFFNrBJCZuaAN9UmSJHWVCXvOMnNFZj4M+GpmLsjMgxr/bGONkiRJXaOVJwScFhGHAieWTT/OzO3VliVJktSdprxbs7wh4ErgRcCLgSsj4oVVFyZJktSNpuw5A94OnJiZdwBExCHAN4A1VRYmSZLUjVqZ52y/sWBWuqvF/SRJkjRNrfScXR4R64ALy+WXAJdVV5IkSVL3auWGgBURcTrwFIppNM7PzEsqr0ySJKkLtdJzRmauBdZWXIskSVLX89oxSZKkGjGcSZIk1Ugr85w9LyIMcZIkSW3QSug6A/iviPjfEfEnVRckSZLUzaYMZ5n5CmAp8DPgMxHxo4g4MyIOqrw6SZKkLtPScGVm7gC+BFwELAReAPwkIl5XYW2SJEldp5Vrzp4fEZcA3wJ6gZMy8znAscCbK65PkiSpq7Qyz9kLgX/IzO81NmbmbyPi/6mmLEmSpO7UyrDmtvHBLCLeD5CZ35xsx4g4NSI2R8RNEfHWJutPiYjfRMS15c/fN6y7NSI2lu1Xt/h+JM3Q4Pohnvy+b3HUW7/Kk9/3LQbXD3W6JEnqKq30nD0LeMu4tuc0adtLRPQAHyv33wJcFRGXZuamcZt+PzOfN8FhnpaZd7ZQo6RZMLh+iFVrNzKyaxSAoeERVq3dCMDypQOdLE2SusaE4SwiXgO8Fjg6IjY0rDoI+EELxz4JuCkzby6PdxFwGjA+nEmqidXrNu8JZmNGdo2ycs0GLrzytg5VpVZs2raDJQsXdLoMSbNgsp6zC4B/B84FGock787MX7Vw7AHg9oblLcATm2x3ckRcB2wF3pyZN5TtCXwtIhL4RGae3+xFIuJM4EyAI444ooWyJE1k6/BI0/ado7vbXImma8nCBZx2nL2b0nwwWTjLzLw1Iv52/IqIeEQLAS2aHXPc8k+AR2XmPRHxXGAQOKZc9+TM3BoRjwS+HhE/HX/tW1nk+cD5ACeccML440uahkX9fQw1CWgD/X1cfNbJHahIkrrPZDcEXFD+eQ1wdfnnNQ3LU9kCHN6wfBhF79gembkjM+8pf78M6I2Ig8vlreWfdwCXUAyTSqrQimWL6evt2autr7eHFcsWd6giSeo+E/acZebzIiKA/5aZ+3KxyVXAMRFxFDBE8RiolzVuEBF/CPwyMzMiTqIIi3dFxIHAfpl5d/n7s4F37UMNkqZh7KL/lWs2sHN0NwP9faxYttibASSpjSa9W7MMTZcAx0/3wJl5X0ScDawDeoBPZ+YNEfHqcv15FHOovSYi7gNGgDPK1zwUuKTIhuwPXJCZl0+3BknTt3zpwJ6L/x3KlKT2a2UqjSsi4sTMvGq6By+HKi8b13Zew+8fBT7aZL+bKZ5AIEmS1FVaCWdPA86KiJ8D91Jc6J+Z+fhKK5MkSepCrYSz51RehSRJkoDWwpnTU0iSJLVJK+HsqxQBLYAHA0cBm4HHVliXJElSV5oynGXm4xqXI+IJwFmVVSRJktTFJpuEtqnM/AlwYgW1SJIkdb0pe84i4k0Ni/sBTwC2V1aRJElSF2vlmrODGn6/j+IatC9VU44kSVJ3a+Was3cCRMSCYjHvrrwqSZKkLjXlNWcRcUJEbAQ2ABsj4rqImPbjnCRJkjS1VoY1Pw28NjO/DxARTwE+A/iEAEmSpFnWyt2ad48FM4DM/A/AoU1JkqQKtNJzdmVEfAK4kGIy2pcA3ynnOxubWkOSJEmzoJVwdlz55zvGtf8ZRVh7+mwWJEmS1M1auVvzae0oRJIkSa1NQtsPvBI4snH7zHx9ZVVJkiR1qVaGNS8DrgA2ArurLUeSJKm7tRLOHpyZb5p6M0mSJM1UK1Np/GtEvCoiFkbEI8Z+Kq9MkiSpC7XSc7YTWA28jeLuTMo/H11VUZIkSd2qlXD2JuAxmXln1cVIkiR1u1aGNW8Aflt1IZIkSWqt52wUuDYivg38fqzRqTQkSZJmXyvhbLD8kSRJUsVaeULAZ9tRiCRJkiYJZxHxhcx8cURs5P67NPfIzMdXWpkkSVIXmqzn7A3ln89rRyGSJEmaJJxl5rbyz5+3rxxJkqTu1spUGpIkSWqTVu7WlOatwfVDrF63ma3DIyzq72PFssUsXzrQ6bIkSV3McKauNbh+iFVrNzKyaxSAoeERVq3dCGBAkyR1zGR3aza9SxMIIL1bU3Pd6nWb9wSzMSO7Rlm5ZgMXXnlbh6qqh03bdrBk4YJOlyFJXWmynjPv0tS8tnV4pGn7ztHdba6kfpYsXMBpx9l7KEmdMNndmt6lqXltUX8fQ00C2kB/HxefdXIHKpIkqYW7NSPiSRFxVUTcExE7I2I0Ina0ozipSiuWLaavt2evtr7eHlYsW9yhiiRJau2GgI8CZwBfBE4AXgk8psqipHYYu+h/5ZoN7BzdzYB3a0qSaqCluzUz86aI6MnMUeAzEfHDiuuS2mL50oE9F/87lClJqoNWwtlvI+IA4NqI+N/ANuDAasuSJEnqTq08IeAvy+3OBu4FDgdOr7IoSZKkbtVKOFuemb/LzB2Z+c7MfBNOsyFJklSJVsLZXzVp++tZrkOSJElM/oSAlwIvA46KiEsbVi0A7qq6MEmSpG402Q0BP6S4+P9g4AMN7XcDG6osSpIkqVtN9YSAnwMnR8ShwInlqhsz8752FCdJktRtWnlCwIuAK4EXAS8GfhwRL6y6MEmSpG7UyjxnbwdOzMw7ACLiEOAbwJoqC5MkSepGrdytud9YMCvd1eJ+kiRJmqZWes4uj4h1wIXl8kuAf6+uJEmSpO41ZTjLzBURcTrwFCCA8zPzksorkyRJ6kJThrOIeH9mvgVY26RNkiRJs6iVa8ee1aTtObNdiCRJkiZ/QsBrgNcCj46IxklnDwJ+UHVhkiRJ3WiyYc0LKC78Pxd4a0P73Zn5q0qrkmbR4PohVq/bzNbhERb197Fi2WKWLx3odFmSJDU12RMCfgP8Bnhp+8qRZtfg+iFWrd3IyK5RAIaGR1i1diOAAU2SVEutTKUhzVmr123eE8zGjOwaZeWaDVx45W0AbNq2gyULF3SiPEmSHsDJZDWvbR0eadq+c3T3nt+XLFzAacfZiyZJqgd7zjSvLervY6hJQBvo7+Pis07uQEWSJE3OnjPNayuWLaavt2evtr7eHlYsW9yhiiRJmpw9Z5rXxi76X7lmAztHdzPg3ZqSpJoznGneW750YM/F/w5lSpLqzmFNSZKkGjGcSZIk1YjhTJIkqUYMZ5IkSTViOJMkSaoRw5kkSVKNGM4kSZJqxHAmSZJUI4YzSZKkGjGcSZIk1YjhTJIkqUYMZ5IkSTViOJMkSaoRw5kkSVKNGM4kSZJqZP9OF6DuMrh+iNXrNrN1eIRF/X2sWLaY5UsHOl2WJEm1YThT2wyuH2LV2o2M7BoFYGh4hFVrNwIY0CRJKhnO1Dar123eE8zGjOwaZeWaDVx45W2VvvambTtYsnBBpa8hSdJs8Joztc3W4ZGm7TtHd1f+2ksWLuC04+ydkyTVnz1naptF/X0MNQloA/19XHzWyR2oSJKk+rHnTG2zYtli+np79mrr6+1hxbLFHapIkqT6sedMbTN20f/KNRvYObqbAe/WlCTpAQxnaqvlSwf2XPzvUKYkSQ/ksKYkSVKNGM4kSZJqxHAmSZJUI4YzSZKkGjGcSZIk1YjhTJIkqUYMZ5IkSTViOJMkSaoRw5kkSVKN+IQAzdjg+iFWr9vM1uERFvlIJkmSZsRwphkZXD/EqrUbGdk1CsDQ8Air1m4EMKBJkrQPDGeakdXrNu8JZmNGdo2ycs2GPc/QHG/Tth0sWbigHeVJkjTneM2ZZmTr8EjT9p2juyfcZ8nCBZx2nL1qkiQ1Y8+ZZmRRfx9DTQLaQH8fF591cgcqkiRpbrPnTDOyYtli+np79mrr6+1hxbLFHapIkqS5zZ4zzcjYRf8r12xg5+huBrxbU5KkGTGcacaWLx3Yc/G/Q5mSJM2Mw5qSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0mSpBpxKg1NanD9EKvXbWbr8AiLnMNMkqTKGc40ocH1Q6xau3HPg82HhkdYtXYjgAFNkqSKGM40odXrNu8JZmNGdo2ycs2GPZPOjtm0bQdLFi5oZ3mSJM1LXnOmCW1t8kBzgJ2jux/QtmThAk47zt40SZJmqtKes4g4Ffgw0AN8MjPfN279KcCXgVvKprWZ+a5W9lX1FvX3MdQkoA309/mYJkmSKlJZz1lE9AAfA54DLAFeGhFLmmz6/cw8rvx51zT3VYVWLFtMX2/PXm19vT2sWLa4QxVJkjT/VTmseRJwU2benJk7gYuA09qwr2bJ8qUDnHv64zigp/iaDPT3ce7pj/NmAEmSKlTlsOYAcHvD8hbgiU22OzkirgO2Am/OzBumsS8RcSZwJsARRxwxC2Wr0fKlA3su/ncoU5Kk6lXZcxZN2nLc8k+AR2XmscA/AoPT2LdozDw/M0/IzBMOOeSQfa1VkiSpFqoMZ1uAwxuWD6PoHdsjM3dk5j3l75cBvRFxcCv7SpIkzUdVhrOrgGMi4qiIOAA4A7i0cYOI+MOIiPL3k8p67mplX0mSpPmosmvOMvO+iDgbWEcxHcanM/OGiHh1uf484IXAayLiPmAEOCMzE2i6b1W1SpIk1UWl85yVQ5WXjWs7r+H3jwIfbXVfSZKk+c7HN+kBxj/s/MH778fBBz2o02VJktQVDGfaS7OHne/X7N5ZSZJUCZ+tqb00e9j57oTbf938OZuSJGl2Gc60l+k87FySJM0+w5n2sqi/r2n7wATtkiRpdhnOtBcfdi5JUmd5Q4D2MvZQ85VrNrBzdDcD/X2sWLbYh51LktQmhjM9gA87lySpcxzWlCRJqhHDmSRJUo0YziRJkmrEcCZJklQj3hAwz41/TqZ3XkqSVG+Gs3ms2XMyV63dCGBAkySppgxn81iz52SO7Bpl5ZoNe6bKmMimbTtYsnBBleVJkqQmvOZsHpvJczKXLFzAacfZuyZJUrvZczaPLervY6hJQBvo73NyWUmSasqes3nM52RKkjT32HM2j/mcTEmS5h7D2TznczIlSZpbHNaUJEmqEXvO5plmk85KkqS5w56zeWRs0tmh4RGS+yedvfPu33e6NEmS1CLD2Twy0aSzN995b4cqkiRJ02U4m0cmmnQ2wQllJUmaIwxn88ii/r6m7QP9fbzsiUe0uRpJkrQvDGfziJPOSpI093m35jzipLOSJM19hrN5xklnJUma2wxnc0izOczsFZMkaX4xnM0RY3OYjU2VMTaHGWBAkyRpHjGczRETzWG2cs2GPcOYYzZt28GShQvaWZ4kSZol3q05R0w0h9nO0d0PaFuycIHzmkmSNEfZczZHLOrvY6hJQBvo7/PCf0mS5hF7zuYI5zCTJKk72HM2RziHmSRJ3cFwNgc0TqHR27MfRx98IN988ymdLkuSJFXAYc2aG5tCY2h4hKS4AeCWu+5lcP1Qp0uTJEkVMJzVXLMpNHZn0S5JkuYfw1nNTTSFxkTtkiRpbjOc1dyi/r5ptUuSpLnNcFZzTqEhSVJ3MZzNAQ/a//7T9PCH9HLu6Y9zCg1JkuYpp9KosfEPOwf43a4HPq5JkiTNH/ac1dhEDzv3Tk1JkuYvw1mNeaemJEndx3BWY96pKUlS9zGc1Zh3akqS1H28IaDGfNi5JEndx3DWAY0PMl80ReBavnSAC6+8DYCLzzq5nWVKkqQOMJy12fjpMYaGR1i1diOAPWKSJMlw1m4TTY+xcs2GPT1k423atoMlCxe0ozxJktRh3hDQRoPrhxiaYBqMnaMTTy67ZOECTjvOXjVJkrqBPWdtMjacOZGB/j6vKZMkSfactUuz4cwxTo8hSZLGGM7aZLJZ/X2QuSRJGmM4a4PB9UPsF9F03UB/n8FMkiTtYTir2Ni1ZqOZD1jncKYkSRrPcFaxia4164lwOFOSJD2A4axiE11rtjvTYCZJkh7AcFax/of0TqtdkiR1N8NZxZpcajZpuyRJ6m6Gs4r9ZmTXtNolSVJ3M5xVzGFNSZI0HYazijmsKUmSpsNwVrFhhzUlSdI0GM4qNLh+iObPBYBF/X1trUWSJM0NhrMKrV63mWajlwE+GUCSJDVlOKvQRBPQJjgBrSRJaspwVqGJhi4HHNKUJEkTMJxVaMWyxfT19uzV5sPOJUnSZPbvdAHz0eD6IVav28zW4REe1tfLrtHd3Lc7GejvY8WyxQ5pSpKkCRnOZtng+iFWrd3IyK5RoJhKY7+Aow8+kG+++ZTOFidJkmrPYc1Z9s5/u2FPMBuzO+H2Xze/OUCSJKmR4WwWDa4f4te/bT657M7R3W2uRpIkzUWGs1m0et3mCdd5h6YkSWqF4WwWTTSvGTjprCRJao3hbBb1P6S3eXtfr3doSpKklhjOZsng+iHu+d19D2jv7QnOef5jO1CRJEmaiwxns2T1us3s2v3AJ2keeMD+9ppJkqSWGc5mydAE15sNjzS/e1OSJKkZw9ks2S+at/fEBCskSZKaMJzNgsH1QzQZ0QRgNCdYIUmS1IThbBa8899umHCd85tJkqTpMJzNgomeCgDObyZJkqbHcFYx79SUJEnTYTibBf19E08+K0mSNB2Gs1nwvGMXTqtdkiRpIoazWfDVDduatn/7p9vbXIkkSZrrDGcz9PbBjRPeEDDZg9AlSZKaMZzNwOD6IT53xW0Trl/kNBqSJGmaDGczMNn8ZuA0GpIkafoMZzMw2fxm/X29TqMhSZKmzXBWkXOe/9hOlyBJkuYgw1lF7DWTJEn7wnAmSZJUI4YzSZKkGjGcSZIk1YjhbAZimu2SJElTMZzNQE6zXZIkaSqGs300uH5ownU9Yd+ZJEnaN4azfTTZ0wFG074zSZK0bwxn+2iypwMM+ExNSZK0jwxnFfCZmpIkaV8ZzvbBZNebgU8HkCRJ+85wtg9Wr9vc6RIkSdI8ZTjbB0PDIxOu6+/rbWMlkiRpvjGc7YPJJso45/mPbVsdkiRp/jGcTdPg+qFJJ5n1ejNJkjQThrNpetslGztdgiRJmscqDWcRcWpEbI6ImyLirZNsd2JEjEbECxvabo2IjRFxbURcXWWd03HvztEJ13m9mSRJmqn9qzpwRPQAHwOeBWwBroqISzNzU5Pt3g+sa3KYp2XmnVXVONu83kySJM1UlT1nJwE3ZebNmbkTuAg4rcl2rwO+BNxRYS1t4fVmkiRppqoMZwPA7Q3LW8q2PSJiAHgBcF6T/RP4WkRcExFnTvQiEXFmRFwdEVdv3759Fsqe2FSTz0qSJM1UleGs2YwT4290/BDwlsxsdiHXkzPzCcBzgL+NiKc2e5HMPD8zT8jMEw455JAZFTwVbwaQJElVq+yaM4qessMblg8Dto7b5gTgoogAOBh4bkTcl5mDmbkVIDPviIhLKIZJv1dhvVOa7GYASZKk2VBlz9lVwDERcVREHACcAVzauEFmHpWZR2bmkcAa4LWZORgRB0bEQQARcSDwbOD6CmudMe/UlCRJs6GynrPMvC8izqa4C7MH+HRm3hARry7XN7vObMyhwCVlj9r+wAWZeXlVtc4G79SUJEmzocphTTLzMuCycW1NQ1lm/nXD7zcDx1ZZ23RNdTOAd2pKkqTZ4BMCWjC4fog3XXxtp8uQJEldwHDWgtXrNrO700VIkqSuYDhrwdbhkU6XIEmSuoThrAWL+vsmXf/kox/RpkokSdJ8ZzhrwYpliyf8oA496AA+/6qT21qPJEmavyq9W3O+GLsTc9XaDYzsKq4+2y/gZU88gvcsf1wnS5MkSfOM4axFy5cOOF2GJEmqnMOakiRJNWI4kyRJqhHDmSRJUo0YziRJkmrEcCZJklQjhjNJkqQaMZxJkiTViOFMkiSpRgxnkiRJNWI4kyRJqhHDmSRJUo0YziRJkmrEcCZJklQjhjNJkqQaMZxJkiTViOFMkiSpRgxnkiRJNWI4kyRJqhHDmSRJUo0YziRJkmokMrPTNcyaiNgO/LzilzkYuLPi11B7eU7nJ8/r/OM5nZ+6+bw+KjMPGd84r8JZO0TE1Zl5Qqfr0OzxnM5Pntf5x3M6P3leH8hhTUmSpBoxnEmSJNWI4Wz6zu90AZp1ntP5yfM6/3hO5yfP6zhecyZJklQj9pxJkiTViOFMkiSpRgxnpYg4NSI2R8RNEfHWJusjIj5Srt8QEU9odV91zgzP66cj4o6IuL69VWsy+3pOI+LwiPh2RNwYETdExBvaX70mMoPz+uCIuDIirivP6zvbX72amcm/v+X6nohYHxFfaV/VNZGZXf8D9AA/Ax4NHABcBywZt81zgX8HAngS8ONW9/Vn7p3Xct1TgScA13f6vfgz83MKLASeUP5+EPCf/l2tx88Mz2sADy1/7wV+DDyp0++p239m+u9vuf5NwAXAVzr9ftr9Y89Z4STgpsy8OTN3AhcBp43b5jTgX7JwBdAfEQtb3FedMZPzSmZ+D/hVWyvWVPb5nGbmtsz8CUBm3g3cCAy0s3hNaCbnNTPznnKb3vLHO906b0b//kbEYcD/BXyynUXXheGsMADc3rC8hQf+oz3RNq3sq86YyXlVPc3KOY2II4GlFL0s6rwZnddy+Ota4A7g65npee28mf5d/RCwEthdUX21ZjgrRJO28f/nNdE2reyrzpjJeVU9zficRsRDgS8Bb8zMHbNYm/bdjM5rZo5m5nHAYcBJEfGns1ue9sE+n9OIeB5wR2ZeM/tlzQ2Gs8IW4PCG5cOArS1u08q+6oyZnFfV04zOaUT0UgSzz2fm2grr1PTMyt/VzBwGvgOcOusVarpmck6fDDw/Im6lGA59ekR8rrpS68dwVrgKOCYijoqIA4AzgEvHbXMp8Mry7pInAb/JzG0t7qvOmMl5VT3t8zmNiAA+BdyYmR9sb9mawkzO6yER0Q8QEX3AM4GftrF2NbfP5zQzV2XmYZl5ZLnftzLzFW2tvsP273QBdZCZ90XE2cA6ijtMPp2ZN0TEq8v15wGXUdxZchPwW+BvJtu3A29D48zkvAJExIXAKcDBEbEFeEdmfqq970KNZnhOnwz8JbCxvD4J4O8y87I2vgU1McPzuhD4bET0UHQ4fCEzu2/qhZqZ6b+/3c7HN0mSJNWIw5qSJEk1YjiTJEmqEcOZJElSjRjOJEmSasRwJkmSVCOGM0lzQkS8KCJujIhvl8sXRsSGiPif0zxOf0S8tmF5UUSsme16x73mPbOxjaTu4FQakuaEiLgceH9mfjsi/hD4cWY+ah+OcyTwlcxs2yN+IuKezHzoTLeR1B3sOZNUKxHxioi4MiKujYhPlA+1/nvgKcB5EbEa+BrwyHKbP4+IoyPi8oi4JiK+HxF/XB7r0Ii4JCKuK3/+DHgfcHS57+qIODIiri+3/3FEPLahlu9ExPERcWBEfDoiroqI9RFxWpO6HxoR34yIn0TExgm2OSUivlfWtCkizouI/RrWv7es84qIOLRs+4uyrvUR8Y2xdknzl+FMUm1ExJ8ALwGeXD7IehR4eWa+C7i6/H0F8HzgZ5l5XGZ+HzgfeF1mHg+8Gfh4eciPAN/NzGOBJwA3AG9t2HfFuBIuAl5c1rIQWFQ+fPltFI+QORF4GrA6Ig4ct+/vgBdk5hPKbT5QPjJqvJOA/xd4HHA0cHrZfiBwRVnr94BXle3/ATwpM5eW9a2c8oOUNKf5+CZJdfIM4HjgqjLX9AF3TLZDRDwU+DPgiw1Z6EHln08HXgmQmaPAbyLi4ZMc7gvA14F3UIS0L5btz6Z4EPOby+UHA0cANzaWAvyviHgqsBsYAA4FfjHuNa7MzJvL2i+k6BFcA+wExh47dA3wrPL3w4CLy7B4AHDLJPVLmgcMZ5LqJIDPZuaqaeyzHzBc9rTNSGYORcRdEfF4ih68sxrq+r8zc/Mku78cOAQ4PjN3RcStFCHuAS8zwfKuvP8i4FHu//f5H4EPZualEXEKcE7r70jSXOSwpqQ6+Sbwwoh4JEBEPCIiJr3oPzN3ALdExIvKfSIijm043mvK9p6IWADcDRw0ySHHhg4flpkby7Z1wOvGhikjYmmT/R4G3FEGs6cBE9V9UkQcVV5r9hKKYcvJPAwYKn//qym2lTQPGM4k1UZmbgLeDnwtIjZQDDEubGHXlwP/PSKuo7iubOxi/DcAT4uIjRRDhY/NzLuAH0TE9eXNBeOtAc6gGOIc826gF9hQ3jzw7ib7fR44ISKuLuv56QS1/ojipoTrKYYoL5nivZ1DMWT7feDOKbaVNA84lYYktUk5LPnmzHxeh0uRVGP2nEmSJNWIPWeSJEk1Ys+ZJElSjRjOJEmSasRwJkmSVCOGM0mSpBoxnEmSJNXI/w/gsqIYW1TY0AAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light",
            "tags": []
          },
          "output_type": "display_data"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fL73mGetZajH"
      },
      "source": [
        "Next, we train a decision tree using the effective ``alphas``. The last value in ``ccp_alphas`` is the alpha value that prunes the whole tree, leaving the tree, ``clfs[-1]``, with one node."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HyY3Ir8-ZajI",
        "outputId": "1e25d502-3ee8-4f24-894e-8e55089badf2"
      },
      "source": [
        "clfs = []\n",
        "for ccp_alpha in ccp_alphas:\n",
        "    clf = DecisionTreeClassifier(random_state=0, ccp_alpha=ccp_alpha)\n",
        "    clf.fit(X_train, y_train)\n",
        "    clfs.append(clf)\n",
        "print(\"Number of nodes in the last tree is: {} with ccp_alpha: {}\".format(\n",
        "      clfs[-1].tree_.node_count, ccp_alphas[-1]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of nodes in the last tree is: 1 with ccp_alpha: 0.15397075459323895\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tW-1uafhZajN"
      },
      "source": [
        "Now, we remove the last element in ``clfs`` and ``ccp_alphas``, because it is the trivial tree with only one node.\n",
        "\n",
        "\n",
        "Here we show that the number of nodes and tree depth decreases as alpha increases."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9TBgdT7sZajO",
        "outputId": "1fa4b095-1509-4d15-e168-5034c24e807e"
      },
      "source": [
        "clfs = clfs[:-1]\n",
        "ccp_alphas = ccp_alphas[:-1]\n",
        "\n",
        "node_counts = [clf.tree_.node_count for clf in clfs]\n",
        "depth = [clf.tree_.max_depth for clf in clfs]\n",
        "\n",
        "fig, ax = plt.subplots(2, 1,figsize=(10,10))\n",
        "ax[0].plot(ccp_alphas, node_counts, marker='o', drawstyle=\"steps-post\")\n",
        "ax[0].set_xlabel(\"alpha\")\n",
        "ax[0].set_ylabel(\"number of nodes\")\n",
        "ax[0].set_title(\"Number of nodes vs alpha\")\n",
        "ax[1].plot(ccp_alphas, depth, marker='o', drawstyle=\"steps-post\")\n",
        "ax[1].set_xlabel(\"alpha\")\n",
        "ax[1].set_ylabel(\"depth of tree\")\n",
        "ax[1].set_title(\"Depth vs alpha\")\n",
        "fig.tight_layout()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAALICAYAAABiqwZ2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABDQ0lEQVR4nO3de7hddX3n8feHECSKGijBgQAGLcUqKmBEKNbxRqHeoJ1R8VLROqIdKzq2WGg7tba10tJaa+epFbylBRW0NFJBkUHREbkYDBIBKSAISSgEYwQ1cgnf+WOvg4vDuezk7H323ue8X8+zn7P2b63fWt999iL55MdvrZWqQpIkSVLHdoMuQJIkSRomBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSNIUkn0jyFwM6dpJ8PMkPk1ze52M9N8nafh6jH5K8PsnXe72tpPnNgCxppCS5OcntSR7VavsfSS4aYFn98mzgcGDPqjp40MVI0nxhQJY0irYH3j7oIrZWkgVb2eXxwM1V9ZN+1CNJmpgBWdIoOgX4/SSLx69IsixJJdm+1XZRkv/RLL8+ycVJ/i7JpiTfS/IrTfutSe5Icuy43e6a5IIkdyf5apLHt/b9pGbdxiTXJXlFa90nknwoyXlJfgI8b4J690hyTtP/hiRvatrfCHwEODTJj5O8Z4K+r0/y9SR/00zDuCnJr0+372bdoqa+Hya5BnjmBHX9a5INzX6Pb607OMmqJHc1o/nvn+A7Ism1SV7Ser99kjuTHJRkxySnJ/lB8z18M8njJtnPiUlubH7/1yT5jYm2a7atJMc33+udSU5Jst24bSb7fb2hqfnupv+bJzuOpLnNgCxpFK0CLgJ+fxv7Pwu4CvgF4JPAp+kExF8EXgv8nyQ7tbZ/DfDnwK7AlcAZAM00jwuafewGvAr4xyRPafV9NfBe4NHARPNfPwWsBfYA/jvwl0leUFUfBd4CXFJVO1XVu6f4LNc1tf018NEkmWrfzbp3A09sXkcAD/6joAmU/w58G1gKvAB4R5Ijmk3+Hvj7qnpM0/+sSWr7VPM7GXMEcGdVfas53mOBveh8D28BNk+ynxuBX222fw9wepLdJ9kW4DeA5cBBwFHAb7fWTfX7ugN4CfAY4A3A3yU5aIrjSJqjDMiSRtWfAG9LsmQb+t5UVR+vqi3AmXRC2p9V1T1V9SXgXjphecy5VfW1qroH+CM6o7p70QlTNzf7ur8Jfv9KJ4yO+VxVXVxVD1TVz9pFNPt4NvAHVfWzqrqSzqjxb23FZ/l+VZ3WfJYVwO7A47rY9yuA91bVxqq6Ffhga5/PBJZU1Z9V1b1V9T3gNOCYZv19wC8m2bWqflxVl05S2yeBlyV5ZPP+1U3b2D5+AfjFqtpSVVdU1V0T7aSqPlNV65vf4ZnA9cBUc7L/qvlctwAf4KEhfcLfV3Occ6vqxur4KvAlOsFc0jxjQJY0kqrqO8DngRO3ofvtreXNzf7Gt7VHkG9tHffHwEY6o7KPB57VTBHYlGQTndHm/zJR3wnsAWysqrtbbd+nM2rbrf9s1fbTZnGnLva9x7javt9afjywx7jP9Yc0QRJ4I/BLwHebqREvYQJVdQNwLfDSJiS/jJ8H5H8Bzgc+nWR9kr9OsnCi/SR5XZIrW7XsT2cEeDLjP9cerfeT/b5I8utJLm2mpGwCXjTNcSTNUdtPv4kkDa13A98C/rbVNnZB2yOBsRHJdmDdFnuNLTRTL3YB1tMJYl+tqsOn6FtTrFsP7JLk0a0guzewbob1drPv2+h8rqtb68bcSmeUfd+JdlxV1wOvaqZi/Cbw2SS/MMnFhGPTLLYDrmlCM1V1H53pEu9Jsgw4j87Uh4+2OzfzvU+jM83jkqrakuRKIExu/OdaP8W2Y8d5BJ3R/9fRGfW/L8nKaY4jaY5yBFnSyGrC1pnA8a22DXRC4GuTLEjy23Tmyc7Ei5I8O8kOdOYiX9ZMS/g88EtJfivJwub1zCS/3GX9twLfAN7XXLT2NDqjs2fMsN5u9n0WcFKSnZPsCbyt1f1y4K4kf9BczLcgyf5JngmQ5LVJllTVA8Cmps+WSUr5NPBrwO/w89FjkjwvyVPTubPHXXSmXEy0j0fR+UfGhqbfG+iMIE/lhOZz7UXnbidnTrM9wA7AI5rj3N9cvPdrXfSTNAcZkCWNuj+jE6La3gScAPwAeAqdoDgTn6QzWr0ReAadaRQ0I7O/Rmdu7no6//v+r+gErW69CljW9P834N1VdcEM6+1m3++hM/3gJjpzbf9lrFMzP/elwAHN+jvpzF9+bLPJkcDVSX5M54K9Y8bPr27t6zbgEuBXeGhQ/S/AZ+mE42uBrwKnT9D/Gjr/h+ASOlNjngpcPM3n/hxwBZ0LKs9l3Kj0JHXeTecfWmcBP6QzX/qc6fpJmptSNdX//ZMkaXQkKWDfsakckrQtHEGWJEmSWgzIkiRJUotTLCRJkqQWR5AlSZKkljl7H+Rdd921li1bNugyJEmSNKSuuOKKO6vqYU9knbMBedmyZaxatWrQZUiSJGlIJfn+RO1OsZAkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJklrm7JP0ZtPK1es45fzrWL9pM3ssXsQJR+zH0QcuHXRZkiRJ2gYG5BlauXodJ529hs33bQFg3abNnHT2GgBDsiRJ0ghyisUMnXL+dQ+G4zGb79vCKedfN6CKJEmSNBMG5Blav2nzVrVLkiRpuBmQZ2iPxYu2ql2SJEnDzYA8QyccsR+LFi54SNt26bRLkiRp9HiR3gyNXYg3dheLhQu2Y6+dF3mBniRJ0ogyIPfA0QcufTAQv/LDlwy4GkmSJM2EUywkSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktfQ1ICf5X0muTvKdJJ9KsmOSXZJckOT65ufOre1PSnJDkuuSHNFqf0aSNc26DyZJP+uWJEnS/NW3gJxkKXA8sLyq9gcWAMcAJwIXVtW+wIXNe5I8uVn/FOBI4B+TLGh29yHgOGDf5nVkv+qWJEnS/NbvKRbbA4uSbA88ElgPHAWsaNavAI5ulo8CPl1V91TVTcANwMFJdgceU1WXVFUB/9zqI0mSJPVU3wJyVa0D/ga4BbgN+FFVfQl4XFXd1mxzG7Bb02UpcGtrF2ubtqXN8vj2h0lyXJJVSVZt2LChlx9HkiRJ80Q/p1jsTGdUeB9gD+BRSV47VZcJ2mqK9oc3Vp1aVcuravmSJUu2tmRJkiSpr1MsXgjcVFUbquo+4GzgV4Dbm2kTND/vaLZfC+zV6r8nnSkZa5vl8e2SJElSz/UzIN8CHJLkkc1dJ14AXAucAxzbbHMs8Llm+RzgmCSPSLIPnYvxLm+mYdyd5JBmP69r9ZEkSZJ6avt+7biqLkvyWeBbwP3AauBUYCfgrCRvpBOiX95sf3WSs4Brmu3fWlVbmt39DvAJYBHwheYlSZIk9VzfAjJAVb0bePe45nvojCZPtP17gfdO0L4K2L/nBUqSJEnj+CQ9SZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLU0teAnGRxks8m+W6Sa5McmmSXJBckub75uXNr+5OS3JDkuiRHtNqfkWRNs+6DSdLPuiVJkjR/9XsE+e+BL1bVk4CnA9cCJwIXVtW+wIXNe5I8GTgGeApwJPCPSRY0+/kQcBywb/M6ss91S5IkaZ7qW0BO8hjgOcBHAarq3qraBBwFrGg2WwEc3SwfBXy6qu6pqpuAG4CDk+wOPKaqLqmqAv651UeSJEnqqX6OID8B2AB8PMnqJB9J8ijgcVV1G0Dzc7dm+6XAra3+a5u2pc3y+PaHSXJcklVJVm3YsKG3n0aSJEnzQj8D8vbAQcCHqupA4Cc00ykmMdG84pqi/eGNVadW1fKqWr5kyZKtrVeSJEnqa0BeC6ytqsua95+lE5hvb6ZN0Py8o7X9Xq3+ewLrm/Y9J2iXJEmSeq5vAbmq/hO4Ncl+TdMLgGuAc4Bjm7Zjgc81y+cAxyR5RJJ96FyMd3kzDePuJIc0d694XauPJEmS1FPb93n/bwPOSLID8D3gDXRC+VlJ3gjcArwcoKquTnIWnRB9P/DWqtrS7Od3gE8Ai4AvNC9JkiSp5/oakKvqSmD5BKteMMn27wXeO0H7KmD/nhYnSZIkTcAn6UmSJEktBmRJkiSpxYAsSZIktUwbkJM8Ksl2zfIvJXlZkoX9L02SJEmafd2MIH8N2DHJUuBCOnei+EQ/i5IkSZIGpZuAnKr6KfCbwD9U1W8AT+5vWZIkSdJgdBWQkxwKvAY4t2nr9/2TJUmSpIHoJiC/AzgJ+LfmYR5PAL7S16okSZKkAZl2JLiqvgp8NcmjmvffA47vd2GSJEnSIHRzF4tDk1wDXNu8f3qSf+x7ZZIkSdIAdDPF4gPAEcAPAKrq28Bz+liTJEmSNDBdPSikqm4d17SlD7VIkiRJA9fN3ShuTfIrQCXZgc7842v7W5YkSZI0GN2MIL8FeCuwFFgLHNC8lyRJkuacbu5icSedeyBLkiRJc96kATnJPwA12fqq8lZvkiRJmnOmmmKxCrgC2BE4CLi+eR2AF+lJkiRpjpp0BLmqVgAkeT3wvKq6r3n/T8CXZqU6SZIkaZZ1c5HeHsCjW+93atokSZKkOaeb27ydDKxO8pXm/X8F/rRvFUmSJEkD1M1dLD6e5AvAs+hctHdiVf1n3yuTJEmSBqCbEWSAg4FfbZYL+Pf+lCNJkiQN1rRzkJOcDLwduKZ5HZ/kff0uTJIkSRqEbkaQXwQcUFUPACRZAawGTupnYZIkSdIgdHMXC4DFreXH9qEOSZIkaSh0M4L8Pn5+F4sAz8HRY0mSJM1R3dzF4lNJLgKeSScg/4F3sZAkSdJc1e0Ui+2AO4EfAr+U5Dn9K0mSJEkanGlHkJP8FfBK4Grggaa5gK/1sS5JkiRpILqZg3w0sF9V3dPnWiRJkqSB62aKxfeAhf0uRJIkSRoG3Ywg/xS4MsmFwIOjyFV1fN+qkiRJkgakm4B8TvOSJEmS5rxubvO2YjYKkSRJkoZBt7d5kyRJkuYFA7IkSZLUMmlATvIvzc+3z145kiRJ0mBNNYL8jCSPB347yc5Jdmm/ZqtASZIkaTZNdZHePwFfBJ4AXAGkta6adkmSJGlOmXQEuao+WFW/DHysqp5QVfu0XoZjSZIkzUnd3Obtd5I8HfjVpulrVXVVf8uSJEmSBmPau1gkOR44A9iteZ2R5G39LkySJEkahG6epPc/gGdV1U8AkvwVcAnwD/0sTJIkSRqEbu6DHGBL6/0WHnrBniRJkjRndDOC/HHgsiT/1rw/Gvho3yqSJEmSBqibi/Ten+Qi4Nl0Ro7fUFWr+12YJEmSNAjdjCBTVd8CvtXnWiRJkqSB62YOsiRJkjRvGJAlSZKklikDcpIFSf7vbBUjSZIkDdqUAbmqtgA/TfLYWapHkiRJGqhuLtL7GbAmyQXAT8Yaq+r4vlUlSZIkDUg3Afnc5iVJkiTNed3cB3lFkkXA3lV13dYeIMkCYBWwrqpekmQX4ExgGXAz8Iqq+mGz7UnAG+k8re/4qjq/aX8G8AlgEXAe8Paqqq2tRZIkSZrOtHexSPJS4Ergi837A5KcsxXHeDtwbev9icCFVbUvcGHzniRPBo4BngIcCfxjE64BPgQcB+zbvI7ciuNLkiRJXevmNm9/ChwMbAKoqiuBfbrZeZI9gRcDH2k1HwWsaJZX0Hl09Vj7p6vqnqq6CbgBODjJ7sBjquqSZtT4n1t9JEmSpJ7qJiDfX1U/GtfW7fSGDwDvAh5otT2uqm4DaH7u1rQvBW5tbbe2aVvaLI9vf5gkxyVZlWTVhg0buixRkiRJ+rluAvJ3krwaWJBk3yT/AHxjuk5JXgLcUVVXdFlLJmirKdof3lh1alUtr6rlS5Ys6fKwkiRJ0s91E5DfRmde8D3Ap4C7gHd00e8w4GVJbgY+DTw/yenA7c20CZqfdzTbrwX2avXfE1jftO85QbskSZLUc9MG5Kr6aVX9EfAC4HlV9UdV9bMu+p1UVXtW1TI6F999uapeC5wDHNtsdizwuWb5HOCYJI9Isg+di/Eub6Zh3J3kkCQBXtfqI0mSJPXUtLd5S/JM4GPAo5v3PwJ+eyumTox3MnBWkjcCtwAvB6iqq5OcBVwD3A+8tXmSH8Dv8PPbvH2heUmSJEk9182DQj4K/M+q+n8ASZ4NfBx4WrcHqaqLgIua5R/QGY2eaLv3Au+doH0VsH+3x5MkSZK2VTdzkO8eC8cAVfV14O7+lSRJkiQNzqQjyEkOahYvT/JhOhfoFfBKmtFgSZIkaa6ZaorF3457/+7Wso95liRJ0pw0aUCuqufNZiGSJEnSMOjmLhaL6dxabVl7+6o6vm9VSZIkSQPSzV0szgMuBdbw0EdGS5IkSXNONwF5x6p6Z98rkSRJkoZAN7d5+5ckb0qye5Jdxl59r0ySJEkagG5GkO8FTgH+iJ/fvaKAJ/SrKEmSJGlQugnI7wR+saru7HcxkiRJ0qB1M8XiauCn/S5EkiRJGgbdjCBvAa5M8hXgnrFGb/MmSZKkuaibgLyyeUmSJElz3rQBuapWzEYhkiRJ0jDo5kl6N/Hzu1c8qKq8i4UkSZLmnG6mWCxvLe8IvBzwPsiSJEmak6a9i0VV/aD1WldVHwCe3//SJEmSpNnXzRSLg1pvt6MzovzovlUkSZIkDVA3Uyz+trV8P3Az8Iq+VCNJkiQNWDd3sXjebBQiSZIkDYNuplg8AvhvwLL29lX1Z/0rS5IkSRqMbqZYfA74EXAFrSfpSZIkSXNRNwF5z6o6su+VSJIkSUNg2tu8Ad9I8tS+VyJJkiQNgW5GkJ8NvL55ot49QICqqqf1tTJJkiRpALoJyL/e9yokSZKkIdHNbd6+PxuFSJIkScOgmznIkiRJ0rxhQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktTSt4CcZK8kX0lybZKrk7y9ad8lyQVJrm9+7tzqc1KSG5Jcl+SIVvszkqxp1n0wSfpVtyRJkua3fo4g3w/8XlX9MnAI8NYkTwZOBC6sqn2BC5v3NOuOAZ4CHAn8Y5IFzb4+BBwH7Nu8juxj3ZIkSZrH+haQq+q2qvpWs3w3cC2wFDgKWNFstgI4ulk+Cvh0Vd1TVTcBNwAHJ9kdeExVXVJVBfxzq48kSZLUU7MyBznJMuBA4DLgcVV1G3RCNLBbs9lS4NZWt7VN29JmeXz7RMc5LsmqJKs2bNjQ088gSZKk+aHvATnJTsC/Au+oqrum2nSCtpqi/eGNVadW1fKqWr5kyZKtL1aSJEnzXl8DcpKFdMLxGVV1dtN8ezNtgubnHU37WmCvVvc9gfVN+54TtEuSJEk918+7WAT4KHBtVb2/teoc4Nhm+Vjgc632Y5I8Isk+dC7Gu7yZhnF3kkOafb6u1UeSJEnqqe37uO/DgN8C1iS5smn7Q+Bk4KwkbwRuAV4OUFVXJzkLuIbOHTDeWlVbmn6/A3wCWAR8oXlJkiRJPde3gFxVX2fi+cMAL5ikz3uB907QvgrYv3fVSZIkSRPzSXqSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkHto5ep1rL5lE5fdtJHDTv4yK1evG3RJkiRJ2koG5B5ZuXodJ529hnu3PADAuk2bOensNYZkSZKkEWNA7pFTzr+OzfdteUjb5vu2cMr51w2oIkmSJG0LA3KPrN+0eavaJUmSNJwMyD2yx+JFW9UuSZKk4WRA7pETjtiPRQsXPKRt0cIFnHDEfgOqSJIkSdti+0EXMFccfeBSAN712au4d8sDLF28iBOO2O/BdkmSJI0GR5AlSZKkFkeQe2Tl6nW888wreaB5v27TZt555pUAjiJLkiSNEEeQe+Sks696MByPeaBplyRJ0ugwIPfI5vvGx+Op2yVJkjScDMiSJElSiwF5FrzmtEsGXYIkSZK6ZECeBRffuNGQLEmSNCK8i0WPbBd4oCZff/GNG1l24rkAPGL77fir//Y0724hSZI0hFI1RaobYcuXL69Vq1bN2vH+eOUaTr/0lm3quyDhkCfszOU3bWT8NX0LErZUdf3gkZWr13HK+dexbtPmB/t2s4+xfus3bWaPZjvgYW2GekmSNFckuaKqlj+sfVQCcpIjgb8HFgAfqaqTp9p+tgMy8OAIcb8sWriA9/3mUycNqStXr+Oks9ew+b4tW7WPifot3C4QuG9LTdl3MuMD9/OetISvfHfDtGF7oqDe7T8KBh3kp6tjWOocVf7+5je/f82E58/wGvR3M9IBOckC4D+Aw4G1wDeBV1XVNZP1GURAPvz9F3H9HT/p6zF2WLAdB+69eMJ1q2/ZxL1bpr+t3Ph9dNtvuuOPufPue7jpBz+ZcspJt0F9W/5RsDVBvlemq2NY6hxV/v7mN79/zYTnz/Aahu9m1APyocCfVtURzfuTAKrqfZP1GURAhv6PIgM8a59dJmy/7KaN27SPrek31fHH9Dqob8s/CroJ8r00XR3DUueo8vc3v/n9ayY8f4bXZN/N0sWLuPjE589KDZMF5FG5SG8pcGvr/VrgWeM3SnIccBzA3nvvPTuVjXPzyS/mSX90Hj/b0p9/eCxdvIgz33zohOsOO/nLrNu0eav30W2/6Y4/Zp8u/5Ew/j+KyUL1VGF7W/r0w3R1DEudo8rf3/zm96+Z8PwZXpN9B+u7zCT9NCoBORO0PSyBVtWpwKnQGUHud1GT+e57XwT0fsrFooULHrx4biInHLFfV3OQx+9jon6TzUGe6vhj9li8qKdBfVv+UdBNkO+l6eoYljpHlb+/+c3vXzPh+TO8Jvtu9li8aADVPNSo3Ad5LbBX6/2ewPoB1dK1C975XG4++cUPvg574sRTExYkHPbEXVg4wbexIJ1/GyxdvGjaOTlHH7iU9/3mU1nanFhjfafbR7tfmu1OefnTOeW/P/0hbd3OCTrhiP1YtHDBlNtMFtTH9+vmHwVb26cfpqtjWOocVf7+5je/f82E58/wGubvZlTmIG9P5yK9FwDr6Fyk9+qqunqyPoOag6wO72LhXSx6zd/f/Ob3r5nw/Bleg/5uRvoiPYAkLwI+QOc2bx+rqvdOtb0BWZIkSVMZ9Yv0qKrzgPMGXYckSZLmtlGZgyxJkiTNCgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqWVkHhSytZJsAL4/y4fdFbhzlo+p0eN5oul4jmg6niOajudIdx5fVUvGN87ZgDwISVZN9DQWqc3zRNPxHNF0PEc0Hc+RmXGKhSRJktRiQJYkSZJaDMi9deqgC9BI8DzRdDxHNB3PEU3Hc2QGnIMsSZIktTiCLEmSJLUYkCVJkqQWA3KXkhyZ5LokNyQ5cYL1SfLBZv1VSQ7qtq/mhhmeIx9LckeS78xu1ZpN23qOJNkryVeSXJvk6iRvn/3qNVtmcJ7smOTyJN9uzpP3zH71mg0z+fumWb8gyeokn5+9qkdMVfma5gUsAG4EngDsAHwbePK4bV4EfAEIcAhwWbd9fY3+aybnSLPuOcBBwHcG/Vl8Dd85AuwOHNQsPxr4D/8cmZuvGZ4nAXZqlhcClwGHDPoz+Rqec6S1/p3AJ4HPD/rzDOvLEeTuHAzcUFXfq6p7gU8DR43b5ijgn6vjUmBxkt277KvRN5NzhKr6GrBxVivWbNvmc6SqbquqbwFU1d3AtcDS2Sxes2Ym50lV1Y+bbRY2L6/En3tm9PdNkj2BFwMfmc2iR40BuTtLgVtb79fy8L+cJtumm74afTM5RzQ/9OQcSbIMOJDO6KDmnhmdJ83/Or8SuAO4oKo8T+aemf5Z8gHgXcADfapvTjAgdycTtI3/V/lk23TTV6NvJueI5ocZnyNJdgL+FXhHVd3Vw9o0PGZ0nlTVlqo6ANgTODjJ/r0tT0Ngm8+RJC8B7qiqK3pf1txiQO7OWmCv1vs9gfVdbtNNX42+mZwjmh9mdI4kWUgnHJ9RVWf3sU4NVk/+LKmqTcBFwJE9r1CDNpNz5DDgZUlupjM14/lJTu9fqaPLgNydbwL7JtknyQ7AMcA547Y5B3hdc+XoIcCPquq2Lvtq9M3kHNH8sM3nSJIAHwWurar3z27ZmmUzOU+WJFkMkGQR8ELgu7NYu2bHNp8jVXVSVe1ZVcuafl+uqtfOavUjYvtBFzAKqur+JL8LnE/n6tGPVdXVSd7SrP8n4Dw6V43eAPwUeMNUfQfwMdRHMzlHAJJ8CngusGuStcC7q+qjs/sp1E8zPEcOA34LWNPMLwX4w6o6bxY/gmbBDM+T3YEVSRbQGQA7q6q8jdccM9O/b9QdHzUtSZIktTjFQpIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiTNEUluTrLrTLeRpPnOgCxJkiS1GJAlaQQlWZnkiiRXJzlu3LplSb6bZEWSq5J8NskjW5u8Lcm3kqxJ8qSmz8FJvpFkdfNzv1n9QJI0RAzIkjSafruqngEsB45P8gvj1u8HnFpVTwPuAv5na92dVXUQ8CHg95u27wLPqaoDgT8B/rKv1UvSEDMgS9JoOj7Jt4FLgb2Afcetv7WqLm6WTwee3Vp3dvPzCmBZs/xY4DNJvgP8HfCUfhQtSaPAgCxJIybJc4EXAodW1dOB1cCO4zarKd7f0/zcAmzfLP858JWq2h946QT7k6R5w4AsSaPnscAPq+qnzRziQybYZu8khzbLrwK+3sU+1zXLr+9JlZI0ogzIkjR6vghsn+QqOiO/l06wzbXAsc02u9CZbzyVvwbel+RiYEEvi5WkUZOq8f8XTpI0ypIsAz7fTJeQJG0lR5AlSZKkFkeQJUmSpBZHkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJakEZHk5iQvHHQdY5L8aZLTe72tJA2aAVmStkETVjcnuTvJpiTfSPKWJD35czXJJ5L8RS/2JUnaOgZkSdp2L62qRwOPB04G/gD46GBLkiTNlAFZkmaoqn5UVecArwSOTbI/QJJHJPmbJLckuT3JPyVZ1Kx7bpK1Sf4wyZ3NiPRrmnXHAa8B3pXkx0n+vXW4A5JcleRHSc5MsuP4eprjbhqro2lb0ox475Zk1ySfb7bZmOT/TTbyneTvk9ya5K4kVyT51Um2W5akkhyXZH2S25L83rjNdkjyz82o+9VJlrf6n5jkxmbdNUl+o5vfvST1gwFZknqkqi4H1gJjIfKvgF8CDgB+EVgK/Emry38Bdm3ajwVOTbJfVZ0KnAH8dVXtVFUvbfV5BXAksA/wNOD1E9RxD3A28Kpx/b5aVXcAv9fUuQR4HPCHQE3ysb7Z1L8L8EngMxOF8pbnAfsCvwacOG7O9MuATwOLgXOA/9NadyOd39tjgfcApyfZfYrjSFLfGJAlqbfWA7skCfAm4H9V1caquhv4S+CYcdv/76q6p6q+CpxLJ8hO5YNVtb6qNgL/Tie8TuSTPDQgv7ppA7gP2B14fFXdV1X/r6omDMhVdXpV/aCq7q+qvwUeAew3RX3vqaqfVNUa4OPjavh6VZ1XVVuAfwGe3jrOZ5rP9UBVnQlcDxw8xXEkqW8MyJLUW0uBjXRGZx8JXNFMZdgEfLFpH/PDqvpJ6/33gT2m2f9/tpZ/Cuw0yXZfBhYleVaSx9MJ0v/WrDsFuAH4UpLvJTlxsoMl+b0k1zZTOjbRGeHddYr6bm0tj/8842vfMcn2zXFel+TK1u9q/2mOI0l9Y0CWpB5J8kw6AfnrwJ3AZuApVbW4eT22qtqBduckj2q935vOCDRMPuWhK1X1AHAWnRHcVwOfb0axqaq7q+r3quoJwEuBdyZ5wQSf51fpXHj4CmDnqloM/AjIFIfea5LPM6kmwJ8G/C7wC81xvjPNcSSpbwzIkjRDSR6T5CV05teeXlVrmoB6GvB3SXZrtlua5Ihx3d+TZIcmjL4E+EzTfjvwhBmW9kk6Fw6+hp9PryDJS5L8YjMN5C5gS/Ma79HA/cAGYPskfwI8Zppj/u8kj0zyFOANwJld1PkoOv8g2NDU9wY6I8iSNBAGZEnadv+e5G460wr+CHg/nVA45g/oTGW4NMldwP/lofN3/xP4IZ1R1jOAt1TVd5t1HwWe3Ew5WLktxVXVZcBP6Exz+EJr1b5NLT8GLgH+saoummAX5zf9/oPOdImf8dApFBP5Kp3PfCHwN1X1pS7qvAb426aW24GnAhdP10+S+iWTXJchSeqjJM+lM9q854BL6Ykky4CbgIVVdf+Ay5GkGXEEWZIkSWoxIEuSJEktTrGQJEmSWhxBliRJklq2H3QB/bLrrrvWsmXLBl2GJEmShtQVV1xxZ1UtGd8+ZwPysmXLWLVq1aDLkCRJ0pBK8v2J2p1iIUmSJLUYkCVJkqQWA7IkSZLUMnRzkJPcDNwNbAHur6rlSXYBzgSWATcDr6iqHw6qRkmSJM1dwzqC/LyqOqCqljfvTwQurKp9gQub95IkSVLPDWtAHu8oYEWzvAI4enClSJIkaS4bxoBcwJeSXJHkuKbtcVV1G0Dzc7eJOiY5LsmqJKs2bNgwS+VKkiRpLhm6OcjAYVW1PsluwAVJvtttx6o6FTgVYPny5T5DW5IkSVtt6AJyVa1vft6R5N+Ag4Hbk+xeVbcl2R24Y6BFjrNy9TpOOf861m/azB6LF3HCEftx9IFLB12WJEmStsFQTbFI8qgkjx5bBn4N+A5wDnBss9mxwOcGU+HDrVy9jpPOXsO6TZspYN2mzZx09hpWrl436NIkSZK0DYYqIAOPA76e5NvA5cC5VfVF4GTg8CTXA4c374fCKedfx+b7tjykbfN9Wzjl/OsGVJEkSZJmYqimWFTV94CnT9D+A+AFs1/R9NZv2rxV7ZIkSRpuwzaCPHL2WLxoq9olSZI03AzIM3TCEfuxaOGCh7Rtl067JEmSRs9QTbEYRWN3qxi7i8XCBdux186LvIuFJEnSiDIg98DRBy59MBC/8sOXDLgaSZIkzYRTLCRJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJklqGLiAnWZBkdZLPN+93SXJBkuubnzsPukZJkiTNXUMXkIG3A9e23p8IXFhV+wIXNu8lSZKkvhiqgJxkT+DFwEdazUcBK5rlFcDRs1yWJEmS5pGhCsjAB4B3AQ+02h5XVbcBND93m6xzkuOSrEqyasOGDX0tVJIkSXPT0ATkJC8B7qiqK7Z1H1V1alUtr6rlS5Ys6WF1kiRJmi+2H3QBLYcBL0vyImBH4DFJTgduT7J7Vd2WZHfgjoFWKUmSpDltaEaQq+qkqtqzqpYBxwBfrqrXAucAxzabHQt8bkAlSpIkaR4YmoA8hZOBw5NcDxzevJckSZL6YpimWDyoqi4CLmqWfwC8YJD1SJIkaf4YhRFkSZIkadYYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLQZkSZIkqcWALEmSJLUYkCVJkqQWA7IkSZLUYkCWJEmSWgzIkiRJUosBWZIkSWoxIEuSJEktBmRJkiSpxYAsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJEmS1GJAliRJkloMyJIkSVKLAVmSJElqMSBLkiRJLUMVkJPsmOTyJN9OcnWS9zTtuyS5IMn1zc+dB12rJEmS5qa+BeQkj0/ywmZ5UZJHd9HtHuD5VfV04ADgyCSHACcCF1bVvsCFzXtJkiSp5/oSkJO8Cfgs8OGmaU9g5XT9quPHzduFzauAo4AVTfsK4OgelitJkiQ9qF8jyG8FDgPuAqiq64HduumYZEGSK4E7gAuq6jLgcVV1W7Ov2ybbV5LjkqxKsmrDhg0z/xSSJEmad/oVkO+pqnvH3iTZns5I8LSqaktVHUBn1PngJPt3e9CqOrWqllfV8iVLlmxtzZIkSVLfAvJXk/whsCjJ4cBngH/fmh1U1SbgIuBI4PYkuwM0P+/oabWSJElSo18B+URgA7AGeDNwHvDH03VKsiTJ4mZ5EfBC4LvAOcCxzWbHAp/rfcmSJEkSbN+PnVbVA0lOB75WVddtRdfdgRVJFtAJ72dV1eeTXAKcleSNwC3Ay3tftSRJktSngJzkZcApwA7APkkOAP6sql42Vb+qugo4cIL2HwAv6EOpkiRJ0kP0a4rFu4GDgU0AVXUlsKxPx5IkSZJ6pl8B+f6q+lGf9i1JkiT1TV+mWADfSfJqYEGSfYHjgW/06ViSJElSz/RrBPltwFPoPDr6k8CPgHf06ViSJElSz/R8BLm5A8U5VfVC4I96vX9JkiSpn3o+glxVW4CfJnlsr/ctSZIk9Vu/5iD/DFiT5ALgJ2ONVXV8n44nSZIk9US/AvK5zaut+nQsSZIkqWf6FZAXV9XftxuSvL1Px5IkSZJ6pl93sTh2grbX9+lYkiRJUs/0dAQ5yauAV9N5vPQ5rVWPBn7Qy2NJkiRJ/dDrKRbfAG4DdgX+ttV+N3BVj48lSZIk9VxPA3JVfR/4PnBoL/crSZIkzZZ+zUGWJEmSRpIBuYdWrl7H6ls2cdlNGzns5C+zcvW6QZckSZKkrdTTgJzkwubnX/Vyv6Ng5ep1nHT2Gu7d8gAA6zZt5qSz1xiSJUmSRkyvR5B3T/JfgZclOTDJQe1Xj481VE45/zo237flIW2b79vCKedfN6CKJEmStC16fReLPwFOBPYE3j9uXQHP7/Hxhsb6TZu3ql2SJEnDqdd3sfgs8Nkk/7uq/ryX+x52eyxexLoJwvAeixcNoBpJkiRtq75cpFdVf57kZUn+pnm9pB/HGSYnHLEfixYueEjbooULOOGI/QZUkSRJkrZFr6dYAJDkfcDBwBlN09uTHFZVJ/XjeMPg6AOXAvCuz17FvVseYOniRZxwxH4PtkuSJGk09CUgAy8GDqiqBwCSrABWA3M2IEMnJH/q8lsAOPPNPitFkiRpFPXzPsiLW8uP7eNxJEmSpJ7p1wjy+4DVSb4CBHgOc3z0WJIkSXNDXwJyVX0qyUXAM+kE5D+oqv/sx7EkSZKkXurXCDJVdRtwTr/2L0mSJPVDP+cgS5IkSSPHgCxJkiS19G2KRZIFwOPax6iqW/p1PEmSJKkX+vWgkLcB7wZuBx5omgt4Wj+OJ0mSJPVKv0aQ3w7sV1U/6NP+JUmSpL7o1xzkW4Ef9WnfkiRJUt/0dAQ5yTubxe8BFyU5F7hnbH1Vvb+Xx5MkSZJ6rddTLB7d/Lylee3QvKAzB1mSJEkaaj0NyFX1HoAkL6+qz7TXJXl5L48lSZIk9UO/5iCf1GWbJEmSNFR6PQf514EXAUuTfLC16jHA/b08liRJktQPvZ6DvB5YBbwMuKLVfjfwv3p8LEmSJKnnej0H+dvAt5N8EgjwJDoX511XVff28liSJElSP/TrQSGHAx8GbqQTlPdJ8uaq+kKfjidJkiT1RL8C8vuB51XVDQBJngicCxiQJUmSNNT6dReLO8bCceN7wB19OpYkSZLUM/0aQb46yXnAWXTmIL8c+GaS3wSoqrP7dNyBes1pl3DZTRsBWHbiuRz2xF04402HDrgqSZIkbY1+jSDvCNwO/FfgucAGYBfgpcBL+nTMgXrNaZdw8Y0bH9J28Y0bec1plwyoIkmSJG2LvowgV9Ub+rHfYTY+HE/XLkmSpOHUlxHkJL+U5MIk32nePy3JH/fjWJIkSVIv9WuKxWl0Hi19H0BVXQUc06djSZIkST3Tr4D8yKq6fFzbtI+aTrJXkq8kuTbJ1Une3rTvkuSCJNc3P3fuS9UzcNgTd9mqdkmSJA2nfgXkO5t7HxdAkv8O3NZFv/uB36uqXwYOAd6a5MnAicCFVbUvcGHzfqic8aZDHxaGvYuFJEnS6OnXbd7eCpwKPCnJOuAm4LXTdaqq22iCdFXdneRaYClwFJ27YQCsAC4C/qDnVc/QGW86lFd+uHPXijPfbDCWJEkaRf26i8X3gBcmeRSwXVXdvbX7SLIMOBC4DHhcE56pqtuS7NbLeiVJkqQxPQ3ISd45STsAVfX+LvezE/CvwDuq6q6x/l30Ow44DmDvvffuqo8kSZLU1us5yI9uXsuB36EzPWIp8Bbgyd3sIMlCOuH4jNYT925PsnuzfncmeWx1VZ1aVcuravmSJUtm9EEkSZI0P/V0BLmq3gOQ5EvAQWNTK5L8KfCZ6fqnM1T8UeDacaPN5wDHAic3Pz/Xy7p7ZfyjpgEe84gFXPWeIwdZliRJkrZCv+5isTdwb+v9vcCyLvodBvwW8PwkVzavF9EJxocnuR44vHk/VCZ61DTAXfds4Wnv/uIAKpIkSdK26NddLP4FuDzJv9G51dtv0Ln7xJSq6uvAZBOOX9C78npvqkdK33XPllmsRJIkSTPRr7tYvDfJF4BfbZreUFWr+3EsSZIkqZf6NYJMVX0L+Fa/9i9JkiT1Q7/mIM87Uz1S+jGPWDCLlUiSJGkmDMg9MtGjpsG7WEiSJI2avk2xmI/OeFPn8dI+blqSJGl0OYIsSZIktRiQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1+CS9Hlu5eh2X37SRApadeC7bBV79rL35i6OfOujSJEmS1AVHkHto5ep1vPPMK6lW2wMFp196C3+8cs3A6pIkSVL3DMg9dMr51/HAJOs+ddmts1qLJEmSto0BuYfWb9o86botVZOukyRJ0vAwIPfQHosXTbpuQTKLlUiSJGlbGZB76IQj9pv0F/qqZ+01q7VIkiRp23gXix46+sClAJx09lVsvq8zG9m7WEiSJI0WA3KPHX3gUo4+cCmv/PAlAJz55kMHXJEkSZK2hlMsJEmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgNyH7zmtEu47KaNXHbTRpadeC6vOe2SQZckSZKkLhmQe+w1p13CxTdufEjbxTduNCRLkiSNCANyj40Px9O1S5IkabgYkCVJkqQWA7IkSZLUYkDuscOeuMuE7Y95xIJZrkSSJEnbwoDcY/ss2WnC9rvu2eKFepIkSSPAgNxjn7rs1knXeaGeJEnS8DMg99iWqkGXIEmSpBkwIPfYgmTQJUiSJGkGDMg99qpn7TXpusku4JMkSdLwMCD32F8c/VRee8jeD2s/7Im7cMabDh1ARZIkSdoaqTk6Z3b58uW1atWqQZfBKz/cuXPFmW82HEuSJA2TJFdU1fLx7Y4gS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKklu0HXUBbko8BLwHuqKr9m7ZdgDOBZcDNwCuq6oeDqnFrHP7+i7j+jp8AsOzEcx9s944WkiRJw2vYRpA/ARw5ru1E4MKq2he4sHk/9NrheLyLb9zIa067ZJYrkiRJUjeGKiBX1deAjeOajwJWNMsrgKNns6ZtNVk4HnPxjeM/piRJkobBUAXkSTyuqm4DaH7uNtmGSY5LsirJqg0bNsxagZIkSZo7RiEgd62qTq2q5VW1fMmSJYMuR5IkSSNoFALy7Ul2B2h+3jHgerqy726PmtF6SZIkDcYoBORzgGOb5WOBzw2wlq5d8M7nThmC1/7wZ6xcvW4WK5IkSVI3hiogJ/kUcAmwX5K1Sd4InAwcnuR64PDm/Ui44J3P5eaTX8zSxYsetm7zfVs45fzrBlCVJEmSpjJU90GuqldNsuoFs1pIj63ftHmr2iVJkjQ4QzWCPFftMcEI8lTtkiRJGhwD8iw44Yj9WLRwwUPaFi1cwAlH7DegiiRJkjSZVNWga+iL5cuX16pVqwZdxoNWrl7HCZ+5kvseePi6BQmvetZe/MXRT539wiRJkuapJFdU1fLx7Y4gz5LPrLplwnAMsKWK0y+9hT9euWZ2i5IkSdLDGJBnSTePlj790lv45GW3zEI1kiRJmowBech87krvjSxJkjRIBmRJkiSpxYA8Sw574i7TbrPbTjvMQiWSJEmainexmEWvOe2SCeciL0g45Ak7T7puSxVLFy/ihCP24+gDl85GqZIkSXPeZHexGKon6c11Z7zp0AnbV65exzvOvHLCdVuaf8Cs27SZk87u3OXCkCxJktQ/BuQhcMr513W13eb7tvCuz17Fpy6f/k4XRx2wlFc/a++ZliZJkjTvOAd5CKzftLnrbe/dMsnNlFuuue0u74YhSZK0jRxBHgJ7LF7Eui5D8tLFizjzzRNP1Rjzyg9f0ouyJEmS5iVHkIfACUfs19V2ixYumHbblavXsfqWTVx200YOO/nLrFztSLIkSdLWcAR5CIxddHfCZ6582OOot+YuFitXr+Oks9c8OA3DC/skSZK2ngF5SBx94NIZh9hTzr+OzfdteUjb5vu2cMr51xmQJUmSuuQUizlksov9tuYiQEmSpPnOgDyH7LF40Va1S5Ik6eEMyHPICUfsx6KFCx7Stl26vwhQkiRJPmp6zlm5eh2nnH/dQ24b182FfmP91m/azB7NdsDD2pzLLEmS5gofNT1PjAXYk85e8+AFe9M9rnrs7hdj26/btJkTPvNtCNy3xUddS5Kk+cUR5DnosJO/POWDR3ZYsB0H7r34wferb9nU1RP6Juo7HR95LUmShtVkI8jOQZ6Dprtrxfgw3G043tptfeS1JEkaRU6xmIOme3T1+MdVTzfiPFXfqfjIa0mSNIocQZ6DJrqbxZiJHlc90fYLtwsLF2TavpIkSXONI8hz0NhFdGN3s5juLhbt7b2LhSRJmu8MyHPU1j66erLtDcSSJGm+cYqFJEmS1GJAliRJkloMyJIkSVKLc5DVFytXr3vwASSHnfxlnvekJXzluxumveBvokdeTzcPelv69MN0dQxLnZIkaWoGZPXc2KOrxx4qsm7TZk6/9JYH12/NI6+ne7z1tvTph+nqGJY6JUnS9HzUtHqu2wePdPvI66keb70tffphujqGpc65wMeXS5J6xUdNa9ZM96jrMd0+8nqqx1tvS59+mK6OYalz1Pn4cknSbHCKhXpuukddj+n2kddTPd56W/r0w3R1DEudo87Hl0uSZoMjyOq5qR51PabbR15P93jrbenTD9PVMSx1SpKk6TmCrJ6b6NHV3dzFYrJHXk91Edu29OmH6eoYljpH2fg7o/j7m1+8C4yk2eRFepKG3vi7gEBnBP59v/lUQ9I84PcvqV8mu0jPgCxp6E02h9u7gMwP3gVGveJdcDSed7GQNLImuzOKdwGZH7wLjHrBu+BoazgHWdLQm+zOKN4FZH7wLjDqBe+Co63hCLKkoeddQOY3v3/N1NhFvpfdtJHDTv4yK1c7kjwsVq5ex2Enf5l9Tjx3qL4bR5AlDT3vAjK/+f1rJsYu8hybkrNu02ZOOnsNgOfQgI2/AHeYvhsv0pMkSXOWF/kOr8kuwF26eBEXn/j8WanBi/QkSdK840W+w2uy72Cy72w2OcVCkiTNWV7kO7wmG93fY/GiAVTzUI4gS5KkOcuLPIfXMH83IxOQkxyZ5LokNyQ5cdD1SJKk4Xf0gUt5328+laWLFxE6I8c+hXE4DPN3MxIX6SVZAPwHcDiwFvgm8KqqumayPl6kJ0mSpKmM+kV6BwM3VNX3qupe4NPAUQOuSZIkSXPQqATkpcCtrfdrm7aHSHJcklVJVm3YsGHWipMkSdLcMSoBORO0PWxuSFWdWlXLq2r5kiVLZqEsSZIkzTWjEpDXAnu13u8JrB9QLZIkSZrDRiUgfxPYN8k+SXYAjgHOGXBNkiRJmoNG4kEhVXV/kt8FzgcWAB+rqqsHXJYkSZLmoJEIyABVdR5w3qDrkCRJ0tw2KlMsJEmSpFlhQJYkSZJaRuJJetsiyQbg+7N82F2BO2f5mBo9nieajueIpuM5oul4jnTn8VX1sHsDz9mAPAhJVk30uEKpzfNE0/Ec0XQ8RzQdz5GZcYqFJEmS1GJAliRJkloMyL116qAL0EjwPNF0PEc0Hc8RTcdzZAacgyxJkiS1OIIsSZIktRiQJUmSpBYDcpeSHJnkuiQ3JDlxgvVJ8sFm/VVJDuq2r+aGGZ4jH0tyR5LvzG7Vmk3beo4k2SvJV5Jcm+TqJG+f/eo1W2ZwnuyY5PIk327Ok/fMfvWaDTP5+6ZZvyDJ6iSfn72qR0xV+ZrmBSwAbgSeAOwAfBt48rhtXgR8AQhwCHBZt319jf5rJudIs+45wEHAdwb9WXwN3zkC7A4c1Cw/GvgP/xyZm68ZnicBdmqWFwKXAYcM+jP5Gp5zpLX+ncAngc8P+vMM68sR5O4cDNxQVd+rqnuBTwNHjdvmKOCfq+NSYHGS3bvsq9E3k3OEqvoasHFWK9Zs2+ZzpKpuq6pvAVTV3cC1wNLZLF6zZibnSVXVj5ttFjYvr8Sfe2b0902SPYEXAx+ZzaJHjQG5O0uBW1vv1/Lwv5wm26abvhp9MzlHND/05BxJsgw4kM7ooOaeGZ0nzf86vxK4A7igqjxP5p6Z/lnyAeBdwAN9qm9OMCB3JxO0jf9X+WTbdNNXo28m54jmhxmfI0l2Av4VeEdV3dXD2jQ8ZnSeVNWWqjoA2BM4OMn+vS1PQ2Cbz5EkLwHuqKorel/W3GJA7s5aYK/W+z2B9V1u001fjb6ZnCOaH2Z0jiRZSCccn1FVZ/exTg1WT/4sqapNwEXAkT2vUIM2k3PkMOBlSW6mMzXj+UlO71+po8uA3J1vAvsm2SfJDsAxwDnjtjkHeF1z5eghwI+q6rYu+2r0zeQc0fywzedIkgAfBa6tqvfPbtmaZTM5T5YkWQyQZBHwQuC7s1i7Zsc2nyNVdVJV7VlVy5p+X66q185q9SNi+0EXMAqq6v4kvwucT+fq0Y9V1dVJ3tKs/yfgPDpXjd4A/BR4w1R9B/Ax1EczOUcAknwKeC6wa5K1wLur6qOz+ynUTzM8Rw4DfgtY08wvBfjDqjpvFj+CZsEMz5PdgRVJFtAZADurqryN1xwz079v1B0fNS1JkiS1OMVCkiRJajEgS5IkSS0GZEmSJKnFgCxJkiS1GJAlSZKkFgOyJM0RSW5OsutMt5Gk+c6ALEmSJLUYkCVpBCVZmeSKJFcnOW7cumVJvptkRZKrknw2ySNbm7wtybeSrEnypKbPwUm+kWR183O/Wf1AkjREDMiSNJp+u6qeASwHjk/yC+PW7wecWlVPA+4C/mdr3Z1VdRDwIeD3m7bvAs+pqgOBPwH+sq/VS9IQMyBL0mg6Psm3gUuBvYB9x62/taoubpZPB57dWnd28/MKYFmz/FjgM0m+A/wd8JR+FC1Jo8CALEkjJslzgRcCh1bV04HVwI7jNqsp3t/T/NwCbN8s/znwlaraH3jpBPuTpHnDgCxJo+exwA+r6qfNHOJDJthm7ySHNsuvAr7exT7XNcuv70mVkjSiDMiSNHq+CGyf5Co6I7+XTrDNtcCxzTa70JlvPJW/Bt6X5GJgQS+LlaRRk6rx/xdOkjTKkiwDPt9Ml5AkbSVHkCVJkqQWR5AlSZKkFkeQJUmSpBYDsiRJktRiQJYkSZJaDMiSJElSiwFZkiRJavn/twa2pVbf0+UAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x720 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light",
            "tags": []
          },
          "output_type": "display_data"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UDS2Pt2yZajQ",
        "outputId": "16483dfd-56f4-4a86-cdf5-c2378f8857df"
      },
      "source": [
        "train_scores = [clf.score(X_train, y_train) for clf in clfs]\n",
        "test_scores = [clf.score(X_test, y_test) for clf in clfs]\n",
        "\n",
        "fig, ax = plt.subplots(figsize=(10,10))\n",
        "ax.set_xlabel(\"alpha\")\n",
        "ax.set_ylabel(\"accuracy\")\n",
        "ax.set_title(\"Accuracy vs alpha for training and testing sets\")\n",
        "ax.plot(ccp_alphas, train_scores, marker='o', label=\"train\",\n",
        "        drawstyle=\"steps-post\")\n",
        "ax.plot(ccp_alphas, test_scores, marker='o', label=\"test\",\n",
        "        drawstyle=\"steps-post\")\n",
        "ax.legend()\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAJcCAYAAAC8DwN/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA3UElEQVR4nO3de5xddX3v/9eHYQLDxQSSiOQCSZUToYAEI0rDz6JUA1o01jZeSq1YBVsRPG2DwbZ4bZtTTsXy84J6xMtBgRQhgKBBQEpFEJImJtxSkFtmohICE0AGkky+54+1Juxs9p7Zk5k9+5uZ1/PxmEf2+q7vWuuz15rLO+u79lqRUkKSJEl52K3VBUiSJOkFhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJA1JRLw/In463H0bXN/nIuLxiPj1cK1zqCLihxHx58Pdt5Ui4uGI+IMWbPdPI+L6kd6u1Gq7t7oAqZUi4mbgVcDLUkrPt7gcDUJETAf+Bjg4pfTYMK0zAYeklB7Y2XWklE5qRt9cRcS3gM6U0t8PcT0zgIeA9pTSVoCU0neB7w61xuFSq0apGTxzpjGr/EX7/wEJeNsIb9v/GA3dwcDGnQlmO7v/PW6SRoLhTGPZ+4DbgW8BOwwtRcT0iLgiIjZExMaI+GLFvA9FxL0R8XRE3BMRR5ftKSJeUdHvWxHxufL18RHRGREfL4fgvhkR+0XED8ptPFm+nlax/P4R8c2IWF/OX1q23xURJ1f0ay+H9o6qfoNlnX9YMb172ffoiNgzIi4u3193RNwZEQfU2lERsSgiflnxnt9Rb6eW++HMiHiw3NZ5EbFbVZ//Xb6nhyLipIr2Uyv27YMRcXqdbfwB8GNgSkQ8U569ISLeFhF3l+/n5og4tGKZh8v9vxr4bXXQiohbype/KNf5rp08bjdHxAfL1++PiJ/2834H03dmRNxS7psbIuJLEXFxnf3TSI2fjYhby/VdHxGTKub/WUQ8Un5v/F2tbZT9TgP+FDi73GfXlO1TIuL75fYfiogzK5Y5JiKWR8RTEfGbiPh8Oatv/3eX6zo2qobBy++tD0fE/eX7+lJERDmvLSL+tfyeeygizij71wzU5THtKt//2og4oWzfreL7fWNELImI/fup8RUR8R8Rsanc9mX19pfUsJSSX36NyS/gAeCvgFcDW4ADyvY24BfA+cDewJ7AceW8PwG6gNcAAbyCYlgNijNwr6hY/7eAz5Wvjwe2Av8L2APoACYC7wT2AvYF/h1YWrH8tcBlwH5AO/D7ZfvZwGUV/d4OrKnzHs8Fvlsx/VbgvvL16cA15fbbyv3wkjrr+RNgCsV/6N4F/BY4sJz3fuCnFX0T8BNgf+Ag4L+BD1b03QJ8qNzmXwLrgaio7+Xlvv194Fng6Do1HU8xnNY3/T/Kut5U7q+zy2M8rpz/MLAKmA501Fln9THcmeN28yDe72D63gb8b2AccBzwFHBxnffRSI2/LPdZRzm9uJx3GPAM8PryPX++3Ad/UGdb36L8Pi+ndwNWUHzvjQN+B3gQmFfxPv6sfL0P8Lry9Yxy/+9esa738+LvrR8AEyi+tzYAJ5bzPgzcA0yj+Jm5oXp9FeuZBawDplRs++Xl649R/KdtWvn+vwpc0k+NlwB/V77v7b8r/PJrKF8tL8Avv1rxVf5x2wJMKqfvA/5n+frY8pd+rV/qy4Cz6qxzoHC2Gdizn5qOAp4sXx8IbAP2q9FvCvA0ZZACLgfOrrPOV5R99yqnvwucW77+APAz4Mid2H+rgLeXr2v9AT2xYvqvgBsr+j5QMW+vsv/L6mxnaT/7+3h2DGf/ACypmN6NIkgfX04/DHxggPdVK5w1fNzK6ZvZMXDVfb+N9qUIIlv7jmM5/2LqhLMGa/z7qmP0o/L1ucClFfP2LvdBo+HstcCjVX3OAb5Zvr4F+DTlz15Fnxk0Fs6Oq5heAiwqX98EnF4x7w+q11f1c/FY2ae9at69wAkV0wdS/K7YvU6N3wG+Bkwb7M+RX37V+3JYU2PVnwPXp5QeL6e/xwtDm9OBR1LtC36nU5xx2BkbUkrP9U1ExF4R8dVy+Ogpij9aEyKirdzOEymlJ6tXklJaD9wKvDMiJgAnUeei6VRc2H4vcHJE7EVxbd33ytn/lyJsXhrF0Om/RER7rfVExPsiYlU5XNgNHA5MqtW3tK7i9SMUgbLP9k9WppSeLV/uU27npIi4PSKeKLfzlgG2U2lKua2+dW8r65hap65GDea41VL3/Q6i7xSK74dnK/rWfS8N1lj5CddnK2qaUrnulNJvgY31tlXDwRTDzd0V3y+fAPqGzP+C4ozdfVEMpf9hnfXU01Dd9LN/yp+LjwGfAh6LiEsjou979GDgyora7wV6K+qvdjbFmd47ohhS/8Cg3o1Ug+FMY05EdAALgN+PiF9HcS3R/wReFRGvovilflCda1XWUQy71fIsxdmOPi+rmp+qpv+GYnjltSmll1AMI0Hxi34dsH8Zvmr5NnAKxXDjbSmlrjr9oBh2eQ/F8Oc95R8mUkpbUkqfTikdBvwe8IcU1+HtICIOBr4OnAFMTClNAO4q66xnesXrgyiG5/oVEXsA36cYujug3M51A2yn0nqKP6x964uyjsp9U30MGjGY49Ysv6L4fqj8/pperzNDq/FXlesutzmxn/7V+2cd8FBKaULF174ppbcApJTuTym9B3gpxXDx5RGxd431DNavKIYi+/S3f0gpfS+ldBzF90wqa+mr/6Sq+vcsf8ZeVGNK6dcppQ+llKZQXCrw5ai49lTaGYYzjUXzKf4nfBjFcM9RwKHAf1KEkzsoftEvjoi9o7hwfm657P8B/jYiXh2FV5ThBYqhvveWFyafSHHNVH/2BXooLi7eH/hk34yU0q+AH1L8ot8viov+X1+x7FLgaOAsimGV/lwKvJniGqa+s2ZExBsi4ojybMpTFEM3vTWW7/vDuaFc7lSKM2f9WVjWPb2ssZGLpMdRXOOzAdgaxcXwb25guT5LgLdGxAnlGcC/AZ6nGLpt1G8orpHqT93j1iwppUeA5cCnImJcRBwLnNzPIkOp8XLgDyPiuIgYB3yG/v9WVO+zO4CnygvuO8qfh8Mj4jUAEXFKREwuz2x2l8v0Uhz3bQy8/+tZApwVEVPL/9R8vF7HiJgVEW8s/0PwHMW+6vvevxD4x76f64iYHBFvL+e9qMaI+JN44cMWT1L8rNT6OZIaZjjTWPTnFNe/PFr+r/fXKaVfA1+k+ORZUPzhewXwKNBJcRE8KaV/B/6RIuQ8TRGS+j7JdVa5XHe5nqUD1PEFiouxH6e4APlHVfP/jCIw3UdxfczH+maklHoozjLNBK7obyNl0LuN4uxYZUh6GcUf4qcohm7+g+I6purl7wH+tVzHb4AjKIZV+3MVxUXhqyg+2PCNAfqTUnoaOJPij+yTwHuBqwdarmL5tRRnE/9/in16MnBySmlzo+ugGOb6djmktaBOny/Q/3Frlj+luB5yI/A5imNZ7958X2Ana0wp3Q18hOJ7/FcUx6Kzn0W+ARxW7rOlKaVein1/FMU9wR6n+E/N+LL/icDdEfEM8G/Au1NKz5VDtv8I3Fqu63WN1lz6OnA9sBpYSXHWdSu1g9IewOKytl9TnMX7RDnv3yi+766PiKcp9t9rYftQc3WNrwF+Xr6fqymukXxokLVLO+j7FJCkXUxEnAv8j5TSKa2upVIMw41cNbDylg33pZSafuZuV1Seeb0wpXTwgJ2lzHjmTNoFlUNVf0HxKTGNARHxmoh4eRT34TqR4hrCpS0uKxvlEOpboriX31SKodwrW12XtDMMZ9IuJiI+RHHR8g9TSrcM1F+jxssoboHxDHAB8JcppZUtrSgvQXGLjicphjXvpbgtiLTLcVhTkiQpI545kyRJysioeojvpEmT0owZM1pdhiRJ0oBWrFjxeEppcnX7qApnM2bMYPny5a0uQ5IkaUAR8Uitdoc1JUmSMmI4kyRJyojhTJIkKSOj6pozSZK0a9iyZQudnZ0899xzrS6l6fbcc0+mTZtGe3t7Q/0NZ5IkacR1dnay7777MmPGDCKi1eU0TUqJjRs30tnZycyZMxtaxmFNSZI04p577jkmTpw4qoMZQEQwceLEQZ0hNJxJkqSWGO3BrM9g36fhTJIkKSOGM0mSNOZ0d3fz5S9/edDLveUtb6G7u3v4C6pgOJMkSdlburKLuYtvYuaia5m7+CaWruwa0vrqhbPe3t5+l7vuuuuYMGHCkLY9ED+tKUmSsrZ0ZRfnXLGGni1FcOrq7uGcK9YAMH/21J1a56JFi/jlL3/JUUcdRXt7O/vssw8HHnggq1at4p577mH+/PmsW7eO5557jrPOOovTTjsNeOFRkc888wwnnXQSxx13HD/72c+YOnUqV111FR0dHUN+v4YzSZLUUp++5m7uWf9U3fkrH+1mc++2Hdp6tvRy9uWrueSOR2suc9iUl/DJk3+37joXL17MXXfdxapVq7j55pt561vfyl133bX9dhcXXXQR+++/Pz09PbzmNa/hne98JxMnTtxhHffffz+XXHIJX//611mwYAHf//73OeWUUxp923UZziRJUtaqg9lA7TvjmGOO2eE+ZBdccAFXXnklAOvWreP+++9/UTibOXMmRx11FACvfvWrefjhh4elFsOZJElqqf7OcAHMXXwTXd09L2qfOqGDy04/dlhq2Hvvvbe/vvnmm7nhhhu47bbb2GuvvTj++ONr3qdsjz322P66ra2Nnp4X17gz/ECAJEnK2sJ5s+hob9uhraO9jYXzZu30Ovfdd1+efvrpmvM2bdrEfvvtx1577cV9993H7bffvtPb2RmeOZMkSVnru+j/vGVrWd/dw5QJHSycN2unPwwAMHHiRObOncvhhx9OR0cHBxxwwPZ5J554IhdeeCFHHnkks2bN4nWve92Q38NgREppRDfYTHPmzEnLly9vdRmSJGkA9957L4ceemiryxgxtd5vRKxIKc2p7uuwpiRJUkYMZ5IkSRkxnEmSJGXEDwQ0aOnKrmG9EFGSJKkWw1kDmvHYCEmSpFoc1mzAecvWbg9mfXq29HLesrUtqkiSJI1WhrMGrK9xV+L+2iVJUt66u7v58pe/vFPLfuELX+DZZ58d5opeYDhrwJQJtZ8wX69dkiQNs9VL4PzD4VMTin9XLxnS6nIOZ15z1oCF82btcM0ZDP2xEZIkqUGrl8A1Z8KWcsRq07piGuDIBTu1ykWLFvHLX/6So446ije96U289KUvZcmSJTz//PO84x3v4NOf/jS//e1vWbBgAZ2dnfT29vIP//AP/OY3v2H9+vW84Q1vYNKkSfzkJz8Zpjf5AsNZAyofG9HV3cO4tt345z86wg8DSJI0HH64CH69pv78zjuh9/kd27b0wFVnwIpv117mZUfASYvrrnLx4sXcddddrFq1iuuvv57LL7+cO+64g5QSb3vb27jlllvYsGEDU6ZM4dprrwWKZ26OHz+ez3/+8/zkJz9h0qRJg32nDXFYs0HzZ0/l1kVv5LUz92f2QRMMZpIkjZTqYDZQ+yBdf/31XH/99cyePZujjz6a++67j/vvv58jjjiCG264gY9//OP853/+J+PHjx+W7Q3EM2eSJKm1+jnDBRTXmG1a9+L28dPh1GuHvPmUEueccw6nn376i+atWLGC6667jnPOOYc3v/nNnHvuuUPe3kA8cyZJkvJ2wrnQXvUhvPaOon0n7bvvvjz99NMAzJs3j4suuohnnnkGgK6uLh577DHWr1/PXnvtxSmnnMLf/u3f8l//9V8vWrYZPHMmSZLy1nfR/42fgU2dMH5aEcx28sMAABMnTmTu3LkcfvjhnHTSSbz3ve/l2GOPBWCfffbh4osv5oEHHmDhwoXstttutLe385WvfAWA0047jZNOOokDDzywKR8IiJTSsK+0VebMmZOWL1/e1G2866u3AXDZ6cc2dTuSJI1m9957L4ceemiryxgxtd5vRKxIKc2p7uuwpiRJUkYMZ5IkSRkxnEmSpJYYTZdW9Wew79NwJkmSRtyee+7Jxo0bR31ASymxceNG9txzz4aX8dOakiRpxE2bNo3Ozk42bNjQ6lKabs8992TatGkN9zecSZKkEdfe3s7MmTNbXUaWHNaUJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKSFPDWUScGBFrI+KBiFhUY/5+EXFlRKyOiDsi4vCKeQ9HxJqIWBURy5tZpyRJUi52b9aKI6IN+BLwJqATuDMirk4p3VPR7RPAqpTSOyLilWX/EyrmvyGl9HizapQkScpNM8+cHQM8kFJ6MKW0GbgUeHtVn8OAGwFSSvcBMyLigCbWJEmSlLVmhrOpwLqK6c6yrdIvgD8CiIhjgIOBaeW8BFwfESsi4rR6G4mI0yJieUQs37Bhw7AVL0mS1ArNDGdRoy1VTS8G9ouIVcBHgZXA1nLe3JTS0cBJwEci4vW1NpJS+lpKaU5Kac7kyZOHp3JJkqQWado1ZxRnyqZXTE8D1ld2SCk9BZwKEBEBPFR+kVJaX/77WERcSTFMeksT65UkSWq5Zp45uxM4JCJmRsQ44N3A1ZUdImJCOQ/gg8AtKaWnImLviNi37LM38GbgribWKkmSlIWmnTlLKW2NiDOAZUAbcFFK6e6I+HA5/0LgUOA7EdEL3AP8Rbn4AcCVxck0dge+l1L6UbNqlSRJykUzhzVJKV0HXFfVdmHF69uAQ2os9yDwqmbWJkmSlCOfECBJkpQRw5kkSVJGDGeSJEkZMZxJkiRlxHAmSZKUEcOZJElSRgxnkiRJGTGcSZIkZcRwJkmSlBHDmSRJUkYMZ5IkSRkxnEmSJGXEcCZJkpQRw5kkSVJGDGeSJEkZMZxJkiRlxHAmSZKUEcOZJElSRgxnkiRJGTGcSZIkZcRwJkmSlBHDmSRJUkYMZ5IkSRkxnEmSJGXEcCZJkpQRw5kkSVJGDGeSJEkZMZxJkiRlxHAmSZKUEcOZJElSRgxnkiRJGTGcSZIkZcRwJkmSlBHDmSRJUkYMZ5IkSRkxnEmSJGXEcCZJkpQRw5kkSVJGDGeSJEkZMZxJkiRlxHAmSZKUEcOZJElSRgxnkiRJGTGcSZIkZcRwJkmSlBHDmSRJUkYMZ5IkSRkxnEmSJGXEcCZJkpQRw5kkSVJGDGeSJEkZMZxJkiRlxHAmSZKUEcOZJElSRgxnkiRJGTGcSZIkZcRwJkmSlBHDmSRJUkYMZ5IkSRkxnEmSJGXEcCZJkpQRw5kkSVJGDGeSJEkZMZxJkiRlxHAmSZKUEcOZJElSRgxnkiRJGTGcSZIkZcRwJkmSlBHDmSRJUkYMZ5IkSRkxnEmSJGXEcCZJkpQRw5kkSVJGDGeSJEkZMZxJkiRlxHAmSZKUEcOZJElSRgxnkiRJGTGcSZIkZcRwJkmSlBHDmSRJUkYMZ5IkSRkxnEmSJGXEcCZJkpQRw5kkSVJGDGeSJEkZMZxJkiRlxHAmSZKUEcOZJElSRgxnkiRJGTGcSZIkZcRwJkmSlBHDmSRJUkYMZ5IkSRkxnEmSJGWkqeEsIk6MiLUR8UBELKoxf7+IuDIiVkfEHRFxeKPLSpIkjUZNC2cR0QZ8CTgJOAx4T0QcVtXtE8CqlNKRwPuAfxvEspIkSaNOM8+cHQM8kFJ6MKW0GbgUeHtVn8OAGwFSSvcBMyLigAaXlSRJGnWaGc6mAusqpjvLtkq/AP4IICKOAQ4GpjW4LOVyp0XE8ohYvmHDhmEqXZIkqTWaGc6iRluqml4M7BcRq4CPAiuBrQ0uWzSm9LWU0pyU0pzJkycPoVxJkqTW272J6+4EpldMTwPWV3ZIKT0FnAoQEQE8VH7tNdCykiRJo1Ezz5zdCRwSETMjYhzwbuDqyg4RMaGcB/BB4JYysA24rCRJ0mjUtDNnKaWtEXEGsAxoAy5KKd0dER8u518IHAp8JyJ6gXuAv+hv2WbVKkmSlItmDmuSUroOuK6q7cKK17cBhzS6rCRJ0mjnEwIkSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOBmHpyi5WPtrNzx96grmLb2Lpyq5WlyRJkkYZw1mDlq7s4pwr1rC5dxsAXd09nHPFGgOaJEkaVoazBp23bC09W3p3aOvZ0st5y9a2qCJJkjQaGc4atL67Z1DtkiRJO8Nw1qApEzoG1S5JkrQzDGcNWjhvFh3tbTu0dbS3sXDerBZVJEmSRqPdW13ArmL+7KkAnH35ajb3bmPqhA4Wzpu1vV2SJGk4GM4GYf7sqVxyx6MAXHb6sS2uRpIkjUYOa0qSJGXEcCZJkpQRw5kkSVJGDGeSJEkZMZxJkiRlxHAmSZKUEcOZJElSRgxnkiRJGTGcSZIkZcRwJkmSlBHDmSRJUkYMZ5IkSRkxnEmSJGXEcCZJkpSRhsJZRHw/It4aEYY5SZKkJmo0bH0FeC9wf0QsjohXNrGmbC1d2cXKR7v5+UNPMHfxTSxd2dXqkiRJ0ijTUDhLKd2QUvpT4GjgYeDHEfGziDg1ItqbWWAulq7s4pwr1rC5dxsAXd09nHPFGgOaJEkaVg0PU0bEROD9wAeBlcC/UYS1Hzelssyct2wtPVt6d2jr2dLLecvWtqgiSZI0Gu3eSKeIuAJ4JfB/gZNTSr8qZ10WEcubVVxO1nf31GzvqtMuSZK0Mxo9c/bFlNJhKaV/rghmAKSU5jShruxMmdBRd97fL10zgpVIkqTRrNFwdmhETOibiIj9IuKvmlNSnhbOm1V33iU/XzeClUiSpNGs0XD2oZRSd99ESulJ4ENNqShT82dPrTuvN6URrESSJI1mjYaz3SIi+iYiog0Y15yS8tX2wi5oqF2SJGmwGg1ny4AlEXFCRLwRuAT4UfPKytN7Xjt9UO2SJEmD1Wg4+zhwE/CXwEeAG4Gzm1VUruYcvD/V58iibJckSRoODd1KI6W0jeIpAV9pbjl5+9TVd1N9dVkq2/u7Jk2SJKlRjd7n7BDgn4HDgD372lNKv9OkurLU3bNlUO2SJEmD1eiw5jcpzpptBd4AfIfihrSSJEkaRo2Gs46U0o1ApJQeSSl9Cnhj88qSJEkamxoa1gSei4jdgPsj4gygC3hp88qSJEkamxo9c/YxYC/gTODVwCnAnzepJkmSpDFrwDNn5Q1nF6SUFgLPAKc2vapM7T2ujbO3fZ0/bbuJNrbRy258t/eN/MtuY+phCZIkqYkGDGcppd6IeHVEREpj+zlFN00+n5c+fjt9DwTYnW28r+0GjpsxCTixpbVJkqTRodFrzlYCV0XEvwO/7WtMKV3RlKpytHoJB2y8neq70EbAyx/9d+CrLSlLkiSNLo2Gs/2Bjez4Cc0EjJ1wduNn6s9LvSNXhyRJGtUafULAmL3ObLtNnXVnbaPxT1ZIkiT1p9EnBHwTXvTkIlJKHxj2inI1fhpsWldz1rZkOJMkScOj0UzxA+Da8utG4CUUn9wcO044l3ofh2gb2UokSdIo1uiw5vcrpyPiEuCGplSUqyMXwPe9ZYYkSWqunR2NOwQ4aDgLkSRJUuPXnD3Njtec/Rr4eFMqylkMsl2SJGmQGh3W3LfZhWRv9RKzmSRJarqGhjUj4h0RMb5iekJEzG9aVTnq7z5nAN9+28jUIUmSRrVGrzn7ZEppU99ESqkb+GRTKspVP/c5A+Ch/zCgSZKkIWs0nNXq1+jTBUaH8dMG7mNAkyRJQ9RoOFseEZ+PiJdHxO9ExPnAimYWlp0TzoX2joH7GdAkSdIQNBrOPgpsBi4DlgA9wEeaVVSWjlwAJ19Q90a0O3joP5pejiRJGp0a/bTmb4FFTa4lf96IVpIkNVmjn9b8cURMqJjeLyKWNa0qSZKkMarRYc1J5Sc0AUgpPQm8tCkVSZIkjWGNhrNtEbH9cU0RMYMdnxigSpNe2eoKJEnSLqrR22H8HfDTiOi70v31wGnNKSlv3amDCfQQFY8FqPyQwFP7vJzxZ/x85AuTJEmjQkNnzlJKPwLmAGspPrH5NxSf2BxzuvedRXfqICW2f/3ntt/l5+lQfp4O5aiNn2Xpyq5WlylJknZRjT74/IPAWcA0YBXwOuA24I1NqyxTMyftw4pnZ/LOnr/fof3ScZ8FirHe85atZf7sqS2oTpIk7eoavebsLOA1wCMppTcAs4ENTasqc68+aP9+53d1j8mTipIkaRg0es3Zcyml5yKCiNgjpXRfRMxqamW7kLft9lNeE/eyG/DQHu8tGj9VzuzYH076X8U90iRJkgbQ6JmzzvI+Z0uBH0fEVcD6ZhW1KzjkpXsDRTD7fPuXaQuIiq/tep6Aqz4Cq5e0plBJkrRLafQDAe9IKXWnlD4F/APwDWB+E+vK3o//+ngAzt59CbtH/33p3Qw/PLvpNUmSpF1fo8Oa26WUfHBkhSnxeGMde55sbiGSJGlUaHRYUzXMffn+rE+TGuvctkdzi5EkSaOC4WwIvvuhY7lq/w+wdcBnJQRMOHgkSpIkSbu4poaziDgxItZGxAMRsajG/PERcU1E/CIi7o6IUyvmPRwRayJiVUQsb2adQ/GRj/0du7/z6zxPW+3nWXXsDxMPgX18FKkkSRrYoK85a1REtAFfAt4EdAJ3RsTVKaV7Krp9BLgnpXRyREwG1kbEd1NKm8v5b0gpNXhRVwsduYD33TYdgMtOP3bHeauXwBUfgo3/DZ8a/0J7tEHqhfHT4YRzvdWGJEkCmhjOgGOAB1JKDwJExKXA24HKcJaAfSMigH2AJ4CtTaypKZau7OKOh54gATMWXctuAe997UF87nfuLYJZLam3+HfTOrjmzOK1AU2SpDGvmeFsKrCuYroTeG1Vny8CV1PcM21f4F0ppW3lvARcHxEJ+GpK6Wu1NhIRp1E+hP2ggw4avuobtHRlF3992aodhjS3Jbj49kc5a83fMbmRlWzpgavOgBXfrt/niD+GOafWny9JkkaFZl5zVuvuX9WXZc2jeFbnFOAo4IsR8ZJy3tyU0tHAScBHIuL1tTaSUvpaSmlOSmnO5MkNRaFhdd6ytWyrM2/i1scaX1Hv87Xbn3kMHrkVfvAxOP9wb2YrSdIo18wzZ53A9Irpabz4qQKnAotTSgl4ICIeAl4J3JFSWg+QUnosIq6kGCa9pYn17pT1/TxHc32axLRG74M2fjqceu2ObauXlEOeZaZ1CFSSpFGvmWfO7gQOiYiZETEOeDfFEGalR4ETACLiAGAW8GBE7B0R+5btewNvBu5qYq07bcqEjrrz/mXrggZuswG0dxQfCqh242eKIc9KW3qKdkmSNCo1LZyllLYCZwDLgHuBJSmluyPiwxHx4bLbZ4Hfi4g1wI3Ax8tPZx4A/DQifgHcAVybUvpRs2odioXz6j///eptx/HXW/6Knm1tpOqQFm3Fv+Onw8kX1D4Ttqmz9orrtUuSpF1eM4c1SSldB1xX1XZhxev1FGfFqpd7EHhVM2sbLvNnT2X5I09w8e2P1px/9bbjuHrzcTu0zX35/nz3Q8fW7L+D8dOKocxa7ZIkaVRqajgbKz43/wiAugGt2q2/fIIZi4rry6ZO6GDhvFnMnz31xR1POLe4xqxyaLPeEKgkSRoVfHzTMPnc/CP4wruOYkJH+6CW6+ru4Zwr1rB0ZdeLZx65oBjyHF9+rqJtj/pDoJIkaVTwzNkwmj976g5nwGYuurb2I52q9Gzp5ezLV3PJHS8+8/b40y9l3ZPn8Z22TzMuduPR3rnMH76SJUlSZjxz1kT9fZKz2ubeF98t7fGnn+ehjb/dPm9z77b6Z9kkSdKo4JmzJlo4bxYfu2xVQ32nTuh40XM55y6+iW1Vp956tvRy3rK1ta9RkyRJuzzPnDXR/NlT+cK7jqJ9gL3c0d5W85Yc9W5w29+NbyVJ0q7NM2dNVnkd2tKVXZy3bC1d3T20RdCbUr+f1pwyoYOuGkFsMMOlkiRp12I4G0HVHxgYyMJ5szjnijX0bOnd3lbvLJskSRodDGcZ6wtyq679GrO33M84tnJ323vY7aptcPP04n5n1bfVWL2keLzTps7iZrWHvBnuv/6F6VrLVKtcR8d+RVvPk/WXr95mf9sYTN/h0t82W1GPJEn9MJxlbn7brcyPr0JsBSBS+anOWg9B73tQet9Nazetg+XfeGFljTw4vXodPU/0v3ytbdbbxmD6Dpf+tgkjX48kSQOI9KKHPu665syZk5YvX97cjXzzrcW/p17b3O30Of/w2o9w6tO2B0x7TfG6807ofX7gdVYuU62RdTSyzVrbGEzf4dLfNmHk69mVHPHHMOfUVlchSaNWRKxIKc2pbvfTmrkb6CHnleGikWA2UL9G1tHINmu1D6bvcOlvm62oZ1fx6zWw5vJWVyFJY5LDmrmr9/Dz7fOnv3AWb6CzbLWWqdbIOhrZZq1tDKbvcOlvmzDy9ewq+s4QS5JGnGfOMnfnyz9KTxpXc15PGsedL/9ov32rR62rlxnM9hrdZr1tDKbvcOlvm62oR5KkgXjmLHMfu+cQXr3lg5y9+xKmxONsYzfa2EZXmsS/bF3Aj35+ELN/dRsAKx89iBNTX9+NrE8TuXHbUZyw26rt09XLVKtex5NpbyJgAr+tuXytbdbbxmD6Dpf+tgmMeD27inM3bmLSPntwQKsLkaQxyHCWufXdPXRxHFdvPq5Ojxeeybm5dxtX1+j7yX6WqVZvHYPdZq1tDKbvcBlomyNdz67i2c29PP7M84YzSWoBw1nm6j0loE/lMznnLr6p3761lqnWyDoa2Wa9Z4U22ne49LdNYMTr2VXc/U9trS5BksYsrznL3MJ5s+hor/2HsvppAf31rbfMYLbX6DbrbWMwfYdLf9tsRT2SJA3EM2eZ63tKQCPP5Kzsu767hykTOnjDKyfzk/s2bJ+u9xzPeusY39FOBHQ/u6Xm8rW2WW8bg+k7XBrZ5kjWI0nSQLwJ7WCN9E1opRa4+5+K6/B+9xM/bXElkjR6eRNaSZKkXYDhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJO0o9VLOGTzfRy2eQ2cfzisXtLqijSQ1UuKY/WpCR4zaRTYvdUFSMrI6iVwzZmMY0sxvWkdXHNm8frIBa2rS/WVx4wtPcW0x0za5RnOJL3gxs+88Ee+z5YeuOoMWPHt1tSk/nXeCb3P79jmMRv9jvhjmHNqq6tQkzisKekFmzprt1f/8Vc+6h0bj9no9es1sObyVlehJvLMmaQXjJ9WDIu9qH06nHrtyNejgZ1/uMdsrPnmW1tdgZrMM2eSXnDCuWxt23OHpq1te8IJ57aoIA3IYyaNOoYzSdst7Z3Loi0fpHPbJLaloHPbJBZt+SBLe+e2ujTV4TGTRh+HNSVtd96ytXRt/j0u5/d2aL/68tVccsejLapK/Vn5aDebez1mY8m5GzcxaZ89OKDVhahpPHMmabv13T012zf3bhvhStSoesfGYzZ6Pbu5l8ef8QMfo5lnziRtN2VCB101AtrUCR1cdvqxLahIA5m7+CaP2Rhz9z+1tboENZlnziRtt3DeLDrad/zF39HexsJ5s1pUkQbiMZNGH8+cSdpu/uypQHHt2fruHqZM6GDhvFnb25Ufj5k0+hjOJO1g/uyp/mHfxXjMpNHFYU1JkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOGrV6CZx/ODzyU+i8s5iWJEkaZt5KoxGrl8A1Z8KW8i7cvc8X0wBHLmhdXZIkadTxzFkjbvzMC8Gsz5aeol2SJGkYGc4asalzcO2SJEk7yXDWiPHTBtcuSZK0kwxnjTjhXGjv2LGtvaNolyRJGkaGs0YcuQBOvgDGTwei+PfkC/wwgCRJGnZ+WrNRRy4wjEmSpKbzzJkkSVJGDGeSJEkZMZxJkiRlxHAmSZKUEcOZJElSRgxnkiRJGTGcSZIkZcRwJkmSlBHDmSRJUkYMZ5IkSRlpajiLiBMjYm1EPBARi2rMHx8R10TELyLi7og4tdFlJUmSRqOmhbOIaAO+BJwEHAa8JyIOq+r2EeCelNKrgOOBf42IcQ0uK0mSNOo088zZMcADKaUHU0qbgUuBt1f1ScC+ERHAPsATwNYGl5UkSRp1mhnOpgLrKqY7y7ZKXwQOBdYDa4CzUkrbGlwWgIg4LSKWR8TyDRs2DFftkiRJLdHMcBY12lLV9DxgFTAFOAr4YkS8pMFli8aUvpZSmpNSmjN58uSdr1aSJCkDzQxnncD0iulpFGfIKp0KXJEKDwAPAa9scFlJkqRRp5nh7E7gkIiYGRHjgHcDV1f1eRQ4ASAiDgBmAQ82uKwkSdKos3uzVpxS2hoRZwDLgDbgopTS3RHx4XL+hcBngW9FxBqKocyPp5QeB6i1bLNqlSRJykXTwhlASuk64LqqtgsrXq8H3tzospIkSaOdTwiQJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkScqI4UySJCkjhjNJkqSMGM4kSZIyYjiTJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTJIkKSOGM0mSpIwYziRJkjJiOJMkaRexdGUXzzy3laef28rcxTexdGVXq0saVZau7GLu4puYuejalu5fw5kkSbuApSu7OOeKNaRyuqu7h3OuWGNAGyZ9+7eru4dEa/fv7iO+RUmSNGjnLVtLz5ZeGPdCW8+WXs6+fDWX3PFo6wobJVY+2s3m3m07tPVs6eW8ZWuZP3vqiNbimTNJknYB67t7arZXBwrtnHr7sd5+bybPnEmStAuYMqGDrhpBYeqEDi47/dgWVDS6zF18U839O2VCx4jX0tQzZxFxYkSsjYgHImJRjfkLI2JV+XVXRPRGxP7lvIcjYk05b3kz65QkKXcL582io71th7aO9jYWzpvVoopGl5z2b9POnEVEG/Al4E1AJ3BnRFydUrqnr09K6TzgvLL/ycD/TCk9UbGaN6SUHm9WjZIk7Sr6rnsad/VubO7dxtQJHSycN2vEr4carfr243nL1rK+u4cpLdy/zRzWPAZ4IKX0IEBEXAq8HbinTv/3AJc0sR5JknZp82dPhVX7AXDrqW9scTWjz/zZU7MIu80c1pwKrKuY7izbXiQi9gJOBL5f0ZyA6yNiRUScVm8jEXFaRCyPiOUbNmwYhrIlSZJap5nhLGq0pRptACcDt1YNac5NKR0NnAR8JCJeX2vBlNLXUkpzUkpzJk+ePLSKJUmSWqyZ4awTmF4xPQ1YX6fvu6ka0kwprS//fQy4kmKYVJKksWv1Eui8Ex75KZx/eDGt4bN6SbFfPzWhpfu3meHsTuCQiJgZEeMoAtjV1Z0iYjzw+8BVFW17R8S+fa+BNwN3NbFWSZLytnoJXHMm9D5fTG9aV0wb0IZH3/7dtA5ILd2/TftAQEppa0ScASwD2oCLUkp3R8SHy/kXll3fAVyfUvptxeIHAFdGRF+N30sp/ahZtUqSlL0bPwNbqu7DtaUHrjoDVny7NTWNJp13vhB8+2zpKfb7kQtGtJSm3oQ2pXQdcF1V24VV098CvlXV9iDwqmbWJknSLmVTZ+326kChnVNvP9bb703kEwIkSdoVjJ9WDrlVt0+HU68d+XpGm/MPr7N/p414KT5bU5KkXcEJ50J71aOE2juKdg1dRvvXcCZJ0q7gyAVw8gXFmTKi+PfkC0b8eqhRK6P9GynVu/XYrmfOnDlp+XIfwylJkvIXEStSSnOq2z1zJkmSlBHDmSRJUkYMZ5IkSRkxnEmSJGXEcCZJkpQRw5kkSVJGDGeSJEkZMZxJkiRlxHAmSZKUEcOZJElSRgxnkiRJGTGcSZIkZcRwJkmSlBHDmSRJUkYMZ5IkSRkxnEmSJGXEcCZJkpQRw5kkSVJGDGeSJEkZMZxJkiRlJFJKra5h2ETEBuCRJm9mEvB4k7ehkeUxHZ08rqOPx3R0GsvH9eCU0uTqxlEVzkZCRCxPKc1pdR0aPh7T0cnjOvp4TEcnj+uLOawpSZKUEcOZJElSRgxng/e1VhegYecxHZ08rqOPx3R08rhW8ZozSZKkjHjmTJIkKSOGM0mSpIwYzkoRcWJErI2IByJiUY35EREXlPNXR8TRjS6r1hnicb0oIh6LiLtGtmr1Z2ePaURMj4ifRMS9EXF3RJw18tWrniEc1z0j4o6I+EV5XD898tWrlqH8/i3nt0XEyoj4wchVnYmU0pj/AtqAXwK/A4wDfgEcVtXnLcAPgQBeB/y80WX92vWOaznv9cDRwF2tfi9+Df2YAgcCR5ev9wX+25/VPL6GeFwD2Kd83Q78HHhdq9/TWP8a6u/fcv5fA98DftDq9zPSX545KxwDPJBSejCltBm4FHh7VZ+3A99JhduBCRFxYIPLqjWGclxJKd0CPDGiFWsgO31MU0q/Sin9F0BK6WngXmDqSBavuoZyXFNK6ZmyT3v55SfdWm9Iv38jYhrwVuD/jGTRuTCcFaYC6yqmO3nxL+16fRpZVq0xlOOqPA3LMY2IGcBsirMsar0hHddy+GsV8Bjw45SSx7X1hvqz+gXgbGBbk+rLmuGsEDXaqv/nVa9PI8uqNYZyXJWnIR/TiNgH+D7wsZTSU8NYm3bekI5rSqk3pXQUMA04JiIOH97ytBN2+phGxB8Cj6WUVgx/WbsGw1mhE5heMT0NWN9gn0aWVWsM5bgqT0M6phHRThHMvptSuqKJdWpwhuVnNaXUDdwMnDjsFWqwhnJM5wJvi4iHKYZD3xgRFzev1PwYzgp3AodExMyIGAe8G7i6qs/VwPvKT5e8DtiUUvpVg8uqNYZyXJWnnT6mERHAN4B7U0qfH9myNYChHNfJETEBICI6gD8A7hvB2lXbTh/TlNI5KaVpKaUZ5XI3pZROGdHqW2z3VheQg5TS1og4A1hG8QmTi1JKd0fEh8v5FwLXUXyy5AHgWeDU/pZtwdtQlaEcV4CIuAQ4HpgUEZ3AJ1NK3xjZd6FKQzymc4E/A9aU1ycBfCKldN0IvgXVMMTjeiDw7YhoozjhsCSlNPZuvZCZof7+Het8fJMkSVJGHNaUJEnKiOFMkiQpI4YzSZKkjBjOJEmSMmI4kyRJyojhTNKYFxEPR8SkofaRpOFgOJMkScqI4UzSmBIRSyNiRUTcHRGnVc2bERH3RcS3I2J1RFweEXtVdPloRPxXRKyJiFeWyxwTET+LiJXlv7NG9A1JGnUMZ5LGmg+klF4NzAHOjIiJVfNnAV9LKR0JPAX8VcW8x1NKRwNfAf62bLsPeH1KaTZwLvBPTa1e0qhnOJM01pwZEb8Abqd46PIhVfPXpZRuLV9fDBxXMa/vYekrgBnl6/HAv0fEXcD5wO82o2hJY4fhTNKYERHHUzwY+9iU0quAlcCeVd2qn2lXOf18+W8vLzyb+LPAT1JKhwMn11ifJA2K4UzSWDIeeDKl9Gx5zdjravQ5KCKOLV+/B/hpA+vsKl+/f1iqlDSmGc4kjSU/AnaPiNUUZ7xur9HnXuDPyz77U1xf1p9/Af45Im4F2oazWEljU6RUfQZfksamiJgB/KAcopSklvDMmSRJUkY8cyZJkpQRz5xJkiRlxHAmSZKUEcOZJElSRgxnkiRJGTGcSZIkZeT/AczSW+4XtxSKAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light",
            "tags": []
          },
          "output_type": "display_data"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "26fcP9JaZajV"
      },
      "source": [
        "* As we can see, When ``ccp_alpha`` is set to ``zero`` and keeping the other default parameters of DecisionTreeClassifier, the tree overfits, leading to a ``97.5%`` training accuracy and ``80%`` testing accuracy.\n",
        "\n",
        "\n",
        "* As alpha increases, more of the tree is pruned, thus creating a decision tree that generalizes better.Setting ``ccp_alpha=8.70266265e-05`` maximizes the testing accuracy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bl4Z_z9sZajW"
      },
      "source": [
        "# Extra : Gain Ratio\n",
        "Information gain is biased for the attribute with many outcomes. It means it prefers the attribute with a large number of distinct values. For instance, consider an attribute with a unique identifier such as customer_ID has zero info(D) because of pure partition. This maximizes the information gain and creates useless partitioning.\n",
        "\n",
        "C4.5, an improvement of ID3, uses an extension to information gain known as the gain ratio. Gain ratio handles the issue of bias by normalizing the information gain using Split Info"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E1b4H4YRZajX"
      },
      "source": [
        "# Decision Tree Pros and Cons"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y3EbtV9bZajY"
      },
      "source": [
        "\n",
        "| Pros | Cons\n",
        "| --- | --- \n",
        "|Easy to interpret and visualize.| Sensitive to noisy data. It can overfit noisy data.\n",
        "|Has no assumptions about distribution because of the non-parametric nature of the algorithm.|The small variation(or variance) in data can result in the different decision tree. This can be reduced by bagging and boosting algorithms.\n",
        "|Can easily capture Non-linear patterns.|Biased with imbalance dataset, so it is recommended that balance out the dataset before creating the decision tree.\n",
        "|It requires fewer data preprocessing from the user, for example, there is no need to normalize columns.|Greedy algorithms don’t yield the global optimum tree structure.\n",
        "|It can be used for feature engineering such as predicting missing values, suitable for variable selection.| As the branching goes deeper, you get exponentially less data.\n",
        "|It has no problem whether the inputs and outputs are binary, categorical or continuous"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wS7T3bDdZajZ"
      },
      "source": [
        "# Tasks [ 10 points ]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "40YO0fkuZajb"
      },
      "source": [
        "## Task 1 : Work with the COMPLETE [dataset](https://archive.ics.uci.edu/ml/datasets/adult)\n",
        "\n",
        "- **[ 2 points ] :** Load the dataset and perform the proper data preprocessing/ cleaning. Justify all your steps. Elaborate and descripe as much as possible. Don't worry, I will read everything.\n",
        "\n",
        "\n",
        "- **[ 2 point  ] :** Run our implementation on the complete dataset. Compare the accuracy you get with the one we got before. Explain why the results are either better or worse.\n",
        "\n",
        "\n",
        "- **[ 1 point  ] :** Perform hyper-parameter tuning using gridsearch. \n",
        "\n",
        "\n",
        "- **[ 1 point  ] :** Visualize the tree by using the sklearn functions. Discuss the first two features and attributes that the tree were split on. Do you they make sense for you? Why?\n",
        "\n",
        "\n",
        "- **[ 2 point  ] :** Plot the feature importance, and choose the most important 5 of them and run our algorithm only on them. Compare the accuracy you get with the accuracy you got on the full dataset. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yRMF0kriOb-g",
        "outputId": "4aeba375-e114-48b3-c0ab-d4786668b8cd"
      },
      "source": [
        "df = pd.read_csv('adult.data')\n",
        "\n",
        "df_clean = df.copy()\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>39</th>\n",
              "      <th>State-gov</th>\n",
              "      <th>77516</th>\n",
              "      <th>Bachelors</th>\n",
              "      <th>13</th>\n",
              "      <th>Never-married</th>\n",
              "      <th>Adm-clerical</th>\n",
              "      <th>Not-in-family</th>\n",
              "      <th>White</th>\n",
              "      <th>Male</th>\n",
              "      <th>2174</th>\n",
              "      <th>0</th>\n",
              "      <th>40</th>\n",
              "      <th>United-States</th>\n",
              "      <th>&lt;=50K</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>50</td>\n",
              "      <td>Self-emp-not-inc</td>\n",
              "      <td>83311</td>\n",
              "      <td>Bachelors</td>\n",
              "      <td>13</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Exec-managerial</td>\n",
              "      <td>Husband</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>13</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>38</td>\n",
              "      <td>Private</td>\n",
              "      <td>215646</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Divorced</td>\n",
              "      <td>Handlers-cleaners</td>\n",
              "      <td>Not-in-family</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>53</td>\n",
              "      <td>Private</td>\n",
              "      <td>234721</td>\n",
              "      <td>11th</td>\n",
              "      <td>7</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Handlers-cleaners</td>\n",
              "      <td>Husband</td>\n",
              "      <td>Black</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>28</td>\n",
              "      <td>Private</td>\n",
              "      <td>338409</td>\n",
              "      <td>Bachelors</td>\n",
              "      <td>13</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Prof-specialty</td>\n",
              "      <td>Wife</td>\n",
              "      <td>Black</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>Cuba</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>37</td>\n",
              "      <td>Private</td>\n",
              "      <td>284582</td>\n",
              "      <td>Masters</td>\n",
              "      <td>14</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Exec-managerial</td>\n",
              "      <td>Wife</td>\n",
              "      <td>White</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   39          State-gov   77516   Bachelors   13        Never-married  \\\n",
              "0  50   Self-emp-not-inc   83311   Bachelors   13   Married-civ-spouse   \n",
              "1  38            Private  215646     HS-grad    9             Divorced   \n",
              "2  53            Private  234721        11th    7   Married-civ-spouse   \n",
              "3  28            Private  338409   Bachelors   13   Married-civ-spouse   \n",
              "4  37            Private  284582     Masters   14   Married-civ-spouse   \n",
              "\n",
              "         Adm-clerical   Not-in-family   White     Male   2174   0   40  \\\n",
              "0     Exec-managerial         Husband   White     Male      0   0   13   \n",
              "1   Handlers-cleaners   Not-in-family   White     Male      0   0   40   \n",
              "2   Handlers-cleaners         Husband   Black     Male      0   0   40   \n",
              "3      Prof-specialty            Wife   Black   Female      0   0   40   \n",
              "4     Exec-managerial            Wife   White   Female      0   0   40   \n",
              "\n",
              "    United-States   <=50K  \n",
              "0   United-States   <=50K  \n",
              "1   United-States   <=50K  \n",
              "2   United-States   <=50K  \n",
              "3            Cuba   <=50K  \n",
              "4   United-States   <=50K  "
            ]
          },
          "execution_count": 206,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TAQNO2DAOb-h"
      },
      "source": [
        "### A very obvious problem here is that the columns' labels are all incorrect, they are values not labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SdhcrzsXOb-h",
        "outputId": "0c15ccd4-93e2-471d-dd7e-1cad258b4429"
      },
      "source": [
        "col = {'39': 'age', ' State-gov': 'workclass', ' 77516': 'fnlwgt'\n",
        "                             , ' Bachelors': 'education', ' 13': 'education-num', ' Never-married': 'marital-status',\n",
        "                             ' Adm-clerical': 'occupation', ' Not-in-family': 'relationship', ' White': 'race', ' Male': 'sex',\n",
        "                             ' 2174': 'capital-gain', ' 0': 'capital-loss', ' 40': 'hours-per-week',\n",
        "                             ' United-States': 'native-country', ' <=50K': 'eligible'}\n",
        "df.columns"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Index(['39', ' State-gov', ' 77516', ' Bachelors', ' 13', ' Never-married',\n",
              "       ' Adm-clerical', ' Not-in-family', ' White', ' Male', ' 2174', ' 0',\n",
              "       ' 40', ' United-States', ' <=50K'],\n",
              "      dtype='object')"
            ]
          },
          "execution_count": 207,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZsoGcAhHOb-h",
        "outputId": "8042cf54-226f-44f3-b554-74c1b858e093"
      },
      "source": [
        "df_clean = df.rename(columns=col) \n",
        "df_clean.head()\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>workclass</th>\n",
              "      <th>fnlwgt</th>\n",
              "      <th>education</th>\n",
              "      <th>education-num</th>\n",
              "      <th>marital-status</th>\n",
              "      <th>occupation</th>\n",
              "      <th>relationship</th>\n",
              "      <th>race</th>\n",
              "      <th>sex</th>\n",
              "      <th>capital-gain</th>\n",
              "      <th>capital-loss</th>\n",
              "      <th>hours-per-week</th>\n",
              "      <th>native-country</th>\n",
              "      <th>eligible</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>50</td>\n",
              "      <td>Self-emp-not-inc</td>\n",
              "      <td>83311</td>\n",
              "      <td>Bachelors</td>\n",
              "      <td>13</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Exec-managerial</td>\n",
              "      <td>Husband</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>13</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>38</td>\n",
              "      <td>Private</td>\n",
              "      <td>215646</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Divorced</td>\n",
              "      <td>Handlers-cleaners</td>\n",
              "      <td>Not-in-family</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>53</td>\n",
              "      <td>Private</td>\n",
              "      <td>234721</td>\n",
              "      <td>11th</td>\n",
              "      <td>7</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Handlers-cleaners</td>\n",
              "      <td>Husband</td>\n",
              "      <td>Black</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>28</td>\n",
              "      <td>Private</td>\n",
              "      <td>338409</td>\n",
              "      <td>Bachelors</td>\n",
              "      <td>13</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Prof-specialty</td>\n",
              "      <td>Wife</td>\n",
              "      <td>Black</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>Cuba</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>37</td>\n",
              "      <td>Private</td>\n",
              "      <td>284582</td>\n",
              "      <td>Masters</td>\n",
              "      <td>14</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Exec-managerial</td>\n",
              "      <td>Wife</td>\n",
              "      <td>White</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   age          workclass  fnlwgt   education  education-num  \\\n",
              "0   50   Self-emp-not-inc   83311   Bachelors             13   \n",
              "1   38            Private  215646     HS-grad              9   \n",
              "2   53            Private  234721        11th              7   \n",
              "3   28            Private  338409   Bachelors             13   \n",
              "4   37            Private  284582     Masters             14   \n",
              "\n",
              "        marital-status          occupation    relationship    race      sex  \\\n",
              "0   Married-civ-spouse     Exec-managerial         Husband   White     Male   \n",
              "1             Divorced   Handlers-cleaners   Not-in-family   White     Male   \n",
              "2   Married-civ-spouse   Handlers-cleaners         Husband   Black     Male   \n",
              "3   Married-civ-spouse      Prof-specialty            Wife   Black   Female   \n",
              "4   Married-civ-spouse     Exec-managerial            Wife   White   Female   \n",
              "\n",
              "   capital-gain  capital-loss  hours-per-week  native-country eligible  \n",
              "0             0             0              13   United-States    <=50K  \n",
              "1             0             0              40   United-States    <=50K  \n",
              "2             0             0              40   United-States    <=50K  \n",
              "3             0             0              40            Cuba    <=50K  \n",
              "4             0             0              40   United-States    <=50K  "
            ]
          },
          "execution_count": 208,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eh6ThImDOb-i",
        "outputId": "cb45c83b-41f6-4616-926a-344bdd6e07aa"
      },
      "source": [
        "df_clean.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 32560 entries, 0 to 32559\n",
            "Data columns (total 15 columns):\n",
            " #   Column          Non-Null Count  Dtype \n",
            "---  ------          --------------  ----- \n",
            " 0   age             32560 non-null  int64 \n",
            " 1   workclass       32560 non-null  object\n",
            " 2   fnlwgt          32560 non-null  int64 \n",
            " 3   education       32560 non-null  object\n",
            " 4   education-num   32560 non-null  int64 \n",
            " 5   marital-status  32560 non-null  object\n",
            " 6   occupation      32560 non-null  object\n",
            " 7   relationship    32560 non-null  object\n",
            " 8   race            32560 non-null  object\n",
            " 9   sex             32560 non-null  object\n",
            " 10  capital-gain    32560 non-null  int64 \n",
            " 11  capital-loss    32560 non-null  int64 \n",
            " 12  hours-per-week  32560 non-null  int64 \n",
            " 13  native-country  32560 non-null  object\n",
            " 14  eligible        32560 non-null  object\n",
            "dtypes: int64(6), object(9)\n",
            "memory usage: 3.7+ MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZBdw7xbyOb-i"
      },
      "source": [
        "### There are no null values, the data is complete. \n",
        "### The dataypes are right "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TQ4r_T1BOb-i",
        "outputId": "55690a47-57a6-4f2f-f89e-96851a21c525"
      },
      "source": [
        "df_clean.describe()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>fnlwgt</th>\n",
              "      <th>education-num</th>\n",
              "      <th>capital-gain</th>\n",
              "      <th>capital-loss</th>\n",
              "      <th>hours-per-week</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>32560.000000</td>\n",
              "      <td>3.256000e+04</td>\n",
              "      <td>32560.000000</td>\n",
              "      <td>32560.000000</td>\n",
              "      <td>32560.000000</td>\n",
              "      <td>32560.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>38.581634</td>\n",
              "      <td>1.897818e+05</td>\n",
              "      <td>10.080590</td>\n",
              "      <td>1077.615172</td>\n",
              "      <td>87.306511</td>\n",
              "      <td>40.437469</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>13.640642</td>\n",
              "      <td>1.055498e+05</td>\n",
              "      <td>2.572709</td>\n",
              "      <td>7385.402999</td>\n",
              "      <td>402.966116</td>\n",
              "      <td>12.347618</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>17.000000</td>\n",
              "      <td>1.228500e+04</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>28.000000</td>\n",
              "      <td>1.178315e+05</td>\n",
              "      <td>9.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>40.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>37.000000</td>\n",
              "      <td>1.783630e+05</td>\n",
              "      <td>10.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>40.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>48.000000</td>\n",
              "      <td>2.370545e+05</td>\n",
              "      <td>12.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>45.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>90.000000</td>\n",
              "      <td>1.484705e+06</td>\n",
              "      <td>16.000000</td>\n",
              "      <td>99999.000000</td>\n",
              "      <td>4356.000000</td>\n",
              "      <td>99.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                age        fnlwgt  education-num  capital-gain  capital-loss  \\\n",
              "count  32560.000000  3.256000e+04   32560.000000  32560.000000  32560.000000   \n",
              "mean      38.581634  1.897818e+05      10.080590   1077.615172     87.306511   \n",
              "std       13.640642  1.055498e+05       2.572709   7385.402999    402.966116   \n",
              "min       17.000000  1.228500e+04       1.000000      0.000000      0.000000   \n",
              "25%       28.000000  1.178315e+05       9.000000      0.000000      0.000000   \n",
              "50%       37.000000  1.783630e+05      10.000000      0.000000      0.000000   \n",
              "75%       48.000000  2.370545e+05      12.000000      0.000000      0.000000   \n",
              "max       90.000000  1.484705e+06      16.000000  99999.000000   4356.000000   \n",
              "\n",
              "       hours-per-week  \n",
              "count    32560.000000  \n",
              "mean        40.437469  \n",
              "std         12.347618  \n",
              "min          1.000000  \n",
              "25%         40.000000  \n",
              "50%         40.000000  \n",
              "75%         45.000000  \n",
              "max         99.000000  "
            ]
          },
          "execution_count": 210,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ihIjLQ8JOb-i"
      },
      "source": [
        "### The data is not normalized nor standardized "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bTUKGZ1VOb-j",
        "outputId": "dc1f2456-fa1b-49db-d33e-402218a7a1cb"
      },
      "source": [
        "df_clean.sample(20)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>workclass</th>\n",
              "      <th>fnlwgt</th>\n",
              "      <th>education</th>\n",
              "      <th>education-num</th>\n",
              "      <th>marital-status</th>\n",
              "      <th>occupation</th>\n",
              "      <th>relationship</th>\n",
              "      <th>race</th>\n",
              "      <th>sex</th>\n",
              "      <th>capital-gain</th>\n",
              "      <th>capital-loss</th>\n",
              "      <th>hours-per-week</th>\n",
              "      <th>native-country</th>\n",
              "      <th>eligible</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>32514</th>\n",
              "      <td>66</td>\n",
              "      <td>Federal-gov</td>\n",
              "      <td>47358</td>\n",
              "      <td>10th</td>\n",
              "      <td>6</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Craft-repair</td>\n",
              "      <td>Husband</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>3471</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27432</th>\n",
              "      <td>29</td>\n",
              "      <td>Private</td>\n",
              "      <td>187188</td>\n",
              "      <td>Masters</td>\n",
              "      <td>14</td>\n",
              "      <td>Never-married</td>\n",
              "      <td>Exec-managerial</td>\n",
              "      <td>Not-in-family</td>\n",
              "      <td>Asian-Pac-Islander</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>60</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16800</th>\n",
              "      <td>56</td>\n",
              "      <td>Private</td>\n",
              "      <td>111218</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Transport-moving</td>\n",
              "      <td>Husband</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>60</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26931</th>\n",
              "      <td>25</td>\n",
              "      <td>Private</td>\n",
              "      <td>113035</td>\n",
              "      <td>Some-college</td>\n",
              "      <td>10</td>\n",
              "      <td>Never-married</td>\n",
              "      <td>Exec-managerial</td>\n",
              "      <td>Own-child</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29303</th>\n",
              "      <td>24</td>\n",
              "      <td>Private</td>\n",
              "      <td>44738</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Never-married</td>\n",
              "      <td>Handlers-cleaners</td>\n",
              "      <td>Other-relative</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18080</th>\n",
              "      <td>31</td>\n",
              "      <td>Private</td>\n",
              "      <td>192900</td>\n",
              "      <td>Some-college</td>\n",
              "      <td>10</td>\n",
              "      <td>Never-married</td>\n",
              "      <td>Other-service</td>\n",
              "      <td>Not-in-family</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28181</th>\n",
              "      <td>23</td>\n",
              "      <td>Self-emp-inc</td>\n",
              "      <td>39844</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Never-married</td>\n",
              "      <td>Transport-moving</td>\n",
              "      <td>Not-in-family</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>50</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10451</th>\n",
              "      <td>31</td>\n",
              "      <td>Local-gov</td>\n",
              "      <td>213307</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Never-married</td>\n",
              "      <td>Adm-clerical</td>\n",
              "      <td>Own-child</td>\n",
              "      <td>White</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22384</th>\n",
              "      <td>47</td>\n",
              "      <td>Private</td>\n",
              "      <td>246739</td>\n",
              "      <td>Bachelors</td>\n",
              "      <td>13</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Exec-managerial</td>\n",
              "      <td>Husband</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>99999</td>\n",
              "      <td>0</td>\n",
              "      <td>55</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&gt;50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16016</th>\n",
              "      <td>41</td>\n",
              "      <td>Private</td>\n",
              "      <td>207578</td>\n",
              "      <td>Assoc-acdm</td>\n",
              "      <td>12</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Exec-managerial</td>\n",
              "      <td>Husband</td>\n",
              "      <td>Black</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>50</td>\n",
              "      <td>India</td>\n",
              "      <td>&gt;50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17767</th>\n",
              "      <td>37</td>\n",
              "      <td>Private</td>\n",
              "      <td>178948</td>\n",
              "      <td>Bachelors</td>\n",
              "      <td>13</td>\n",
              "      <td>Never-married</td>\n",
              "      <td>Prof-specialty</td>\n",
              "      <td>Not-in-family</td>\n",
              "      <td>White</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>45</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23884</th>\n",
              "      <td>40</td>\n",
              "      <td>Private</td>\n",
              "      <td>231991</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Divorced</td>\n",
              "      <td>Exec-managerial</td>\n",
              "      <td>Unmarried</td>\n",
              "      <td>White</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19713</th>\n",
              "      <td>19</td>\n",
              "      <td>Private</td>\n",
              "      <td>109005</td>\n",
              "      <td>Some-college</td>\n",
              "      <td>10</td>\n",
              "      <td>Never-married</td>\n",
              "      <td>Prof-specialty</td>\n",
              "      <td>Own-child</td>\n",
              "      <td>White</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6069</th>\n",
              "      <td>48</td>\n",
              "      <td>Self-emp-not-inc</td>\n",
              "      <td>185385</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Farming-fishing</td>\n",
              "      <td>Husband</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>98</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14011</th>\n",
              "      <td>28</td>\n",
              "      <td>Private</td>\n",
              "      <td>338376</td>\n",
              "      <td>Some-college</td>\n",
              "      <td>10</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Adm-clerical</td>\n",
              "      <td>Husband</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>50</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&gt;50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5740</th>\n",
              "      <td>46</td>\n",
              "      <td>Private</td>\n",
              "      <td>256522</td>\n",
              "      <td>1st-4th</td>\n",
              "      <td>2</td>\n",
              "      <td>Never-married</td>\n",
              "      <td>Machine-op-inspct</td>\n",
              "      <td>Own-child</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>Puerto-Rico</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23656</th>\n",
              "      <td>37</td>\n",
              "      <td>Private</td>\n",
              "      <td>111499</td>\n",
              "      <td>Some-college</td>\n",
              "      <td>10</td>\n",
              "      <td>Never-married</td>\n",
              "      <td>Exec-managerial</td>\n",
              "      <td>Not-in-family</td>\n",
              "      <td>White</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>40</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24646</th>\n",
              "      <td>20</td>\n",
              "      <td>Private</td>\n",
              "      <td>250630</td>\n",
              "      <td>Some-college</td>\n",
              "      <td>10</td>\n",
              "      <td>Never-married</td>\n",
              "      <td>Sales</td>\n",
              "      <td>Not-in-family</td>\n",
              "      <td>White</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>20</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25140</th>\n",
              "      <td>23</td>\n",
              "      <td>Private</td>\n",
              "      <td>41721</td>\n",
              "      <td>Assoc-voc</td>\n",
              "      <td>11</td>\n",
              "      <td>Married-civ-spouse</td>\n",
              "      <td>Craft-repair</td>\n",
              "      <td>Husband</td>\n",
              "      <td>White</td>\n",
              "      <td>Male</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>55</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&gt;50K</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>230</th>\n",
              "      <td>41</td>\n",
              "      <td>Private</td>\n",
              "      <td>130408</td>\n",
              "      <td>HS-grad</td>\n",
              "      <td>9</td>\n",
              "      <td>Divorced</td>\n",
              "      <td>Sales</td>\n",
              "      <td>Unmarried</td>\n",
              "      <td>Black</td>\n",
              "      <td>Female</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>38</td>\n",
              "      <td>United-States</td>\n",
              "      <td>&lt;=50K</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       age          workclass  fnlwgt      education  education-num  \\\n",
              "32514   66        Federal-gov   47358           10th              6   \n",
              "27432   29            Private  187188        Masters             14   \n",
              "16800   56            Private  111218        HS-grad              9   \n",
              "26931   25            Private  113035   Some-college             10   \n",
              "29303   24            Private   44738        HS-grad              9   \n",
              "18080   31            Private  192900   Some-college             10   \n",
              "28181   23       Self-emp-inc   39844        HS-grad              9   \n",
              "10451   31          Local-gov  213307        HS-grad              9   \n",
              "22384   47            Private  246739      Bachelors             13   \n",
              "16016   41            Private  207578     Assoc-acdm             12   \n",
              "17767   37            Private  178948      Bachelors             13   \n",
              "23884   40            Private  231991        HS-grad              9   \n",
              "19713   19            Private  109005   Some-college             10   \n",
              "6069    48   Self-emp-not-inc  185385        HS-grad              9   \n",
              "14011   28            Private  338376   Some-college             10   \n",
              "5740    46            Private  256522        1st-4th              2   \n",
              "23656   37            Private  111499   Some-college             10   \n",
              "24646   20            Private  250630   Some-college             10   \n",
              "25140   23            Private   41721      Assoc-voc             11   \n",
              "230     41            Private  130408        HS-grad              9   \n",
              "\n",
              "            marital-status          occupation     relationship  \\\n",
              "32514   Married-civ-spouse        Craft-repair          Husband   \n",
              "27432        Never-married     Exec-managerial    Not-in-family   \n",
              "16800   Married-civ-spouse    Transport-moving          Husband   \n",
              "26931        Never-married     Exec-managerial        Own-child   \n",
              "29303        Never-married   Handlers-cleaners   Other-relative   \n",
              "18080        Never-married       Other-service    Not-in-family   \n",
              "28181        Never-married    Transport-moving    Not-in-family   \n",
              "10451        Never-married        Adm-clerical        Own-child   \n",
              "22384   Married-civ-spouse     Exec-managerial          Husband   \n",
              "16016   Married-civ-spouse     Exec-managerial          Husband   \n",
              "17767        Never-married      Prof-specialty    Not-in-family   \n",
              "23884             Divorced     Exec-managerial        Unmarried   \n",
              "19713        Never-married      Prof-specialty        Own-child   \n",
              "6069    Married-civ-spouse     Farming-fishing          Husband   \n",
              "14011   Married-civ-spouse        Adm-clerical          Husband   \n",
              "5740         Never-married   Machine-op-inspct        Own-child   \n",
              "23656        Never-married     Exec-managerial    Not-in-family   \n",
              "24646        Never-married               Sales    Not-in-family   \n",
              "25140   Married-civ-spouse        Craft-repair          Husband   \n",
              "230               Divorced               Sales        Unmarried   \n",
              "\n",
              "                      race      sex  capital-gain  capital-loss  \\\n",
              "32514                White     Male          3471             0   \n",
              "27432   Asian-Pac-Islander     Male             0             0   \n",
              "16800                White     Male             0             0   \n",
              "26931                White     Male             0             0   \n",
              "29303                White     Male             0             0   \n",
              "18080                White     Male             0             0   \n",
              "28181                White     Male             0             0   \n",
              "10451                White   Female             0             0   \n",
              "22384                White     Male         99999             0   \n",
              "16016                Black     Male             0             0   \n",
              "17767                White   Female             0             0   \n",
              "23884                White   Female             0             0   \n",
              "19713                White   Female             0             0   \n",
              "6069                 White     Male             0             0   \n",
              "14011                White     Male             0             0   \n",
              "5740                 White     Male             0             0   \n",
              "23656                White   Female             0             0   \n",
              "24646                White   Female             0             0   \n",
              "25140                White     Male             0             0   \n",
              "230                  Black   Female             0             0   \n",
              "\n",
              "       hours-per-week  native-country eligible  \n",
              "32514              40   United-States    <=50K  \n",
              "27432              60   United-States    <=50K  \n",
              "16800              60   United-States    <=50K  \n",
              "26931              40   United-States    <=50K  \n",
              "29303              40   United-States    <=50K  \n",
              "18080              40   United-States    <=50K  \n",
              "28181              50   United-States    <=50K  \n",
              "10451              40   United-States    <=50K  \n",
              "22384              55   United-States     >50K  \n",
              "16016              50           India     >50K  \n",
              "17767              45   United-States    <=50K  \n",
              "23884              40   United-States    <=50K  \n",
              "19713              40   United-States    <=50K  \n",
              "6069               98   United-States    <=50K  \n",
              "14011              50   United-States     >50K  \n",
              "5740               40     Puerto-Rico    <=50K  \n",
              "23656              40   United-States    <=50K  \n",
              "24646              20   United-States    <=50K  \n",
              "25140              55   United-States     >50K  \n",
              "230                38   United-States    <=50K  "
            ]
          },
          "execution_count": 211,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p-0puXcJOb-j"
      },
      "source": [
        "### Now the `eligible` column's values should be changed to `yes` , `no` ,\n",
        "### or `1` for eligible (>50K) and `0` for not eligible (<=50K)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JL3nc52BOb-j",
        "outputId": "128fe8f8-857e-4ad2-d8d2-6edffcdba8cb"
      },
      "source": [
        "#df_clean[df_clean['eligible'] == \" >50K\"].eligible\n",
        "df_clean['eligible'] = df_clean.eligible.apply(lambda x: 1 if x == \" >50K\" else 0)\n",
        "#df_clean[df_clean['eligible'] == \" <=50K\"].eligible = 0\n",
        "\n",
        "df_clean['eligible'].value_counts()\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0    24719\n",
              "1     7841\n",
              "Name: eligible, dtype: int64"
            ]
          },
          "execution_count": 212,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aE5ouT3bOb-k",
        "outputId": "89988e76-c26f-45c3-81a9-60696703708e"
      },
      "source": [
        "df_clean['workclass'].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              " Private             22696\n",
              " Self-emp-not-inc     2541\n",
              " Local-gov            2093\n",
              " ?                    1836\n",
              " State-gov            1297\n",
              " Self-emp-inc         1116\n",
              " Federal-gov           960\n",
              " Without-pay            14\n",
              " Never-worked            7\n",
              "Name: workclass, dtype: int64"
            ]
          },
          "execution_count": 213,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bp_vn6ocOb-k",
        "outputId": "d6dc3df8-c134-4ca8-c14e-adebde53f807"
      },
      "source": [
        "df_clean['occupation'].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              " Prof-specialty       4140\n",
              " Craft-repair         4099\n",
              " Exec-managerial      4066\n",
              " Adm-clerical         3769\n",
              " Sales                3650\n",
              " Other-service        3295\n",
              " Machine-op-inspct    2002\n",
              " ?                    1843\n",
              " Transport-moving     1597\n",
              " Handlers-cleaners    1370\n",
              " Farming-fishing       994\n",
              " Tech-support          928\n",
              " Protective-serv       649\n",
              " Priv-house-serv       149\n",
              " Armed-Forces            9\n",
              "Name: occupation, dtype: int64"
            ]
          },
          "execution_count": 214,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eQ_t5EhlOb-k",
        "outputId": "c0f87081-768d-41f1-c5b1-eb0c8a9bda8a"
      },
      "source": [
        "v_c = {}\n",
        "for col in df_clean.columns:\n",
        "    v_c [col] = [df_clean[col].value_counts()]\n",
        "v_c"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'age': [36    898\n",
              "  31    888\n",
              "  34    886\n",
              "  23    877\n",
              "  35    876\n",
              "       ... \n",
              "  83      6\n",
              "  85      3\n",
              "  88      3\n",
              "  87      1\n",
              "  86      1\n",
              "  Name: age, Length: 73, dtype: int64],\n",
              " 'workclass': [ Private             22696\n",
              "   Self-emp-not-inc     2541\n",
              "   Local-gov            2093\n",
              "   ?                    1836\n",
              "   State-gov            1297\n",
              "   Self-emp-inc         1116\n",
              "   Federal-gov           960\n",
              "   Without-pay            14\n",
              "   Never-worked            7\n",
              "  Name: workclass, dtype: int64],\n",
              " 'fnlwgt': [123011    13\n",
              "  203488    13\n",
              "  164190    13\n",
              "  121124    12\n",
              "  148995    12\n",
              "            ..\n",
              "  68318      1\n",
              "  284211     1\n",
              "  312881     1\n",
              "  177711     1\n",
              "  229376     1\n",
              "  Name: fnlwgt, Length: 21647, dtype: int64],\n",
              " 'education': [ HS-grad         10501\n",
              "   Some-college     7291\n",
              "   Bachelors        5354\n",
              "   Masters          1723\n",
              "   Assoc-voc        1382\n",
              "   11th             1175\n",
              "   Assoc-acdm       1067\n",
              "   10th              933\n",
              "   7th-8th           646\n",
              "   Prof-school       576\n",
              "   9th               514\n",
              "   12th              433\n",
              "   Doctorate         413\n",
              "   5th-6th           333\n",
              "   1st-4th           168\n",
              "   Preschool          51\n",
              "  Name: education, dtype: int64],\n",
              " 'education-num': [9     10501\n",
              "  10     7291\n",
              "  13     5354\n",
              "  14     1723\n",
              "  11     1382\n",
              "  7      1175\n",
              "  12     1067\n",
              "  6       933\n",
              "  4       646\n",
              "  15      576\n",
              "  5       514\n",
              "  8       433\n",
              "  16      413\n",
              "  3       333\n",
              "  2       168\n",
              "  1        51\n",
              "  Name: education-num, dtype: int64],\n",
              " 'marital-status': [ Married-civ-spouse       14976\n",
              "   Never-married            10682\n",
              "   Divorced                  4443\n",
              "   Separated                 1025\n",
              "   Widowed                    993\n",
              "   Married-spouse-absent      418\n",
              "   Married-AF-spouse           23\n",
              "  Name: marital-status, dtype: int64],\n",
              " 'occupation': [ Prof-specialty       4140\n",
              "   Craft-repair         4099\n",
              "   Exec-managerial      4066\n",
              "   Adm-clerical         3769\n",
              "   Sales                3650\n",
              "   Other-service        3295\n",
              "   Machine-op-inspct    2002\n",
              "   ?                    1843\n",
              "   Transport-moving     1597\n",
              "   Handlers-cleaners    1370\n",
              "   Farming-fishing       994\n",
              "   Tech-support          928\n",
              "   Protective-serv       649\n",
              "   Priv-house-serv       149\n",
              "   Armed-Forces            9\n",
              "  Name: occupation, dtype: int64],\n",
              " 'relationship': [ Husband           13193\n",
              "   Not-in-family      8304\n",
              "   Own-child          5068\n",
              "   Unmarried          3446\n",
              "   Wife               1568\n",
              "   Other-relative      981\n",
              "  Name: relationship, dtype: int64],\n",
              " 'race': [ White                 27815\n",
              "   Black                  3124\n",
              "   Asian-Pac-Islander     1039\n",
              "   Amer-Indian-Eskimo      311\n",
              "   Other                   271\n",
              "  Name: race, dtype: int64],\n",
              " 'sex': [ Male      21789\n",
              "   Female    10771\n",
              "  Name: sex, dtype: int64],\n",
              " 'capital-gain': [0        29849\n",
              "  15024      347\n",
              "  7688       284\n",
              "  7298       246\n",
              "  99999      159\n",
              "           ...  \n",
              "  4931         1\n",
              "  1455         1\n",
              "  6097         1\n",
              "  22040        1\n",
              "  1111         1\n",
              "  Name: capital-gain, Length: 119, dtype: int64],\n",
              " 'capital-loss': [0       31041\n",
              "  1902      202\n",
              "  1977      168\n",
              "  1887      159\n",
              "  1848       51\n",
              "          ...  \n",
              "  1411        1\n",
              "  1539        1\n",
              "  2472        1\n",
              "  1944        1\n",
              "  2201        1\n",
              "  Name: capital-loss, Length: 92, dtype: int64],\n",
              " 'hours-per-week': [40    15216\n",
              "  50     2819\n",
              "  45     1824\n",
              "  60     1475\n",
              "  35     1297\n",
              "        ...  \n",
              "  92        1\n",
              "  94        1\n",
              "  87        1\n",
              "  74        1\n",
              "  82        1\n",
              "  Name: hours-per-week, Length: 94, dtype: int64],\n",
              " 'native-country': [ United-States                 29169\n",
              "   Mexico                          643\n",
              "   ?                               583\n",
              "   Philippines                     198\n",
              "   Germany                         137\n",
              "   Canada                          121\n",
              "   Puerto-Rico                     114\n",
              "   El-Salvador                     106\n",
              "   India                           100\n",
              "   Cuba                             95\n",
              "   England                          90\n",
              "   Jamaica                          81\n",
              "   South                            80\n",
              "   China                            75\n",
              "   Italy                            73\n",
              "   Dominican-Republic               70\n",
              "   Vietnam                          67\n",
              "   Guatemala                        64\n",
              "   Japan                            62\n",
              "   Poland                           60\n",
              "   Columbia                         59\n",
              "   Taiwan                           51\n",
              "   Haiti                            44\n",
              "   Iran                             43\n",
              "   Portugal                         37\n",
              "   Nicaragua                        34\n",
              "   Peru                             31\n",
              "   France                           29\n",
              "   Greece                           29\n",
              "   Ecuador                          28\n",
              "   Ireland                          24\n",
              "   Hong                             20\n",
              "   Cambodia                         19\n",
              "   Trinadad&Tobago                  19\n",
              "   Thailand                         18\n",
              "   Laos                             18\n",
              "   Yugoslavia                       16\n",
              "   Outlying-US(Guam-USVI-etc)       14\n",
              "   Hungary                          13\n",
              "   Honduras                         13\n",
              "   Scotland                         12\n",
              "   Holand-Netherlands                1\n",
              "  Name: native-country, dtype: int64],\n",
              " 'eligible': [0    24719\n",
              "  1     7841\n",
              "  Name: eligible, dtype: int64]}"
            ]
          },
          "execution_count": 215,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F4CXjIirOb-l"
      },
      "source": [
        "### The ? in `workclass`,  `occupation`, ` native-country`, should be changed to Nan values "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ddcx4O12Ob-l",
        "outputId": "d5a0bb14-432e-4130-e66b-22153a0d3d68"
      },
      "source": [
        "df_clean.replace({' ?': np.NaN}, inplace=True)\n",
        "df_clean.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 32560 entries, 0 to 32559\n",
            "Data columns (total 15 columns):\n",
            " #   Column          Non-Null Count  Dtype \n",
            "---  ------          --------------  ----- \n",
            " 0   age             32560 non-null  int64 \n",
            " 1   workclass       30724 non-null  object\n",
            " 2   fnlwgt          32560 non-null  int64 \n",
            " 3   education       32560 non-null  object\n",
            " 4   education-num   32560 non-null  int64 \n",
            " 5   marital-status  32560 non-null  object\n",
            " 6   occupation      30717 non-null  object\n",
            " 7   relationship    32560 non-null  object\n",
            " 8   race            32560 non-null  object\n",
            " 9   sex             32560 non-null  object\n",
            " 10  capital-gain    32560 non-null  int64 \n",
            " 11  capital-loss    32560 non-null  int64 \n",
            " 12  hours-per-week  32560 non-null  int64 \n",
            " 13  native-country  31977 non-null  object\n",
            " 14  eligible        32560 non-null  int64 \n",
            "dtypes: int64(7), object(8)\n",
            "memory usage: 3.7+ MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J_Nz0w-ROb-l"
      },
      "source": [
        "### Apparently, There were missing values in these columns!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PXx3cywnOb-m",
        "outputId": "acc473b7-258f-4af9-e7ac-7041943b73d6"
      },
      "source": [
        "sns.countplot(x='eligible',data=df_clean)\n",
        "\n",
        "##The data is not balanced"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<AxesSubplot:xlabel='eligible', ylabel='count'>"
            ]
          },
          "execution_count": 217,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEGCAYAAABPdROvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAASC0lEQVR4nO3df6zl9V3n8eeLGYpoBfkxsDjDOqSdmAK6dGcyi3b/qEuyjCYuWEGnbmVWMdMQurGJ2aSYzbZZM4nN2qI0QhYFGVALs7Tdsk1RybS2aSTQS0OEAdneSC0jU5gKodSkuDN97x/nc+2Zy7nX2/u555w53ucj+eZ8z/v7+XzP5zu5k1e+n+/3fE+qCkmSVuuUaQ9AkjTbDBJJUheDRJLUxSCRJHUxSCRJXTZOewCTdu6559bWrVunPQxJmimPPfbY16tq06ht6y5Itm7dytzc3LSHIUkzJcnfLLXNqS1JUheDRJLUxSCRJHUZW5AkuTDJZ5M8neRQkl9t9Q8k+dskj7flp4b63JRkPskzSa4cqm9P8kTbdkuStPppSe5r9UeSbB3X8UiSRhvnGckx4Neq6i3A5cCNSS5u226uqsva8mmAtm03cAmwC7g1yYbW/jZgL7CtLbta/Xrg5ap6M3Az8MExHo8kaYSxBUlVHamqL7X1V4Gngc3LdLkKuLeqXquqZ4F5YGeSC4AzqurhGjxh8m7g6qE++9v6/cAVC2crkqTJmMg1kjbl9FbgkVZ6T5K/THJnkrNabTPw3FC3w622ua0vrp/Qp6qOAa8A54z4/L1J5pLMHT16dG0OSpIETCBIkrwR+Bjw3qr6BoNpqjcBlwFHgA8tNB3RvZapL9fnxELV7VW1o6p2bNo08vs0kqRVGmuQJDmVQYj8UVV9HKCqXqiq41X1beD3gJ2t+WHgwqHuW4DnW33LiPoJfZJsBM4EXhrP0UiSRhnbN9vbtYo7gKer6sND9Quq6kh7+zPAk239AeCPk3wY+EEGF9UfrarjSV5NcjmDqbHrgI8M9dkDPAxcA3ymJvBLXdv/y93j/gjNoMf+x3XTHoI0FeN8RMrbgF8EnkjyeKv9OvDOJJcxmIL6CvBugKo6lOQA8BSDO75urKrjrd8NwF3A6cCDbYFBUN2TZJ7BmcjuMR6PJGmEsQVJVX2B0dcwPr1Mn33AvhH1OeDSEfVvAdd2DFOS1MlvtkuSuhgkkqQuBokkqYtBIknqYpBIkroYJJKkLgaJJKmLQSJJ6mKQSJK6GCSSpC4GiSSpi0EiSepikEiSuhgkkqQuBokkqYtBIknqYpBIkroYJJKkLgaJJKmLQSJJ6mKQSJK6GCSSpC4GiSSpi0EiSepikEiSuhgkkqQuBokkqYtBIknqYpBIkroYJJKkLgaJJKmLQSJJ6mKQSJK6jC1IklyY5LNJnk5yKMmvtvrZSR5K8uX2etZQn5uSzCd5JsmVQ/XtSZ5o225JklY/Lcl9rf5Ikq3jOh5J0mjjPCM5BvxaVb0FuBy4McnFwPuAg1W1DTjY3tO27QYuAXYBtybZ0PZ1G7AX2NaWXa1+PfByVb0ZuBn44BiPR5I0wtiCpKqOVNWX2vqrwNPAZuAqYH9rth+4uq1fBdxbVa9V1bPAPLAzyQXAGVX1cFUVcPeiPgv7uh+4YuFsRZI0GRO5RtKmnN4KPAKcX1VHYBA2wHmt2WbguaFuh1ttc1tfXD+hT1UdA14Bzhnx+XuTzCWZO3r06BodlSQJJhAkSd4IfAx4b1V9Y7mmI2q1TH25PicWqm6vqh1VtWPTpk3/1JAlSd+FsQZJklMZhMgfVdXHW/mFNl1Fe32x1Q8DFw513wI83+pbRtRP6JNkI3Am8NLaH4kkaSnjvGsrwB3A01X14aFNDwB72voe4JND9d3tTqyLGFxUf7RNf72a5PK2z+sW9VnY1zXAZ9p1FEnShGwc477fBvwi8ESSx1vt14HfBA4kuR74KnAtQFUdSnIAeIrBHV83VtXx1u8G4C7gdODBtsAgqO5JMs/gTGT3GI9HkjTC2IKkqr7A6GsYAFcs0WcfsG9EfQ64dET9W7QgkiRNh99slyR1MUgkSV0MEklSF4NEktTFIJEkdTFIJEldDBJJUheDRJLUxSCRJHUxSCRJXQwSSVIXg0SS1MUgkSR1MUgkSV0MEklSF4NEktTFIJEkdTFIJEldDBJJUheDRJLUxSCRJHUxSCRJXQwSSVIXg0SS1MUgkSR1MUgkSV0MEklSF4NEktTFIJEkdTFIJEldDBJJUheDRJLUxSCRJHUZW5AkuTPJi0meHKp9IMnfJnm8LT81tO2mJPNJnkly5VB9e5In2rZbkqTVT0tyX6s/kmTruI5FkrS0cZ6R3AXsGlG/uaoua8unAZJcDOwGLml9bk2yobW/DdgLbGvLwj6vB16uqjcDNwMfHNeBSJKWNrYgqarPAy+tsPlVwL1V9VpVPQvMAzuTXACcUVUPV1UBdwNXD/XZ39bvB65YOFuRJE3ONK6RvCfJX7apr7NabTPw3FCbw622ua0vrp/Qp6qOAa8A54z6wCR7k8wlmTt69OjaHYkkaeJBchvwJuAy4AjwoVYfdSZRy9SX6/P6YtXtVbWjqnZs2rTpuxqwJGl5Ew2Sqnqhqo5X1beB3wN2tk2HgQuHmm4Bnm/1LSPqJ/RJshE4k5VPpUmS1shEg6Rd81jwM8DCHV0PALvbnVgXMbio/mhVHQFeTXJ5u/5xHfDJoT572vo1wGfadRRJ0gRtHNeOk3wUeDtwbpLDwPuBtye5jMEU1FeAdwNU1aEkB4CngGPAjVV1vO3qBgZ3gJ0OPNgWgDuAe5LMMzgT2T2uY5EkLW1sQVJV7xxRvmOZ9vuAfSPqc8ClI+rfAq7tGaMkqd+KpraSHFxJTZK0/ix7RpLke4DvZTA9dRbfuVPqDOAHxzw2SdIM+Kemtt4NvJdBaDzGd4LkG8Dvjm9YkqRZsWyQVNXvAL+T5D9X1UcmNCZJ0gxZ0cX2qvpIkh8Htg73qaq7xzQuSdKMWFGQJLmHwTfSHwcWbstdePaVJGkdW+ntvzuAi/3CnyRpsZV+s/1J4F+McyCSpNm00jOSc4GnkjwKvLZQrKr/MJZRSZJmxkqD5APjHIQkaXat9K6tz417IJKk2bTSu7Ze5Tu/9fEG4FTg76vqjHENTJI0G1Z6RvL9w++TXM13fktEkrSOrer3SKrqfwP/bm2HIkmaRSud2nrH0NtTGHyvxO+USJJWfNfWTw+tH2Pwo1RXrfloJEkzZ6XXSH5p3AORJM2mlf6w1ZYkn0jyYpIXknwsyZZxD06SdPJb6cX2PwAeYPC7JJuB/9NqkqR1bqVBsqmq/qCqjrXlLmDTGMclSZoRKw2Sryd5V5INbXkX8HfjHJgkaTasNEh+Gfg54GvAEeAawAvwkqQV3/77G8CeqnoZIMnZwG8xCBhJ0jq20jOSH10IEYCqegl463iGJEmaJSsNklOSnLXwpp2RrPRsRpL0z9hKw+BDwF8kuZ/Bo1F+Dtg3tlFJkmbGSr/ZfneSOQYPagzwjqp6aqwjkyTNhBVPT7XgMDwkSSdY1WPkJUlaYJBIkroYJJKkLgaJJKmLQSJJ6jK2IElyZ/v9kieHamcneSjJl9vr8Jccb0oyn+SZJFcO1bcneaJtuyVJWv20JPe1+iNJto7rWCRJSxvnGcldwK5FtfcBB6tqG3CwvSfJxcBu4JLW59YkG1qf24C9wLa2LOzzeuDlqnozcDPwwbEdiSRpSWMLkqr6PPDSovJVwP62vh+4eqh+b1W9VlXPAvPAziQXAGdU1cNVVcDdi/os7Ot+4IqFsxVJ0uRM+hrJ+VV1BKC9ntfqm4HnhtodbrXNbX1x/YQ+VXUMeAU4Z9SHJtmbZC7J3NGjR9foUCRJcPJcbB91JlHL1Jfr8/pi1e1VtaOqdmza5A87StJamnSQvNCmq2ivL7b6YeDCoXZbgOdbfcuI+gl9kmwEzuT1U2mSpDGbdJA8AOxp63uATw7Vd7c7sS5icFH90Tb99WqSy9v1j+sW9VnY1zXAZ9p1FEnSBI3tN0WSfBR4O3BuksPA+4HfBA4kuR74KnAtQFUdSnKAwUMhjwE3VtXxtqsbGNwBdjrwYFsA7gDuSTLP4Exk97iORZK0tLEFSVW9c4lNVyzRfh8jfuOkquaAS0fUv0ULIknS9JwsF9slSTPKIJEkdTFIJEldDBJJUheDRJLUxSCRJHUxSCRJXQwSSVIXg0SS1MUgkSR1MUgkSV0MEklSl7E9tFHS5H31v//ItIegk9C//G9PjHX/npFIkroYJJKkLgaJJKmLQSJJ6mKQSJK6GCSSpC4GiSSpi0EiSepikEiSuhgkkqQuBokkqYtBIknqYpBIkroYJJKkLgaJJKmLQSJJ6mKQSJK6GCSSpC4GiSSpi0EiSeoylSBJ8pUkTyR5PMlcq52d5KEkX26vZw21vynJfJJnklw5VN/e9jOf5JYkmcbxSNJ6Ns0zkp+oqsuqakd7/z7gYFVtAw629yS5GNgNXALsAm5NsqH1uQ3YC2xry64Jjl+SxMk1tXUVsL+t7weuHqrfW1WvVdWzwDywM8kFwBlV9XBVFXD3UB9J0oRMK0gK+LMkjyXZ22rnV9URgPZ6XqtvBp4b6nu41Ta39cX110myN8lckrmjR4+u4WFIkjZO6XPfVlXPJzkPeCjJXy3TdtR1j1qm/vpi1e3A7QA7duwY2UaStDpTOSOpqufb64vAJ4CdwAttuor2+mJrfhi4cKj7FuD5Vt8yoi5JmqCJB0mS70vy/QvrwL8HngQeAPa0ZnuAT7b1B4DdSU5LchGDi+qPtumvV5Nc3u7Wum6ojyRpQqYxtXU+8Il2p+5G4I+r6k+SfBE4kOR64KvAtQBVdSjJAeAp4BhwY1Udb/u6AbgLOB14sC2SpAmaeJBU1V8D/2pE/e+AK5bosw/YN6I+B1y61mOUJK3cyXT7ryRpBhkkkqQuBokkqYtBIknqYpBIkroYJJKkLgaJJKmLQSJJ6mKQSJK6GCSSpC4GiSSpi0EiSepikEiSuhgkkqQuBokkqYtBIknqYpBIkroYJJKkLgaJJKmLQSJJ6mKQSJK6GCSSpC4GiSSpi0EiSepikEiSuhgkkqQuBokkqYtBIknqYpBIkroYJJKkLgaJJKmLQSJJ6mKQSJK6zHyQJNmV5Jkk80neN+3xSNJ6M9NBkmQD8LvATwIXA+9McvF0RyVJ68tMBwmwE5ivqr+uqn8A7gWumvKYJGld2TjtAXTaDDw39P4w8G8WN0qyF9jb3n4zyTMTGNt6cS7w9WkP4mSQ39oz7SHoRP5tLnh/1mIvP7TUhlkPklH/OvW6QtXtwO3jH876k2SuqnZMexzSYv5tTs6sT20dBi4cer8FeH5KY5GkdWnWg+SLwLYkFyV5A7AbeGDKY5KkdWWmp7aq6liS9wB/CmwA7qyqQ1Me1nrjlKFOVv5tTkiqXndJQZKkFZv1qS1J0pQZJJKkLgaJVsVH0+hkleTOJC8meXLaY1kvDBJ913w0jU5ydwG7pj2I9cQg0Wr4aBqdtKrq88BL0x7HemKQaDVGPZpm85TGImnKDBKtxooeTSNpfTBItBo+mkbSPzJItBo+mkbSPzJI9F2rqmPAwqNpngYO+GganSySfBR4GPjhJIeTXD/tMf1z5yNSJEldPCORJHUxSCRJXQwSSVIXg0SS1MUgkSR1MUikMUrylSTntvW/WEH73194AGaSby7R5q4k16ztSKXVm+mf2pVmSVX9+Ara/MokxiKtJc9IpDWS5F1JHk3yeJL/2R63P7z9m+31lCS3JjmU5FNJPr1whpHkz5PsGOrzoSRfSnIwyaYRn7k9yeeSPJbkT5NcMO7jlBYzSKQ1kOQtwM8Db6uqy4DjwH9covk7gK3AjwC/AvzYEu2+D/hSVf1r4HPA+xd95qnAR4Brqmo7cCewr+tApFVwaktaG1cA24EvJgE4HXhxibb/FvhfVfVt4GtJPrtEu28D97X1PwQ+vmj7DwOXAg+1z9wAHFntAUirZZBIayPA/qq66YRi8p+WaLsai59nFOBQVS11RiNNhFNb0to4CFyT5DyAJGcn+aEl2n4B+Nl2reR84O1LtDsFWLg76xdav2HPAJuS/Fj7zFOTXNJxDNKqeEYirYGqeirJfwX+LMkpwP8Dblyi+ccYTIU9Cfxf4BHglRHt/h64JMljbfvPL/rMf2gX6W9JciaD/8+/DfgkZk2UT/+VpiDJG6vqm0nOAR5lcJH+a9Mel7QanpFI0/GpJD8AvAH4DUNEs8wzEklSFy+2S5K6GCSSpC4GiSSpi0EiSepikEiSuvx/Z46r/Gd5H6kAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h7PCOjB2Ob-m"
      },
      "source": [
        "df_clean.sample(10)\n",
        "\n",
        "## fnlwgt column name is mysterious and vague! It can be removed\n",
        "df_clean.drop('fnlwgt', axis=1, inplace=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1mfnFTK0Ob-m",
        "outputId": "114d2ea8-a136-47f9-8664-1573b6827cde"
      },
      "source": [
        "df_clean.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 32560 entries, 0 to 32559\n",
            "Data columns (total 14 columns):\n",
            " #   Column          Non-Null Count  Dtype \n",
            "---  ------          --------------  ----- \n",
            " 0   age             32560 non-null  int64 \n",
            " 1   workclass       30724 non-null  object\n",
            " 2   education       32560 non-null  object\n",
            " 3   education-num   32560 non-null  int64 \n",
            " 4   marital-status  32560 non-null  object\n",
            " 5   occupation      30717 non-null  object\n",
            " 6   relationship    32560 non-null  object\n",
            " 7   race            32560 non-null  object\n",
            " 8   sex             32560 non-null  object\n",
            " 9   capital-gain    32560 non-null  int64 \n",
            " 10  capital-loss    32560 non-null  int64 \n",
            " 11  hours-per-week  32560 non-null  int64 \n",
            " 12  native-country  31977 non-null  object\n",
            " 13  eligible        32560 non-null  int64 \n",
            "dtypes: int64(6), object(8)\n",
            "memory usage: 3.5+ MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gsUF-TziOb-n"
      },
      "source": [
        "categorical = ['workclass', 'education', 'marital-status', 'occupation', \n",
        "                   'relationship', 'race', 'sex', 'native-country']\n",
        "numerical = ['age', 'education-num','capital-gain', 'capital-loss',\n",
        "                'hours-per-week']\n",
        " \n",
        "X = pd.concat([df_clean[categorical], df_clean[numerical]], axis=1) ## Just sort them, no need for encoding\n",
        "y = df_clean['eligible']\n",
        "X_train , X_test , y_train , y_test = train_test_split(X,y,test_size = 0.2 , random_state = 5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jgrq9C3-Ob-n"
      },
      "source": [
        "#### Convert the dataframes to numpy arrays as our implementation works on NumPy arrays\n",
        "X_train , X_test , y_train , y_test = X_train.to_numpy() , X_test.to_numpy() , y_train.to_numpy() , y_test.to_numpy()\n",
        "max_depth = 7\n",
        "inital_depth = 0\n",
        "id3_dt = MyDecisionTree(max_depth)\n",
        "# Building a tree\n",
        "print(\"fitting the decision tree\")\n",
        "id3_dt.fit(X_train, y_train, inital_depth)\n",
        "\n",
        "# Evaluating the decision tree\n",
        "DecisionTreeEvalution(id3_dt,X_test,y_test, True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aiZ2Pft9Ob-n"
      },
      "source": [
        "id3_dt.tree"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XVzopxcqOb-o"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JUNePDvLOb-o"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ASsyU7JQOb-o"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w1H_13-lOb-o"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J-ZwF88TaQxD"
      },
      "source": [
        "## Task 2 : Work with another classification dataset from your choice\n",
        "- **[ 2 points ] :** Do the same steps we did in Task 1 but with a classification dataset you choose. No minimum accuracy required.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KvoWFpSQaUy7"
      },
      "source": [
        "\n",
        "\n",
        "## Bonus Task : Work with another *regression* dataset from your choice \n",
        "- **[ 3 points ] :** Use sklearn DecisionTree regressor on a regression dataset from your choice. It needs to be a one that we did not use in the course so far. You need to perform all the necessary preprocessing/visualization and justify it."
      ]
    }
  ]
}