Kaggle_loan_default_prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eh0Zhz3xeLaq"
   },
   "source": [
    "# **Validation of Dataset for Defaulter Prediction Based on Customer Behavior**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JPwSVU0UeXZO",
    "tags": []
   },
   "source": [
    "### **Introduction**\n",
    "\n",
    "Given a dataset of the past consumer loans, the potential defaulters should be predicted. The dataset is from Kaggle \n",
    "\n",
    "### **Data Source**\n",
    "\n",
    "https://www.kaggle.com/datasets/subhamjain/loan-prediction-based-on-customer-behavior\n",
    "\n",
    "### **Data Format**\n",
    "\n",
    "* ID: Id of the loanees\n",
    "* Income: Income of the loanees\n",
    "* Age: Age of the loanees\n",
    "* Experience: Professional experience (in years)\n",
    "* Profession: Profession of the loanees \n",
    "* Married/Single: Married or not\n",
    "* House_Ownership: Owned or rented or neither\n",
    "* Car_Ownership: Does loanee own a car or not\n",
    "* STATE: State of residence\n",
    "* CITY: City of residence\n",
    "* CURRENT_JOB_YRS: time with the current job (in years)\n",
    "* CURRENT_HOUSE_YRS: time with the current house (in years)\n",
    "* Risk_Flag: Defaulted on a loan (Target variable) (1 for being a defaulter, and 0 for not)\n",
    "\n",
    "### **Training and Validation**\n",
    "\n",
    "The zip file contains:\n",
    "* Training Data.csv: training data with data format as above\n",
    "* Test Data.csv: test data with the same predictor variables (without Risk_flag colummn)\n",
    "* Sample Prediction Dataset.csv: test data labels (only Risk_flag column)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RJMO-yyHku2b",
    "tags": []
   },
   "source": [
    "## **Importing libraries**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "5xlWt7Sze3Gx",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import precision_score\n",
    "from sklearn.metrics import recall_score\n",
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "from imblearn.over_sampling import RandomOverSampler\n",
    "from imblearn.over_sampling import SMOTENC\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sLzxLAbHm8xR"
   },
   "source": [
    "## **Loading and preprocessing the datasets**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-GD5bYX-szTi"
   },
   "source": [
    "### **Training data inspection**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 270
    },
    "id": "G0mFBvQbwGTf",
    "outputId": "bff008de-7804-46d4-c82a-7c4334c09882",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Income</th>\n",
       "      <th>Age</th>\n",
       "      <th>Experience</th>\n",
       "      <th>Married/Single</th>\n",
       "      <th>House_Ownership</th>\n",
       "      <th>Car_Ownership</th>\n",
       "      <th>Profession</th>\n",
       "      <th>CITY</th>\n",
       "      <th>STATE</th>\n",
       "      <th>CURRENT_JOB_YRS</th>\n",
       "      <th>CURRENT_HOUSE_YRS</th>\n",
       "      <th>Risk_Flag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1303834</td>\n",
       "      <td>23</td>\n",
       "      <td>3</td>\n",
       "      <td>single</td>\n",
       "      <td>rented</td>\n",
       "      <td>no</td>\n",
       "      <td>Mechanical_engineer</td>\n",
       "      <td>Rewa</td>\n",
       "      <td>Madhya_Pradesh</td>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>7574516</td>\n",
       "      <td>40</td>\n",
       "      <td>10</td>\n",
       "      <td>single</td>\n",
       "      <td>rented</td>\n",
       "      <td>no</td>\n",
       "      <td>Software_Developer</td>\n",
       "      <td>Parbhani</td>\n",
       "      <td>Maharashtra</td>\n",
       "      <td>9</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>3991815</td>\n",
       "      <td>66</td>\n",
       "      <td>4</td>\n",
       "      <td>married</td>\n",
       "      <td>rented</td>\n",
       "      <td>no</td>\n",
       "      <td>Technical_writer</td>\n",
       "      <td>Alappuzha</td>\n",
       "      <td>Kerala</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>6256451</td>\n",
       "      <td>41</td>\n",
       "      <td>2</td>\n",
       "      <td>single</td>\n",
       "      <td>rented</td>\n",
       "      <td>yes</td>\n",
       "      <td>Software_Developer</td>\n",
       "      <td>Bhubaneswar</td>\n",
       "      <td>Odisha</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>5768871</td>\n",
       "      <td>47</td>\n",
       "      <td>11</td>\n",
       "      <td>single</td>\n",
       "      <td>rented</td>\n",
       "      <td>no</td>\n",
       "      <td>Civil_servant</td>\n",
       "      <td>Tiruchirappalli[10]</td>\n",
       "      <td>Tamil_Nadu</td>\n",
       "      <td>3</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id   Income  Age  Experience Married/Single House_Ownership Car_Ownership  \\\n",
       "0   1  1303834   23           3         single          rented            no   \n",
       "1   2  7574516   40          10         single          rented            no   \n",
       "2   3  3991815   66           4        married          rented            no   \n",
       "3   4  6256451   41           2         single          rented           yes   \n",
       "4   5  5768871   47          11         single          rented            no   \n",
       "\n",
       "            Profession                 CITY           STATE  CURRENT_JOB_YRS  \\\n",
       "0  Mechanical_engineer                 Rewa  Madhya_Pradesh                3   \n",
       "1   Software_Developer             Parbhani     Maharashtra                9   \n",
       "2     Technical_writer            Alappuzha          Kerala                4   \n",
       "3   Software_Developer          Bhubaneswar          Odisha                2   \n",
       "4        Civil_servant  Tiruchirappalli[10]      Tamil_Nadu                3   \n",
       "\n",
       "   CURRENT_HOUSE_YRS  Risk_Flag  \n",
       "0                 13          0  \n",
       "1                 13          0  \n",
       "2                 10          0  \n",
       "3                 12          1  \n",
       "4                 14          1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data = pd.read_csv('Training Data.csv', on_bad_lines='warn') # make sure all the csv files are in the working directory\n",
    "train_data.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0f_nKEhLqGP8",
    "outputId": "37e80607-2695-41c3-dd3a-9d8ced637a23",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training set size: 252000 data points with 13 columns\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Id                    int64\n",
       "Income                int64\n",
       "Age                   int64\n",
       "Experience            int64\n",
       "Married/Single       object\n",
       "House_Ownership      object\n",
       "Car_Ownership        object\n",
       "Profession           object\n",
       "CITY                 object\n",
       "STATE                object\n",
       "CURRENT_JOB_YRS       int64\n",
       "CURRENT_HOUSE_YRS     int64\n",
       "Risk_Flag             int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('training set size: {} data points with {} columns\\n'.format(train_data.shape[0], train_data.shape[1]))\n",
    "train_data.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6X_mawa2sQno",
    "outputId": "97afe868-500d-4f55-8ee8-925888588c1d",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                   False\n",
       "Income               False\n",
       "Age                  False\n",
       "Experience           False\n",
       "Married/Single       False\n",
       "House_Ownership      False\n",
       "Car_Ownership        False\n",
       "Profession           False\n",
       "CITY                 False\n",
       "STATE                False\n",
       "CURRENT_JOB_YRS      False\n",
       "CURRENT_HOUSE_YRS    False\n",
       "Risk_Flag            False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking for NaN values\n",
    "train_data.isnull().any()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LPhRCydOteDS",
    "outputId": "aca0d819-1321-4217-a4f4-1c2656115d93",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Married/Single: \n",
      " ['single' 'married']\n",
      "House_Ownership: \n",
      " ['rented' 'norent_noown' 'owned']\n",
      "Car_Ownership: \n",
      " ['no' 'yes']\n",
      "Profession: \n",
      " ['Mechanical_engineer' 'Software_Developer' 'Technical_writer'\n",
      " 'Civil_servant' 'Librarian' 'Economist' 'Flight_attendant' 'Architect'\n",
      " 'Designer' 'Physician' 'Financial_Analyst' 'Air_traffic_controller'\n",
      " 'Politician' 'Police_officer' 'Artist' 'Surveyor' 'Design_Engineer'\n",
      " 'Chemical_engineer' 'Hotel_Manager' 'Dentist' 'Comedian'\n",
      " 'Biomedical_Engineer' 'Graphic_Designer' 'Computer_hardware_engineer'\n",
      " 'Petroleum_Engineer' 'Secretary' 'Computer_operator'\n",
      " 'Chartered_Accountant' 'Technician' 'Microbiologist' 'Fashion_Designer'\n",
      " 'Aviator' 'Psychologist' 'Magistrate' 'Lawyer' 'Firefighter' 'Engineer'\n",
      " 'Official' 'Analyst' 'Geologist' 'Drafter' 'Statistician' 'Web_designer'\n",
      " 'Consultant' 'Chef' 'Army_officer' 'Surgeon' 'Scientist' 'Civil_engineer'\n",
      " 'Industrial_Engineer' 'Technology_specialist']\n",
      "CITY: \n",
      " ['Rewa' 'Parbhani' 'Alappuzha' 'Bhubaneswar' 'Tiruchirappalli[10]'\n",
      " 'Jalgaon' 'Tiruppur' 'Jamnagar' 'Kota[6]' 'Karimnagar' 'Hajipur[31]'\n",
      " 'Adoni' 'Erode[17]' 'Kollam' 'Madurai' 'Anantapuram[24]' 'Kamarhati'\n",
      " 'Bhusawal' 'Sirsa' 'Amaravati' 'Secunderabad' 'Ahmedabad' 'Ajmer'\n",
      " 'Ongole' 'Miryalaguda' 'Ambattur' 'Indore' 'Pondicherry' 'Shimoga'\n",
      " 'Chennai' 'Gulbarga' 'Khammam' 'Saharanpur' 'Gopalpur' 'Amravati' 'Udupi'\n",
      " 'Howrah' 'Aurangabad[39]' 'Hospet' 'Shimla' 'Khandwa' 'Bidhannagar'\n",
      " 'Bellary' 'Danapur' 'Purnia[26]' 'Bijapur' 'Patiala' 'Malda' 'Sagar'\n",
      " 'Durgapur' 'Junagadh' 'Singrauli' 'Agartala' 'Thanjavur' 'Hindupur'\n",
      " 'Naihati' 'North_Dumdum' 'Panchkula' 'Anantapur' 'Serampore' 'Bathinda'\n",
      " 'Nadiad' 'Kanpur' 'Haridwar' 'Berhampur' 'Jamshedpur' 'Hyderabad' 'Bidar'\n",
      " 'Kottayam' 'Solapur' 'Suryapet' 'Aizawl' 'Asansol' 'Deoghar' 'Eluru[25]'\n",
      " 'Ulhasnagar' 'Aligarh' 'South_Dumdum' 'Berhampore' 'Gandhinagar'\n",
      " 'Sonipat' 'Muzaffarpur' 'Raichur' 'Rajpur_Sonarpur' 'Ambarnath' 'Katihar'\n",
      " 'Kozhikode' 'Vellore' 'Malegaon' 'Kochi' 'Nagaon' 'Nagpur' 'Srinagar'\n",
      " 'Davanagere' 'Bhagalpur' 'Siwan[32]' 'Meerut' 'Dindigul' 'Bhatpara'\n",
      " 'Ghaziabad' 'Kulti' 'Chapra' 'Dibrugarh' 'Panihati' 'Bhiwandi' 'Morbi'\n",
      " 'Kalyan-Dombivli' 'Gorakhpur' 'Panvel' 'Siliguri' 'Bongaigaon' 'Patna'\n",
      " 'Ramgarh' 'Ozhukarai' 'Mirzapur' 'Akola' 'Satna' 'Motihari[34]' 'Jalna'\n",
      " 'Jalandhar' 'Unnao' 'Karnal' 'Cuttack' 'Proddatur' 'Ichalkaranji'\n",
      " 'Warangal[11][12]' 'Jhansi' 'Bulandshahr' 'Narasaraopet' 'Chinsurah'\n",
      " 'Jehanabad[38]' 'Dhanbad' 'Gudivada' 'Gandhidham' 'Raiganj'\n",
      " 'Kishanganj[35]' 'Varanasi' 'Belgaum' 'Tirupati[21][22]' 'Tumkur'\n",
      " 'Coimbatore' 'Kurnool[18]' 'Gurgaon' 'Muzaffarnagar' 'Aurangabad'\n",
      " 'Bhavnagar' 'Arrah' 'Munger' 'Tirunelveli' 'Mumbai' 'Mango' 'Nashik'\n",
      " 'Kadapa[23]' 'Amritsar' 'Khora,_Ghaziabad' 'Ambala' 'Agra' 'Ratlam'\n",
      " 'Surendranagar_Dudhrej' 'Delhi_city' 'Bhopal' 'Hapur' 'Rohtak' 'Durg'\n",
      " 'Korba' 'Bangalore' 'Shivpuri' 'Thrissur' 'Vijayanagaram' 'Farrukhabad'\n",
      " 'Nangloi_Jat' 'Madanapalle' 'Thoothukudi' 'Nagercoil' 'Gaya'\n",
      " 'Chandigarh_city' 'Jammu[16]' 'Kakinada' 'Dewas' 'Bhalswa_Jahangir_Pur'\n",
      " 'Baranagar' 'Firozabad' 'Phusro' 'Allahabad' 'Guna' 'Thane' 'Etawah'\n",
      " 'Vasai-Virar' 'Pallavaram' 'Morena' 'Ballia' 'Surat' 'Burhanpur'\n",
      " 'Phagwara' 'Mau' 'Mangalore' 'Alwar' 'Mahbubnagar' 'Maheshtala'\n",
      " 'Hazaribagh' 'Bihar_Sharif' 'Faridabad' 'Lucknow' 'Tenali' 'Barasat'\n",
      " 'Amroha' 'Giridih' 'Begusarai' 'Medininagar' 'Rajahmundry[19][20]'\n",
      " 'Saharsa[29]' 'New_Delhi' 'Bhilai' 'Moradabad' 'Machilipatnam'\n",
      " 'Mira-Bhayandar' 'Pali' 'Navi_Mumbai' 'Mehsana' 'Imphal' 'Kolkata'\n",
      " 'Sambalpur' 'Ujjain' 'Madhyamgram' 'Jabalpur' 'Jamalpur[36]' 'Ludhiana'\n",
      " 'Bareilly' 'Gangtok' 'Anand' 'Dehradun' 'Pune' 'Satara' 'Srikakulam'\n",
      " 'Raipur' 'Jodhpur' 'Darbhanga' 'Nizamabad' 'Nandyal' 'Dehri[30]' 'Jorhat'\n",
      " 'Ranchi' 'Kumbakonam' 'Guntakal' 'Haldia' 'Loni' 'Pimpri-Chinchwad'\n",
      " 'Rajkot' 'Nanded' 'Noida' 'Kirari_Suleman_Nagar' 'Jaunpur' 'Bilaspur'\n",
      " 'Sambhal' 'Dhule' 'Rourkela' 'Thiruvananthapuram' 'Dharmavaram'\n",
      " 'Nellore[14][15]' 'Visakhapatnam[4]' 'Karawal_Nagar' 'Jaipur' 'Avadi'\n",
      " 'Bhimavaram' 'Bardhaman' 'Silchar' 'Buxar[37]' 'Kavali' 'Tezpur'\n",
      " 'Ramagundam[27]' 'Yamunanagar' 'Sri_Ganganagar' 'Sasaram[30]' 'Sikar'\n",
      " 'Bally' 'Bhiwani' 'Rampur' 'Uluberia' 'Sangli-Miraj_&_Kupwad' 'Hosur'\n",
      " 'Bikaner' 'Shahjahanpur' 'Sultan_Pur_Majra' 'Vijayawada' 'Bharatpur'\n",
      " 'Tadepalligudem' 'Tinsukia' 'Salem' 'Mathura' 'Guntur[13]'\n",
      " 'Hubliâ€“Dharwad' 'Guwahati' 'Chittoor[28]' 'Tiruvottiyur' 'Vadodara'\n",
      " 'Ahmednagar' 'Fatehpur' 'Bhilwara' 'Kharagpur' 'Bettiah[33]' 'Bhind'\n",
      " 'Bokaro' 'Karaikudi' 'Raebareli' 'Pudukkottai' 'Udaipur'\n",
      " 'Mysore[7][8][9]' 'Panipat' 'Latur' 'Tadipatri' 'Bahraich' 'Orai'\n",
      " 'Raurkela_Industrial_Township' 'Gwalior' 'Katni' 'Chandrapur' 'Kolhapur']\n",
      "STATE: \n",
      " ['Madhya_Pradesh' 'Maharashtra' 'Kerala' 'Odisha' 'Tamil_Nadu' 'Gujarat'\n",
      " 'Rajasthan' 'Telangana' 'Bihar' 'Andhra_Pradesh' 'West_Bengal' 'Haryana'\n",
      " 'Puducherry' 'Karnataka' 'Uttar_Pradesh' 'Himachal_Pradesh' 'Punjab'\n",
      " 'Tripura' 'Uttarakhand' 'Jharkhand' 'Mizoram' 'Assam' 'Jammu_and_Kashmir'\n",
      " 'Delhi' 'Chhattisgarh' 'Chandigarh' 'Uttar_Pradesh[5]' 'Manipur' 'Sikkim']\n"
     ]
    }
   ],
   "source": [
    "columns = ['Married/Single', 'House_Ownership', 'Car_Ownership', 'Profession', 'CITY', 'STATE']\n",
    "for column in columns:\n",
    "    print('{}: \\n {}'.format(column,train_data[column].unique()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "HS7oRlYUuMh9",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# cleaning the categorical vars\n",
    "def cleaner(text):\n",
    "    text = text.lower()\n",
    "    text = ' '.join(re.findall(r'[a-zA-Z]+',text))\n",
    "    return text\n",
    "\n",
    "\n",
    "for i in train_data.select_dtypes(include='object').columns:\n",
    "    train_data[i] =train_data[i].apply(cleaner)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cPs0Xg1RulFM",
    "outputId": "4081e353-c711-4946-ae14-2942edb1ab29",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(43190, 12)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data = train_data.drop('Id', axis=1).drop_duplicates(keep='first')\n",
    "train_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HA92b6gzuw1a",
    "outputId": "caffef54-cb0e-4952-b0b1-2b40318665db",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(40824, 12)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vars = ['Income', 'Age', 'Experience', 'Married/Single', 'House_Ownership', 'Car_Ownership', 'Profession', 'CITY', 'STATE', 'CURRENT_JOB_YRS', 'CURRENT_HOUSE_YRS']\n",
    "train_data = train_data[~train_data.duplicated(subset=vars, keep=False)]\n",
    "train_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S8_uXYtvu-7r"
   },
   "source": [
    "### **Remarks**\n",
    "\n",
    "1. **Duplicate**: The original training data has 252000 observations (shape: (252000,13)) but after removing all duplicates, only 43190 data points remain.\n",
    "2. **Random labeling**: The remaining data points consist cases where all predictor variables are exactly the same but with different label (Risk_flag)\n",
    "\n",
    "With those observations, the integrity of the training data is highly questionable\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YR-Iptwwyu7e"
   },
   "source": [
    "### **Test data inspection**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 305
    },
    "id": "vL5lUejXs_gf",
    "outputId": "f1aadeca-83bb-448e-b563-2432636241a5",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Income</th>\n",
       "      <th>Age</th>\n",
       "      <th>Experience</th>\n",
       "      <th>Married/Single</th>\n",
       "      <th>House_Ownership</th>\n",
       "      <th>Car_Ownership</th>\n",
       "      <th>Profession</th>\n",
       "      <th>CITY</th>\n",
       "      <th>STATE</th>\n",
       "      <th>CURRENT_JOB_YRS</th>\n",
       "      <th>CURRENT_HOUSE_YRS</th>\n",
       "      <th>risk_flag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>7393090</td>\n",
       "      <td>59</td>\n",
       "      <td>19</td>\n",
       "      <td>single</td>\n",
       "      <td>rented</td>\n",
       "      <td>no</td>\n",
       "      <td>Geologist</td>\n",
       "      <td>Malda</td>\n",
       "      <td>West Bengal</td>\n",
       "      <td>4</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1215004</td>\n",
       "      <td>25</td>\n",
       "      <td>5</td>\n",
       "      <td>single</td>\n",
       "      <td>rented</td>\n",
       "      <td>no</td>\n",
       "      <td>Firefighter</td>\n",
       "      <td>Jalna</td>\n",
       "      <td>Maharashtra</td>\n",
       "      <td>5</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>8901342</td>\n",
       "      <td>50</td>\n",
       "      <td>12</td>\n",
       "      <td>single</td>\n",
       "      <td>rented</td>\n",
       "      <td>no</td>\n",
       "      <td>Lawyer</td>\n",
       "      <td>Thane</td>\n",
       "      <td>Maharashtra</td>\n",
       "      <td>9</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1944421</td>\n",
       "      <td>49</td>\n",
       "      <td>9</td>\n",
       "      <td>married</td>\n",
       "      <td>rented</td>\n",
       "      <td>yes</td>\n",
       "      <td>Analyst</td>\n",
       "      <td>Latur</td>\n",
       "      <td>Maharashtra</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>13429</td>\n",
       "      <td>25</td>\n",
       "      <td>18</td>\n",
       "      <td>single</td>\n",
       "      <td>rented</td>\n",
       "      <td>yes</td>\n",
       "      <td>Comedian</td>\n",
       "      <td>Berhampore</td>\n",
       "      <td>West Bengal</td>\n",
       "      <td>13</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   ID   Income  Age  Experience Married/Single House_Ownership Car_Ownership  \\\n",
       "0   1  7393090   59          19         single          rented            no   \n",
       "1   2  1215004   25           5         single          rented            no   \n",
       "2   3  8901342   50          12         single          rented            no   \n",
       "3   4  1944421   49           9        married          rented           yes   \n",
       "4   5    13429   25          18         single          rented           yes   \n",
       "\n",
       "    Profession        CITY        STATE  CURRENT_JOB_YRS  CURRENT_HOUSE_YRS  \\\n",
       "0    Geologist       Malda  West Bengal                4                 13   \n",
       "1  Firefighter       Jalna  Maharashtra                5                 10   \n",
       "2       Lawyer       Thane  Maharashtra                9                 14   \n",
       "3      Analyst       Latur  Maharashtra                3                 12   \n",
       "4     Comedian  Berhampore  West Bengal               13                 11   \n",
       "\n",
       "   risk_flag  \n",
       "0          0  \n",
       "1          0  \n",
       "2          1  \n",
       "3          0  \n",
       "4          0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test = pd.read_csv('Test Data.csv')\n",
    "y_test = pd.read_csv('Sample Prediction Dataset.csv')\n",
    "\n",
    "# merging predictor vars in X_test and target var in y_test to form the complete test set for further analysis\n",
    "y_test = y_test.rename(columns={\"id\": \"ID\"})\n",
    "test_data = X_test.merge(y_test, on='ID', how='inner')\n",
    "test_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kGNLMihNy6ee",
    "outputId": "288ee6d1-e3d0-4193-8a08-323c5ed8f54b",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(28000, 13)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-DlL6UoZzLnn",
    "outputId": "f987493d-cb9a-48ef-c951-84d6cd951ddf",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ID                   False\n",
       "Income               False\n",
       "Age                  False\n",
       "Experience           False\n",
       "Married/Single       False\n",
       "House_Ownership      False\n",
       "Car_Ownership        False\n",
       "Profession           False\n",
       "CITY                 False\n",
       "STATE                False\n",
       "CURRENT_JOB_YRS      False\n",
       "CURRENT_HOUSE_YRS    False\n",
       "risk_flag            False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking for NaN values\n",
    "test_data.isnull().any()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "l1gf2USNzA_q",
    "tags": []
   },
   "outputs": [],
   "source": [
    "for i in test_data.select_dtypes(include='object').columns:\n",
    "    test_data[i] = test_data[i].apply(cleaner)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9mZ4zOVIzWX-",
    "outputId": "52c0c7bc-deb0-48da-98f5-5ec5d22d8363",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(12260, 12)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data = test_data.drop('ID', axis=1).drop_duplicates(keep='first')\n",
    "test_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wbUvcrRIzoyN",
    "outputId": "8bed618b-ce22-48c1-d9a9-e9cf83604357",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7584, 12)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vars = ['Income', 'Age', 'Experience', 'Married/Single', 'House_Ownership', 'Car_Ownership', 'Profession', 'CITY', 'STATE', 'CURRENT_JOB_YRS', 'CURRENT_HOUSE_YRS']\n",
    "test_data = test_data[~test_data.duplicated(subset=vars, keep=False)]\n",
    "test_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZzDVBxCkzxWF"
   },
   "source": [
    "### **Remarks**\n",
    "\n",
    "Duplicated data points and random labeling are also encountered with test data\\\n",
    "Nonetheless, lets see how far we can go with the predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "velZY8_5E4GJ"
   },
   "source": [
    "### **One-hot Encoding categorical variables**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "d8ZK70KAD_iU",
    "outputId": "8586f9fc-b007-4ccb-db3e-aadfd900a910",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((40824, 11), (40824,))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = train_data.drop(columns=['Risk_Flag'])\n",
    "X_oh = pd.get_dummies(X)\n",
    "y = train_data['Risk_Flag']\n",
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P5equ08zMya3"
   },
   "source": [
    "We will use Random Forest Classifier for training and prediction, therefore, rescaling the data is not necessary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YjIWm1G831Dl"
   },
   "source": [
    "## **Defaulter Prediction with Random Forest**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "j35DJA8xEb4G",
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X_oh, y, train_size=0.8, test_size=0.2)  #Split training data to 8:2 proportion for first time validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Ambz82g2Eb1C",
    "outputId": "cfae4c69-fec1-4713-f27f-760506d0a632",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training_acc: 0.8219228413962033\n",
      "f1 score: 0.0013736263736263737\n"
     ]
    }
   ],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=250)\n",
    "rf_model.fit(X_train, y_train)\n",
    "acc_train = accuracy_score(y_valid, rf_model.predict(X_valid))\n",
    "f1 = f1_score(y_valid, rf_model.predict(X_valid))\n",
    "print('training_acc: {}'.format(acc_train))\n",
    "print('f1 score: {}'.format(f1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4TEfnFWqGsOJ"
   },
   "source": [
    "### **Remarks**\n",
    "* The overall acc is about 82%, and with a little tweaking, it can be further increased\n",
    "* However, f1-score is 0%. This signifies that the model performs extremely poor when predicting positive cases, which is more important is case of a defaulter prediction task.\n",
    "* The reason is unbalanced data, i.e., there is significantly less positive-labeled data points to learn from."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Rywjx7ANJ1Kt"
   },
   "source": [
    "### **Dealing with unbalanced data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 406
    },
    "id": "IybSwZZii9Si",
    "outputId": "2558f3c2-3ad2-44d9-8d17-f6bf0b9ffe81",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcoAAAGFCAYAAAB9krNlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA3t0lEQVR4nO3dd3hUVcIG8PfOJDOT3nsCSSAh1FAVpEiVIqzgIuLaoqhrYVf4RFF3QcoKioKCWLAt4K6FFQRWikgV0KUJAUIgEEhPCKT3aff7IxKNkCGBJOfemff3PDyEZHLnDYR5c+4951xJlmUZREREdE0a0QGIiIiUjEVJRERkA4uSiIjIBhYlERGRDSxKIiIiG1iURERENrAoiYiIbGBREhER2cCiJCIisoFFSUREZAOLkoiIyAYWJRERkQ0sSiIiIhtYlERERDawKImIiGxgURIREdnAoiQiIrKBRUlERGQDi5KIiMgGFiUREZENLEoiIiIbWJREREQ2sCiJiIhsYFESERHZwKIkIiKygUVJRERkA4uSiIjIBhYlERGRDSxKIiIiG1iURERENrAoiYiIbGBREhER2cCiJCIisoFFSUREZAOLkoiIyAYWJRERkQ0sSiIiIhtYlERERDawKImIiGxgURIREdnAoiQiIrKBRUlERGQDi5KIiMgGFiUREZENLEoiIiIbnEQHIKKGlVSakFdajbJqE8qqzSitNqG8xoyyajMqa8yoMVt/+WWpe1uWZRictDDotDA4aeGi08DFWQvDb37V/rn2/V6uzgjzdoG3q070l0ukSCxKIoFKKk3ILKpEVlEVsup+r307u7gKZdXmVsviptMi1Nul7le4jwtCvQ0I9XJBmI8Lgj0NcNLyJBQ5HkmWZVl0CCJ7V1RhxLGsYhzPLEFSTgkyClu/CG+WViMhyEOPdoHu6Bbuha5hXuga7o0wbxfR0YhaFIuSqJlV1JhxIrsEx7OKkZhV+3tmYZXoWC3Gz02HLmFXirP291CWJ9kRFiXRTUrOLcXhtMK6UjyXXw6rg/+v8nevLc9u4d7oF+2H3pE+cOZpW1IpFiVRE1UZLdh37jJ2ns7H7jP5yC2pFh1J8dx0Wtwa7YeBMf4YGOOP9oEeoiMRNRqLkqgRMgsrsfN0Pnaczsf/zhfAaLaKjqRqYd4uGBoXiGEdA9GvnR/0TlrRkYgaxKIkugazxYpDaUXYdSYfO0/n41x+uehIdstNp8XAmAAM6xiIEZ2CuEyFFIdFSfQbh9IKsfZIFjafyEWpimak2gudVoPhnQIxsVc4bo8NhFYjiY5ExKIkyiioxNqfs/DN0WxkFFaKjkO/CPTQY0KPMNzTO5zXNEkoFiU5pGqTBVtO5uKLg5k4lFYI/i9QtvgIb0zsFY4/xIfCy8VZdBxyMCxKcijn8svw+YFMrDuaheJKk+g41ER6Jw1GdArCxF7hGBQTAA1PzVIrYFGS3ZNlGdtOXcQney/gYFqh6DjUTCJ8XfDEwGjc0zsCBmfOmqWWw6Iku2Wxyvj2eA7e25WKMxfLRMehFuLvrsejAyLxYN+28DDwtCw1PxYl2R2j2Yp1P2fhgz2pSCvg5BxH4WFwwgN922LKgCj4u+tFxyE7wqIku1FtsuCLgxn48Ifz3C3HgemdNJjUOwJPDIpGhK+r6DhkB1iUpHpl1Sas/ikdn+67gIIKo+g4pBBOGgnj4kPx1OB2iA3i8hK6cSxKUq2KGjNW/HAeK/df4OYA1CBJAsZ2C8VLo+N4VxO6ISxKUh1ZlrH+WDZe23IaF0trRMchlTA4a/DEoHZ46vZ2cNFxliw1HouSVOV4VjHmbEzCzxnFoqOQSoV4GTBzVBzu6h4KSeI6TLo+FiWpwuXyGryx9Qz+cyTT4e/1SM2jZxtvvDKuM+IjvEVHIYVjUZKimSxWrPoxDUt3nEUZr0NSM5MkYEKPMMwcFYcgT4PoOKRQLEpSrN1n8jH/21NIvVQhOgrZOVedFk8PbofHBkZzlx+6CouSFCenuAqzN5zE9uR80VHIwbT1c8UbE+NxS5Sv6CikICxKUpQNx7Ixa/1JLvcgYSQJSLgtEi+MjOPsWALAoiSFKKk04e8bTuK/iTmioxABACL9XPHGPfHoE8nRpaNjUZJw+85exvNfJ3LbOVIcjQQ80j8KL4zqAL0TR5eOikVJwlSbLHhty2ms+imNN04mRYsL9sDSyT3QIZhb4TkiFiUJcTK7BNO/Ooaz+eWioxA1it5Jg5mj4vBI/0huVOBgWJTUqqxWGe/vScXb21NgsvBbj9RnYIw/Fk+KR6AH1106ChYltZr80mpM/fwoDqYVio5CdFOCPPVY8WBvdOeuPg6BRUmt4ueMIjz52RHkl3ETc7IPOicNFk7oij/2ChcdhVoYi5Ja3JcHMzB7QxKMFqvoKETNbsqAKLw8piO0Gl63tFcsSmoxJosVczYm4d8HMkRHIWpRA2P8sfy+nvBydRYdhVoAi5JaRFGFEX/+7AivR5LDaOvnio8f6o2YIC4hsTcsSmp25/LLMWXVIaQXVIqOQtSq3PVOeOve7hjRKUh0FGpGLEpqVvvPXcZT/zrCvVrJYUkSMH14LP4ytD3XW9oJFiU1m88PZGD2hpMw887KRBgXH4olk+LhrNWIjkI3iUVJzeKdHWex+PsU0TGIFGVYXCDee6An94lVORYl3bTF287gnZ3nRMcgUqQB7f3x0UO9ecsuFWNR0k1ZuDkZK344LzoGkaL1ifTBpwl94GHg8hE1YlHSDZFlGXP/eworf0wTHYVIFeLDvbDq0Vvg7aoTHYWaiEVJTSbLMl7+5iS+OMiNBIiaIi7YA/967Fb4u+tFR6EmYFFSk1itMp7/+jjW/pwlOgqRKkUHuOHzx/oi2It3H1ELFiU1mtlixf+tScTGxBzRUYhULcLXBZ8/1hcRvq6io1AjsCipUUwWK/7y+VFsTcoTHYXILoR4GfDlE33R1s9NdBS6Dq6EpeuyWmWWJFEzyy2pxkOfHkRBOW89p3QsSrqued+eYkkStYD0gko8uvIQKo3c8lHJWJRk00c/nOcSEKIWlJhVgqmfH4WFWz8qFouSGrTpeC4WbEkWHYPI7u08nY+/fXNCdAxqAIuSrulQWiGmrzkGTvUiah1fHsrE29u5X7ISsSjpKufyy/H46sMwmq2ioxA5lLe3n8VXh7iRh9KwKKmeS2U1eGTlQRRXmkRHIXJIf/vmJHadzhcdg36DRUl1Ko1mTFl1CJmFVaKjEDkss1XGM5//jMTMYtFR6BcsSgIAWKwypn5+FMezSkRHIXJ4lUYLHl15CJmFlaKjEFiU9IuFm5Oxk6d7iBSjoMKIp//9M2rMFtFRHJ7DFmVeXh5GjBgBNzc3eHt7N9txExISMH78+GY7Xmv4/tRFfLzvgugYRPQ7J7JLMGdjkugYDq9JRZmQkABJkvDaa6/Ve//69eshSVKzBvu9tLQ0SJJU98vDwwOdO3fGM888g7Nnzzb5eG+99RZyc3Nx7NgxpKS03JTswYMHY9q0aS12/JuVXVyFGf9JFB2DiBrwxcFMrDmcKTqGQ2vyiNJgMOD1119HUVFRS+S5ru3btyM3NxeJiYlYsGABkpOTER8fjx07djTpOKmpqejVqxdiYmIQGBjYQmmbj9FobPZjmi1W/PWLoyip4gxXIiWbveEkknI4f0CUJhfl8OHDERwcjIULFzb4mLVr16Jz587Q6/WIjIzE4sWL6308MjISCxYswKOPPgoPDw+0adMGH374YaOe38/PD8HBwYiOjsZdd92F7du349Zbb8WUKVNgsfx6Ln/Dhg3o2bMnDAYDoqOjMXfuXJjN5rrnX7t2LVavXg1JkpCQkAAAWLJkCbp27Qo3NzdERETg6aefRnl5ed0x58yZg+7du9fL8/bbbyMyMvKaWRMSErBnzx4sXbq0biSclpYGADh58iRGjx4Nd3d3BAUF4cEHH8Tly5frPnfw4MGYOnUqpk2bBn9/f4wcObJRfz9N8ea2FBxJF/MDDxE1XrXJiqf//TPKa7gnrAhNLkqtVosFCxbgnXfeQVbW1TfvPXLkCCZNmoTJkyfjxIkTmDNnDmbNmoWVK1fWe9zixYvRu3dvHD16FE8//TSeeuopnDlzpulfgEaDZ599Funp6Thy5AgAYO/evXjooYfw7LPP4tSpU1ixYgVWrlyJV199FQBw6NAhjBo1CpMmTUJubi6WLl1ad6xly5YhKSkJq1atws6dO/HCCy80OdMVS5cuRb9+/fD4448jNzcXubm5iIiIQHFxMYYOHYoePXrg8OHD2Lp1Ky5evIhJkybV+/xVq1ZBp9Nh//79+OCDD244x7XsPpOPFT+kNusxiajlpBdU4uV13OZOhBuazDNhwgR0794dr7zyylUfW7JkCYYNG4ZZs2YhNjYWCQkJmDp1Kt544416jxszZgyefvpptG/fHjNnzoS/vz927dp1Q19EXFwcANSN1ubOnYsXX3wRDz/8MKKjozFixAjMnz8fK1asAAAEBARAr9fDxcUFwcHB8PLyAgBMmzYNQ4YMQWRkJIYOHYp//OMfWLNmzQ1lAgAvLy/odDq4uroiODgYwcHB0Gq1WL58OXr06IEFCxYgLi4OPXr0wKeffopdu3bVu14aExODRYsWoUOHDujQocMN5/i9i6XVeG5NIrenI1KZjYk5vF4pwA3Pen399dexatUqJCfX3zQ7OTkZ/fv3r/e+/v374+zZs/VOjXbr1q3ubUmSEBwcjPz82uUJV05Juru7o3PnztfNcuXe01cmFCUmJmLevHl1x3B3d68b1VVWNrwuafv27Rg2bBjCwsLg4eGBBx98EAUFBTY/50YkJiZi165d9fJdKfvU1F9Heb169WrW5wVq10v+9YujKKho/mueRNTy5mxMwrn88us/kJqN041+4qBBgzBy5Ei89NJLddf4msLZ2bnenyVJgtVau7foxx9/jKqqqms+7lqulHVUVBQAoLy8HHPnzsXdd9991WMNBsM1j5GWloaxY8fiqaeewquvvgpfX1/s27cPU6ZMgdFohKurKzQaTV0pX2EyNX0iTHl5OcaNG4fXX3/9qo+FhITUve3m1vx3Pl+64ywOXChs9uMSUeuoNFrwly+OYv0zt0HvpBUdxyHccFECwGuvvYbu3bvXOy3YsWNH7N+/v97j9u/fj9jYWGi1jftHDQsLa3QGq9WKZcuWISoqCj169AAA9OzZE2fOnEH79u0bfZwjR47AarVi8eLF0GhqB9q/P+0aEBCAvLw8yLJcN3o9duyYzePqdLp6I+kr+dauXYvIyEg4Od3UP0GT/Jh6Gct3Nn0pDREpS3JuKZZ8n4KXRncUHcUh3NSGA127dsX999+PZcuW1b3vueeew44dOzB//nykpKRg1apVWL58OWbMmHHTYQGgoKAAeXl5OH/+PDZu3Ijhw4fj4MGD+OSTT+qKePbs2Vi9ejXmzp2LpKQkJCcn48svv8Tf//73Bo/bvn17mEwmvPPOOzh//jw+++yzqybQDB48GJcuXcKiRYuQmpqKd999F1u2bLGZNzIyEgcOHEBaWhouX74Mq9WKZ555BoWFhbjvvvtw6NAhpKam4rvvvsMjjzxyVak2l4oaM57/z3Hw3rBE9uGTvRdwKqdUdAyHcNM788ybN6/ulClQO1pas2YNvvzyS3Tp0gWzZ8/GvHnzbuj07LUMHz4cISEh6Nq1K1588UV07NgRx48fx5AhQ+oeM3LkSHz77bfYtm0b+vTpg759++Ktt95C27ZtGzxufHw8lixZgtdffx1dunTBv//976uWwHTs2BHvvfce3n33XcTHx+PgwYPX/QFgxowZ0Gq16NSpEwICApCRkYHQ0FDs378fFosFd9xxB7p27Ypp06bB29u7bjTb3N747gyyi7nZOZG9MFtlvLTuOKz86bfFSfLvL7qR3fk5owgT3/+Ro0kiOzRrbCdMGRAlOoZdc9i9Xh2F0WzFzK95ypXIXi3ZxrNFLY1Faefe3XUOZzmVnMhuVRgtmL3+pOgYdo1FacfOXyrH+7u5+w6RvdtxOh/fHs8RHcNusSjt2Csbk2C0WK//QCJSvTkbT/EGBy2ERWmnNh3Pxd6zl6//QCKyC5fLa7Bwc/L1H0hNxqK0QxU1Zsz/9pToGETUyr46nIkD5wtEx7A7LEo7tHTHWeSVVouOQUStTJaB2RuSuLaymbEo7UxmYSVW7k8THYOIBDlzsQwbErNFx7ArLEo7s2zHWU7gIXJwb31/Fia+DjQbFqUdSb1UjnVH+ZMkkaPLKKzEl4d438rmwqK0I299nwILr00QEYB3dpxFlbFlbrLgaFiUdiI5txSbTuSKjkFECpFfVoOVP6aJjmEXWJR2YvG2FHB7eyL6rRU/pKK0mpsQ3CwWpR04llmM7ckXRccgIoUprjThwz3nRcdQPRalHVi87YzoCESkUJ/uv4DL5TWiY6gai1Ll/ne+gFvVEVGDKo0WLN95TnQMVWNRqhxHk0R0PZ8fyOA9K28Ci1LF9p69hENpRaJjEJHCGS1WrOYM2BvGolSxT/ddEB2BiFTiq8OZqDZxXeWNYFGqVEZBJfakXBIdg4hUorjShPXcueuGsChV6l8H0sFNeIioKbgBwY1hUapQtcmCNYe5jyMRNc3pvDL8j/erbDIWpQptPJaD4krutkFETbeKo8omY1Gq0Or/pYmOQEQqte3UReRwqUiTsChV5kh6EU5ml4qOQUQqZbHK+Ox/6aJjqAqLUmU++ylNdAQiUrkvD2ZwqUgTsChV5HJ5DTafyBMdg4hUrqjShI2JOaJjqAaLUkW+PJgBo8UqOgYR2YHVPDvVaE6iA1DjyLKMzw9kiI7RKLLVgpJ9n6P81G5YK4qgdfeFW5dh8LptMiRJgmwxo3jvZ6hKPQxzSR40ejcY2sbD+/YEOHn4NXjckp/WoDLlJ5gKsyA56aAP6wif2xPg7Bde95jCHR+h4uQOSM4GeN/+MNw7D6n7WMXpfag4uQOBE19p0a+fSA1OZpci5WIZYoM8REdRPI4oVeJIehFySqpFx2iU0gNrUXZsC3xHPInQx96H9+0JKD24DmVH/gsAkM01MOalwuu2yQh5eCkCxr8MU2E2Lq2bb/O41Zkn4dHzTgQ/8CaC7p0PWMy4uGYWrMbav5fKcwdQkbwHgZPmw2fwIyjc+g4slSUAAGtNBYp/WA3fO55q2S+eSEU2HONOPY3BolQJNV2brMlOhkv7W+Harg+cvILgFjcALpE9YMxNAQBo9G4ImvwPuHUcCGe/cOjD4uA74kkY887BXJrf4HGDJs2De9fh0AW0hS4wGn53Toel9BKMF2tvIWQqyIQhoiv0ITFw63Q7JJ0rzCW1N7Qu2vVPePQYAyfPwJb/CyBSif8m5oqOoAosShWQZRlbT6rnG1of1hHV6YkwFdb+tGrMP4/qrFMwRPdq8HOsNZUAJGj07o1+HmtNBQBAY6j9HF1AFIx552CpLkdN3jnI5ho4+YSiOisJxoup8Og17sa/KCI7lFFYiZ8zeAei6+E1ShU4llmsmtOuAODZdyKsNZXI+ehJQKMBrFZ4D3qw3vXC35LNRhTv/idcOw2CRu/aqOeQZSuKdnwEfVgn6AIiAQAu0b3g1nkw8lZNh+Skg/+d06Fx1qPwu/fgd+d0lB3djLKfv4XWxRO+I6dCF9C2ub5kItXaeCwHPdv4iI6haCxKFdh6Uj2nXQGgMnkvKk7thv+4GXAOaAvjxfMo2vERtO5+cO86rN5jZYsZlza8BgDwu+OZRj9H4bb3YbyUjuD7F9V7v/eA++E94P66Pxfv+xyGyO6QNFqU/PQVQh99F1XnDqJg0xKEJCy9ia+SyD5sOpGL2WM7QaORREdRLJ56VYEtKivKot3/hFffiXDrdDt0AZFw7zIUHn3uQsn//lPvcVdK0lySj8B75zd6NFn4/fuoSj2EoPsWwMnTv8HHmQoyUXFqF7wHPoDqjBMwhHeB1tULrnEDYbyY+svpXiLHdqmshqdfr4NFqXAns0uQUaiuF3TZVANI9b+1JEkDyL+uAa0ryaIcBE1+FVoXz+sfV5ZR+P37qEz5CUGTX4Wzd7DNxxZ89y58hj4Gjc4FkK2QrebaD175XeaaVCJAfWetWhuLUuE2n1DPJJ4rXNrfgpIfv0Jl6iGYSy6iMuVHlB5aD9fYfgB+Kcn1C2HMOwf/cTMAqxWW8iJYyosgW369K8rFL19G6S9LSoDakWR50m74j3seGp1r3edYTTVXZShP/A5aF0+4tr8VwJUJRsdRk30apYc2wNmvTd0kICJH990pFqUtvEapcGo77QoAvsP/jOK9/0LhtvdgrSyB1t0X7t1Hw7v/ZACApbwAVecOAABy//nXep8bdN8CGNp0AwCYivKgr/p1A/jyo5sBABe/eKne5/iNmQb3rsPr/mypKELJT2sQ/MAbde/Th3aA5y0TkP/1XGhcveB/5/Rm/IqJ1C2zsAons0vQJcxLdBRFkmRZlkWHoGtLzi3F6KV7RccgIgfwl6Ht8dwdHUTHUCSeelUwNY4miUid9qRcEh1BsViUCsZvXCJqLUk5pSirNl3/gQ6IRalQZdUmnMwuER2DiByExSrjcDqXiVwLi1KhDl4ohMXKy8dE1HoOnC8UHUGRWJQK9VNqgegIRORgDlzg6861sCgV6kcWJRG1shNZJag0mkXHUBwWpQKVVJqQnFd6/QcSETUjs1XG4TRep/w9FqUCHckoBFe3EpEIPP16NRalAh3hzDMiEoQTeq7GolQgFiURiXI8qwRVRovoGIrColQYs8WK41lcP0lEYhgtVt5263dYlApzOq8MlfxpjogEOniBp19/i0WpMEczi0VHICIHdyavTHQERWFRKszZi/wGJSKxzubzdei3WJQKk3qpXHQEInJw6QWVMFmsomMoBotSYVLzK0RHICIHZ7bKuHCZr0VXsCgVpKLGjLzSatExiIhwLp9nt65gUSoIT7sSkVKcvcjXoytYlArCoiQipeCEnl+xKBWE1yeJSCl46vVXLEoF4YiSiJTiwuUK3jz+FyxKBWFREpFS1JityCisFB1DEViUCmGxykgr4DclESkHN0CpxaJUiMzCShjNXOBLRMqRVsB5EwCLUjF4ioOIlKagwig6giKwKBWiqJLfkESkLIXlfF0CWJSKUcif3IhIYfi6VItFqRBFlSbREYiI6inkmS4ALErFKOY3JBEpDEeUtViUCsERJREpDa9R1mJRKgRHlESkNGU1Zi5bA4tSMTjrlYiUiK9NLErFKKrgqVciUp4Cnn5lUSoFT70SkRJxQg+LUhGMZisqjBbRMYiIrsIlIixKReBokoiUqrSKl4VYlArA0SQRKRXvScmiVARJdAAiogawKFmUiiCxKYlIoViULEpF0LApiUihLDKLkkWpAOxJIlIqjigBJ9EBCJDYlNSMenuVYbXHckgcCVAzMOmnAGgvOoZQLEoF0LAnqRkdLvFAta83fHP3io5CdsDFUiw6gnA89aoAEue9UjNbXH2X6AhkLzRa0QmEY1EqAEeU1Nz+nRuKkqC+omOQPZBYlCxKBeA1SmoJ75gniI5A9oAjShalErAnqSV8nB2B8sBeomOQ2mmdRScQjkWpAFxHSS3lA/mPoiOQ2hm8RScQjkWpAAZn/jNQy1ieGYlK/3jRMUjNXHxEJxCOr9AK4Kpzgs6J/xTUMj7VcFRJN4FFyaJUCh9XXgeglrE4sx2qfTuKjkFqxaJkUSqFj6tOdASyU7Is4V+6SaJjkFqxKFmUSuHNESW1oIXpMTD6xIiOQWojaQGDl+gUwrEoFYIjSmpJFlmDrwz3iI5BamPw4vo1sCgVw8+dRUkta156Z5i8okTHIDXhaVcALErFCPQwiI5Ads5klfCNO69VUhOwKAGwKBUjyFMvOgI5gFfSusDsES46BqmFq6/oBIrAolQIjiipNVRZtNjsda/oGKQWXhGiEygCi1IhAjmipFbyUlp3WNyCRccgNfCNFp1AEViUChHsyREltY4Ksxbf+3BUSY3AogTAolQMP3c93PVOomOQg5iZ3gtWV3/RMUjpWJQAWJSK0j7QXXQEchAlJifs9uOokmyQNIAvlxMBLEpF6RDkIToCOZCZ6bfA6sJZjdQAj1DAiXMnABalosQEcURJreeS0Rk/+nO3HmoAR5N1WJQKEssRJbWy5zP7QdZ7io5BSsTrk3VYlArSIZhFSa0rt1qHQ4EcVdI1sCjrsCgVJMjTAC8X3kWEWtcLWf0h69xExyClYVHWYVEqTCyvU1IrS6sy4FjwRNExSGmCu4hOoBgsSoWJ4XVKEuCF7EGQnVxExyClcPHliPI3WJQKwyUiJMLZChecCpkgOgYpRVgv0QkUhUWpMFwiQqK8kDcEspbr5ggsyt9hUSoMR5QkSlKZG1JC7xIdg5QgvLfoBIrColQYP3c9ovw5A5HEeCl/GGQNZ147PI4o62FRKlDfaD/REchB/VzigQuhY0XHIJF8onjD5t9hUSpQv3YsShLnbwV3QJa0omOQKBxNXoVFqUB9o/nTHInzU5EXMsNGi45BovD65FVYlAoU6GHgLbdIqDlFoyFLfHlwSBxRXoX/ExSqH69TkkA7C3yQG3qH6BjU2nQeQEh30SkUh0WpUJzQQ6LNLx0DGZLoGNSaom8HnHSiUygOi1Kh+kb7QuJrFAm05ZI/LoUOER2DWlP74aITKBKLUqH83PWIDeTmAyTWaxV/EB2BWlMMT7dfC4tSwbhMhERbdzEQBSGDRMeg1hDYCfAKE51CkViUCsbrlKQEi6u5rZ1D4GnXBjmJDkAN69fOD85aCSaLLDoKObDPc0Mws21feF38n+go+CHdjDd+NOJIjgW55TK+udcF4+N+3XJPmlt6zc9bNFyP5/tfe8N3i1XGnN01+NcJE/LKZYR6SEiI1+Hvg3SQfpko8OaPNVi03wgAmNlfh+du+/VYB7LMeHpzNQ485gYnjYonFvC0a4NYlArm5eKMgTEB2Hk6X3QUcnDLzBMwC+KLssIoIz5Ig0e7O+PuNVVXfTz3ufrrj7ecNWPKxmr8sVPD+9e+vt+I9w+bsGq8AZ0DtTicY8EjG6rgZQD+eqsexy9aMHtXDb79kytkGRj7RSXuaOeErkFamK0yntxUjQ/Huqi7JHUeQJu+olMoFotS4cZ2C2FRknCfZEdgWpve8Mg/LDTH6BhnjI65UnpXF2Wwe/2rSRvOmDEkSoton4avMv2YacFdHZxwZ2ztcSO9NfjipAkHs60AgNOXregWpMXQqNqXy25BGpy+bEXXIC3e2G/EoDZO6BOm8i3/om8HtNwMvyG8RqlwIzoFQe/EfyYS7wP5btERmuRiuRWbzpoxpYftdYG3RWix44IZKQUWAEBingX7MiwY3b62GLsGapBSYEFGiRXpxVakFFjRJVCD1EIr/nnMhH8MtYN7ePK0q00cUSqch8EZgzsE4Luki6KjkIN7NzMSz4THw/VyougojbIq0QQPHXB3R9svcy8O0KG0Rkbc8gpoNYDFCrw6VI/7u9WOsDoGaLFgmAEjPqsEACwcZkDHAC2Gr67AohF6fJdqxpzdNXDWAktHGTCorcpeVjXOQBzvGGOLyv5FHdPYbqEsSlKETzQT8Reooyg/PWrC/V2dYXCyfe1wTZIZ/z5hwud/dEHnAA2O5Vkw7bsahHpIeLh77Wj0yd46PNn715HpqmNGeOgl9AvXosPychx63A1ZpTImf12FC8+6Q3+d51SUmBGAG2fY28JzeiowvGMQXHUqvwZCdmFxRjSq/TqJjnFde9PNOFNgxWM9r78d2/PfV+PF/npM7uKMrkFaPBivw/S+OizcZ7zm4y9XWjF3Tw3eGW3AgWwLYv00iPHTYkiUE0xWIKXA2txfTsvqdq/oBIrHolQBF50WQ+MCRccgAiDhM+d7RIe4rk+OmtArRIP44Ov/gFlpAn4/YVUrAdYGVmVN/64G0/vqEe6pgcUKmH7Ti2arDFWt5jJ4AR14S7XrYVGqxLj4UNERiAAAC9JjUeMTK+S5y40yjuVZcCyvduLNhSIrjuXVTrS5orRGxn9OmRocTQ5bXYHlB38dLY6LdcKre2uwKcWEtGIrvkk2Ycn/jJgQd/WVqe9Tayf9PHNL7fXLPmFanL5sxZazJnx4xAitJKGDn4peVjuNB5zsYDJSC+M1SpUY3CEAHnonlNWYRUchByfLEtYY7sGDeLXVn/twjgVDVlXW/fn/ttUAqMHD8c5YOd4FAPDlSRNkGbivy7WXO6QWWnG58tdifWe0AbN21eDpzdXIr6jdcODPvZwx+/b6BVJlkjF1SzW+mugCzS8bEYR7avDOaAMe2VANvROwarwBLs4quj4ZP1l0AlWQZFlW04kCh/Z/Xx3DuqPZomMQwVkj41TA3+FcckF0FLpR3m2BZxPB2xRdn4rOEdAfuvP0KymDySphnRsngahat3tZko3EolSRQTEBCPdxER2DCAAwO60LzB7homPQjeJp10ZjUaqIRiPhgb5tRccgAgDUWDXY5MkXW1UKvwXwayc6hWqwKFXm3t4R3NKOFOPl9HhY3ENEx6CmuvXPohOoCl9xVcbHTcelIqQYFWYttnlPEh2DmsIronZZCDUai1KFHurH06+kHC+m94LVNUB0DGqsW54AtFwZ2BQsShXqFu6Nnm28RccgAgCUmJyw248zYFVB5wH0elh0CtVhUarU4wOjRUcgqjMzvQ+sLr6iY9D19Higdts6ahIWpUqN7ByMNr6uomMQAQAuGZ3xo7/y94B1aJIG6Puk6BSqxKJUKY1GwpQBUaJjENV5PrMfZL2n6BjUkLg7AZ9I0SlUiUWpYvf0DoeXy7X3syRqbbnVOhwM5AxYxeo3VXQC1WJRqpirzokzYElRns+6DbLOXXQM+r2wXkCbvqJTqBaLUuUeGxjNUSUpRkaVAceC/ig6Bv3eoBdEJ1A1FqXKebk44+nB3IqKlOOFnEGQnTnRTDHa9AM6jBKdQtVYlHbg4dsiEeJlEB2DCABwtsIFScETRMegK4a9IjqB6rEo7YDBWYtnh8WIjkFUZ2beYMha/fUfSC0rdhTQtp/oFKrHorQT9/SOQLsAN9ExiAAASWVuSAm9S3QMxyZpgGGzRaewCyxKO6HVSHh+ZAfRMYjqvJQ/HLKGE82E6XoPENRZdAq7wKK0I6O6hKB7hLfoGEQAgJ9L3HEhbJzoGI5JqwOGvCw6hd1gUdqZmaPiREcgqvO3yyMgS1rRMRxPrwTuwtOMWJR2pl87P9wey1sekTL8VOSFzLAxomM4Fp071002MxalHXphVAdIkugURLVmF46CLPGlptUMmAa484fl5sTvXjvUOdQLk/u0ER2DCACwu9AHuaF3iI7hGPxigNueFZ3C7rAo7dRLY+IQ6MF1bKQM80rvhAye5mhxd74JOOlEp7A7LEo75Wlwxry7uoiOQQQA2HrJD5dCh4qOYd863w1EDxadwi6xKO3YqC7BGN0lWHQMIgDAggouFWkxek9g5ALRKewWi9LOzb2rMzwNTqJjEGH9xUAUhNwuOoZ9Gj4H8AwRncJusSjtXKCHAX+7s6PoGEQAgDeq/yA6gv1pcxvQ+1HRKQAAH374ISIiIqDRaPD22283yzHT0tIgSRKOHTvWLMe7ESxKB3Bvnza4rZ2f6BhE+DI3BMXB3KS72Wj1wB+W4WbWgyUkJECSJEiSBGdnZwQFBWHEiBH49NNPYbVaG32c0tJSTJ06FTNnzkR2djaeeOKJG85ky+7duyFJEoqLi1vk+NfConQQC+/uCoMz/7lJvGWm8aIj2I/bnwf8b/7OQaNGjUJubi7S0tKwZcsWDBkyBM8++yzGjh0Ls9ncqGNkZGTAZDLhzjvvREhICFxdlX1PUlmWG/218ZXTQbT1c8O04bGiYxDh0+wIlAX2Fh1D/cJvAfpPb5ZD6fV6BAcHIywsDD179sTLL7+MDRs2YMuWLVi5ciUAoLi4GI899hgCAgLg6emJoUOHIjExEQCwcuVKdO3aFQAQHR0NSZKQlpaG1NRU3HXXXQgKCoK7uzv69OmD7du313tuSZKwfv36eu/z9vaue97fSktLw5AhQwAAPj4+kCQJCQkJAACr1YqFCxciKioKLi4uiI+Px9dff133uVdGolu2bEGvXr2g1+uxb9++Rv39sCgdyOMDo9ElzFN0DCK8b71bdAR103sBf/wY0LbcRL2hQ4ciPj4e69atAwDcc889yM/Px5YtW3DkyBH07NkTw4YNQ2FhIe699966Ajx48CByc3MRERGB8vJyjBkzBjt27MDRo0cxatQojBs3DhkZGTeUKSIiAmvXrgUAnDlzBrm5uVi6dCkAYOHChVi9ejU++OADJCUlYfr06XjggQewZ8+eesd48cUX8dprryE5ORndunVr1PNyOqQD0WokLJnUHXct348qk0V0HHJg72VFYmp4PFwvJ4qOok7j3gZ82rb408TFxeH48ePYt28fDh48iPz8fOj1tRuZvPnmm1i/fj2+/vprPPHEE/Dzq50HERAQgODg2mVp8fHxiI+Przve/Pnz8c0332Djxo2YOnVqk/NotVr4+voCAAIDA+Ht7Q0AqKmpwYIFC7B9+3b061d7DTw6Ohr79u3DihUrcPvtv862njdvHkaMGNGk52VROpjYIA+8OqEL/m8NX6BIrE+0E/EX8PuwyXo8CHRpnRG5LMuQJAmJiYkoLy+vK8MrqqqqkJqa2uDnl5eXY86cOdi0aRNyc3NhNptRVVV1wyPKhpw7dw6VlZVXFaDRaESPHj3qva9376af9mdROqC7e4bjUFohvjiYKToKObDF6e3wWFhnuBQkiY6iHv6xwOhFrfZ0ycnJiIqKQnl5OUJCQrB79+6rHnNlVHctM2bMwPfff48333wT7du3h4uLCyZOnAij0Vj3GEmSIMtyvc8zmUxNylleXg4A2LRpE8LCwup97MoI+Ao3N7cmHRtgUTqsV8Z1xvGsEiTllIqOQg7sM+eJeAIsykbR6oGJnwK61plNunPnTpw4cQLTp09HeHg48vLy4OTkhMjIyEYfY//+/UhISMCECRMA1BZaWlpavccEBAQgNze37s9nz55FZWVlg8fU6Wr3srVYfr181KlTJ+j1emRkZNQ7zdpcWJQOyuCsxfv398Kd7+xFWXXjpkgTNbeF6bF4ODgW+qIU0VGU7475QHDXFjl0TU0N8vLyYLFYcPHiRWzduhULFy7E2LFj8dBDD0Gj0aBfv34YP348Fi1ahNjYWOTk5GDTpk2YMGFCg6czY2JisG7dOowbNw6SJGHWrFlXrc0cOnQoli9fjn79+sFisWDmzJlwdnZuMGvbtm0hSRK+/fZbjBkzBi4uLvDw8MCMGTMwffp0WK1WDBgwACUlJdi/fz88PT3x8MMP39TfD2e9OrA2fq5485746z+QqIXIsoSvDJNEx1C+2NHArX9uscNv3boVISEhiIyMxKhRo7Br1y4sW7YMGzZsgFarhSRJ2Lx5MwYNGoRHHnkEsbGxmDx5MtLT0xEUFNTgcZcsWQIfHx/cdtttGDduHEaOHImePXvWe8zixYsRERGBgQMH4k9/+hNmzJhhcw1mWFgY5s6dixdffBFBQUF1k4Lmz5+PWbNmYeHChejYsSNGjRqFTZs2ISoq6qb/fiT59yeHyeG8uukUPtp7QXQMclDOGhmnAmbBueS86CjK5N0WeGI34OorOonD4oiSMHNUHHq39REdgxyUySphnRtHldek9wT+tIYlKRiLkuCk1WD5n3rCz403fCUxZqd1gdkzQnQMZZG0wMR/AoFxopM4PBYlAQCCvQxYdl8POGl4F3pqfTVWDb71vFd0DGUZuQCIGS46BYFFSb/Rv70/5o/vIjoGOai/XegOi3uo6BjK0PtRoO+TolPQL1iUVM99t7TBM0PaiY5BDqjCosE2b16rRPRgYPQbolPQb7Ao6SrPj4zD+O78yZ5a34vpPWF1DRAdQxy/GOCeVS262Tk1HYuSrmnRxHj0i+bNnql1lZicsMtvsugYYrj4AH/6CnDxFp2EfodFSdekc9JgxUO90CmEt+Wi1vVCem9YXRxsOYSzGzD5C8CPlz2UiEVJDfI0OGPVo7cg0k/Zdyon+1JgdMZ+fwe6VulkAO77AmjbT3QSagCLkmwK8NDjsym3ItjTIDoKOZDnM/pC1nuJjtHytDrg3n8B0c2/kTc1HxYlXVeErytWT7kF3q4Nb1RM1JzyanQ4GGjno0qNE3DPSiCmaTcRptbHoqRGiQ3ywMpHboGHgbPxqHU8n9UPss5ddIyWIWmBuz8C4u4UnYQagUVJjdY9whtfPN6XW91Rq8ioMuBo0ETRMZqfpAHGvwd0uVt0EmokFiU1SZcwL3z15368Zkmt4vnsgZCd7WkymQSMfQuId9AlMCrFoqQmax/ojv882Q9tfO3pBYyUKLXSBSeDJ4iO0UwkYMwbQK8E0UGoiViUdEMifF3xnyf7ISbQTq8hkWK8kDsEslYvOsbN0eqAP34M3PK46CR0A1iUdMOCPA346s/90CWMmxJQy0kud0VK6HjRMW6czqP2npJd7fB6q4NgUdJN8XXT4YvH+6JPJG/8TC3npfxhkDUqXJ7kFggkfAu0GyI6Cd0EFiXdNA+DM1Y/eisGxTrwZtbUon4uccf5sD+IjtE0vtHAlG1AaHfRSegmsSipWbjotPj4od4Y0zVYdBSyUy9fGgFZ0oqO0Tgh3YFHtwG+UaKTUDNgUVKz0Tlp8O6femL68FhIkug0ZG8OFHsiI0wFC/TbDQUSNgHuPMNiL1iU1KwkScKzw2Pw0YO94aHnLj7UvGYXjYQsKfhlq/v9tRN39JwNbk8kWZZl0SHIPp3LL8cTnx3G+UsVoqOQHdnf7jOEZW8RHaM+rQ4Y9RrQZ4roJNQCFPyjGald+0B3bHimP4bFBYqOQnZkfukYyFDQuX3PMOCRrSxJO8YRJbU4WZax5PsULN91Dvxuo+ZwIPpTBOVsFx0DiBoETPwn4OYvOgm1II4oqcVJkoTn7uiA9+/vBTedSmYtkqItqBgrOgLQfxrw4HqWpAPgiJJaVcrFMjyx+jDSCipFRyGVOxK1An65e1r/ifWetXf/6Diu9Z+bhOCIklpVbJAHNv5lAO7uESY6CqncG9UCNiAI6Ag8vosl6WA4oiRhtiXl4eVvTuJyeY3oKKRSxyKXwzvvx5Z/IkkD9HsGGPJ3wJm3mHM0HFGSMHd0Dsa26YO4mw/dsLeN41v+SXzb1c5qveMfLEkHxRElKcLGxBzM3nASxZUm0VFIZY63eQue+Yda4MgScOufgWGvADree9WRcURJivCH+FBsmz6Iay6pyd6XW+DGzt5ta+/6Mfp1liRxREnKs+ZwJub/9xTKasyio5BKJEUsgtulY81zsN6PAiPmcxs6qsMRJSnOpN4R+G76IAyM4fo0apyPpWa4KbJPJPDgN8DYt1iSVA9HlKRom47nYsHmZGQXV4mOQgqXHPYqXAqSmv6JOndg4HO1s1qd9M0fjFSPRUmKV22y4P3dqVjxQyqqTVbRcUihXo5MwRN5c5rwGRIQfx8w/BXAgzOvqWEsSlKNrKJKLNicjM0n8kRHIQWSJBmng+dBX3Tm+g8OvwUY/RoQ1qvlg5HqsShJdQ6nFWLB5mT8nFEsOgopzJyoZCTkzm/4AR6hwIi5QLdJrReKVI9FSaq1+UQuFm09zX1jqY5WsiI5aDZ0xefrf8DZFeg3FRgwDdC5CclG6sWiJFUzWaz4/EAG3tl5jlvhEQDgtegTmJyzsPYPzq61yz36TwPcA4TmIvViUZJdqDFbsPZINj7aex4XLleIjkMC6TVWnAyeD+cOI4D+zwLu3MSCbg6LkuyK1Srju6Q8fLAnFYlZJaLjUCvzMDjhwb5t8Xj/tvDxcBEdh+wEi5Ls1o+pl7Fiz3nsSbkkOgq1MH93PaYMiMIDfdvAw+AsOg7ZGRYl2b1TOaVY8UMqNh3PhdnKb3d70i7ADQn9o3BPr3AYnLWi45CdYlGSw8gsrMQn+y5g7ZEs7iOrYi7OWtzZLQST+0Sgd6Sv6DjkAFiU5HCqTRZ8l5SHdT9nY9+5y7BwlKkK3cK9cG+fCPwhPpSnV6lVsSjJoeWXVmP9sWysPZKNMxfLRMeh3/Fyccb47qG4t08bdAr1FB2HHBSLkugXJ7NLsPbnLPw3MQeXy42i4zgsSQJujfLF5D5tMKpLMK89knAsSqLfMVus2H3mEtYdzcKO5HzUmLkRe0vTO2nQv70/hnUMxLC4IAR7GURHIqrDoiSyodpkwYELhdhz5hL2pOQj9RI3M2guAR56DIsLxLCOQRjQ3h8uOo4cSZlYlERNkFVUiR9SLmNPSj5+PFfA2bNN1CnEE8M71pZjt3AvSJIkOhLRdbEoiW6Q2WLFkfQi/HD2EvakXEJSTin4v+lXkgRE+7uhe4QPerb1xpAOgQj15m45pD4sSqJmUlhhxLHMIiRmliAxqxjHs0pQWOE4k4L83HToHuFd+6uNN7qFe8PLhcs4SP1YlEQtKLOwEsezSpCcW4rTeaVIzi1DdnGV6Fg3zeCsQacQT3SP8EH3Nt7oEeGNCF9X0bGIWgSLkqiVlVabcDq3DOfyy5FbUoWc4mrkllQhr6QaOSVVqDYpY5ath94Jod4uiPB1RZS/K6L83RHp74oofzcEexp4fZEcBouSSGGKKozILaktz5ySauSVVCG3uBoFFUbUmC2oMVtRY7L++rbZihpT7dtGi7XedVJJAlydtXDTO8Fd7wQ3vRNcddq6t930TnDT1X7c312HUG+Xul88bUpUi0VJZGeM5toS1UgSXHVajvyIbhKLkoiIyAaN6ABERERKxqIkIiKygUVJRERkA4uSiIjIBhYlERGRDSxKIiIiG1iURERENrAoiYiIbGBREhER2cCiJCIisoFFSUREZAOLkoiIyAYWJRERkQ0sSiIiIhtYlERERDawKImIiGxgURIREdnAoiQiIrKBRUlERGQDi5KIiMgGFiUREZENLEoiIiIbWJREREQ2sCiJiIhsYFESERHZwKIkIiKygUVJRERkA4uSiIjIBhYlERGRDSxKIiIiG1iURERENrAoiYiIbGBREhER2cCiJCIisoFFSUREZAOLkoiIyAYWJRERkQ0sSiIiIhtYlERERDawKImIiGxgURIREdnAoiQiIrKBRUlERGQDi5KIiMgGFiUREZEN/w9RGYau1HKj2AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# check for target var proportion\n",
    "r = train_data.groupby('Risk_Flag')['Risk_Flag'].count()\n",
    "plt.pie(r, labels=['Non-Defaulter', 'Defaulter'], radius=1.0, autopct='%1.1f%%');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BJLaCFG154GH"
   },
   "source": [
    "**Unbalance Data**\n",
    "\n",
    "The label, i.e. Risk_flag column, is not evenly distributed and is likely to produce undesired outcome. To solve this issue, an oversampling method should be used:\n",
    "1. **Random Over Sampling**: this method is easy to implement since, it will increase the dataset size simply by duplicating the original data points randomly\n",
    "2. **SMOTE**: creates synthetic data points based on the original ones. Not only SMOTE increases dataset size, the variety will also be increased"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mh2Lgbh_5u21"
   },
   "source": [
    "### **Random Over Sampling**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "O-7izMcKLeOQ",
    "tags": []
   },
   "outputs": [],
   "source": [
    "random_sampler = RandomOverSampler(random_state=30)\n",
    "X_oversampling, y_oversampling = random_sampler.fit_resample(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_oh_oversampling = pd.get_dummies(X_oversampling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "id": "b9RdwjxiL1Xb",
    "outputId": "343aaa12-92a6-44a4-e5e9-1b5930f34bbf",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEWCAYAAACOv5f1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA0rElEQVR4nO3de1hU9b4/8PdwmWEQZ0C5qoAphqkoXgLHMjuJjsYxrZ6jx3jKzDQN09IwaVeatcNT7XZuH7I6JXSxMHfbLookoWgpmRJ4g/CG4S4uqTEjXobb+/eHh/VzBRqj0ojr83qe7/O01vezvvNdzHfejLNWg44kIYQQQlPcXD0BIYQQfz4JfyGE0CAJfyGE0CAJfyGE0CAJfyGE0CAJfyGE0CAJfyGE0CAJfyGE0CAJfyGE0CAJfyGE0CCnwn/FihXo378/TCYTTCYTLBYLNmzYoPTffvvt0Ol0qjZz5kzVGGVlZYiPj4e3tzcCAwORlJSE+vp6VU1ubi4GDRoEg8GAiIgIpKenN5tLamoqunfvDi8vL8TGxuL777935lSEEELTPJwp7tatG5YuXYpevXqBJN577z2MHz8eBQUF6Nu3LwBg+vTpWLJkiXKMt7e38t8NDQ2Ij49HcHAwtm/fjvLycjzwwAPw9PTESy+9BAAoLS1FfHw8Zs6ciVWrViEnJwcPP/wwQkJCYLVaAQCrV6/GvHnz8OabbyI2Nhavv/46rFYrSkpKEBgY2OrzaWxsxC+//IKOHTtCp9M586MQQohrEkmcOnUKXbp0gZvbJd7f8wr5+fnxnXfeIUmOGDGCc+fOvWhtZmYm3dzcWFFRoexbsWIFTSYTHQ4HSXLBggXs27ev6rhJkybRarUq2zExMUxMTFS2Gxoa2KVLF6akpDg192PHjhGANGnSpF137dixY5fMP6fe+V+ooaEBa9aswenTp2GxWJT9q1atwocffojg4GCMGzcOzz77rPLuPy8vD1FRUQgKClLqrVYrZs2ahf3792PgwIHIy8tDXFyc6rGsVisef/xxAEBtbS3y8/ORnJys9Lu5uSEuLg55eXmXnLPD4YDD4VC2+X9faHrs2DGYTKbL+0EIIcQ1xG63IzQ0FB07drx0oVNvlUnu2bOHHTp0oLu7O81mM9evX6/0vfXWW8zKyuKePXv44YcfsmvXrrz77ruV/unTp3P06NGq8U6fPk0AzMzMJEn26tWLL730kqpm/fr1BMAzZ87w559/JgBu375dVZOUlMSYmJhLzn3RokUt/oa02WzO/hg07Y033mBUVBQ7duzIjh07cujQocrzR5IzZsxgjx496OXlRX9/f951110sLi5WjdHS8/Dxxx+ras6dO8enn36aYWFh1Ov1DA8P57vvvtvinD7++GMC4Pjx46/6+QrXkzXXejabrVW55vQ7/8jISBQWFsJms+Gf//wnpkyZgi1btqBPnz6YMWOGUhcVFYWQkBCMHDkShw8fRs+ePZ19qKsuOTkZ8+bNU7abfkMK5/zRtZ/BgwcjISEBYWFhOHnyJBYvXozRo0ejtLQU7u7uyjhpaWkYM2aMsu3r66t6nIkTJ6KyshLvvvsuIiIiUF5ejsbGxmbzOXr0KJ588kkMHz68zc5ZuJasuTZwpb9lRo4cyRkzZrTYV1NTQwDMysoiST777LMcMGCAqubIkSMEwB9++IEkOXz48GbXDVauXEmTyUSSdDgcdHd359q1a1U1DzzwAO+66y6n5t7a35Dij1147ef3du/eTQA8dOiQsg9As+fwQhs2bKDZbOaJEycu+bj19fUcNmwY33nnHU6ZMuWaexcm2o6suZa1Nteu+D7/xsZG1efoFyosLAQAhISEAAAsFgv27t2LqqoqpSY7Oxsmkwl9+vRRanJyclTjZGdnK9cV9Ho9Bg8erKppbGxETk6O6tqD+HM0NDQgIyOj2bWfJqdPn0ZaWhpuuOGGZv/KSkxMhL+/P2JiYrBy5UrlGgwAfPHFFxgyZAhefvlldO3aFTfeeCOefPJJnD17VjXGkiVLEBgYiGnTprXNCYprjqy5q8SZ3ygLFy7kli1bWFpayj179nDhwoXU6XTcuHEjDx06xCVLlnDXrl0sLS3l559/zh49evC2225Tjq+vr2e/fv04evRoFhYWMisriwEBAUxOTlZqjhw5Qm9vbyYlJbG4uJipqal0d3dX/vVAkhkZGTQYDExPT2dRURFnzJhBX19f1V1ErSHv/C/fpa79kGRqaio7dOhAAIyMjFS9AyPJJUuW8Ntvv+UPP/zApUuX0mAwcNmyZUq/1WqlwWBgfHw8d+zYwfXr1zM8PJwPPvigUvPNN9+wa9eu/PXXX0nymnwXJq4eWXOt09pccyr8H3roIYaHh1Ov1zMgIIAjR47kxo0bSZJlZWW87bbb2KlTJxoMBkZERDApKanZBI4ePcqxY8fSaDTS39+f8+fPZ11dnapm8+bNjI6Opl6vZ48ePZiWltZsLsuXL1cuysTExPC7775z5lRISvhfCYfDwYMHD3LXrl1cuHAh/f39uX//fqW/urqaBw4c4JYtWzhu3DgOGjSIZ8+eveh4zz77LLt166Zsjxo1il5eXqyurlb2ffrpp9TpdDxz5gztdju7d++uuuh3Lb4QxdUja6512iT8rzcS/lfPpa79OBwOent786OPPrro8evWrSMAnjt3juT5azg9e/ZU1RQVFREADxw4wIKCAgKgu7u70nQ6HXU6Hd3d3Zu96xPXH1lzLWuzu32EaMmlrv3w/JuMi/YD568P+fn5wWAwAABuueUWrFmzBjU1NfDx8QEAHDhwAG5ubujWrRt0Oh327t2rGuOZZ57BqVOnsGzZMrmLSwNkzV0ZCX/htOTkZIwdOxZhYWE4deoUPvroI+Tm5uKrr77CkSNHsHr1aowePRoBAQH497//jaVLl8JoNOLOO+8EAHz55ZeorKzE0KFD4eXlhezsbLz00kt48sknlce477778MILL2Dq1Kl4/vnncfz4cSQlJeGhhx6C0WgEAPTr1081r6bb9n6/X7R/subawJ/wr5Br1hV97ANotj0EMBygHmAAwJEAN/5f388AxwIMBOgJsBvA+wD+eMHxGwBGA/QB2AHgAIBvAmz43eMUA4wDaPy/ceYBPHOJeU0BOL4tz92FroGn3cXtIQLhBPQEAgiMJLDx//p+JjCWQCABTwLdCNxH4McLjt9AIJqAD4EOBAYQeJNAw+8ep5hAHAHj/40zj8CZS8xrCoHx19SSa22u6c4vLG2y2+0wm82w2WzOf72DfBGc9rjwpSLLTZsuZ8m1Ntfk+/yFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDnAr/FStWoH///jCZTDCZTLBYLNiwYYPSf+7cOSQmJqJz587w8fHBvffei8rKStUYZWVliI+Ph7e3NwIDA5GUlIT6+npVTW5uLgYNGgSDwYCIiAikp6c3m0tqaiq6d+8OLy8vxMbG4vvvv3fmVIQQQtuc+cPAX3zxBdevX88DBw6wpKSETz/9ND09Pblv3z6S5MyZMxkaGsqcnBzu2rWLQ4cO5bBhw5Tj6+vr2a9fP8bFxbGgoICZmZn09/dncnKyUnPkyBF6e3tz3rx5LCoq4vLly+nu7s6srCylJiMjg3q9nitXruT+/fs5ffp0+vr6srKy0pnTkT/gLq3t/5r2VeLqU5fWfpZca3Ptile0n58f33nnHVZXV9PT05Nr1qxR+oqLiwmAeXl5JMnMzEy6ubmxoqJCqVmxYgVNJhMdDgdJcsGCBezbt6/qMSZNmkSr1apsx8TEMDExUdluaGhgly5dmJKS4tTcJfyltfkr8Spx9alLaz9LrrW5dtmf+Tc0NCAjIwOnT5+GxWJBfn4+6urqEBcXp9T07t0bYWFhyMvLAwDk5eUhKioKQUFBSo3VaoXdbsf+/fuVmgvHaKppGqO2thb5+fmqGjc3N8TFxSk1F+NwOGC321VNCCG0yOnw37t3L3x8fGAwGDBz5kysXbsWffr0QUVFBfR6PXx9fVX1QUFBqKioAABUVFSogr+pv6nvUjV2ux1nz57F8ePH0dDQ0GJN0xgXk5KSArPZrLTQ0FBnT18IIa4LTod/ZGQkCgsLsWPHDsyaNQtTpkxBUVFRW8ztqktOTobNZlPasWPHXD0lIYRwCQ9nD9Dr9YiIiAAADB48GDt37sSyZcswadIk1NbWorq6WvXuv7KyEsHBwQCA4ODgZnflNN0NdGHN7+8QqqyshMlkgtFohLu7O9zd3VusaRrjYgwGAwwGg7OnLIQQ150rvs+/sbERDocDgwcPhqenJ3JycpS+kpISlJWVwWKxAAAsFgv27t2LqqoqpSY7Oxsmkwl9+vRRai4co6mmaQy9Xo/BgwerahobG5GTk6PUCCGE+APOXEVeuHAht2zZwtLSUu7Zs4cLFy6kTqfjxo0bSZ6/1TMsLIybNm3irl27aLFYaLFYlOObbvUcPXo0CwsLmZWVxYCAgBZv9UxKSmJxcTFTU1NbvNXTYDAwPT2dRUVFnDFjBn19fVV3EbWG3O0jrc1vvbhKXH3q0trPkmuTWz0feughhoeHU6/XMyAggCNHjlSCnyTPnj3LRx99lH5+fvT29ubdd9/N8vJy1RhHjx7l2LFjaTQa6e/vz/nz57Ourk5Vs3nzZkZHR1Ov17NHjx5MS0trNpfly5czLCyMer2eMTEx/O6775w5FZIS/tL+hFfiVeLqU5fWfpZca3NNd35haZPdbofZbIbNZoPJZHLuYJ2ubSYlrl0ufKnIctOmy1lyrc01+W4fIYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIKfCPyUlBTfffDM6duyIwMBATJgwASUlJaqa22+/HTqdTtVmzpypqikrK0N8fDy8vb0RGBiIpKQk1NfXq2pyc3MxaNAgGAwGREREID09vdl8UlNT0b17d3h5eSE2Nhbff/+9M6cjhBDaRSdYrVampaVx3759LCws5J133smwsDDW1NQoNSNGjOD06dNZXl6uNJvNpvTX19ezX79+jIuLY0FBATMzM+nv78/k5GSl5siRI/T29ua8efNYVFTE5cuX093dnVlZWUpNRkYG9Xo9V65cyf3793P69On09fVlZWVlq8/HZrMRgGp+rQZI01pzIVefurT2s+Ram2tXtKKrqqoIgFu2bFH2jRgxgnPnzr3oMZmZmXRzc2NFRYWyb8WKFTSZTHQ4HCTJBQsWsG/fvqrjJk2aRKvVqmzHxMQwMTFR2W5oaGCXLl2YkpJy0cc+d+4cbTab0o4dO9aqH1KLXL0qpLWPV+JV4upTl9Z+llxrw/+KPvO32WwAgE6dOqn2r1q1Cv7+/ujXrx+Sk5Nx5swZpS8vLw9RUVEICgpS9lmtVtjtduzfv1+piYuLU41ptVqRl5cHAKitrUV+fr6qxs3NDXFxcUpNS1JSUmA2m5UWGhp6mWcuhBDtm8flHtjY2IjHH38ct9xyC/r166fsv++++xAeHo4uXbpgz549eOqpp1BSUoJ//etfAICKigpV8ANQtisqKi5ZY7fbcfbsWfz2229oaGhosebHH3+86JyTk5Mxb948Zdtut8svACGEJl12+CcmJmLfvn349ttvVftnzJih/HdUVBRCQkIwcuRIHD58GD179rz8mV4FBoMBBoPBpXMQQohrwWV97DN79mysW7cOmzdvRrdu3S5ZGxsbCwA4dOgQACA4OBiVlZWqmqbt4ODgS9aYTCYYjUb4+/vD3d29xZqmMYQQQlycU+FPErNnz8batWuxadMm3HDDDX94TGFhIQAgJCQEAGCxWLB3715UVVUpNdnZ2TCZTOjTp49Sk5OToxonOzsbFosFAKDX6zF48GBVTWNjI3JycpQaIYQQl+DMVeRZs2bRbDYzNzdXdSvnmTNnSJKHDh3ikiVLuGvXLpaWlvLzzz9njx49eNtttyljNN3qOXr0aBYWFjIrK4sBAQEt3uqZlJTE4uJipqamtnirp8FgYHp6OouKijhjxgz6+vqq7iL6I3Krp7Q2v/XiKnH1qUtrP0uuTW71BNBiS0tLI0mWlZXxtttuY6dOnWgwGBgREcGkpKRmkzh69CjHjh1Lo9FIf39/zp8/n3V1daqazZs3Mzo6mnq9nj169FAe40LLly9nWFgY9Xo9Y2Ji+N133zlzOhL+0tr+lXiVuPrUpbWfJdfaXNOdX1jaZLfbYTabYbPZYDKZnDtYp2ubSYlrlwtfKrLctOlyllxrc02+20cIITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITTIqfBPSUnBzTffjI4dOyIwMBATJkxASUmJqubcuXNITExE586d4ePjg3vvvReVlZWqmrKyMsTHx8Pb2xuBgYFISkpCfX29qiY3NxeDBg2CwWBAREQE0tPTm80nNTUV3bt3h5eXF2JjY/H99987czpCCKFddILVamVaWhr37dvHwsJC3nnnnQwLC2NNTY1SM3PmTIaGhjInJ4e7du3i0KFDOWzYMKW/vr6e/fr1Y1xcHAsKCpiZmUl/f38mJycrNUeOHKG3tzfnzZvHoqIiLl++nO7u7szKylJqMjIyqNfruXLlSu7fv5/Tp0+nr68vKysrW30+NpuNAGiz2Zz5MZwHSNNacyFXn7q09rPkWptrV7Siq6qqCIBbtmwhSVZXV9PT05Nr1qxRaoqLiwmAeXl5JMnMzEy6ubmxoqJCqVmxYgVNJhMdDgdJcsGCBezbt6/qsSZNmkSr1apsx8TEMDExUdluaGhgly5dmJKS0ur5S/hLc6q5kKtPXVr7WXKtzbUr+szfZrMBADp16gQAyM/PR11dHeLi4pSa3r17IywsDHl5eQCAvLw8REVFISgoSKmxWq2w2+3Yv3+/UnPhGE01TWPU1tYiPz9fVePm5oa4uDilpiUOhwN2u13VhBBCiy47/BsbG/H444/jlltuQb9+/QAAFRUV0Ov18PX1VdUGBQWhoqJCqbkw+Jv6m/ouVWO323H27FkcP34cDQ0NLdY0jdGSlJQUmM1mpYWGhjp/4kIIcR247PBPTEzEvn37kJGRcTXn06aSk5Nhs9mUduzYMVdPSQghXMLjcg6aPXs21q1bh61bt6Jbt27K/uDgYNTW1qK6ulr17r+yshLBwcFKze/vymm6G+jCmt/fIVRZWQmTyQSj0Qh3d3e4u7u3WNM0RksMBgMMBoPzJyyEENcZp975k8Ts2bOxdu1abNq0CTfccIOqf/DgwfD09EROTo6yr6SkBGVlZbBYLAAAi8WCvXv3oqqqSqnJzs6GyWRCnz59lJoLx2iqaRpDr9dj8ODBqprGxkbk5OQoNUIIIS7BmavIs2bNotlsZm5uLsvLy5V25swZpWbmzJkMCwvjpk2buGvXLlosFlosFqW/6VbP0aNHs7CwkFlZWQwICGjxVs+kpCQWFxczNTW1xVs9DQYD09PTWVRUxBkzZtDX11d1F9Efkbt9pLX5rRdXiatPXVr7WXJtcqsngBZbWlqaUnP27Fk++uij9PPzo7e3N++++26Wl5erxjl69CjHjh1Lo9FIf39/zp8/n3V1daqazZs3Mzo6mnq9nj169FA9RpPly5czLCyMer2eMTEx/O6775w5HQl/aW3/SrxKXH3q0trPkmttrunOLyxtstvtMJvNsNlsMJlMzh2s07XNpMS1y4UvFVlu2nQ5S661uSbf7SOEEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBrkdPhv3boV48aNQ5cuXaDT6fDZZ5+p+h988EHodDpVGzNmjKrm5MmTSEhIgMlkgq+vL6ZNm4aamhpVzZ49ezB8+HB4eXkhNDQUL7/8crO5rFmzBr1794aXlxeioqKQmZnp7OkIIYQmOR3+p0+fxoABA5CamnrRmjFjxqC8vFxpH3/8sao/ISEB+/fvR3Z2NtatW4etW7dixowZSr/dbsfo0aMRHh6O/Px8vPLKK1i8eDHefvttpWb79u2YPHkypk2bhoKCAkyYMAETJkzAvn37nD0lIYTQHl4BAFy7dq1q35QpUzh+/PiLHlNUVEQA3Llzp7Jvw4YN1Ol0/Pnnn0mSb7zxBv38/OhwOJSap556ipGRkcr2xIkTGR8frxo7NjaWjzzySKvnb7PZCIA2m63VxygAaVprLuTqU5fWfpZca3OtTT7zz83NRWBgICIjIzFr1iycOHFC6cvLy4Ovry+GDBmi7IuLi4Obmxt27Nih1Nx2223Q6/VKjdVqRUlJCX777TelJi4uTvW4VqsVeXl5F52Xw+GA3W5XNSGE0KKrHv5jxozB+++/j5ycHPzP//wPtmzZgrFjx6KhoQEAUFFRgcDAQNUxHh4e6NSpEyoqKpSaoKAgVU3T9h/VNPW3JCUlBWazWWmhoaFXdrJCCNFOeVztAf/7v/9b+e+oqCj0798fPXv2RG5uLkaOHHm1H84pycnJmDdvnrJtt9vlF4AQQpPa/FbPHj16wN/fH4cOHQIABAcHo6qqSlVTX1+PkydPIjg4WKmprKxU1TRt/1FNU39LDAYDTCaTqgkhhBa1efj/+9//xokTJxASEgIAsFgsqK6uRn5+vlKzadMmNDY2IjY2VqnZunUr6urqlJrs7GxERkbCz89PqcnJyVE9VnZ2NiwWS1ufkhBCtH/OXkk+deoUCwoKWFBQQAB87bXXWFBQwJ9++omnTp3ik08+yby8PJaWlvLrr7/moEGD2KtXL547d04ZY8yYMRw4cCB37NjBb7/9lr169eLkyZOV/urqagYFBfH+++/nvn37mJGRQW9vb7711ltKzbZt2+jh4cFXX32VxcXFXLRoET09Pbl3795Wn4vc7SPNqeZCrj51ae1nybU215wefvPmzQTQrE2ZMoVnzpzh6NGjGRAQQE9PT4aHh3P69OmsqKhQjXHixAlOnjyZPj4+NJlMnDp1Kk+dOqWq2b17N2+99VYaDAZ27dqVS5cubTaXTz75hDfeeCP1ej379u3L9evXO3UuEv7SnGou5OpTl9Z+llxrc013fmFpk91uh9lshs1mc/7zf52ubSYlrl0ufKnIctOmy1lyrc01+W4fIYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIKfDf+vWrRg3bhy6dOkCnU6Hzz77TNVPEs899xxCQkJgNBoRFxeHgwcPqmpOnjyJhIQEmEwm+Pr6Ytq0aaipqVHV7NmzB8OHD4eXlxdCQ0Px8ssvN5vLmjVr0Lt3b3h5eSEqKgqZmZnOno4QQmiS0+F/+vRpDBgwAKmpqS32v/zyy/jHP/6BN998Ezt27ECHDh1gtVpx7tw5pSYhIQH79+9HdnY21q1bh61bt2LGjBlKv91ux+jRoxEeHo78/Hy88sorWLx4Md5++22lZvv27Zg8eTKmTZuGgoICTJgwARMmTMC+ffucPSUhhNAeXgEAXLt2rbLd2NjI4OBgvvLKK8q+6upqGgwGfvzxxyTJoqIiAuDOnTuVmg0bNlCn0/Hnn38mSb7xxhv08/Ojw+FQap566ilGRkYq2xMnTmR8fLxqPrGxsXzkkUdaPX+bzUYAtNlsrT5GAUjTWnMhV5+6tPaz5Fqba1f1M//S0lJUVFQgLi5O2Wc2mxEbG4u8vDwAQF5eHnx9fTFkyBClJi4uDm5ubtixY4dSc9ttt0Gv1ys1VqsVJSUl+O2335SaCx+nqabpcVricDhgt9tVTQghtOiqhn9FRQUAICgoSLU/KChI6auoqEBgYKCq38PDA506dVLVtDTGhY9xsZqm/pakpKTAbDYrLTQ01NlTFEKI64Km7vZJTk6GzWZT2rFjx1w9JSGEcImrGv7BwcEAgMrKStX+yspKpS84OBhVVVWq/vr6epw8eVJV09IYFz7GxWqa+ltiMBhgMplUTQghtOiqhv8NN9yA4OBg5OTkKPvsdjt27NgBi8UCALBYLKiurkZ+fr5Ss2nTJjQ2NiI2Nlap2bp1K+rq6pSa7OxsREZGws/PT6m58HGaapoeRwghxCU4eyX51KlTLCgoYEFBAQHwtddeY0FBAX/66SeS5NKlS+nr68vPP/+ce/bs4fjx43nDDTfw7NmzyhhjxozhwIEDuWPHDn777bfs1asXJ0+erPRXV1czKCiI999/P/ft28eMjAx6e3vzrbfeUmq2bdtGDw8PvvrqqywuLuaiRYvo6enJvXv3tvpc5G4faU41F3L1qUtrP0uutbnm9PCbN28mgGZtypQpJM/f7vnss88yKCiIBoOBI0eOZElJiWqMEydOcPLkyfTx8aHJZOLUqVN56tQpVc3u3bt566230mAwsGvXrly6dGmzuXzyySe88cYbqdfr2bdvX65fv96pc5Hwl+ZUcyFXn7q09rPkWptruvMLS5vsdjvMZjNsNpvzn//rdG0zKXHtcuFLRZabNl3Okmttrmnqbh8hhBDnSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGXfXwX7x4MXQ6nar17t1b6T937hwSExPRuXNn+Pj44N5770VlZaVqjLKyMsTHx8Pb2xuBgYFISkpCfX29qiY3NxeDBg2CwWBAREQE0tPTr/apCCHEdatN3vn37dsX5eXlSvv222+VvieeeAJffvkl1qxZgy1btuCXX37BPffco/Q3NDQgPj4etbW12L59O9577z2kp6fjueeeU2pKS0sRHx+P//iP/0BhYSEef/xxPPzww/jqq6/a4nSEEOL6w6ts0aJFHDBgQIt91dXV9PT05Jo1a5R9xcXFBMC8vDySZGZmJt3c3FhRUaHUrFixgiaTiQ6HgyS5YMEC9u3bVzX2pEmTaLVanZqrzWYjANpsNqeOI0kC0rTWXMjVpy6t/Sy51uZam7zzP3jwILp06YIePXogISEBZWVlAID8/HzU1dUhLi5Oqe3duzfCwsKQl5cHAMjLy0NUVBSCgoKUGqvVCrvdjv379ys1F47RVNM0xsU4HA7Y7XZVE0IILbrq4R8bG4v09HRkZWVhxYoVKC0txfDhw3Hq1ClUVFRAr9fD19dXdUxQUBAqKioAABUVFargb+pv6rtUjd1ux9mzZy86t5SUFJjNZqWFhoZe6ekKIUS75HG1Bxw7dqzy3/3790dsbCzCw8PxySefwGg0Xu2Hc0pycjLmzZunbNvtdvkFIITQpDa/1dPX1xc33ngjDh06hODgYNTW1qK6ulpVU1lZieDgYABAcHBws7t/mrb/qMZkMl3yF4zBYIDJZFI1IYTQojYP/5qaGhw+fBghISEYPHgwPD09kZOTo/SXlJSgrKwMFosFAGCxWLB3715UVVUpNdnZ2TCZTOjTp49Sc+EYTTVNYwghhPgDl3c9+eLmz5/P3NxclpaWctu2bYyLi6O/vz+rqqpIkjNnzmRYWBg3bdrEXbt20WKx0GKxKMfX19ezX79+HD16NAsLC5mVlcWAgAAmJycrNUeOHKG3tzeTkpJYXFzM1NRUuru7Mysry6m5yt0+0tr81ourxNWnLq39LLnW5tpVX9GTJk1iSEgI9Xo9u3btykmTJvHQoUNK/9mzZ/noo4/Sz8+P3t7evPvuu1leXq4a4+jRoxw7diyNRiP9/f05f/581tXVqWo2b97M6Oho6vV69ujRg2lpaU7PVcJfWpu/Eq8SV5+6tPaz5Fqba7rzC0ub7HY7zGYzbDab85//63RtMylx7XLhS0WWmzZdzpJrba7Jd/sIIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGtfvwT01NRffu3eHl5YXY2Fh8//33rp6SEEJc89p1+K9evRrz5s3DokWL8MMPP2DAgAGwWq2oqqpy9dSEEOKa1q7D/7XXXsP06dMxdepU9OnTB2+++Sa8vb2xcuVKV09NCCGuaR6unsDlqq2tRX5+PpKTk5V9bm5uiIuLQ15eXovHOBwOOBwOZdtmswEA7HZ7205WXB9knYg/2eUsuaY8I3nJunYb/sePH0dDQwOCgoJU+4OCgvDjjz+2eExKSgqef/75ZvtDQ0PbZI7iOmM2u3oGQmOuZMmdOnUK5ksM0G7D/3IkJydj3rx5ynZjYyNOnjyJzp07Q6fTuXBm7YPdbkdoaCiOHTsGk8nk6ukIDZA15zySOHXqFLp06XLJunYb/v7+/nB3d0dlZaVqf2VlJYKDg1s8xmAwwGAwqPb5+vq21RSvWyaTSV6I4k8la845l3rH36TdXvDV6/UYPHgwcnJylH2NjY3IycmBxWJx4cyEEOLa127f+QPAvHnzMGXKFAwZMgQxMTF4/fXXcfr0aUydOtXVUxNCiGtauw7/SZMm4ddff8Vzzz2HiooKREdHIysrq9lFYHF1GAwGLFq0qNlHZ0K0FVlzbUfHP7ofSAghxHWn3X7mL4QQ4vJJ+AshhAZJ+AshhAZJ+AshhAZJ+F/ncnNzodPpUF1dfcm67t274/XXX/9T5iTEhRYvXozo6GhXT0Nz5G6f61xtbS1OnjyJoKAg6HQ6pKen4/HHH2/2y+DXX39Fhw4d4O3t7ZqJCk3Q6XRYu3YtJkyYoOyrqamBw+FA586dXTcxDWrX9/mLP6bX6y/6dRcXCggI+BNmI0RzPj4+8PHxcfU0NEc+9nGx22+/HbNnz8bs2bNhNpvh7++PZ599VvV1rL/99hseeOAB+Pn5wdvbG2PHjsXBgweV/p9++gnjxo2Dn58fOnTogL59+yIzMxOA+mOf3NxcTJ06FTabDTqdDjqdDosXLwag/tjnvvvuw6RJk1TzrKurg7+/P95//30A579KIyUlBTfccAOMRiMGDBiAf/7zn5c8V4fDgaeeegqhoaEwGAyIiIjAu+++CwBoaGjAtGnTlPEiIyOxbNky1fG5ubmIiYlBhw4d4Ovri1tuuQU//fST0v/5559j0KBB8PLyQo8ePfD888+jvr7eiWfj+nT77bdjzpw5WLBgATp16oTg4GDleW9SXV2Nhx9+GAEBATCZTLjjjjuwe/duVc2LL76IwMBAdOzYEQ8//DAWLlyo+rhm586dGDVqFPz9/WE2mzFixAj88MMPSn/37t0BAHfffTd0Op2yfeHHPhs3boSXl1ezf5nOnTsXd9xxh7L97bffYvjw4TAajQgNDcWcOXNw+vTpS/4cvvzyS9x8883w8vKCv78/7r77bqXvgw8+wJAhQ9CxY0cEBwfjvvvuU/1RqN9++w0JCQkICAiA0WhEr169kJaWpvQfO3YMEydOhK+vLzp16oTx48fj6NGjl5yPy1G41IgRI+jj48O5c+fyxx9/5Icffkhvb2++/fbbSs1dd93Fm266iVu3bmVhYSGtVisjIiJYW1tLkoyPj+eoUaO4Z88eHj58mF9++SW3bNlCkty8eTMB8LfffqPD4eDrr79Ok8nE8vJylpeX89SpUyTJ8PBw/v3vfydJrlu3jkajUekjyS+//JJGo5F2u50k+eKLL7J3797Mysri4cOHmZaWRoPBwNzc3Iue68SJExkaGsp//etfPHz4ML/++mtmZGSQJGtra/ncc89x586dPHLkiPJzWL16NUmyrq6OZrOZTz75JA8dOsSioiKmp6fzp59+Iklu3bqVJpOJ6enpPHz4MDdu3Mju3btz8eLFV+NpatdGjBhBk8nExYsX88CBA3zvvfeo0+m4ceNGpSYuLo7jxo3jzp07eeDAAc6fP5+dO3fmiRMnSJIffvghvby8uHLlSpaUlPD555+nyWTigAEDlDFycnL4wQcfsLi4mEVFRZw2bRqDgoKUNVNVVUUATEtLY3l5OauqqkiSixYtUsapr69nUFAQ33nnHWXc3+87dOgQO3TowL///e88cOAAt23bxoEDB/LBBx+86M9g3bp1dHd353PPPceioiIWFhbypZdeUvrfffddZmZm8vDhw8zLy6PFYuHYsWOV/sTEREZHR3Pnzp0sLS1ldnY2v/jiC5Ln1+5NN93Ehx56iHv27GFRURHvu+8+RkZG0uFwXM5T9qeQ8HexESNG8KabbmJjY6Oy76mnnuJNN91Ekjxw4AABcNu2bUr/8ePHaTQa+cknn5Ako6KiLhpyF4Y/SaalpdFsNjeruzD86+rq6O/vz/fff1/pnzx5MidNmkSSPHfuHL29vbl9+3bVGNOmTePkyZNbnEdJSQkBMDs7+xI/DbXExETee++9JMkTJ04QwEV/uYwcOVL1YibJDz74gCEhIa1+vOvViBEjeOutt6r23XzzzXzqqadIkt988w1NJhPPnTunqunZsyffeustkmRsbCwTExNV/bfccosq/H+voaGBHTt25JdffqnsA8C1a9eq6i4Mf5KcO3cu77jjDmX7q6++osFgUNbwtGnTOGPGDNUY33zzDd3c3Hj27NkW52KxWJiQkHDRuf7ezp07CUB5AzRu3DhOnTq1xdoPPviAkZGRqteww+Gg0WjkV1991erH/LPJxz7XgKFDh6r+noDFYsHBgwfR0NCA4uJieHh4IDY2Vunv3LkzIiMjUVxcDACYM2cOXnzxRdxyyy1YtGgR9uzZc0Xz8fDwwMSJE7Fq1SoAwOnTp/H5558jISEBAHDo0CGcOXMGo0aNUj6v9fHxwfvvv4/Dhw+3OGZhYSHc3d0xYsSIiz5uamoqBg8ejICAAPj4+ODtt99GWVkZAKBTp0548MEHYbVaMW7cOCxbtgzl5eXKsbt378aSJUtU85k+fTrKy8tx5syZK/p5XA/69++v2g4JCVE+1ti9ezdqamrQuXNn1c+vtLRUeT5LSkoQExOjGuP325WVlZg+fTp69eoFs9kMk8mEmpoa5TlsrYSEBOTm5uKXX34BAKxatQrx8fHK16/v3r0b6enpqrlarVY0NjaitLS0xTELCwsxcuTIiz5mfn4+xo0bh7CwMHTs2FFZp01znzVrFjIyMhAdHY0FCxZg+/btyrG7d+/GoUOH0LFjR2U+nTp1wrlz5y76ergWyAXf68DDDz8Mq9WK9evXY+PGjUhJScHf/vY3PPbYY5c9ZkJCAkaMGIGqqipkZ2fDaDRizJgxAM7fnQEA69evR9euXVXHXewLuIxG4yUfLyMjA08++ST+9re/wWKxoGPHjnjllVewY8cOpSYtLQ1z5sxBVlYWVq9ejWeeeQbZ2dkYOnQoampq8Pzzz+Oee+5pNraXl5dT53498vT0VG3rdDo0NjYCOP98hoSEIDc3t9lxzvy9iylTpuDEiRNYtmwZwsPDYTAYYLFYUFtb69Rcb775ZvTs2RMZGRmYNWsW1q5di/T0dKW/pqYGjzzyCObMmdPs2LCwsBbHvNT6O336NKxWK6xWK1atWoWAgACUlZXBarUqcx87dix++uknZGZmIjs7GyNHjkRiYiJeffVV1NTUYPDgwcqbpQtdyzdSSPhfAy4MOAD47rvv0KtXL7i7u+Omm25CfX09duzYgWHDhgEATpw4gZKSEvTp00c5JjQ0FDNnzsTMmTORnJyM//3f/20x/PV6PRoaGv5wTsOGDUNoaChWr16NDRs24L/+67+UAOnTpw8MBgPKysou+U7+QlFRUWhsbMSWLVsQFxfXrH/btm0YNmwYHn30UWVfS++aBg4ciIEDByI5ORkWiwUfffQRhg4dikGDBqGkpAQRERGtmo/4/wYNGoSKigp4eHgoF2F/LzIyEjt37sQDDzyg7Nu5c6eqZtu2bXjjjTdw5513Ajh/EfT48eOqGk9Pz1atv4SEBKxatQrdunWDm5sb4uPjVfMtKipy6rnu378/cnJyWvy69x9//BEnTpzA0qVLlT/pumvXrmZ1AQEBmDJlCqZMmYLhw4cjKSkJr776KgYNGoTVq1cjMDCwff3BGVd/7qR1TRd8n3jiCf7444/86KOP2KFDB7755ptKzfjx49mnTx9+8803LCws5JgxY1QXfOfOncusrCweOXKE+fn5jI2N5cSJE0k2/8x/27ZtBMCvv/6av/76K0+fPk1S/Zl/k7/85S/s06cPPTw8+M033zTr69y5M9PT03no0CHm5+fzH//4B9PT0y96rg8++CBDQ0O5du1aHjlyhJs3b1Yu6C5btowmk4lZWVksKSnhM888o7qgeOTIES5cuJDbt2/n0aNH+dVXX7Fz58584403SJJZWVn08PDg4sWLuW/fPhYVFfHjjz/mX/7yl8t7Yq4jI0aM4Ny5c1X7xo8fzylTppAkGxsbeeutt3LAgAH86quvWFpaym3btvHpp5/mzp07SZ6/4Gs0Gpmens4DBw7whRdeoMlkYnR0tDLmwIEDOWrUKBYVFfG7777j8OHDaTQaVeuqV69enDVrFsvLy3ny5EmSzT/zJ8mDBw8SAPv3789p06ap+nbv3k2j0cjExEQWFBTwwIED/Oyzz5pdk7jQ5s2b6ebmplzw3bNnD5cuXUry/IVovV7PpKQkHj58mJ9//jlvvPFGAmBBQQFJ8tlnn+Vnn33GgwcPct++ffzP//xPxsTEkCRPnz7NXr168fbbb+fWrVuVtf3YY4/x2LFjrXqOXEHC38VGjBjBRx99lDNnzqTJZKKfnx+ffvpp1cWjkydP8v7776fZbKbRaKTVauWBAweU/tmzZ7Nnz540GAwMCAjg/fffz+PHj5NsHv4kOXPmTHbu3JkAuGjRIpIth39RUREBMDw8XDUf8nxgvP7664yMjKSnpycDAgJotVqVu4xacvbsWT7xxBMMCQmhXq9nREQEV65cSfL8ReQHH3yQZrOZvr6+nDVrFhcuXKiEQkVFBSdMmKAcGx4ezueee44NDQ3K+FlZWRw2bBiNRiNNJhNjYmJUd01p1R+FP0na7XY+9thj7NKlCz09PRkaGsqEhASWlZUpNUuWLKG/vz99fHz40EMPcc6cORw6dKjS/8MPP3DIkCH08vJir169uGbNmmbr6osvvmBERAQ9PDwYHh5OsuXwJ8mYmBgC4KZNm5r1ff/99xw1ahR9fHzYoUMH9u/fn3/9618v+XP49NNPGR0dTb1eT39/f95zzz1K30cffcTu3bvTYDDQYrHwiy++UIX/Cy+8wJtuuolGo5GdOnXi+PHjeeTIEeX48vJyPvDAA/T396fBYGCPHj04ffp02my2S87JleT/8HWx22+/HdHR0fLVCqLdGTVqFIKDg/HBBx+4eiriMshn/kKIP3TmzBm8+eabsFqtcHd3x8cff4yvv/4a2dnZrp6auEwS/kKIP6TT6ZCZmYm//vWvOHfuHCIjI/Hpp5+2ePFetA/ysY8QQmiQ/E9eQgihQRL+QgihQRL+QgihQRL+QgihQRL+QgihQRL+QgihQRL+QgihQRL+QgihQf8PPlSKTX1wZVMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 400x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pos_samples = y_oversampling.value_counts()[1]\n",
    "neg_samples = y_oversampling.value_counts()[0]\n",
    "fig = plt.figure(figsize=(4, 3))\n",
    "\n",
    "cases = ['positive case', 'negative case']\n",
    "numbers = [pos_samples, neg_samples]\n",
    "plt.bar(cases, numbers, color=['red', 'blue'])\n",
    "for i, number in enumerate(numbers):\n",
    "    plt.text(i, number, number)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "HotRj6U0Lnu5",
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X_oh_oversampling, y_oversampling, train_size=0.8, test_size=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9o8OOx5VLr0D",
    "outputId": "84231da1-be80-4ccc-ae3f-413aa9143d53",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training_acc: 0.9839118129003426\n",
      "f1 score: 0.9837054918527458\n"
     ]
    }
   ],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=250)\n",
    "rf_model.fit(X_train, y_train)\n",
    "acc_train = accuracy_score(y_valid, rf_model.predict(X_valid))\n",
    "f1 = f1_score(y_valid, rf_model.predict(X_valid))\n",
    "print('training_acc: {}'.format(acc_train))\n",
    "print('f1 score: {}'.format(f1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wj9c3JAXOeyW"
   },
   "source": [
    "The result seems really good, lets hope the model performs well on the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "KmpULZD6OeEs",
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_test = test_data.drop(columns=['risk_flag'])\n",
    "X_test = pd.get_dummies(X_test)\n",
    "y_test = test_data['risk_flag']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "nS6CX3EUMMzH",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "testing_acc: 0.9182489451476793\n",
      "f1 score: 0.025157232704402514\n"
     ]
    }
   ],
   "source": [
    "acc_test = accuracy_score(y_test, rf_model.predict(X_test))\n",
    "f1_test = f1_score(y_test, rf_model.predict(X_test))\n",
    "print('testing_acc: {}'.format(acc_test))\n",
    "print('f1 score: {}'.format(f1_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Cq0REM12PiTQ"
   },
   "source": [
    "### **Remarks**\n",
    "* Again overall accuracy is good but the model still performs poorly with positive cases on the test set\n",
    "* Let's try SMOTE instead of Random Over Sampling this time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9EKWPXMPP7Wg",
    "tags": []
   },
   "source": [
    "### **SMOTE**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "T978ToqRPhkx",
    "tags": []
   },
   "outputs": [],
   "source": [
    "smote_sampler = SMOTENC(random_state=30, categorical_features=[3, 4, 5, 6, 7, 8])\n",
    "X_smote, y_smote = smote_sampler.fit_resample(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_oh_smote = pd.get_dummies(X_smote)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "id": "e4jUkVxzMMqy",
    "outputId": "2dcb450d-b029-4790-e761-323f7d1b107f",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEWCAYAAACOv5f1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA0rElEQVR4nO3de1hU9b4/8PdwmWEQZ0C5qoAphqkoXgLHMjuJjsYxrZ6jx3jKzDQN09IwaVeatcNT7XZuH7I6JXSxMHfbLookoWgpmRJ4g/CG4S4uqTEjXobb+/eHh/VzBRqj0ojr83qe7/O01vezvvNdzHfejLNWg44kIYQQQlPcXD0BIYQQfz4JfyGE0CAJfyGE0CAJfyGE0CAJfyGE0CAJfyGE0CAJfyGE0CAJfyGE0CAJfyGE0CAJfyGE0CCnwn/FihXo378/TCYTTCYTLBYLNmzYoPTffvvt0Ol0qjZz5kzVGGVlZYiPj4e3tzcCAwORlJSE+vp6VU1ubi4GDRoEg8GAiIgIpKenN5tLamoqunfvDi8vL8TGxuL777935lSEEELTPJwp7tatG5YuXYpevXqBJN577z2MHz8eBQUF6Nu3LwBg+vTpWLJkiXKMt7e38t8NDQ2Ij49HcHAwtm/fjvLycjzwwAPw9PTESy+9BAAoLS1FfHw8Zs6ciVWrViEnJwcPP/wwQkJCYLVaAQCrV6/GvHnz8OabbyI2Nhavv/46rFYrSkpKEBgY2OrzaWxsxC+//IKOHTtCp9M586MQQohrEkmcOnUKXbp0gZvbJd7f8wr5+fnxnXfeIUmOGDGCc+fOvWhtZmYm3dzcWFFRoexbsWIFTSYTHQ4HSXLBggXs27ev6rhJkybRarUq2zExMUxMTFS2Gxoa2KVLF6akpDg192PHjhGANGnSpF137dixY5fMP6fe+V+ooaEBa9aswenTp2GxWJT9q1atwocffojg4GCMGzcOzz77rPLuPy8vD1FRUQgKClLqrVYrZs2ahf3792PgwIHIy8tDXFyc6rGsVisef/xxAEBtbS3y8/ORnJys9Lu5uSEuLg55eXmXnLPD4YDD4VC2+X9faHrs2DGYTKbL+0EIIcQ1xG63IzQ0FB07drx0oVNvlUnu2bOHHTp0oLu7O81mM9evX6/0vfXWW8zKyuKePXv44YcfsmvXrrz77ruV/unTp3P06NGq8U6fPk0AzMzMJEn26tWLL730kqpm/fr1BMAzZ87w559/JgBu375dVZOUlMSYmJhLzn3RokUt/oa02WzO/hg07Y033mBUVBQ7duzIjh07cujQocrzR5IzZsxgjx496OXlRX9/f951110sLi5WjdHS8/Dxxx+ras6dO8enn36aYWFh1Ov1DA8P57vvvtvinD7++GMC4Pjx46/6+QrXkzXXejabrVW55vQ7/8jISBQWFsJms+Gf//wnpkyZgi1btqBPnz6YMWOGUhcVFYWQkBCMHDkShw8fRs+ePZ19qKsuOTkZ8+bNU7abfkMK5/zRtZ/BgwcjISEBYWFhOHnyJBYvXozRo0ejtLQU7u7uyjhpaWkYM2aMsu3r66t6nIkTJ6KyshLvvvsuIiIiUF5ejsbGxmbzOXr0KJ588kkMHz68zc5ZuJasuTZwpb9lRo4cyRkzZrTYV1NTQwDMysoiST777LMcMGCAqubIkSMEwB9++IEkOXz48GbXDVauXEmTyUSSdDgcdHd359q1a1U1DzzwAO+66y6n5t7a35Dij1147ef3du/eTQA8dOiQsg9As+fwQhs2bKDZbOaJEycu+bj19fUcNmwY33nnHU6ZMuWaexcm2o6suZa1Nteu+D7/xsZG1efoFyosLAQAhISEAAAsFgv27t2LqqoqpSY7Oxsmkwl9+vRRanJyclTjZGdnK9cV9Ho9Bg8erKppbGxETk6O6tqD+HM0NDQgIyOj2bWfJqdPn0ZaWhpuuOGGZv/KSkxMhL+/P2JiYrBy5UrlGgwAfPHFFxgyZAhefvlldO3aFTfeeCOefPJJnD17VjXGkiVLEBgYiGnTprXNCYprjqy5q8SZ3ygLFy7kli1bWFpayj179nDhwoXU6XTcuHEjDx06xCVLlnDXrl0sLS3l559/zh49evC2225Tjq+vr2e/fv04evRoFhYWMisriwEBAUxOTlZqjhw5Qm9vbyYlJbG4uJipqal0d3dX/vVAkhkZGTQYDExPT2dRURFnzJhBX19f1V1ErSHv/C/fpa79kGRqaio7dOhAAIyMjFS9AyPJJUuW8Ntvv+UPP/zApUuX0mAwcNmyZUq/1WqlwWBgfHw8d+zYwfXr1zM8PJwPPvigUvPNN9+wa9eu/PXXX0nymnwXJq4eWXOt09pccyr8H3roIYaHh1Ov1zMgIIAjR47kxo0bSZJlZWW87bbb2KlTJxoMBkZERDApKanZBI4ePcqxY8fSaDTS39+f8+fPZ11dnapm8+bNjI6Opl6vZ48ePZiWltZsLsuXL1cuysTExPC7775z5lRISvhfCYfDwYMHD3LXrl1cuHAh/f39uX//fqW/urqaBw4c4JYtWzhu3DgOGjSIZ8+eveh4zz77LLt166Zsjxo1il5eXqyurlb2ffrpp9TpdDxz5gztdju7d++uuuh3Lb4QxdUja6512iT8rzcS/lfPpa79OBwOent786OPPrro8evWrSMAnjt3juT5azg9e/ZU1RQVFREADxw4wIKCAgKgu7u70nQ6HXU6Hd3d3Zu96xPXH1lzLWuzu32EaMmlrv3w/JuMi/YD568P+fn5wWAwAABuueUWrFmzBjU1NfDx8QEAHDhwAG5ubujWrRt0Oh327t2rGuOZZ57BqVOnsGzZMrmLSwNkzV0ZCX/htOTkZIwdOxZhYWE4deoUPvroI+Tm5uKrr77CkSNHsHr1aowePRoBAQH497//jaVLl8JoNOLOO+8EAHz55ZeorKzE0KFD4eXlhezsbLz00kt48sknlce477778MILL2Dq1Kl4/vnncfz4cSQlJeGhhx6C0WgEAPTr1081r6bb9n6/X7R/subawJ/wr5Br1hV97ANotj0EMBygHmAAwJEAN/5f388AxwIMBOgJsBvA+wD+eMHxGwBGA/QB2AHgAIBvAmz43eMUA4wDaPy/ceYBPHOJeU0BOL4tz92FroGn3cXtIQLhBPQEAgiMJLDx//p+JjCWQCABTwLdCNxH4McLjt9AIJqAD4EOBAYQeJNAw+8ep5hAHAHj/40zj8CZS8xrCoHx19SSa22u6c4vLG2y2+0wm82w2WzOf72DfBGc9rjwpSLLTZsuZ8m1Ntfk+/yFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDJPyFEEKDnAr/FStWoH///jCZTDCZTLBYLNiwYYPSf+7cOSQmJqJz587w8fHBvffei8rKStUYZWVliI+Ph7e3NwIDA5GUlIT6+npVTW5uLgYNGgSDwYCIiAikp6c3m0tqaiq6d+8OLy8vxMbG4vvvv3fmVIQQQtuc+cPAX3zxBdevX88DBw6wpKSETz/9ND09Pblv3z6S5MyZMxkaGsqcnBzu2rWLQ4cO5bBhw5Tj6+vr2a9fP8bFxbGgoICZmZn09/dncnKyUnPkyBF6e3tz3rx5LCoq4vLly+nu7s6srCylJiMjg3q9nitXruT+/fs5ffp0+vr6srKy0pnTkT/gLq3t/5r2VeLqU5fWfpZca3Ptile0n58f33nnHVZXV9PT05Nr1qxR+oqLiwmAeXl5JMnMzEy6ubmxoqJCqVmxYgVNJhMdDgdJcsGCBezbt6/qMSZNmkSr1apsx8TEMDExUdluaGhgly5dmJKS4tTcJfyltfkr8Spx9alLaz9LrrW5dtmf+Tc0NCAjIwOnT5+GxWJBfn4+6urqEBcXp9T07t0bYWFhyMvLAwDk5eUhKioKQUFBSo3VaoXdbsf+/fuVmgvHaKppGqO2thb5+fmqGjc3N8TFxSk1F+NwOGC321VNCCG0yOnw37t3L3x8fGAwGDBz5kysXbsWffr0QUVFBfR6PXx9fVX1QUFBqKioAABUVFSogr+pv6nvUjV2ux1nz57F8ePH0dDQ0GJN0xgXk5KSArPZrLTQ0FBnT18IIa4LTod/ZGQkCgsLsWPHDsyaNQtTpkxBUVFRW8ztqktOTobNZlPasWPHXD0lIYRwCQ9nD9Dr9YiIiAAADB48GDt37sSyZcswadIk1NbWorq6WvXuv7KyEsHBwQCA4ODgZnflNN0NdGHN7+8QqqyshMlkgtFohLu7O9zd3VusaRrjYgwGAwwGg7OnLIQQ150rvs+/sbERDocDgwcPhqenJ3JycpS+kpISlJWVwWKxAAAsFgv27t2LqqoqpSY7Oxsmkwl9+vRRai4co6mmaQy9Xo/BgwerahobG5GTk6PUCCGE+APOXEVeuHAht2zZwtLSUu7Zs4cLFy6kTqfjxo0bSZ6/1TMsLIybNm3irl27aLFYaLFYlOObbvUcPXo0CwsLmZWVxYCAgBZv9UxKSmJxcTFTU1NbvNXTYDAwPT2dRUVFnDFjBn19fVV3EbWG3O0jrc1vvbhKXH3q0trPkmuTWz0feughhoeHU6/XMyAggCNHjlSCnyTPnj3LRx99lH5+fvT29ubdd9/N8vJy1RhHjx7l2LFjaTQa6e/vz/nz57Ourk5Vs3nzZkZHR1Ov17NHjx5MS0trNpfly5czLCyMer2eMTEx/O6775w5FZIS/tL+hFfiVeLqU5fWfpZca3NNd35haZPdbofZbIbNZoPJZHLuYJ2ubSYlrl0ufKnIctOmy1lyrc01+W4fIYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIKfCPyUlBTfffDM6duyIwMBATJgwASUlJaqa22+/HTqdTtVmzpypqikrK0N8fDy8vb0RGBiIpKQk1NfXq2pyc3MxaNAgGAwGREREID09vdl8UlNT0b17d3h5eSE2Nhbff/+9M6cjhBDaRSdYrVampaVx3759LCws5J133smwsDDW1NQoNSNGjOD06dNZXl6uNJvNpvTX19ezX79+jIuLY0FBATMzM+nv78/k5GSl5siRI/T29ua8efNYVFTE5cuX093dnVlZWUpNRkYG9Xo9V65cyf3793P69On09fVlZWVlq8/HZrMRgGp+rQZI01pzIVefurT2s+Ram2tXtKKrqqoIgFu2bFH2jRgxgnPnzr3oMZmZmXRzc2NFRYWyb8WKFTSZTHQ4HCTJBQsWsG/fvqrjJk2aRKvVqmzHxMQwMTFR2W5oaGCXLl2YkpJy0cc+d+4cbTab0o4dO9aqH1KLXL0qpLWPV+JV4upTl9Z+llxrw/+KPvO32WwAgE6dOqn2r1q1Cv7+/ujXrx+Sk5Nx5swZpS8vLw9RUVEICgpS9lmtVtjtduzfv1+piYuLU41ptVqRl5cHAKitrUV+fr6qxs3NDXFxcUpNS1JSUmA2m5UWGhp6mWcuhBDtm8flHtjY2IjHH38ct9xyC/r166fsv++++xAeHo4uXbpgz549eOqpp1BSUoJ//etfAICKigpV8ANQtisqKi5ZY7fbcfbsWfz2229oaGhosebHH3+86JyTk5Mxb948Zdtut8svACGEJl12+CcmJmLfvn349ttvVftnzJih/HdUVBRCQkIwcuRIHD58GD179rz8mV4FBoMBBoPBpXMQQohrwWV97DN79mysW7cOmzdvRrdu3S5ZGxsbCwA4dOgQACA4OBiVlZWqmqbt4ODgS9aYTCYYjUb4+/vD3d29xZqmMYQQQlycU+FPErNnz8batWuxadMm3HDDDX94TGFhIQAgJCQEAGCxWLB3715UVVUpNdnZ2TCZTOjTp49Sk5OToxonOzsbFosFAKDX6zF48GBVTWNjI3JycpQaIYQQl+DMVeRZs2bRbDYzNzdXdSvnmTNnSJKHDh3ikiVLuGvXLpaWlvLzzz9njx49eNtttyljNN3qOXr0aBYWFjIrK4sBAQEt3uqZlJTE4uJipqamtnirp8FgYHp6OouKijhjxgz6+vqq7iL6I3Krp7Q2v/XiKnH1qUtrP0uuTW71BNBiS0tLI0mWlZXxtttuY6dOnWgwGBgREcGkpKRmkzh69CjHjh1Lo9FIf39/zp8/n3V1daqazZs3Mzo6mnq9nj169FAe40LLly9nWFgY9Xo9Y2Ji+N133zlzOhL+0tr+lXiVuPrUpbWfJdfaXNOdX1jaZLfbYTabYbPZYDKZnDtYp2ubSYlrlwtfKrLctOlyllxrc02+20cIITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITRIwl8IITTIqfBPSUnBzTffjI4dOyIwMBATJkxASUmJqubcuXNITExE586d4ePjg3vvvReVlZWqmrKyMsTHx8Pb2xuBgYFISkpCfX29qiY3NxeDBg2CwWBAREQE0tPTm80nNTUV3bt3h5eXF2JjY/H99987czpCCKFddILVamVaWhr37dvHwsJC3nnnnQwLC2NNTY1SM3PmTIaGhjInJ4e7du3i0KFDOWzYMKW/vr6e/fr1Y1xcHAsKCpiZmUl/f38mJycrNUeOHKG3tzfnzZvHoqIiLl++nO7u7szKylJqMjIyqNfruXLlSu7fv5/Tp0+nr68vKysrW30+NpuNAGiz2Zz5MZwHSNNacyFXn7q09rPkWptrV7Siq6qqCIBbtmwhSVZXV9PT05Nr1qxRaoqLiwmAeXl5JMnMzEy6ubmxoqJCqVmxYgVNJhMdDgdJcsGCBezbt6/qsSZNmkSr1apsx8TEMDExUdluaGhgly5dmJKS0ur5S/hLc6q5kKtPXVr7WXKtzbUr+szfZrMBADp16gQAyM/PR11dHeLi4pSa3r17IywsDHl5eQCAvLw8REVFISgoSKmxWq2w2+3Yv3+/UnPhGE01TWPU1tYiPz9fVePm5oa4uDilpiUOhwN2u13VhBBCiy47/BsbG/H444/jlltuQb9+/QAAFRUV0Ov18PX1VdUGBQWhoqJCqbkw+Jv6m/ouVWO323H27FkcP34cDQ0NLdY0jdGSlJQUmM1mpYWGhjp/4kIIcR247PBPTEzEvn37kJGRcTXn06aSk5Nhs9mUduzYMVdPSQghXMLjcg6aPXs21q1bh61bt6Jbt27K/uDgYNTW1qK6ulr17r+yshLBwcFKze/vymm6G+jCmt/fIVRZWQmTyQSj0Qh3d3e4u7u3WNM0RksMBgMMBoPzJyyEENcZp975k8Ts2bOxdu1abNq0CTfccIOqf/DgwfD09EROTo6yr6SkBGVlZbBYLAAAi8WCvXv3oqqqSqnJzs6GyWRCnz59lJoLx2iqaRpDr9dj8ODBqprGxkbk5OQoNUIIIS7BmavIs2bNotlsZm5uLsvLy5V25swZpWbmzJkMCwvjpk2buGvXLlosFlosFqW/6VbP0aNHs7CwkFlZWQwICGjxVs+kpCQWFxczNTW1xVs9DQYD09PTWVRUxBkzZtDX11d1F9Efkbt9pLX5rRdXiatPXVr7WXJtcqsngBZbWlqaUnP27Fk++uij9PPzo7e3N++++26Wl5erxjl69CjHjh1Lo9FIf39/zp8/n3V1daqazZs3Mzo6mnq9nj169FA9RpPly5czLCyMer2eMTEx/O6775w5HQl/aW3/SrxKXH3q0trPkmttrunOLyxtstvtMJvNsNlsMJlMzh2s07XNpMS1y4UvFVlu2nQ5S661uSbf7SOEEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBok4S+EEBrkdPhv3boV48aNQ5cuXaDT6fDZZ5+p+h988EHodDpVGzNmjKrm5MmTSEhIgMlkgq+vL6ZNm4aamhpVzZ49ezB8+HB4eXkhNDQUL7/8crO5rFmzBr1794aXlxeioqKQmZnp7OkIIYQmOR3+p0+fxoABA5CamnrRmjFjxqC8vFxpH3/8sao/ISEB+/fvR3Z2NtatW4etW7dixowZSr/dbsfo0aMRHh6O/Px8vPLKK1i8eDHefvttpWb79u2YPHkypk2bhoKCAkyYMAETJkzAvn37nD0lIYTQHl4BAFy7dq1q35QpUzh+/PiLHlNUVEQA3Llzp7Jvw4YN1Ol0/Pnnn0mSb7zxBv38/OhwOJSap556ipGRkcr2xIkTGR8frxo7NjaWjzzySKvnb7PZCIA2m63VxygAaVprLuTqU5fWfpZca3OtTT7zz83NRWBgICIjIzFr1iycOHFC6cvLy4Ovry+GDBmi7IuLi4Obmxt27Nih1Nx2223Q6/VKjdVqRUlJCX777TelJi4uTvW4VqsVeXl5F52Xw+GA3W5XNSGE0KKrHv5jxozB+++/j5ycHPzP//wPtmzZgrFjx6KhoQEAUFFRgcDAQNUxHh4e6NSpEyoqKpSaoKAgVU3T9h/VNPW3JCUlBWazWWmhoaFXdrJCCNFOeVztAf/7v/9b+e+oqCj0798fPXv2RG5uLkaOHHm1H84pycnJmDdvnrJtt9vlF4AQQpPa/FbPHj16wN/fH4cOHQIABAcHo6qqSlVTX1+PkydPIjg4WKmprKxU1TRt/1FNU39LDAYDTCaTqgkhhBa1efj/+9//xokTJxASEgIAsFgsqK6uRn5+vlKzadMmNDY2IjY2VqnZunUr6urqlJrs7GxERkbCz89PqcnJyVE9VnZ2NiwWS1ufkhBCtH/OXkk+deoUCwoKWFBQQAB87bXXWFBQwJ9++omnTp3ik08+yby8PJaWlvLrr7/moEGD2KtXL547d04ZY8yYMRw4cCB37NjBb7/9lr169eLkyZOV/urqagYFBfH+++/nvn37mJGRQW9vb7711ltKzbZt2+jh4cFXX32VxcXFXLRoET09Pbl3795Wn4vc7SPNqeZCrj51ae1nybU215wefvPmzQTQrE2ZMoVnzpzh6NGjGRAQQE9PT4aHh3P69OmsqKhQjXHixAlOnjyZPj4+NJlMnDp1Kk+dOqWq2b17N2+99VYaDAZ27dqVS5cubTaXTz75hDfeeCP1ej379u3L9evXO3UuEv7SnGou5OpTl9Z+llxrc013fmFpk91uh9lshs1mc/7zf52ubSYlrl0ufKnIctOmy1lyrc01+W4fIYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIAl/IYTQIKfDf+vWrRg3bhy6dOkCnU6Hzz77TNVPEs899xxCQkJgNBoRFxeHgwcPqmpOnjyJhIQEmEwm+Pr6Ytq0aaipqVHV7NmzB8OHD4eXlxdCQ0Px8ssvN5vLmjVr0Lt3b3h5eSEqKgqZmZnOno4QQmiS0+F/+vRpDBgwAKmpqS32v/zyy/jHP/6BN998Ezt27ECHDh1gtVpx7tw5pSYhIQH79+9HdnY21q1bh61bt2LGjBlKv91ux+jRoxEeHo78/Hy88sorWLx4Md5++22lZvv27Zg8eTKmTZuGgoICTJgwARMmTMC+ffucPSUhhNAeXgEAXLt2rbLd2NjI4OBgvvLKK8q+6upqGgwGfvzxxyTJoqIiAuDOnTuVmg0bNlCn0/Hnn38mSb7xxhv08/Ojw+FQap566ilGRkYq2xMnTmR8fLxqPrGxsXzkkUdaPX+bzUYAtNlsrT5GAUjTWnMhV5+6tPaz5Fqba1f1M//S0lJUVFQgLi5O2Wc2mxEbG4u8vDwAQF5eHnx9fTFkyBClJi4uDm5ubtixY4dSc9ttt0Gv1ys1VqsVJSUl+O2335SaCx+nqabpcVricDhgt9tVTQghtOiqhn9FRQUAICgoSLU/KChI6auoqEBgYKCq38PDA506dVLVtDTGhY9xsZqm/pakpKTAbDYrLTQ01NlTFEKI64Km7vZJTk6GzWZT2rFjx1w9JSGEcImrGv7BwcEAgMrKStX+yspKpS84OBhVVVWq/vr6epw8eVJV09IYFz7GxWqa+ltiMBhgMplUTQghtOiqhv8NN9yA4OBg5OTkKPvsdjt27NgBi8UCALBYLKiurkZ+fr5Ss2nTJjQ2NiI2Nlap2bp1K+rq6pSa7OxsREZGws/PT6m58HGaapoeRwghxCU4eyX51KlTLCgoYEFBAQHwtddeY0FBAX/66SeS5NKlS+nr68vPP/+ce/bs4fjx43nDDTfw7NmzyhhjxozhwIEDuWPHDn777bfs1asXJ0+erPRXV1czKCiI999/P/ft28eMjAx6e3vzrbfeUmq2bdtGDw8PvvrqqywuLuaiRYvo6enJvXv3tvpc5G4faU41F3L1qUtrP0uutbnm9PCbN28mgGZtypQpJM/f7vnss88yKCiIBoOBI0eOZElJiWqMEydOcPLkyfTx8aHJZOLUqVN56tQpVc3u3bt566230mAwsGvXrly6dGmzuXzyySe88cYbqdfr2bdvX65fv96pc5Hwl+ZUcyFXn7q09rPkWptruvMLS5vsdjvMZjNsNpvzn//rdG0zKXHtcuFLRZabNl3Okmttrmnqbh8hhBDnSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGXfXwX7x4MXQ6nar17t1b6T937hwSExPRuXNn+Pj44N5770VlZaVqjLKyMsTHx8Pb2xuBgYFISkpCfX29qiY3NxeDBg2CwWBAREQE0tPTr/apCCHEdatN3vn37dsX5eXlSvv222+VvieeeAJffvkl1qxZgy1btuCXX37BPffco/Q3NDQgPj4etbW12L59O9577z2kp6fjueeeU2pKS0sRHx+P//iP/0BhYSEef/xxPPzww/jqq6/a4nSEEOL6w6ts0aJFHDBgQIt91dXV9PT05Jo1a5R9xcXFBMC8vDySZGZmJt3c3FhRUaHUrFixgiaTiQ6HgyS5YMEC9u3bVzX2pEmTaLVanZqrzWYjANpsNqeOI0kC0rTWXMjVpy6t/Sy51uZam7zzP3jwILp06YIePXogISEBZWVlAID8/HzU1dUhLi5Oqe3duzfCwsKQl5cHAMjLy0NUVBSCgoKUGqvVCrvdjv379ys1F47RVNM0xsU4HA7Y7XZVE0IILbrq4R8bG4v09HRkZWVhxYoVKC0txfDhw3Hq1ClUVFRAr9fD19dXdUxQUBAqKioAABUVFargb+pv6rtUjd1ux9mzZy86t5SUFJjNZqWFhoZe6ekKIUS75HG1Bxw7dqzy3/3790dsbCzCw8PxySefwGg0Xu2Hc0pycjLmzZunbNvtdvkFIITQpDa/1dPX1xc33ngjDh06hODgYNTW1qK6ulpVU1lZieDgYABAcHBws7t/mrb/qMZkMl3yF4zBYIDJZFI1IYTQojYP/5qaGhw+fBghISEYPHgwPD09kZOTo/SXlJSgrKwMFosFAGCxWLB3715UVVUpNdnZ2TCZTOjTp49Sc+EYTTVNYwghhPgDl3c9+eLmz5/P3NxclpaWctu2bYyLi6O/vz+rqqpIkjNnzmRYWBg3bdrEXbt20WKx0GKxKMfX19ezX79+HD16NAsLC5mVlcWAgAAmJycrNUeOHKG3tzeTkpJYXFzM1NRUuru7Mysry6m5yt0+0tr81ourxNWnLq39LLnW5tpVX9GTJk1iSEgI9Xo9u3btykmTJvHQoUNK/9mzZ/noo4/Sz8+P3t7evPvuu1leXq4a4+jRoxw7diyNRiP9/f05f/581tXVqWo2b97M6Oho6vV69ujRg2lpaU7PVcJfWpu/Eq8SV5+6tPaz5Fqba7rzC0ub7HY7zGYzbDab85//63RtMylx7XLhS0WWmzZdzpJrba7Jd/sIIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGSfgLIYQGtfvwT01NRffu3eHl5YXY2Fh8//33rp6SEEJc89p1+K9evRrz5s3DokWL8MMPP2DAgAGwWq2oqqpy9dSEEOKa1q7D/7XXXsP06dMxdepU9OnTB2+++Sa8vb2xcuVKV09NCCGuaR6unsDlqq2tRX5+PpKTk5V9bm5uiIuLQ15eXovHOBwOOBwOZdtmswEA7HZ7205WXB9knYg/2eUsuaY8I3nJunYb/sePH0dDQwOCgoJU+4OCgvDjjz+2eExKSgqef/75ZvtDQ0PbZI7iOmM2u3oGQmOuZMmdOnUK5ksM0G7D/3IkJydj3rx5ynZjYyNOnjyJzp07Q6fTuXBm7YPdbkdoaCiOHTsGk8nk6ukIDZA15zySOHXqFLp06XLJunYb/v7+/nB3d0dlZaVqf2VlJYKDg1s8xmAwwGAwqPb5+vq21RSvWyaTSV6I4k8la845l3rH36TdXvDV6/UYPHgwcnJylH2NjY3IycmBxWJx4cyEEOLa127f+QPAvHnzMGXKFAwZMgQxMTF4/fXXcfr0aUydOtXVUxNCiGtauw7/SZMm4ddff8Vzzz2HiooKREdHIysrq9lFYHF1GAwGLFq0qNlHZ0K0FVlzbUfHP7ofSAghxHWn3X7mL4QQ4vJJ+AshhAZJ+AshhAZJ+AshhAZJ+F/ncnNzodPpUF1dfcm67t274/XXX/9T5iTEhRYvXozo6GhXT0Nz5G6f61xtbS1OnjyJoKAg6HQ6pKen4/HHH2/2y+DXX39Fhw4d4O3t7ZqJCk3Q6XRYu3YtJkyYoOyrqamBw+FA586dXTcxDWrX9/mLP6bX6y/6dRcXCggI+BNmI0RzPj4+8PHxcfU0NEc+9nGx22+/HbNnz8bs2bNhNpvh7++PZ599VvV1rL/99hseeOAB+Pn5wdvbG2PHjsXBgweV/p9++gnjxo2Dn58fOnTogL59+yIzMxOA+mOf3NxcTJ06FTabDTqdDjqdDosXLwag/tjnvvvuw6RJk1TzrKurg7+/P95//30A579KIyUlBTfccAOMRiMGDBiAf/7zn5c8V4fDgaeeegqhoaEwGAyIiIjAu+++CwBoaGjAtGnTlPEiIyOxbNky1fG5ubmIiYlBhw4d4Ovri1tuuQU//fST0v/5559j0KBB8PLyQo8ePfD888+jvr7eiWfj+nT77bdjzpw5WLBgATp16oTg4GDleW9SXV2Nhx9+GAEBATCZTLjjjjuwe/duVc2LL76IwMBAdOzYEQ8//DAWLlyo+rhm586dGDVqFPz9/WE2mzFixAj88MMPSn/37t0BAHfffTd0Op2yfeHHPhs3boSXl1ezf5nOnTsXd9xxh7L97bffYvjw4TAajQgNDcWcOXNw+vTpS/4cvvzyS9x8883w8vKCv78/7r77bqXvgw8+wJAhQ9CxY0cEBwfjvvvuU/1RqN9++w0JCQkICAiA0WhEr169kJaWpvQfO3YMEydOhK+vLzp16oTx48fj6NGjl5yPy1G41IgRI+jj48O5c+fyxx9/5Icffkhvb2++/fbbSs1dd93Fm266iVu3bmVhYSGtVisjIiJYW1tLkoyPj+eoUaO4Z88eHj58mF9++SW3bNlCkty8eTMB8LfffqPD4eDrr79Ok8nE8vJylpeX89SpUyTJ8PBw/v3vfydJrlu3jkajUekjyS+//JJGo5F2u50k+eKLL7J3797Mysri4cOHmZaWRoPBwNzc3Iue68SJExkaGsp//etfPHz4ML/++mtmZGSQJGtra/ncc89x586dPHLkiPJzWL16NUmyrq6OZrOZTz75JA8dOsSioiKmp6fzp59+Iklu3bqVJpOJ6enpPHz4MDdu3Mju3btz8eLFV+NpatdGjBhBk8nExYsX88CBA3zvvfeo0+m4ceNGpSYuLo7jxo3jzp07eeDAAc6fP5+dO3fmiRMnSJIffvghvby8uHLlSpaUlPD555+nyWTigAEDlDFycnL4wQcfsLi4mEVFRZw2bRqDgoKUNVNVVUUATEtLY3l5OauqqkiSixYtUsapr69nUFAQ33nnHWXc3+87dOgQO3TowL///e88cOAAt23bxoEDB/LBBx+86M9g3bp1dHd353PPPceioiIWFhbypZdeUvrfffddZmZm8vDhw8zLy6PFYuHYsWOV/sTEREZHR3Pnzp0sLS1ldnY2v/jiC5Ln1+5NN93Ehx56iHv27GFRURHvu+8+RkZG0uFwXM5T9qeQ8HexESNG8KabbmJjY6Oy76mnnuJNN91Ekjxw4AABcNu2bUr/8ePHaTQa+cknn5Ako6KiLhpyF4Y/SaalpdFsNjeruzD86+rq6O/vz/fff1/pnzx5MidNmkSSPHfuHL29vbl9+3bVGNOmTePkyZNbnEdJSQkBMDs7+xI/DbXExETee++9JMkTJ04QwEV/uYwcOVL1YibJDz74gCEhIa1+vOvViBEjeOutt6r23XzzzXzqqadIkt988w1NJhPPnTunqunZsyffeustkmRsbCwTExNV/bfccosq/H+voaGBHTt25JdffqnsA8C1a9eq6i4Mf5KcO3cu77jjDmX7q6++osFgUNbwtGnTOGPGDNUY33zzDd3c3Hj27NkW52KxWJiQkHDRuf7ezp07CUB5AzRu3DhOnTq1xdoPPviAkZGRqteww+Gg0WjkV1991erH/LPJxz7XgKFDh6r+noDFYsHBgwfR0NCA4uJieHh4IDY2Vunv3LkzIiMjUVxcDACYM2cOXnzxRdxyyy1YtGgR9uzZc0Xz8fDwwMSJE7Fq1SoAwOnTp/H5558jISEBAHDo0CGcOXMGo0aNUj6v9fHxwfvvv4/Dhw+3OGZhYSHc3d0xYsSIiz5uamoqBg8ejICAAPj4+ODtt99GWVkZAKBTp0548MEHYbVaMW7cOCxbtgzl5eXKsbt378aSJUtU85k+fTrKy8tx5syZK/p5XA/69++v2g4JCVE+1ti9ezdqamrQuXNn1c+vtLRUeT5LSkoQExOjGuP325WVlZg+fTp69eoFs9kMk8mEmpoa5TlsrYSEBOTm5uKXX34BAKxatQrx8fHK16/v3r0b6enpqrlarVY0NjaitLS0xTELCwsxcuTIiz5mfn4+xo0bh7CwMHTs2FFZp01znzVrFjIyMhAdHY0FCxZg+/btyrG7d+/GoUOH0LFjR2U+nTp1wrlz5y76ergWyAXf68DDDz8Mq9WK9evXY+PGjUhJScHf/vY3PPbYY5c9ZkJCAkaMGIGqqipkZ2fDaDRizJgxAM7fnQEA69evR9euXVXHXewLuIxG4yUfLyMjA08++ST+9re/wWKxoGPHjnjllVewY8cOpSYtLQ1z5sxBVlYWVq9ejWeeeQbZ2dkYOnQoampq8Pzzz+Oee+5pNraXl5dT53498vT0VG3rdDo0NjYCOP98hoSEIDc3t9lxzvy9iylTpuDEiRNYtmwZwsPDYTAYYLFYUFtb69Rcb775ZvTs2RMZGRmYNWsW1q5di/T0dKW/pqYGjzzyCObMmdPs2LCwsBbHvNT6O336NKxWK6xWK1atWoWAgACUlZXBarUqcx87dix++uknZGZmIjs7GyNHjkRiYiJeffVV1NTUYPDgwcqbpQtdyzdSSPhfAy4MOAD47rvv0KtXL7i7u+Omm25CfX09duzYgWHDhgEATpw4gZKSEvTp00c5JjQ0FDNnzsTMmTORnJyM//3f/20x/PV6PRoaGv5wTsOGDUNoaChWr16NDRs24L/+67+UAOnTpw8MBgPKysou+U7+QlFRUWhsbMSWLVsQFxfXrH/btm0YNmwYHn30UWVfS++aBg4ciIEDByI5ORkWiwUfffQRhg4dikGDBqGkpAQRERGtmo/4/wYNGoSKigp4eHgoF2F/LzIyEjt37sQDDzyg7Nu5c6eqZtu2bXjjjTdw5513Ajh/EfT48eOqGk9Pz1atv4SEBKxatQrdunWDm5sb4uPjVfMtKipy6rnu378/cnJyWvy69x9//BEnTpzA0qVLlT/pumvXrmZ1AQEBmDJlCqZMmYLhw4cjKSkJr776KgYNGoTVq1cjMDCwff3BGVd/7qR1TRd8n3jiCf7444/86KOP2KFDB7755ptKzfjx49mnTx9+8803LCws5JgxY1QXfOfOncusrCweOXKE+fn5jI2N5cSJE0k2/8x/27ZtBMCvv/6av/76K0+fPk1S/Zl/k7/85S/s06cPPTw8+M033zTr69y5M9PT03no0CHm5+fzH//4B9PT0y96rg8++CBDQ0O5du1aHjlyhJs3b1Yu6C5btowmk4lZWVksKSnhM888o7qgeOTIES5cuJDbt2/n0aNH+dVXX7Fz58584403SJJZWVn08PDg4sWLuW/fPhYVFfHjjz/mX/7yl8t7Yq4jI0aM4Ny5c1X7xo8fzylTppAkGxsbeeutt3LAgAH86quvWFpaym3btvHpp5/mzp07SZ6/4Gs0Gpmens4DBw7whRdeoMlkYnR0tDLmwIEDOWrUKBYVFfG7777j8OHDaTQaVeuqV69enDVrFsvLy3ny5EmSzT/zJ8mDBw8SAPv3789p06ap+nbv3k2j0cjExEQWFBTwwIED/Oyzz5pdk7jQ5s2b6ebmplzw3bNnD5cuXUry/IVovV7PpKQkHj58mJ9//jlvvPFGAmBBQQFJ8tlnn+Vnn33GgwcPct++ffzP//xPxsTEkCRPnz7NXr168fbbb+fWrVuVtf3YY4/x2LFjrXqOXEHC38VGjBjBRx99lDNnzqTJZKKfnx+ffvpp1cWjkydP8v7776fZbKbRaKTVauWBAweU/tmzZ7Nnz540GAwMCAjg/fffz+PHj5NsHv4kOXPmTHbu3JkAuGjRIpIth39RUREBMDw8XDUf8nxgvP7664yMjKSnpycDAgJotVqVu4xacvbsWT7xxBMMCQmhXq9nREQEV65cSfL8ReQHH3yQZrOZvr6+nDVrFhcuXKiEQkVFBSdMmKAcGx4ezueee44NDQ3K+FlZWRw2bBiNRiNNJhNjYmJUd01p1R+FP0na7XY+9thj7NKlCz09PRkaGsqEhASWlZUpNUuWLKG/vz99fHz40EMPcc6cORw6dKjS/8MPP3DIkCH08vJir169uGbNmmbr6osvvmBERAQ9PDwYHh5OsuXwJ8mYmBgC4KZNm5r1ff/99xw1ahR9fHzYoUMH9u/fn3/9618v+XP49NNPGR0dTb1eT39/f95zzz1K30cffcTu3bvTYDDQYrHwiy++UIX/Cy+8wJtuuolGo5GdOnXi+PHjeeTIEeX48vJyPvDAA/T396fBYGCPHj04ffp02my2S87JleT/8HWx22+/HdHR0fLVCqLdGTVqFIKDg/HBBx+4eiriMshn/kKIP3TmzBm8+eabsFqtcHd3x8cff4yvv/4a2dnZrp6auEwS/kKIP6TT6ZCZmYm//vWvOHfuHCIjI/Hpp5+2ePFetA/ysY8QQmiQ/E9eQgihQRL+QgihQRL+QgihQRL+QgihQRL+QgihQRL+QgihQRL+QgihQRL+QgihQf8PPlSKTX1wZVMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 400x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pos_samples = y_smote.value_counts()[1]\n",
    "neg_samples = y_smote.value_counts()[0]\n",
    "fig = plt.figure(figsize=(4, 3))\n",
    "\n",
    "cases = ['positive case', 'negative case']\n",
    "numbers = [pos_samples, neg_samples]\n",
    "plt.bar(cases, numbers, color=['red', 'blue'])\n",
    "for i, number in enumerate(numbers):\n",
    "    plt.text(i, number, number)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "pDVHfTw3Lrxn",
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X_oh_smote, y_smote, train_size=0.8, test_size=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "aoE1OvtyLrvN",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training_acc: 0.7285863250409653\n",
      "f1 score: 0.7293925441853557\n"
     ]
    }
   ],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=250)\n",
    "rf_model.fit(X_train, y_train)\n",
    "acc_train = accuracy_score(y_valid, rf_model.predict(X_valid))\n",
    "f1 = f1_score(y_valid, rf_model.predict(X_valid))\n",
    "print('training_acc: {}'.format(acc_train))\n",
    "print('f1 score: {}'.format(f1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "4cz96Hg3Lrsi",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "testing_acc: 0.7792721518987342\n",
      "f1 score: 0.13711340206185565\n"
     ]
    }
   ],
   "source": [
    "acc_test = accuracy_score(y_test, rf_model.predict(X_test))\n",
    "f1_test = f1_score(y_test, rf_model.predict(X_test))\n",
    "print('testing_acc: {}'.format(acc_test))\n",
    "print('f1 score: {}'.format(f1_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Remarks**\n",
    "* This time the performance is not as good for the training set, but the model performs significantly better with positive cases on the test set\n",
    "* The increase of the dataset in terms of variability provided by SMOTE clearly helps in this case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Conclusion**\n",
    "\n",
    "* A performance of a predictive ML model strongly depends on how well the datast is prepared\n",
    "* Unbalanced data is a challenging problem, and the best way is usually to gather more data.\n",
    "* For a temporary solution to unbalanced data, over sampling method like SMOTE can help in some cases.  "
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "default:Python",
   "language": "python",
   "name": "conda-env-default-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}