06. Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Grand Poobah Classification Notebook\n",
    "In this notebook we will cover uses of scikit-learn for many different purposes including:\n",
    "- Cross Validation and metrics for scoring (including ROC and confusion matrices)\n",
    "- Pipelines\n",
    "- Common (nonlinear model) Classification:\n",
    " - K-Nearest Neighbors\n",
    " - Naive Bayes\n",
    " - Decision Trees\n",
    "- Bagging (Random Forests)\n",
    "- Boosting (Gradient of Error)\n",
    "- Statistical Comparison Questions\n",
    "\n",
    "We will look at some of the advanced features of scikit and look at many different ways of going about using the system to its fullest potential.\n",
    "\n",
    "- http://en.wikipedia.org/wiki/Grand_Poobah\n",
    "\n",
    "For this assignment we will use the LFW faces dataset and try to perform recognition of different individual's faces with several different classifiers. \n",
    "\n",
    "You can look up the dataset here:\n",
    " - http://vis-www.cs.umass.edu/lfw/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fetch the images for the dataset\n",
    "# this will take a long time the first run because it needs to download\n",
    "# after the first time, the dataset will be save to your disk (in sklearn package somewhere) \n",
    "# if this does not run, you may need additional libraries installed on your system (you will need to figure out which ones!!)\n",
    "from sklearn.datasets import fetch_lfw_people\n",
    "\n",
    "# only load people that have N or more examples in the dataset\n",
    "lfw_people = fetch_lfw_people(min_faces_per_person=20, resize=0.4) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of the dataset: (2489, 1850)\n",
      "Number of unique classes: 43\n",
      "['George Robertson' 'George W Bush' 'Gerhard Schroeder'\n",
      " 'Gloria Macapagal Arroyo' 'Gray Davis' 'Hamid Karzai' 'Hans Blix'\n",
      " 'Hugo Chavez' 'Igor Ivanov' 'Jack Straw']\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGFdJREFUeJzt3XuUZWV55/Hvj6YRRQSBlmG4pBHbUeISNB2DYgygY7wQIUuJulBb7bEnE0ww6iAmZoyznBnQjKiJ0ekBpHWpgLfAKIoMFy8xKt1cRSS02EIvEHAEvCAq+Mwf+y27LHZVnb6cOqervp+1zjp7v/vdez+1q+o8593v3u9OVSFJ0lQ7jDoASdJ4MkFIknqZICRJvUwQkqReJghJUi8ThCSplwlCktTLBCFJ6mWCkCT12nHUAWyNvfbaq5YuXTrqMCRpu7Ju3bofVNWS2ept1wli6dKlrF27dtRhSNJ2Jcn3BqnnKSZJUi8ThCSplwlCktTLBCFJ6mWCkCT1MkFIknqZICRJvUwQkqReJghJUq/t+k5qzb2lJ3922mUbTnn+HEYiadhsQUiSepkgJEm9TBCSpF4mCElSLxOEJKmXCUKS1MsEIUnqNdQEkWRDkmuTXJVkbSvbI8lFSW5s749s5Uny3iTrk1yT5MnDjE2SNLO5aEEcWVWHVtXyNn8ycHFVLQMubvMAzwWWtdcq4P1zEJskaRqjOMV0DLCmTa8Bjp1U/qHqfA3YPck+I4hPksTwE0QBX0iyLsmqVrZ3Vd0G0N4f1cr3BW6ZtO7GVvYbkqxKsjbJ2jvvvHOIoUvSwjbssZgOr6pbkzwKuCjJt2eom56yelBB1WpgNcDy5csftFyStG0MtQVRVbe29zuATwNPAW6fOHXU3u9o1TcC+09afT/g1mHGJ0ma3tASRJJdkuw6MQ08G/gmcD6wolVbAZzXps8HXtGuZjoMuGfiVJQkae4N8xTT3sCnk0zs56NV9fkklwPnJlkJ3Awc1+pfADwPWA/cC7xqiLFJkmYxtARRVTcBh/SU/z/gmT3lBZwwrHgkSZvHO6klSb1MEJKkXiYISVIvE4QkqZcJQpLUywQhSeplgpAk9TJBSJJ6mSAkSb1MEJKkXiYISVIvE4QkqZcJQpLUywQhSeplgpAk9TJBSJJ6mSAkSb1MEJKkXiYISVIvE4QkqZcJQpLUywQhSeplgpAk9TJBSJJ6mSAkSb1MEJKkXiYISVIvE4QkqZcJQpLUywQhSeo19ASRZFGSK5N8ps0fmOTrSW5Mck6SnVr5Q9r8+rZ86bBjkyRNby5aECcC10+aPxU4raqWAXcBK1v5SuCuqnoMcFqrJ0kakaEmiCT7Ac8HTm/zAY4CPtGqrAGObdPHtHna8me2+pKkERh2C+LdwEnAr9r8nsDdVXV/m98I7Num9wVuAWjL72n1JUkjMLQEkeRo4I6qWje5uKdqDbBs8nZXJVmbZO2dd965DSKVJPUZZgvicOAFSTYAZ9OdWno3sHuSHVud/YBb2/RGYH+Atnw34IdTN1pVq6tqeVUtX7JkyRDDl6SFbWgJoqreXFX7VdVS4CXAJVV1PHAp8KJWbQVwXps+v83Tll9SVQ9qQUiS5sYo7oN4E/D6JOvp+hjOaOVnAHu28tcDJ48gNklSs+PsVbZeVV0GXNambwKe0lPnPuC4uYhHkjQ776SWJPUyQUiSepkgJEm9Zk0QSf4uyW/PRTCSpPExSAvi28DqNoDenybZbdhBSZJGb9YEUVWnV9XhwCuApcA1ST6a5MhhBydJGp2B+iCSLAIe114/AK6mu5fh7CHGJkkaoVnvg0jyLuAFwMXAf6+qb7RFpya5YZjBSZJGZ5Ab5b4JvKWq7u1Z9qAb3iRJ88Mgp5juAhZPzCTZPcmxAFV1z7ACkySN1iAJ4q2TE0FV3Q28dXghSZLGwSAJoq/OnIzhJEkanUESxNok70pyUJJHJzkNWDfrWpKk7dogCeLPgV8A5wAfB+4DThhmUJKk0Zv1VFFV/RSfzSBJC84g90E8Fngj3V3Uv65fVUcNLyxJ0qgN0tn8ceADwOnAA8MNR5I0LgZJEPdX1fuHHokkaawM0kn9f5L8WZJ9kuwx8Rp6ZJKkkRqkBbGivf/nSWUFPHrbhyNJGheDXMV04FwEIkkaL4M8Ue5hSd6SZHWbX5bk6OGHJkkapUH6ID5Id6Pc09r8RuDtQ4tIkjQWBkkQB1XVO4BfAlTVz4AMNSpJ0sgNkiB+keShdB3TJDkI+PlQo5IkjdwgVzG9Ffg8sH+SjwCHA68cZlCSpNEb5Cqmi5JcARxGd2rpxKr6wdAjkySN1CBjMT2jTf64vR+chKr60vDCkiSN2iCnmCbfILcz3XOo1wEO1idJ89ggp5j+aPJ8kv2BdwwtIknSWBjkKqapNgJPmK1Skp2TfCPJ1UmuS/K2Vn5gkq8nuTHJOUl2auUPafPr2/KlWxCbJGkbGaQP4u9pl7jSJZRDgasH2PbPgaOq6idJFgNfSfI54PXAaVV1dpIPACuB97f3u6rqMUleApwKvHizfyJJ0jYx0DOp6foc1gH/Arypql4220rV+UmbXdxeRdd38YlWvgY4tk0f0+Zpy5+ZxBvyJGlEBumDWDNbnekkWUSXWB4DvA/4DnB3Vd3fqmwE9m3T+wK3tH3en+QeYE/AS2olaQQGOcV0LZtOMf3GIrqGwhOnW7eqHgAOTbI78Gng8X3VJm1vumWT41kFrAI44IADZg5ekrTFBrnM9XPt/cPt/XjgXjadDppVVd2d5DK6m+12T7Jja0XsB9zaqm0E9gc2JtkR2A34Yc+2VgOrAZYvX96XuCRJ28AgfRCHV9VJVXVte50M/GFVfa+qvjfdSkmWtJYDbSynZwHXA5cCL2rVVgDntenz2fRwohcBl1SVCUCSRmSQFsQuSZ5eVV8BSPI0YJcB1tsHWNP6IXYAzq2qzyT5FnB2krcDVwJntPpnAB9Osp6u5fCSzfxZJEnb0CAJYiVwZpLd6PoE7gFePdtKVXUN8KSe8pvo7saeWn4fcNwA8UiS5sAgVzGtAw5J8gggVXXP8MOSJI3aII8c3TvJGcA5VXVPkoOTrJyD2CRJIzRIJ/VZwIXAv23z/wq8blgBSZLGwyAJYq+qOhf4FXQ3sQEPDDUqSdLIDZIgfppkTzY9cvQwuo5qSdI8NshVTK+nu0fhoCT/DCxh030MkqR5asYEkWQHuocE/QHw7+iGw7ihqn45B7FJkkZoxgRRVb9K8j+r6qnAdXMUkyRpDAzSB/GFJC906G1JWlgG7YPYBbg/yX1sGsX1EUONTJI0UtO2IJIc3iaXVNUOVbVTVT2iqnY1OUjS/DfTKab3tvevzkUgkqTxMtMppl8m+SCwX5L3Tl1YVX8xvLAkSaM2U4I4mu4ZDkfRPTZUkrSATJsgquoHdM9tuL6qrp7DmCRJY2DWy1xNDpK0MA1yH4QkaQEyQUiSeg3ywKATkzwinTOSXJHk2XMRnCRpdAZpQby6qn4EPJtuJNdXAacMNSpJ0sgNMtTGxBhMzwM+WFVXOy7T+Fh68menXbbhlOfPYSSS5ptBWhDrknyBLkFcmGRX2tPlJEnz1yAtiJXAocBNVXVvkj3oTjNJkuaxQVoQT6V7SNDdSV4GvAUfOSpJ894gCeL9wL1JDgFOAr4HfGioUUmSRm6QBHF/VRVwDPCeqnoPsOtww5IkjdogfRA/TvJm4GXAM5IsAhYPNyxJ0qgN0oJ4MfBzYGVVfR/YF3jnUKOSJI3crC2IlhTeNWn+ZuyDkKR5b5ChNg5LcnmSnyT5RZIHkngVkyTNc4OcYvoH4KXAjcBDgf8AvG+2lZLsn+TSJNcnuS7Jia18jyQXJbmxvT+ylSfJe5OsT3JNkidv+Y8lSdpaA43mWlXrgUVV9UBVfRA4YoDV7gfeUFWPBw4DTkhyMHAycHFVLQMubvMAzwWWtdcqustrJUkjMshVTPcm2Qm4Ksk7gNuAXWZbqapua3Wpqh8nuZ6ug/sYNiWYNcBlwJta+YfaJbVfS7J7kn3adiRJc2yQFsTLgUXAa4GfAvsDL9ycnSRZCjwJ+Dqw98SHfnt/VKu2L3DLpNU2tjJJ0ggMchXT99rkz4C3be4Okjwc+CTwuqr60QwDwfYtqJ7traI7BcUBBxywueFIkgY0bYJIci09H9ATquqJs208yWK65PCRqvpUK7594tRRkn2AO1r5RrrWyYT9gFt79rsaWA2wfPnyaeOTJG2dmVoQR2/NhtszI84Arq+qd01adD6wgu6hQyuA8yaVvzbJ2cDvAffY/yBJozNTglhM11/wz5MLk/w+Pd/sexxO139xbZKrWtlf0SWGc5OsBG4GjmvLLqB75sR64F4cUlySRmqmBPFuug/0qX7Wlv3RTBuuqq/Q368A8Mye+gWcMNM2JUlzZ6armJZW1TVTC6tqLbB0aBFJksbCTAli5xmWPXRbByJJGi8zJYjLk7xmamHrO1g3vJAkSeNgpj6I1wGfTnI8mxLCcmAn4I+HHZgkabSmTRBVdTvwtCRHAk9oxZ+tqkvmJDJJ0kgNcif1pcClcxCLJGmMDDSaqyRp4TFBSJJ6mSAkSb1MEJKkXiYISVIvE4QkqZcJQpLUywQhSeplgpAk9TJBSJJ6mSAkSb1mHYtJkrTllp782RmXbzjl+XMUyeazBSFJ6mWCkCT1MkFIknqZICRJvUwQkqReJghJUi8ThCSplwlCktTLG+UWqJlu3hnnG3ckzR1bEJKkXiYISVIvE4QkqdfQ+iCSnAkcDdxRVU9oZXsA5wBLgQ3An1TVXUkCvAd4HnAv8MqqumJYsWk47NeQ5pdhtiDOAp4zpexk4OKqWgZc3OYBngssa69VwPuHGJckaQBDSxBV9SXgh1OKjwHWtOk1wLGTyj9Una8BuyfZZ1ixSZJmN9d9EHtX1W0A7f1RrXxf4JZJ9Ta2MknSiIxLJ3V6yqq3YrIqydoka++8884hhyVJC9dcJ4jbJ04dtfc7WvlGYP9J9fYDbu3bQFWtrqrlVbV8yZIlQw1WkhayuU4Q5wMr2vQK4LxJ5a9I5zDgnolTUZKk0RjmZa4fA44A9kqyEXgrcApwbpKVwM3Aca36BXSXuK6nu8z1VcOKS5I0mKEliKp66TSLntlTt4AThhWLJGnzjUsntSRpzJggJEm9TBCSpF4mCElSLx8YNIcczE7S9sQWhCSply0IzVu22KStYwtCktTLBCFJ6mWCkCT1sg9Cmofsf9G2YAtCktTLFoS0jfitXfONCULaDDMlgbnen0lny3hMB+cpJklSL1sQkjSmRt3asQUhSeplC0JzYtTfhMbZXPdrSIMyQUj6tdmS1UzJ3C8B88+CTRD+Mc8PfvuWhmfBJoiZbE3y8ANL0nxhgpAWmFF8iZnvLfatOabj/KXSBLEd2NI/oHH+w5M0/kwQ0hTbS2LdXuLU9ssEMSYW8j/7fD/9oOGY67+bhfg/aoLQg4zTP8I4xaLxGotqnNabr1JVo45hiy1fvrzWrl27Rev6hyBpe7Y1raQk66pq+Wz1HGpDktTLBCFJ6mWCkCT1GqsEkeQ5SW5Isj7JyaOOR5IWsrFJEEkWAe8DngscDLw0ycGjjUqSFq6xSRDAU4D1VXVTVf0COBs4ZsQxSdKCNU4JYl/glknzG1uZJGkExulGufSUPegmjSSrgFVt9idJbtjC/e0F/GAL110oPEYz8/jMzmM0sy0+Pjl1q/b7W4NUGqcEsRHYf9L8fsCtUytV1Wpg9dbuLMnaQW4UWcg8RjPz+MzOYzSzcT8+43SK6XJgWZIDk+wEvAQ4f8QxSdKCNTYtiKq6P8lrgQuBRcCZVXXdiMOSpAVrbBIEQFVdAFwwR7vb6tNUC4DHaGYen9l5jGY21sdnux6sT5I0POPUByFJGiMLMkE4pMeDJTkzyR1JvjmpbI8kFyW5sb0/cpQxjlKS/ZNcmuT6JNclObGVe4yAJDsn+UaSq9vxeVsrPzDJ19vxOaddgLJgJVmU5Mokn2nzY318FlyCcEiPaZ0FPGdK2cnAxVW1DLi4zS9U9wNvqKrHA4cBJ7S/G49R5+fAUVV1CHAo8JwkhwGnAqe143MXsHKEMY6DE4HrJ82P9fFZcAkCh/ToVVVfAn44pfgYYE2bXgMcO6dBjZGquq2qrmjTP6b7J98XjxEA1flJm13cXgUcBXyilS/Y4wOQZD/g+cDpbT6M+fFZiAnCIT0Gt3dV3QbdByTwqBHHMxaSLAWeBHwdj9GvtdMnVwF3ABcB3wHurqr7W5WF/r/2buAk4Fdtfk/G/PgsxAQx0JAeUp8kDwc+Cbyuqn406njGSVU9UFWH0o2C8BTg8X3V5jaq8ZDkaOCOqlo3ubin6lgdn7G6D2KODDSkhwC4Pck+VXVbkn3ovhkuWEkW0yWHj1TVp1qxx2iKqro7yWV0fTW7J9mxfUteyP9rhwMvSPI8YGfgEXQtirE+PguxBeGQHoM7H1jRplcA540wlpFq54vPAK6vqndNWuQxApIsSbJ7m34o8Cy6fppLgRe1agv2+FTVm6tqv6paSveZc0lVHc+YH58FeaNcy+LvZtOQHv9txCGNXJKPAUfQjS55O/BW4J+Ac4EDgJuB46pqakf2gpDk6cCXgWvZdA75r+j6IRb8MUryRLpO1kV0XzzPrar/muTRdBeC7AFcCbysqn4+ukhHL8kRwBur6uhxPz4LMkFIkma3EE8xSZIGYIKQJPUyQUiSepkgJEm9TBCSpF4mCA1Fkn+T5Owk30nyrSQXJHlskqWTR4wdpSR/m+SN22hbD03yxTYY5HYhyXFtdNpLt+E2X5vkVdtqexotE4S2uXZT2aeBy6rqoKo6mO6egb1HG9lQvRr4VFU9MOpA+kyTuFYCf1ZVR27DXZ0J/MU23J5GyAShYTgS+GVVfWCioKquqqovT67UWhNfTnJFez2tle+T5EtJrkryzSS/3waCO6vNX5vkL6dsa7ckG5Ls0OYfluSWJIuTvCbJ5e1ZBZ9M8rCpASe5LMnyNr1Xkg1telGSd7b1r0nyH6f5mY+n3QWbzjsnxfriVn5E288nknw7yUdaMiXJ77QWyLokF7ZhO6bGeFaSD7Rj9q9tfJ9pY2z7uzTJR+lu8Ju8rf8CPB34QFu393fR6p7Ufo6rk5zSyg5K8vkW75eTPK79nu8FNiR5yjTHSduTqvLla5u+6L5BnjbNsqXAN9v0w4Cd2/QyYG2bfgPw1216EbAr8DvARZO2s3vPts8DjmzTLwZOb9N7TqrzduDP2/Tf0t3RCnAZsLxN7wVsaNOrgLe06YcAa4EDp+x3J+D7k+ZfSDea6SK6VtPNwD50d6rfQzfmzg7Av9B9SC8GvgosmRT7mT0/31nA59u6y+jGFdt5uhjb/n46Nd5J25v8M0/3u3hui+1hbX6P9n4xsKxN/x7d0BET2/1rumdnjPxv0dfWvRbiYH0aH4uBf0hyKPAA8NhWfjlwZrrB8f6pqq5KchPw6CR/D3wW+ELP9s6h+3C9lG68m39s5U9I8nZgd+DhwIWbEeOzgScmmRgvZze6D9DvTqqzF3D3pPmnAx+r7nTT7Um+CPwu8CPgG1W1ESDd0NhL27pPAC5qDYpFwG3TxHNuVf0KuLEdk8fNEOMv2v6+27+p3zDd7+JZwAeraxlQVT9MN6Lt04CPt3ihS0wT7mhxaTtngtAwXMemAchm8pd04z4dQvet+D7oHl6U5Bl0D1f5cJJ3VtWHkhwC/CFwAvAndOf9Jzsf+B9J9qBrcVzSys8Cjq2qq5O8ku6b9VT3s+mU686TykPX4pgpqfysZ53pTB5n5wG6/8EA11XVU2dYb8LUsXFquhjTjfnz0wG2CdP8Ltq2p+5zB7rnGBw6zbZ2pjsm2s7ZB6FhuAR4SJLXTBQk+d0kfzCl3m7Abe0b8cvpvjmT5Lfoxs7/33QjqD45yV7ADlX1SeBvgCdP3Wl1TzT7BvAe4DO1qcN4V+C21iI5fpqYN9AlFfjN5HYh8J/auqS7EmuXKfu9C1iUZCJJfAl4cesbWAI8o8U1nRuAJUme2vaxOMlvT1P3uCQ7JDkIeHRbd9YYB9D7u6Brqb16ot8myR7VPQfju0mOa2VpyXvCY4GxuFJNW8cEoW2uqgr4Y+Dfp7vM9Tq68/1Tx7r/R2BFkq/RfahMfNs9ArgqyZV05/PfQ/ekrcvaaZmzgDdPs/tzgJe19wl/Qzfq6kXAt6dZ7+/oPmS/SnfKaMLpwLeAK9Jdnvu/6G95f4Hu1BJ0V3BdA1xNlyxPqqrvT7Nfqnv07YuAU5NcDVxFdwqnzw3AF4HPAX9aVfdtRowz6f1dVNXn6Vpma9uxn7gs+HhgZYv3On7zsb2HA/93M/evMeRortI2kORJwOur6uVD3MdZdC2jT8xWd1Tm4jho7tiCkLaBqroSuDTb0Y1yQ7IXXYtN84AtCElSL1sQkqReJghJUi8ThCSplwlCktTLBCFJ6mWCkCT1+v+qaYOiI3vfQgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print ('Size of the dataset:', lfw_people.data.shape)\n",
    "print ('Number of unique classes:', len(lfw_people.target_names))\n",
    "print (lfw_people.target_names[10:20])\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "\n",
    "plt.hist(lfw_people.target, bins=len(lfw_people.target_names))\n",
    "plt.xlabel('Class value (one per face)')\n",
    "plt.ylabel('Class frequency')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 29.666666,  24.333334,  15.666667, ..., 134.33333 , 137.33333 ,\n",
       "       129.      ], dtype=float32)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lfw_people.data[25]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJ4AAADFCAYAAAClzEtPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGExJREFUeJztXWuIXdd1/tbM6GVL1sN6jTSyJdnBsjBVEoTjklLiNIHglqY/WnAoJQWDoVBwSKGRW2gJ9EfSH6l+FBpEY+pCiOs2gZiQEkRqU+IffkS2YzvC0cPYHkvWWJZl+W3PzO6Pe+7Rd765a8+ZGc3ZlWZ9ILTvuWefs8+5e/a3XnstSykhEOgaQ6UHEFiaiIkXKIKYeIEiiIkXKIKYeIEiiIkXKIKYeIEiWNDEM7MvmdkLZnbczA5cqkEFrnzYfA3IZjYM4DcAvghgHMATAL6SUvr1pRte4ErFyAL63grgeErpJACY2QMAvgzAnXgrVqxIV111FQBgamqqOZCRi0Ppn6PHzazRZ2hoaGBbz2PwH9r09PTA43oN79p6H+87b5xtnyF3nxz4mbg9OTlZt99///1Gnw8//LBuf/zxx3Wbfy99Br72xMTE2ZTSptnGtpCJtx3AK/R5HMBn9CQzuxvA3QCwatUq3H777QCAd955p3He2rVr6/b+/fvr9oYNG+r28uXLG31WrVpVt1evXj3wPJ3gH330Ud3ml8w/BtB8ufyHsHLlyro9PDzc6MOf+bwVK1YMvBYAXH311QO/42fgPz79zOPUPx5+Jn7WN954o24/++yzjT4vvvhi3T59+nTdfvPNNweOGWi+44MHD76EFljIxBv0ZzeDt1NKhwAcAoC1a9emDz74AMDMH3p0dLRub9mypW7rD8VYtmxZ3eaH5x+AJ6dej8/TCcqrCv/QPLm0D3/HE4LP4wmg1/YmEa/M+l1u4nE/bvMfAr9rADh37lzd5sm2devWur1v375Gn9dffx1zxUKUi3EAO+jzGIBTC7heYAlhIRPvCQCfMLNdZrYcwJ0AHro0wwpc6Zg31aaUJs3sLwH8DMAwgPtSSs9fspEFrmgsRMZDSumnAH46n76sDABNGY8Fa08zA3y5LidTsVzIQrIqLiwTsTzKx1W74/uq/NcHKzd6Pe7D9+QxD/o86FpAU05lDZWvvX79+kafTZsuKqTvvfde3WZZ+frrr2/02bx588Dx5BCei0ARxMQLFMGCqHauWLlyJfbs2QNgJi0wbb377rt12zNlAE364OsxFbHpQM9jalNbGdMU0zhfT5+BaZ2fh6+t4oLXh+lQKV3H6oGfwRMD1Bh97bXX1m2mWn5u/n0AoG8imwtixQsUQUy8QBF0SrXDw8O45pprAMykD16uz58/X7fZlaY0xRoiUxN7J9RDwvTIlKVaLX/2aFepn2mT6YxpSmnSc+Hl+rTxCet3nliSE0X4u40bN9Zt9e+eOjV3v0GseIEiiIkXKIJOqXZycrKOjFDnPdMW0xwv6xow4Dni2bmtNMXXVpphME0xXb/99tt1W59h3bp1A8fDkTgqLvAzeNq4Goz5mXJaPz+rpzGrhsqU3BeLgOa7zxnl2yJWvEARxMQLFEFMvEARdCrjpZRqGYJlCQBYs2ZN3WYZ5K233qrbOQs5mwE4AIHlLqApE+XCuT1LP7f1GfizJ7PmZE7+Lid/8n3YHKPPwLIYjycXzMrvkcfG8qzeR01WbRArXqAIYuIFiqBTqjWzGct0H6ySsymDaUGd8t7ONN5oo/Tc1oLPHhN2nDPlKG222ePgPT/QfFYWPXRs3n4MfVY243i0m4tD9EQHNrPofdoiVrxAEcTECxRB51ptX4tSa7endeWc8vzZCzJQrY0pmcO+efse0Nz2x9TNVKnUz14Wpiluq9Xfo1cv9B7wqTsXt8exddxWzZzB7563PaqlQH+XNogVL1AEMfECRRATL1AExTwXKnt52wE98wfQlKnYBMNb9DTPB8tBLLtxG2h6BFiGYflKzRz8meWo3N4O/o5NNdzOJenh/vp+WI7m5/OS8ShYthwfH6/b/Gx6n7aIFS9QBLNOPDO7z8wmzOw5OrbBzA6b2bHq//W5awQCijZU+28A/hnAv9OxAwB+nlL6VpUJ9ACAb8x2oenp6drKrfShVNcHq/S5HfVsimjr7WATjJosvIwD3NZnYC8Am3Ta7nHgoEw2+2jWBR5bjub02QeNM5frjk0/L710MfsYpy8DgB07dmCumHXFSyn9L4BzcvjLAO6v2vcD+KM53zmwpDFfGW9LSuk0AFT/u8kzzOxuM3vSzJ5U42lg6WLRtVpNzNiPr1Nq9RLjMC0wnQK+FujRpH5mSlYLPmu1TJtnz54dODagadFnjdujU72Pp73y1kKgGbTADvucN4ivncuY6i0OvI+F3wfQbTzeGTMbBYDq/4l5XiewRDHfifcQgK9W7a8C+PGlGU5gqWBWqjWzHwD4HICNZjYO4O8BfAvAg2Z2F4CXAfxJm5tNTU3VoezqWPac3R5dAD5lMJ1xomm9Ri75DVMl0ysbTzW3HOf483Ibcyg/0KQtpl2mNk5ADgC7du2q2xzcoHFyLFbweFgsUXGBRQ5vG+XERJPg2mahZ8w68VJKX3G++r053y0QqBCei0ARxMQLFEGnQQLT09Ouuu5Z4FnGU5mMZTkOVHzllYt1Xy5cuNDo45XQUis/yzrs4eAx8L4MADhz5kzdZnmLTUcqc7L8yPdkGZOfBwBeffXVus1eDTUd8fi4vX379rp9ww03NPqwZ4Wvp3Img5+7LWLFCxRBTLxAEXRKtcBFqlPPBS/rXu0vpUN2VnMNLl76c9sbmfaVgj1TAosE6u1gWudnYJrTcgOcfYrHw23twyWccsklme7Z+8H0rs+tpQT64D0oGujA1N8WseIFiiAmXqAIOs8k0Kcq1ZI0AKAPphytEsi5d1mr9QIOAL/CjZ7nVdZhalKNmT0UTHte6DzQfD6mXdZqdac+92GqVa2WtX7vWdWawF4IjrPj30s9NkG1gcsGMfECRdAp1Y6MjNRLNseUAU3HNdMMO8s1rb0XXu71B/xKOuro9nZ/MWWpRsiaOtMe73pTbZ5pirV0fgbd1eXFK+by4/HzsbbLGi7QfMdMqTxuDoYAZhq42yBWvEARxMQLFEFMvEARdCrjDQ0N1WYTlXVYXmKTB5ss1OTBfdj5zrve1RTBpg0OolS5xcuulEusyHIh77/YvXt33datii+//PLANkPlTx4De3a2bdvWOI/lRA7eZBOMytpeliq+j2bW8kxhOcSKFyiCmHiBIui8emOfanLp9L3SShqP56XdZ1pQxzmDKYNj1IAmxTPVMu2p857BVv8bb7yxbqvHhumVTRu5IAGmQzaNqIOf3yObnnjfB3tLAL/AM4soKi5oXGIbxIoXKIKYeIEi6Dwer09Vqh16jnQv/zDQ1KZuvvnmgfdTmmJNLVetxkuaw5q0aszcZ+fOnXWbaVzzB99yyy1129Ok9Rm81P9K4+x54POYXtVz4W3LZOQS/bRFrHiBIoiJFyiCNokZd5jZw2Z21MyeN7N7quORnDEwb7SR8SYB/FVK6YiZrQHwSzM7DODPMcfkjENDQ7XMpiksPDmB5TiWm4CmaYNNByzH6Z4Ltuaz7KRZk7wgSpZv1IzAUSjc5mdQUwSbQNjM4W3dBJpmJN4LwW2g6ZXwElfq+2EZj+XCXCoRLwFkDm0SM55OKR2p2m8DOApgOyI5Y2ABmJOMZ2Y7AXwKwGNomZyREzPyahNY2mhtTjGz1QB+COBrKaULbTMEcWLGLVu2pL4JI6eCcxAlmwRygY48qfl4Llkhm0PUZMFmHK+yJJtjgOYWQu7PNK7PzXTN3g6mL/UusLeDaVi3W/J74PvwOFkkAPwyBzxuFTE0G0EbtFrxzGwZepPu+ymlH1WHIzljYN5oo9UagO8BOJpS+g59FckZA/NGG6r9LIA/A/CsmT1dHfsbzCM5IwcJ5LYTsueCNSsNEvCs+0xtSqFe8V/2OgBND4NSah8aU8gaq5d9QKmfRQmv/IHSIWurHIeo+ZX5XvwM/Gy6JZL78LvzPD4AMDY2hrmiTWLGXwDwBLpIzhiYF8JzESiCYkl7cpV0PI1XaYqpzSuOpxoha7+5Hf5e4Wamce3jxQG2LXrMFMq0qUZeDtNnDVVzD/JYvQLRGq/I92JRhsetZjENlmiDWPECRRATL1AEMfECRdCpjDc5OVmnlNCMQ2x+YGc5yxyqxrMsx1Z2r2wU4JdgUvOOt7eCx6ayKMtrXrCljoefgWUl3iOhaTg2b77onfRKaQF+EeXXXnsNHlhOZNMTy9D6rjRrVhvEihcogph4gSLolGpTSjV1qpXdKwmVKzfg7c3IbYlk80EunzGbDDiDEtOKei543EzJnqlH78smCxYJuEgx0MzxvHfv3rrteVj02kz9SpP8O3ilvdTLo+aeNogVL1AEMfECRdA51fY1otwufNaaWHPUeDzWPNmLwRqz0gJTLV9brf6cJJGp6aabbqrbe/bsafTxtFIep3pSeDxMr3xcQ9offfTRus1ihW4NYDFDRYk+lCbZW+Fto9RtAl0WSg4EFoSYeIEiiIkXKILOo1P68psmlGa5gdve1kKFV2PCuz/QlL1UxmN5zYtOOX78eKMPm2C8gFNNVM3yG+9laFsWi0tpqeeCr+ft+9AIG0/25v5aZVx/yzaIFS9QBDHxAkXQOdV6dMlUx/TKbVXjmSq9KobqyPeom80fQJNq+T5Hjhyp2ydPnmz04fF55h0VAziggU0t7C3hxI6An/lJadyrh5HbWuoF5Hp7SICm96QtYsULFEFMvEARdF5uwMt97FEqa4RqIefzmKpzWpa3F0K1WqZKduwzJeveAw4S4EqTvNufaRLwy1Xxvop9+/Y1+jA9s1arz81j5T4sYiht8u/DGq6XD1n7tEWseIEiaJNJYKWZPW5mz1T58b5ZHd9lZo9V+fH+w8x841kgIGhDtR8C+HxK6Z0qh8ovzOy/AXwdwD+llB4ws+8CuAvAv8x2MS8HMoO1qVxyII+SvTgy/Y7pVbVNDi/n3MIcdKDX5l39HK7OlKziAmcJuO666+o2a7JcGQho0h5TtWq1bHj24ho1hs/LJMDvWvP1zQdt8uOllFLftrCs+pcAfB7Af1XHIz9eYE5omy1quMqbMgHgMIATAM6nlPp/vuPoJWsc1Dfy4wVmoNXESylNpZQ+CWAMwK0ABuX2H2iVTCkdSintTyntz4VmB5YW5mROSSmdN7NHANwGYJ2ZjVSr3hiAU9nOApXdWO5gOSgX68/X4O+4nStDxXKLZmTykixy8kT1pLCMxWYTliU1OxPnQOZATh6PmitYDtPgTwYHGvA7zXkxPA+Q5wUBZu4jaYM2Wu0mM1tXtVcB+AJ6eZAfBvDH1WmRHy8wJ7RZ8UYB3G9mw+hN1AdTSj8xs18DeMDM/gHAU+glbwwEWqFNfrxfoZdwW4+fRE/emxP65oxc8V+vrZTj5SnmPkolbGLg/lrs19snwSYGpRjO9sSxcDlPCp/HwQR8/5zHhsej5ao40IG9KnxtNQnltpN691mUcgOBwGIgJl6gCDoNEjCzmgbVc+GFrucc2kwFTB+89CuVeMEIuqOeaYr7MLVpYAGDn0E1WYZXMSc3Nj6Px6nvlMfgJVlUzdzzAPF5GruYew8eYsULFEFMvEARxMQLFEHnKSz6ngOV11hu8cwpbStBswyjfbzqjbplj79jucfbhwDMfKY+OKhUTUIsj05MXCyOxKaa3HZNDjhVE5XnocjVAfFMIyzHjY+Pt+qTQ6x4gSKIiRcogk6pdnp6ulbr1eHv0WhbevUCSzUUi00B7MVQkwBTEFMW06ZSoEfdTMG654LHx2Pja2/f3ow427Zt28B7qoeDPSEsFnhmEr0Gn8fBFZp5qm0lT0aseIEiiIkXKIJOqXZqaqp2uKtHwdNqPWs+0HR8ew5t1drYgs8aoZ7H1Mt0z+1cZUgvl7DSM2u5XomDEydONPpwrCBTvwY68Fi3bt1at/nd58pQefsvcrGUbRErXqAIYuIFiqBzrbavSaq2ybTAVJAzTnoG21zFHs8grQZkb2MS04zSJlMdb4nk8ag2z9soWePNVeJhAy6XItBgBDZCe+PRWEEWF1gMmE+e4xxixQsUQUy8QBHExAsUQeeJGfvquspUrNazDMIylXox2qStUI+GV5Vb5SOWxbwqkWpWYLnMs/SrCYZlJ97bwV4HTm0BNPMmHzt2rG6zeQhoyqksy/E70eranick50Fq611ixIoXKIKYeIEi6Jxq+1SnKfR5WWea83azA0368PIhq1Wdz2MK1S173nlMK2qK8GL9cpTF9OqZcNRsw2YXziSl2y35fXEmK2+cCq84snqQIh4vcNmg9cSrMkY9ZWY/qT5HYsbAvDEXqr0HvZwp/XX+25hjYkYzq6lWl2te8j2nvIKDBHi556xUOQc206mGpLP26xVUzhVu5nFzn1z5A9bsmQ6Vgr1iz5yVAGhWnWSNlwMLVOtnum6r1S4a1ZrZGIDfB/Cv1WdDJGYMLABtqfYggL8G0J/a12IeiRlzgmxgaWFWqjWzPwAwkVL6pZl9rn94wKluYkYAhwBg7dq1qT/5VKv1wtCZklW78wy2DA0k8DIT5OLkvCJ6GlPo5ZZrG/Sg1+tDac7LlKDP6okIXuIivZ4XI3kp0EbG+yyAPzSzOwCsRE/GO4gFJmYMLG20Sb59b0ppLKW0E8CdAP4npfSniMSMgQVgIevnNwB83cyOoyfzRWLGQGvMNQfyIwAeqdpzTsw4NTVVm0By2wlZhmH5Rh35LEd55aVU7vECTnOlnryyVmzO0c8sI7bNhsAeG07yyG2g+R74+VRG9Ao387tW2W0++yfmg/BcBIogJl6gCDpPzNing5zznsGqv3o7lB77YJpTj4RXooCt+UCTzjzTiNolvdJRuZzDbM7g8fD+DTV5ePGK+g69zAa50gFtxAK9T8TjBS4bxMQLFEGnVDsyMlJvs9Ml3gsVZ4rQXMB8HlMjU5bSAlOdZ6UfNO5B98lpzEy1ucACFhdYLMhpl0yV3M7lQObrsVih2y09qs3RaSTtCVw2iIkXKIKYeIEi6FTGGxoaqs0Euf0TXgSJbsXjRIZeigWVTTyPQi7PsLddMpc/2JPR1CSUCxIddE8Fj0HrT3CVcAabZ7yImNx4LgVixQsUQUy8QBF0npixbxLRQFBvWffMH3qNc+fODTyuVcG9LY1KZ0yJTONeDmVFzuzC4HF7HpJcf6ZazSrlbRnlsgacrUrP84JW1Xyi4kMbxIoXKIKYeIEi6LyyTz9WTpdnT6tkrU81V17y2avBdKjJgZhqWUtW2uSkOV4mgVxySe7DbaUvfg9McywiKNXye+Dn02fgZD/8fjiZo/YZGxubddwRJBC4bBETL1AEnSft6S/TuaLHrGFqnByDaYrbTAUans6GatYouQ00U/oz7eY0TC+hj7c9EvAN32zoVjGE3xVTpebR45g+zqOX08yZUjkhkBcDCMxv62OseIEiiIkXKIKYeIEi6FzG60PlFjYf5PYRMFg+4TabJVT+YBOKJ+/pdywTebmRAV9O9VJOAM3n4/NY9lO5kmtWcB8+ruDEjGyCURnz1KmLCSG4DNXo6Kh77QgEDVw2iIkXKAJbzJirGTczex3AuwDOznbuImNjjGHR7n99SmnTbCd1OvEAwMyeTCnt7/SmMYb/d/cPqg0UQUy8QBGUmHiHCtxTEWMofP/OZbxAAAiqDRRCTLxAEXQ68czsS2b2gpkdN7MDHd3zPjObMLPn6NgGMztcVSU6bGbrc9dY4P13mNnDZnbUzJ43s3sKjGGlmT1uZs9UY/hmdbxcdaaUUif/AAwDOAFgN4DlAJ4BsLeD+/4ugE8DeI6O/SOAA1X7AIBvL+L9RwF8umqvAfAbAHs7HoMBWF21lwF4DMBtAB4EcGd1/LsA/qKz+dDhxPttAD+jz/cCuLeje++UifcCgFGaGC90+B5+DOCLpcYA4CoARwB8Bj3Pxcig32ex/3VJtdsBvEKf3WpAHWBLSuk0AFT/b57l/EsCM9sJ4FPorTidjqEqgvg0gAkAh9Fjn1bVmRYDXU681tWArkSY2WoAPwTwtZTShdnOv9RIKU2llD6JXjGcWwHcPOi0rsbT5cQbB7CDPpesBnTGzEYBoPp/YpbzFwQzW4bepPt+SulHJcbQR0rpPHolI25DVZ2p+qrT36PLifcEgE9UmtRy9KoEPdTh/RkPoVeNCFjkqkRVpcvvATiaUvpOoTFsMrN1VXsVgC+gVwK2XHWmroTJSoC9Az2t7gSAv+3onj8AcBrAx+itunehV4no5wCOVf9vWMT7/w56FPYrAE9X/+7oeAy/BeCpagzPAfi76vhuAI8DOA7gPwGs6GouhMssUAThuQgUQUy8QBHExAsUQUy8QBHExAsUQUy8QBHExAsUwf8BUZF7ALFySFEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 504x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "h = 50\n",
    "w = 37\n",
    "fig = plt.figure(figsize=(7,3))\n",
    "plt.imshow(lfw_people.data[243].reshape((h, w)), \n",
    "           cmap=plt.cm.gray)\n",
    "plt.grid(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How should we choose training and testing splits??\n",
    "- cross validation, how many folds?\n",
    "- repeated holdout?\n",
    "- stratified?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accuracy, precision, recall?\n",
    "- How do you perform precision or recall when there are mutliple classes?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recognizing faces is a classification procedure that we want to be accurate. **Detecting faces** might be something where we want low false positives (i.e., when we think there is a face, there should truly be one). Whereas we might be okay with more false negatives (i.e., we miss a few faces in the image). But this is **classification, not face detection**. Therefore, for this dataset it probably makes sense to be accurate overall--because we have many classes. However, we have such a class imbalance in this dataset that we also want to be aware of false positives and false negatives. \n",
    "\n",
    "Namely, an argument can be made for looking at false negatives/positives on a **per class basis**. For instance, an individual with lots of examples (i.e., a person with many examples in the dataset), we might prefer low false positives for that person. For a person with relatively few examples in the dataset we might prefer low false negatives (when we see an image of them, we identify them correctly). As such, we might breakup the dataset into different populations\n",
    "- (1) those that occur frequently (prefer low false positives or high precision)\n",
    "- (2) those that occur less frequently (prefer low false negatives or high recall)\n",
    "- (3) the entire dataset (prefer high accuracy)\n",
    "- (4) average accuracy per class (prefer everyone to have high true positive rate)\n",
    "- (5) NOTE: that recall and precision are not always dealt with like this! You should look at the structure of your application to decide how to apply accuracy, precision, recall or whatever validation measure best defines your data.\n",
    "\n",
    "Let's start with a simple classifier and ten fold cross validation, calculating accuracy to start."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KNN accuracy 0.3844917637605464\n",
      "Wall time: 19.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import numpy as np\n",
    "from sklearn import metrics as mt\n",
    "\n",
    "\n",
    "# create variables we are more familiar with\n",
    "X = lfw_people.data\n",
    "y = lfw_people.target\n",
    "yhat = np.zeros(y.shape) # we will fill this with predictions\n",
    "\n",
    "scl = StandardScaler()\n",
    "X = scl.fit_transform(X)\n",
    "\n",
    "# create cross validation iterator\n",
    "cv = StratifiedKFold(n_splits=10)\n",
    "\n",
    "# get a handle to the classifier object, which defines the type\n",
    "clf = KNeighborsClassifier(n_neighbors=3)\n",
    "\n",
    "# now iterate through and get predictions, saved to the correct row in yhat\n",
    "# NOTE: you can parallelize this using the cross_val_predict method\n",
    "for train, test in cv.split(X,y):\n",
    "    clf.fit(X[train],y[train])\n",
    "    yhat[test] = clf.predict(X[test])\n",
    "\n",
    "total_accuracy = mt.accuracy_score(y, yhat)\n",
    "print ('KNN accuracy', total_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, that was not highly accurate. Let's get the accuracy up before measuring other performance. One thing that might help is reducing the dimensionality of the faces like we have done before. however, instead of manually coding the PCA step and then classifying, let's train a PipeLine in sklearn.\n",
    "\n",
    "A pipeline allows us to cascade operations. The pipeline can be setup to run PCA, then fit the reduced data with a classifier. The beauty of pipelines comes through when performing different cross validations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KNN, pipeline accuracy 0.40658899156287664\n",
      "Wall time: 3.36 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from sklearn.decomposition import PCA \n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# setup pipeline to take PCA, then fit a KNN classifier\n",
    "clf_pipe = Pipeline(\n",
    "    [('PCA_Eric',PCA(n_components=100,svd_solver='randomized')),\n",
    "     ('CLF_Eric',KNeighborsClassifier(n_neighbors=1))]\n",
    ")\n",
    "\n",
    "# now iterate through and get predictions, saved to the correct row in yhat\n",
    "for train, test in cv.split(X,y):\n",
    "    clf_pipe.fit(X[train],y[train])\n",
    "    yhat[test] = clf_pipe.predict(X[test])\n",
    "\n",
    "total_accuracy = mt.accuracy_score(y, yhat)\n",
    "print ('KNN, pipeline accuracy', total_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is performing about the same overall. Lets take a look at the accurayc per class to get a better idea about how this is performing. To do this, we need to look at the confusion matrix. What classes are getting classified well?\n",
    "\n",
    "Scikit-learn implements confusion matrices as follows: \n",
    "\n",
    "![Confusion Matirx](http://scikit-learn.org/stable/_images/sphx_glr_plot_confusion_matrix_001.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYHFW9//H3J2ELDiRgMHIhGOAGEUFAhkVAnQBKQAW9gsKDCLLkqkT5KahRERG5LiDihgsiRhEICIpREERggPtTlIQtJoBENsMqAglBEALf+0edLjpNd0/NUlOzfF7P0890VZ2q+vbpnv72OVV1ShGBmZkZwJiqAzAzs6HDScHMzHJOCmZmlnNSMDOznJOCmZnlnBTMzCznpGA2ACQ9JGnXquMw6y8nBSuFpHsk7VE3fYCkxyW9WdIUSSHpkoZ1fibphPS8K5U5vaHM/0o6tMD+F0panh7PS3qmbvozPay7uaQVvXm9RUj6SnpNWw/0tgvu/7y0/w3r5q0n6deSnpJ0t6T92qx/VV0dLpf0nKQbBid6GyxOClY6SYcApwNvi4hr6hbtJGmXNqs+Bbxf0pTe7jMiXhsRHRHRAVwHzKxNR8SXeru9/pI0Bngf8Bjw/gr2vzuwQZNFZwCPA68ADgfOkjS12TYiYre6OuwA5gM/Lytmq4aTgpVK0gzgVGDPiPhDw+KTgZParP4EMBv4fAlxjZX0BUn3SXpY0lmS1kqLrwXG1v0i3ja1HrolPSbpH5J+Ule+iD2A8cAxwEGSxjbE82FJt0t6UtICSVul+VMk/UrSo+lxah9e62rAN4CPNsxfB3gH8LmIeCoirgIuBw4qsM1XA53Az3objw1tTgpWpg8BXwR2j4h5TZafDmxW383UxP8A705fQgPpv4H3AG8EppL9Uv56WvYm4Pm6X8U3pfknAq8EtgJeDXy2F/s7BPglcD6wJrBnbYGkg4FPAQcCawP7AY9LWhX4LXAbsBEwGbgorbO7pCfaPDrr9v0p4FLg9oaYNgeejIh76+bdAry24Ov5fUQ8UPD12zDhpGBlegtwPbCgxfJnyL70W7YWIuIh4PtkX8gD6SDglIi4NyKWkX3BHyRJLeK4PSKuiohnU0zfAN5cZEeS1gbeBZwbEU+TJYdD6oocAXwpIm6KzB0RsQTYlSxJfCYi/hURT9daWxFxZURMaPOYl/a9CVl3VbP66wCWNsxbCrRtAdV1hc0u8vpteHFSsDJ9ENgMOLPVly3wQ2CSpHe02c5XgT0H+ADtfwD1v5DvBcYB6zYrLOk/JP1c0v2SlgFnAhML7mt/4Eng92n6HGBfSRPS9GTgb03WmwzcHREvFNxPM98GPhsRTzVZtpws6dRbO8Xazm6p3MX9iMuGKCcFK9MjwO5kXTTfbVYgIp4DvkDWzdTqV/o/yX6Zf3EAY3sAeFXd9EbA02QHgpsNHXwK2YHvLSNibbJf960SXaNDgAnAEkkPAWcDq5N1XwH8Hdi0yXp/B6akX+YrkbRHw5lAjY/tU9HdgG+l/d6T5t0o6d1k3UlrS9qobtNbAwsLvJ4LIuKZnl64DT9OClaq1Oe8GzBd0mktitW+JKe32dTXgZ2B19Rm1J3aOqUPoZ0HHCtpo3TA+CSy7p0gS2ZjG74s1yL7Zb0szf94kZ2k7ptdgbcC26TH1mRJrtaFdCYwS9LWymyWThv9X7Jf7V+UtKakcZJ2BoiI39efCdTkUTtVdErdfndM8/YELomIx4HfACem7XeRvQfntHk9HcB/4a6jEctJwUoXEX8nSwz7Sfpyk+XPk51h1LTrJpVZRna2Un2ZyWTdPvf3IazvAb8A/kDWdfMY6Ys+fVmeDMxPB223AY4n+3JfSnZM4KKC+3k/8MeIuCYiHqo9gG8CO0qaGhFnkyW9C4Fl6e+E1IramyyJLAHuI/tCLiwiHq7b58Np9j/qfuUfSdYN9ijZF/3hEXEn5K2RRxs2uT/wQJMzyWyEkG+yY8OVpOPIvuB+UHUsZiOFk4KZmeVK6z5KFwM9IukvLZZL0rckLZZ0q6TXlxWLmZkVU+Yxhdm0P3C4F9lFQ1OBGWR9vGZmVqHSkkJEXEt28K6VfYGfpot1rgcmSFq/rHjMzKxnq1S47w3IzsOuWZLmPdhYMI2fMwNg3Lhx202ePLnfO3/hhRcYM8YnXzXjumnNddOa66a9quvnr3/966MRsV5P5apMCs0u/Gl61DsiziAbzZHOzs6YN6/ZMDq9093dTVdXV7+3MxK5blpz3bTmummv6vqRdG/Ppaq9TmEJ2XnmNRuSXWVqZmYVqTIpzCUbK1+SdgKWRsRLuo7MzGzwlNZ9JOk8oAuYKGkJ2RWrqwJExPfJhvLdG1gM/Av4QFmxmJlZMaUlhYg4sIflARxV1v7NzKz3fKqAmZnlnBTMzCznpGBmZjknBTMzyzkpmJlZzknBzMxyTgpmZpZzUjAzs5yTgpmZ5ZwUzMws56RgZmY5JwUzM8s5KZiZWc5JwczMck4KZmaWc1IwM7Ock4KZmeWcFMzMLOekYGZmOScFMzPLOSmYmVnOScHMzHJOCmZmlnNSMDOznJOCmZnlnBTMzCznpGBmZjknBTMzyzkpmJlZzknBzMxyTgpmZpZzUjAzs5yTgpmZ5ZwUzMws56RgZma5UpOCpOmS7pC0WNKsJss3knS1pJsk3Spp7zLjMTOz9kpLCpLGAqcDewFbAAdK2qKh2HHABRGxLXAA8N2y4jEzs56V2VLYAVgcEXdFxLPAHGDfhjIBrJ2ejwceKDEeMzPrgSKinA1L+wHTI+KINH0wsGNEzKwrsz7wO2Ad4GXAHhExv8m2ZgAzACZNmrTdnDlz+h3f8uXL6ejo6Pd2RiLXTWuum9ZcN+1VXT/Tpk2bHxGdPZVbpcQY1GReYwY6EJgdEadKegNwtqQtI+KFlVaKOAM4A6CzszO6urr6HVx3dzcDsZ2RyHXTmuumNddNe8OlfsrsPloCTK6b3pCXdg8dDlwAEBF/BNYAJpYYk5mZtVFmUrgBmCppY0mrkR1InttQ5j5gdwBJryFLCv8oMSYzM2ujtKQQESuAmcDlwG1kZxktlHSipH1SsWOAIyXdApwHHBplHeQwM7MelXlMgYi4FLi0Yd7xdc8XAbuUGYOZmRXnK5rNzCznpGBmZjknBTMzyzkpmJlZzknBzMxyTgpmZpZzUjAzs1yvkoKkMZLW7rmkmZkNRz0mBUnnSlpb0suARcAdkj5RfmhmZjbYirQUtoiIZcA7ya5O3gg4uNSozMysEkWGuVhV0qpkSeE7EfGcJI9PZGY9mjLrkpfMu+crb6sgEiuqSEvhB8A9ZDfBuVbSq4BlZQZlZmbV6LGlEBHfAr5VN+teSdPKC8nMzKpS5EDz0elAsyT9SNKNwG6DEJuZmQ2yIt1Hh6UDzW8F1gM+AHyl1KjMzKwSRZJC7V7LewM/johbaH7/ZTMzG+aKJIX5kn5HlhQul7QW8EK5YZmZWRWKnJJ6OLANcFdE/EvSy8m6kMzMbIQpcvbRC5LuBjaTtMYgxGRmJWh2zQD4ugFbWY9JQdIRwNHAhsDNwE7AH/EZSGZmI06R7qOjge2B6yNimqTNgS+UG5aNFL6i1Wx4KXKg+ZmIeAZA0uoRcTvw6nLDMjOzKhRpKSyRNAG4GLhC0uPAA+WGZWZmVShyoPld6ekJkq4GxgOXlRqVmZlVomVSkLRuk9kL0t8O4LFSIjIzs8q0aynMB4KVr16uTQewSYlxmdkg8umqVtMyKUTExoMZiJmZVa/IKKnvkjS+bnqCpHeWG5aZmVWhyCmpn4+IpbWJiHgC+Hx5IZmZWVWKJIVmZYqcympmZsNMkaQwT9LXJW0qaRNJp5EdhDYzsxGmSFL4CPAscD5wAfA0cFSZQZmZWTWKXLz2FDBrEGIxM7OKFWkpmJnZKFFqUpA0XdIdkhZLatrakPQeSYskLZR0bpnxmJlZe6WdRSRpLHA68BZgCXCDpLkRsaiuzFTg08AuEfG4pFeUFY+ZmfWsyE121gOOBKbUl4+Iw3pYdQdgcUTclbYzB9gXWFRX5kjg9Ih4PG3zkd4Eb2ZmA0sR0b6A9AfgOrLTUJ+vzY+Ii3pYbz9gekQckaYPBnaMiJl1ZS4G/grsAowFToiIl4zAKmkGMANg0qRJ282ZM6fQi2tn+fLldHR09Hs7I1Gzullw/9KXlNtqg/Evmdeor+sNVcP5c9PsvYDs/Wi3rKiB/NyMRFV/dqZNmzY/Ijp7Klek+2jNiPhUH2JQk3mNGWgVYCrQRXa7z+skbZmumn5xpYgzgDMAOjs7o6urqw/hrKy7u5uB2M5I1KxuDm12B7WDul4yr1Ff1xuqhvPnptl7Adn70W5ZUQP5uRmJhstnp8iB5t9I2rsP214CTK6b3pCX3pxnCfCriHguIu4G7iBLEmZmVoEiSeFossTwtKRlkp6UtKzAejcAUyVtLGk14ABgbkOZi4FpAJImApsBdxUP38zMBlKRi9fW6suGI2KFpJnA5WTHC86KiIWSTgTmRcTctOytkhaRHa/4RET8sy/7MzOz/mt357XNI+J2Sa9vtjwibuxp4xFxKXBpw7zj654H8PH0MDOzirVrKXyc7IyfU5ssC2C3UiIyM7PKtLvz2oz0d9rghWNmZlUqdEWzpJ156cVrPy0pJjMzq0iRK5rPBjYFbubFi9cCcFKoiG+ybmZlKdJS6AS2iJ4ufTYzs2GvyHUKfwFeWXYgZmZWvXanpP6arJtoLWCRpD8D/64tj4h9yg/PzMwGU7vuo68NWhRmZjYktDsl9RoASV9tHBBP0leBa0qOzczMBlmRA81vARpHSd2ryTwzq5jPTLP+andM4UPAh4FNJN1at2gt4P+XHZiZmQ2+di2Fc4HfAl8G6u+v/GREPFZqVGZmVol2SSEi4h5JRzUukLSuE4OZ2cjTU0vh7WS34QxWvpNaAJuUGJeZmVWg3dlHb09/Nx68cMzMrEo9XtEs6aeSjpS0+WAEZGZm1SlySupsYFfg25I2IRsY79qI+GaZgZmZDSSfrltMkdtxXiXpGmB7svspfxB4LeCkYGY2whQZOvtK4GXAH4HrgO0j4pGyAzMzs8FXZJTUW4FngS2B1wFbShpXalRmZlaJIt1HHwOQ1AF8APgx2VDaq5cbmpmZDbYi3UczgTcC2wH3AmeRdSOZmQ0pPpjcf0XOPhoHfB2YHxErSo7HzMwqVKT76JTBCMTMzKpX5ECzmZmNEk4KZmaWKzLMxUxJ6wxGMGZmVq0iLYVXAjdIukDSdEnqcQ0zMxuWekwKEXEcMBX4EXAocKekL0natOTYzMxskBU6phARATyUHiuAdYALJZ1cYmxmZjbIily89lHgEOBR4EzgExHxnKQxwJ3AJ8sN0QaKL+wxs54UuXhtIvBfEXFv/cyIeEHS28sJy8zMqlCk++hSIL8fs6S1JO0IEBG3lRWYmZkNviJJ4XvA8rrpp9I8MzMbYYokBaUDzUDWbUSxbiczMxtmiiSFuyR9VNKq6XE0cFeRjafrGu6QtFjSrDbl9pMUkjqLBm5mZgOvSFL4ILAzcD+wBNgRmNHTSpLGAqcDewFbAAdK2qJJubWAjwJ/Kh62mdnQN2XWJfljwf1LW54BOJQUGSX1EeCAPmx7B2BxRNwFIGkOsC+wqKHcF4GTgWP7sA8zMxtAqjtc0LyAtAZwOPBaYI3a/Ig4rIf19gOmR8QRafpgYMeImFlXZlvguIh4t6Ru4NiImNdkWzNIrZNJkyZtN2fOnGKvro3ly5fT0dHR7+1UYcH9S5vO32qD8QOyXrO6abZuT/vrz3pD1VD/3LR7j/u6rKiB/Nz0VdmvsT/xTBoHDz9d3ed/2rRp8yOixy76IgeMzwZuB/YETgQOAoqcitpsjKQ8A6WL304jGzqjrYg4AzgDoLOzM7q6ugrsvr3u7m4GYjtVOLTVRWgHdQ3Ies3qptm6Pe2vP+sNVUP9c9PuPe7rsqIG8nPTV2W/xv7Ec8xWKzh1wSpD/vNfJCn8Z0TsL2nfiPiJpHOBywustwSYXDe9IfBA3fRawJZAdxpj75XAXEn7NGstmJmVxVf7v6jIgebn0t8nJG0JjAemFFjvBmCqpI0lrUZ2XGJubWFELI2IiRExJSKmANcDTghmZhUq0lI4I91P4TiyL/UO4HM9rRQRKyTNJGtVjAXOioiFkk4E5kXE3PZbMBsemv3KrP3CbLfMbChqmxRSv/+yiHgcuBbYpDcbj4hLyYbJqJ93fIuyXb3ZtpmZDby23Ufp6uWZ7cqYmdnIUeSYwhWSjpU0WdK6tUfpkZmZ2aArckyhdj3CUXXzgl52JZmZ2dBX5IrmjQcjEDN7kU+RtKoUufPa+5vNj4ifDnw4Zv3jL1Oz/inSfbR93fM1gN2BGwEnBTOzEaZI99FH6qcljScb+sLMzEaYvtws51/A1IEOZKTyxUtmNpwUOabwa14cyG4M2b0RLigzKDMzq0aRlsLX6p6vAO6NiCUlxWNmZhUqkhTuAx6MiGcAJI2TNCUi7ik1MjMzG3RFrmj+OfBC3fTzaZ6ZmY0wRVoKq0TEs7WJiHg2DYVtZlYKX29SnSIthX9I2qc2IWlf4NHyQjIzs6oUaSl8EDhH0nfS9BKg6VXOZmY2vBW5eO1vwE6SOgBFxJPlh2VmZlXosftI0pckTYiI5RHxpKR1JJ00GMGZmdngKnJMYa+IeKI2ke7Ctnd5IZnZcDJl1iVMmXUJC+5fmj+34atIUhgrafXahKRxwOptypuZ2TBV5EDzz4ArJf2YbLiLwximI6TW/4I5ZqsVHJqmfZqbmVmmyIHmkyXdCuwBCPhiRFxeemRmZjboCo2SGhGXAZcBSNpF0ukRcVQPq5mZ2TBTKClI2gY4EHgvcDfwizKDMjOzarRMCpI2Aw4gSwb/BM4nu05h2iDFZmZmg6xdS+F24DrgHRGxGEDSxwYlKjMzq0S7U1LfDTwEXC3ph5J2JzvQbGZmI1TLlkJE/BL4paSXAe8EPgZMkvQ94JcR8btBinHI8y03hwePvGnWsyKnpD4FnEM2KN66wP7ALMBJwQAnRbORpMgVzbmIeCwifhARu5UVkJmZVadXScHMzEY2JwUzM8sVunjNzEa3oXTcyCcMlMtJYYTxP4yZ9Ye7j8zMLFdqUpA0XdIdkhZLmtVk+cclLZJ0q6QrJb2qzHjMzKy90pKCpLHA6cBewBbAgZK2aCh2E9AZEa8DLgROLises/7w3cVstCjzmMIOwOKIuAtA0hxgX2BRrUBEXF1X/nrgfSXGM+S06//3sQEzq4IiopwNS/sB0yPiiDR9MLBjRMxsUf47wEMRcVKTZTOAGQCTJk3abs6cOX2KacH9S/Pnk8bBw09nz7faYHyfttdsuzW1bfZ2WW35YC+rt3z5cjo6OlaaN5Cvsb/13c5AvP522232uSnj9fc11rI+G0VeY2/rpp2h+r/RW82+c8r8/Lczbdq0+RHR2VO5MpPC/sCeDUlhh4j4SJOy7wNmAm+OiH+3225nZ2fMmzevTzE13o7z1AVZQ6m/v77bna7X22W15YO9rF53dzddXV0rzRvI11hma6es1ldt3WafmzJef19jLeuzUeQ19rZu2hmq/xu91ew7p6rWvqRCSaHM7qMlwOS66Q2BBxoLSdoD+CwFEkKVhtJ52jYyuIvQhqIyzz66AZgqaWNJq5HdsGdufQFJ2wI/APaJiEdKjMXMzAooraUQESskzQQuB8YCZ0XEQkknAvMiYi5wCtAB/FwSwH0RsU9ZMVlrjc3cQ9O0f7WajS6lXtEcEZcClzbMO77u+R5l7t/MzHrHVzSbmVnOYx8lPuhnZuakYGY25FT5I9XdR2ZmlnNLwYYdd/WZlcctBTMzyzkpmJlZzt1HZsOMu8+sTG4pmJlZzknBzMxyTgpmZpbzMQUzs34Y7HtmlM1JwcysB6PpfiruPjIzs5yTgpmZ5ZwUzMws52MKZlaJoXqgdbRzS8HMzHJOCmZmlnNSMDOznJOCmZnlfKDZKuMDjWZDj5OCGU5QZjXuPjIzs5yTgpmZ5ZwUzMws56RgZmY5JwUzM8s5KZiZWc5JwczMck4KZmaWc1IwM7Ock4KZmeWcFMzMLOekYGZmuVKTgqTpku6QtFjSrCbLV5d0flr+J0lTyozHzMzaKy0pSBoLnA7sBWwBHChpi4ZihwOPR8R/AqcBXy0rHjMz61mZLYUdgMURcVdEPAvMAfZtKLMv8JP0/EJgd0kqMSYzM2tDEVHOhqX9gOkRcUSaPhjYMSJm1pX5SyqzJE3/LZV5tGFbM4AZafLVwB0DEOJE4NEeS41OrpvWXDetuW7aq7p+XhUR6/VUqMyb7DT7xd+YgYqUISLOAM4YiKDyHUvzIqJzILc5UrhuWnPdtOa6aW+41E+Z3UdLgMl10xsCD7QqI2kVYDzwWIkxmZlZG2UmhRuAqZI2lrQacAAwt6HMXOCQ9Hw/4Kooqz/LzMx6VFr3UUSskDQTuBwYC5wVEQslnQjMi4i5wI+AsyUtJmshHFBWPE0MaHfUCOO6ac1105rrpr1hUT+lHWg2M7Phx1c0m5lZzknBzMxyozIp9DT8xmgi6SxJj6RrRmrz1pV0haQ70991qoyxKpImS7pa0m2SFko6Os0f9fUjaQ1Jf5Z0S6qbL6T5G6cha+5MQ9isVnWsVZE0VtJNkn6TpodF3Yy6pFBw+I3RZDYwvWHeLODKiJgKXJmmR6MVwDER8RpgJ+Co9Flx/cC/gd0iYmtgG2C6pJ3Ihqo5LdXN42RD2YxWRwO31U0Pi7oZdUmBYsNvjBoRcS0vvTakfviRnwDvHNSghoiIeDAibkzPnyT7B98A1w+RWZ4mV02PAHYjG7IGRmndAEjaEHgbcGaaFsOkbkZjUtgA+Hvd9JI0z140KSIehOyLEXhFxfFULo3guy3wJ1w/QN49cjPwCHAF8DfgiYhYkYqM5v+tbwCfBF5I0y9nmNTNaEwKhYbWMKuR1AFcBPy/iFhWdTxDRUQ8HxHbkI1WsAPwmmbFBjeq6kl6O/BIRMyvn92k6JCsmzLHPhqqigy/Mdo9LGn9iHhQ0vpkvwRHJUmrkiWEcyLiF2m266dORDwhqZvsuMsESaukX8Sj9X9rF2AfSXsDawBrk7UchkXdjMaWQpHhN0a7+uFHDgF+VWEslUn9wD8CbouIr9ctGvX1I2k9SRPS83HAHmTHXK4mG7IGRmndRMSnI2LDiJhC9v1yVUQcxDCpm1F5RXPK4N/gxeE3/qfikCoj6Tygi2xY34eBzwMXAxcAGwH3AftHxKgbqFDSrsB1wAJe7Bv+DNlxhVFdP5JeR3awdCzZj8sLIuJESZuQnbyxLnAT8L6I+Hd1kVZLUhdwbES8fbjUzahMCmZm1txo7D4yM7MWnBTMzCznpGBmZjknBTMzyzkpmJlZzknBBoykV0qaI+lvkhZJulTSZpKm1I/CWiVJJ0g6doC2NU7SNWmQxWFB0v5p1NerB3CbMyV9YKC2Z9VyUrABkS70+iXQHRGbRsQWZOf0T6o2slIdBvwiIp6vOpBmWiSrw4EPR8S0AdzVWcBHB3B7ViEnBRso04DnIuL7tRkRcXNEXFdfKLUarpN0Y3rsnOavL+laSTdL+oukN6YB12an6QWSPtawrfGS7pE0Jk2vKenvklaVdKSkG9J4/xdJWrMxYEndkjrT84mS7knPx0o6Ja1/q6T/bvGaDyJdlarMKXWxvjfN70r7uVDS7ZLOSQkUSdullsZ8SZenITMaY5wt6fupzv6axtVpGWPa39WSziW76K5+W8cDuwLfT+s2fS9S2U+m13GLpK+keZtKuizFe52kzdP7/C/gHkk7tKgnG04iwg8/+v0g+6V4WotlU4C/pOdrAmuk51OBeen5McBn0/OxwFrAdsAVdduZ0GTbvwKmpefvBc5Mz19eV+Yk4CPp+QlkV5gCdAOd6flE4J70fAZwXHq+OjAP2Lhhv6sBD9VNv5tspNCxZK2j+4D1ya4WX0o21s0Y4I9kX8yrAn8A1quL/awmr282cFladyrZ2F1rtIox7e+pxnjrtlf/mlu9F3ul2NZM0+umv1cCU9PzHcmGb6ht97Nk956o/LPoR/8eo3FAPKvWqsB3JG0DPA9slubfAJylbAC6iyPiZkl3AZtI+jZwCfC7Jts7n+wL9WqycWa+m+ZvKekkYALQAVzeixjfCrxOUm2cmvFkX5p315WZCDxRN70rcF5kXUkPS7oG2B5YBvw5IpYAKBtqekpad0vgitRwGAs82CKeCyLiBeDOVCebt4nx2bS/u5tvaiWt3os9gB9H1gIgIh5TNlLszsDPU7yQJaOaR1JcNsw5KdhAWciLg3218zGyMZa2Jvv1+wxkN/uR9CayG5OcLemUiPippK2BPYGjgPeQ9ePXmwt8WdK6ZC2Lq9L82cA7I+IWSYeS/YJutIIXu1DXqJsvspZFu0TydJN1Wqkf3+Z5sv87AQsj4g1t1qtpHIsmWsWobKydpwpsE1q8F2nbjfscQ3Y/gG1abGsNsjqxYc7HFGygXAWsLunI2gxJ20t6c0O58cCD6ZfvwWS/kJH0KrIx6H9INjLp6yVNBMZExEXA54DXN+40srt//Rn4JvCbePGg71rAg6nlcVCLmO8hSySwckK7HPhQWhdlZ1C9rGG/jwNjJdUSw7XAe1Nf/3rAm1JcrdwBrCfpDWkfq0p6bYuy+0saI2lTYJO0bo8xFtD0vSBrkR1WOw4jad3I7iNxt6T90zylhF2zGTAkzjCz/nFSsAEREQG8C3iLslNSF5L13zeOGf9d4BBJ15N9kdR+1XYBN0u6iax//ptkd6bqTl0us4FPt9j9+cD70t+az5GNZnoFcHuL9b5G9sX6B7LuoJozgUXAjcpOpf0BzVvVvyPrNoLszKtbgVvIEuQnI+KhFvslslvB7gd8VdItwM1k3TPN3AFcA/wW+GBEPNOLGNtp+l5ExGVkLbB5qe5rp/AeBBye4l3Iyrex3QX4fS/3b0OQR0k16yNJ2wIfj4iDS9zHbLIW0IU9la3KYNSDDR63FMz6KCJuAq7WMLp4rSQTyVpmNgK4pWBmZjm3FMzMLOekYGZmOScFMzPLOSmYmVnOScF6WniGAAAACUlEQVTMzHL/B8EhqkOzmkRUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def per_class_accuracy(ytrue,yhat):\n",
    "    conf = mt.confusion_matrix(ytrue,yhat)\n",
    "    norm_conf = conf.astype('float') / conf.sum(axis=1)[:, np.newaxis]\n",
    "    return np.diag(norm_conf)\n",
    "\n",
    "def plot_class_acc(ytrue,yhat, title=''):\n",
    "    acc_list = per_class_accuracy(ytrue,yhat)\n",
    "    plt.bar(range(len(acc_list)), acc_list)\n",
    "    plt.xlabel('Class value (one per face)')\n",
    "    plt.ylabel('Accuracy within class')\n",
    "    plt.title(title+\", Total Acc=%.1f\"%(100*mt.accuracy_score(ytrue,yhat)))\n",
    "    plt.grid()\n",
    "    plt.ylim([0,1])\n",
    "    plt.show()\n",
    "    \n",
    "plot_class_acc(y,yhat,title=\"KNN\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't seem to be doing very well... Perhaps we should change the classifier we are using--why would this be the case?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline accuracy 0.3893129770992366\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XucHFWZ//HPNyGQSELCzYgkGsAgIsgtgIquE0QMlwVdQWHDJSBkdYmwivLD1RWXdV0QFXQFFRGjrBCjuBgFQX7AALuCQLgngIQQJNyRa0CEkGf/OGc6Rae7pyaZmp7L9/169WvqXk+f7umn6pyqU4oIzMzMAIa1OwAzM+s/nBTMzKzGScHMzGqcFMzMrMZJwczMapwUzMysxklhCJHUIWlpu+OwgUHSKZLOaXcc1recFNpM0hJJf5G0TNKjkmZLGt3uuNaUpJD0Qn5fyyQ908f7rywBSuqU9FJ+X09K+qWkTQrzd5F0iaRnJD0l6QZJR9RtYzNJKySd1YP9/nOhPF+S9GphfEGJ9a+XdEjP3m2329wqf9bf7M3tdrPPEfkzeELSs5JukbR33TLHS3pA0nOS/iDpnS22t62ka/Kyf5S0T/Xvov9yUugf/jYiRgPbAzsAn29zPL1lu4gYnV/jerqypLWqCKrJvjokdfZglVn5M9sSGAecnrfzLuBK4GrgLcCGwCeBverWPwx4GjhI0jpldhgRX+0qT+ATwHWF8n17D2LvTYcDTwHT+/DzehWYBWwSEWOBTwFzJW0IIOlvgJOA/UifzQXAhY02lMt+HvAzYH3g2LytSdW+hf7LSaEfiYhHgctIyQEASfvkI6HnJD0o6cuFeZPyUdrhkv6Uj1q/UJg/Kp95PC1pIbBzcX+S3paPuJ6RtEDSfoV5syWdJem3+Uj0fyW9QdIZeXt3S9phdd6npKMlLcpH0fMkvbEwLyQdI+le4N48bStJl+fl75H00cLye0taKOl5SQ9J+qykdYHfAm8sHEm/cZVAekFEPEX6wdkmTzoN+HFEnBoRT0YyPyI+WrfqYcAXgVeAv+2teCS9T9LN+Qj6ekk75+nfIH3+5+Ty+Eae/l1JS/P364ZWR9QN9jUMOAQ4AVgbmFY3fztJV+bvy6OSjs/T15J0kqTFeb83SnpD2f1GxIqIuDMilksSsAJYB9g0L7IZcGtE3BYRK4DzSN+F9RtsbltgvYg4MyJejYhLgfnA9LLxDDoR4VcbX8ASYI88PAG4A/hWYX4H6Ys7DHgH8BjwoTxvEhDAD4BRwHbAX4G35fmnANcCGwATgTuBpXneCGAR8M+kf+jdgeeBt+b5s4EngZ2AkaSj3/tJP2bDga8AV7V4XwG8pcH03fN2dyT9I/8ncE3depfnmEcB6wIPAkcAa+X1ngTenpd/BHhvHl4f2LFQbkt78Dl0AJ0ll+0EjsrDG+WyOQ94Hekodmo36783f07r5/c/bzW+NzOA/6mb9nrgOeCjuaxmAE8AY/P864FD6tY5LMcxAvhCLusRhe/POS1i+ADwIjA6fwfnFuatn/c9K3/O6wE753n/AtxCOpMaRjo7Hpfn3QM80+T1zbr9X57LMYBfAcrTNwBuzt+VtYDPAdc3eQ87A3+um3YtcEG7fxva9Wp7AEP9RUoKy0g/yAFc0fUP0mT5M4DT8/CkvM6EwvwbgIPy8GJgWmHeTFYmhfcCjwLDCvMvAL6ch2cDPyjM+xRwV2F8W+CZFnFG/oHq+of+dp7+Q+BrheVGk46WJxXW270w/2PAtXXb/j5wUh7+E/APpKO94jIdVJsUXszv6yHgp8DGpCPVALbqZv1zgIvy8Lvy+399D783M1g1KRxNIcHmabcUvg+rJIW6ZZXfV9eBQXdJ4b+AOXl4KvASsH4eP4JUvdVovQeAD/bS/8/apDOtYwvThpGqj5bn12PA9k3WH0lKhMeREuM+eZ1f9UZ8A/Hl6qP+4UMRMYb0w7QV6egTAEm7Srqqq1GNVJe8Ud36jxaGu47cAN5I+sJ3eaAw/EbgwUin18X5mxbGHysM/6XBeHcN4jtGxLj8Oraw31ocEbEM+HPdfosxvxnYNVdxPaPUYD0d6Kpu+AiwN/CApKtznX4pkk4sbPM3wHvq9tPKsfl9bRoR0yPiCVIbwQpgk2YrSRoFHEhKJETEdaTE9vdl427hNWWb1X+m9fF8PlfJPUuKfySrfr8arTcG+DD5fZDaUJ4gJXFIZ6b3NVhPOZ5V5q2OiHg5In4N/J2kPfPkY3IcbyWdpRwNXCpp4wbrvwTsDxxA+j/6R+CXwJC9Ss9JoR+JiKtJR+hfL0w+n9QQNjFSo9r3SEd0ZTxC+ufs8qbC8MPAxFwvXJz/UA/D7qmHST/0AOT6/w3r9lvsuvdB4OpCchkXqWH1kwARcWNE7E+qOrkImNtgGw1FxCld2wT2JR15jytM65GIeBG4jpSomvkwqSrlrFzP/ijpR/Kwnu6vgdeUbVb8TF9TJpI+QDoD/DCpQXYDUrIv8/06kFRd9sP8Hh4mnS11vY8HgS3qV4p0eP5Qo3k5pvsK7UD1rzNaxLNWYZvbkY7074vUTjCPdFa3a6MVI+LmiHhvRGwYEfvk7dzQzfsftJwU+p8zgA9I6mpsHgM8FREvSdqFnh1RzgU+L2l9SRNIPwBd/gC8AJygdIlfB+k0fM4av4PWzgeOkLS90pUfXwX+EBFLmiz/G2BLSYfmOEdI2jk3kq8tabqksRHxCqm66tW83mPAhpLGVvx+6p0AzJD0Oa28GmY7SV3lejhwLqn6bfv82g3YXtK2eflOFS4o6IF5wA6SDsiNuYeRksKlef5jwOaF5ceQqq6eIFXDnEw6UyjjcOC7pHaurvfRAbxT0pakBP0WSZ/Mn9N6XY3epOqzr0raXMkOksYBRMQWsfKKqvrXPwFI2kbSnpJG5m0fSWobuDZv/0ZgP0lvztvfm5QsFzZ6I5LeIWkdSesqXagxmpVnQEOOk0I/k6shfkJqjIN0OnuypOeBL7HySLiMfyVVH9wP/I7UGNq1n5dJl+ztRWq4PQs4LCLuXtP30EpEXEF6bxeSzmS2AA5qsfzzwJ55mYdJp/inkqoFAA4Flkh6jlS1dkhe725SG8niXB1UydVHDeL9Pakxffe876eAs4FLJG0KvB84IyIeLbzmk364D8+bmQj872rs+zHSZ/oFUpXcLGDfiOiqCjsdOCxfDfQ14NfANaSqnMWk78ET3e1H0mbAexq8j+tJ7S2HRcTTpIbog4DHSQ3I78mbOAW4mNRA/xzp7LfUZbnZcODfc6yPk9rKPhIRd+b5P8jv7X/y9k8DZkTE4hz/v0r678L2jiJ9rx4ltfF8MCKW9yCeQaWrtd7M+oF8RvfziCjdNmLWm5wUzMysprLqI0nnSnpc0p1N5kvSt5VuYrpd0o5VxWJmZuVU2aYwm7o7HOvsBUzOr5mkRiszM2ujypJCRFxD6hOlmf2Bn0RyPTBOhU7FzMys7/VZh2MNbMprb1Jamqc9Ur+gpJmkswlGjRq108SJE+sX6bEVK1YwbJgvvmrEZdOcy6Y5l01r7S6fP/7xj09GxCo38NVrZ1JodINMw1bviDibdFkfU6ZMiZtuummNd97Z2UlHR8cab2cwctk057JpzmXTWrvLR1L93e4NtTOtL+W1d9tOIF2HbmZmbdLOpDCPdCONlLrrfTYiVqk6MjOzvlNZ9ZGkC0i3vW+k9ASsk0i9EBIR3wMuIXVktojUidsRjbdk/cGkEy9eZdqSU4b0A6rMBqXKkkJEHNzN/CD1ZmhmZv2ELxUwM7MaJwUzM6txUjAzsxonBTMzq3FSMDOzGicFMzOrcVIwM7MaJwUzM6txUjAzsxonBTMzq3FSMDOzGicFMzOrcVIwM7MaJwUzM6txUjAzsxonBTMzq3FSMDOzGicFMzOrcVIwM7MaJwUzM6txUjAzsxonBTMzq3FSMDOzGicFMzOrcVIwM7MaJwUzM6txUjAzsxonBTMzq3FSMDOzGicFMzOrcVIwM7MaJwUzM6txUjAzsxonBTMzq3FSMDOzmkqTgqRpku6RtEjSiQ3mv0nSVZJukXS7pL2rjMfMzFqrLClIGg6cCewFbA0cLGnrusW+CMyNiB2Ag4CzqorHzMy6V+WZwi7AoohYHBEvA3OA/euWCWC9PDwWeLjCeMzMrBuKiGo2LB0ATIuIo/L4ocCuETGrsMwmwO+A9YF1gT0iYn6Dbc0EZgKMHz9+pzlz5qxxfMuWLWP06NFrvJ3BqFHZ3PHQs6sst+2mY/sqpH7D35vmXDattbt8pk6dOj8ipnS33FoVxqAG0+oz0MHA7Ij4hqR3AedJ2iYiVrxmpYizgbMBpkyZEh0dHWscXGdnJ72xncGoUdnMOPHiVZZbMr1jlWmDnb83zblsWhso5VNl9dFSYGJhfAKrVg99HJgLEBHXASOBjSqMyczMWqgyKdwITJa0maS1SQ3J8+qW+RPwfgBJbyMlhScqjMnMzFqoLClExHJgFnAZcBfpKqMFkk6WtF9e7HjgaEm3ARcAM6KqRg4zM+tWlW0KRMQlwCV1075UGF4I7FZlDGZmVp7vaDYzsxonBTMzq3FSMDOzGicFMzOrcVIwM7MaJwUzM6up9JJUq8akBl1OACw5ZZ8+jsTMBpsenSlIGiZpve6XNDOzgajbpCDpfEnrSVoXWAjcI+lz1YdmZmZ9rcyZwtYR8RzwIdLdyW8CDq00KjMza4sybQojJI0gJYXvRMQrktw/kZXSqP3DbR9m/VeZM4XvA0tID8G5RtKbgeeqDMrMzNqj2zOFiPg28O3CpAckTa0uJDMza5cyDc3H5YZmSfqhpJuB3fsgNjMz62Nlqo+OzA3NewIbA0cAp1QalZmZtUWZpND1rOW9gR9FxG00fv6ymZkNcGWSwnxJvyMlhcskjQFWVBuWmZm1Q5lLUj8ObA8sjogXJW1IqkIyM7NBpszVRysk3Q9sKWlkH8RkZmZt0m1SkHQUcBwwAbgVeCdwHb4Cycxs0CnTpnAcsDPwQERMBXYAnqg0KjMza4sySeGliHgJQNI6EXE38NZqwzIzs3Yo09C8VNI44CLgcklPAw9XG5aZmbVDmYbmD+fBL0u6ChgLXFppVGZm1hZNk4KkDRpMviP/HQ08VUlEZmbWNq3OFOYDwWvvXu4aD2DzCuMyM7M2aJoUImKzvgzEzMzar0wvqR+WNLYwPk7Sh6oNy8zM2qHMJaknRcSzXSMR8QxwUnUhmZlZu5RJCo2WKXMpq5mZDTBlksJNkr4paQtJm0s6ndQIbWZmg0yZpPAp4GXgZ8Bc4C/AMVUGZWZm7VHm5rUXgBP7IBYzM2uzMmcKZmY2RFSaFCRNk3SPpEWSGp5tSPqopIWSFkg6v8p4zMystcquIpI0HDgT+ACwFLhR0ryIWFhYZjLweWC3iHha0uurisfMBr9JJ17ccPqSU/bp40gGrjIP2dkYOBqYVFw+Io7sZtVdgEURsThvZw6wP7CwsMzRwJkR8XTe5uM9Cd7MzHqXIqL1AtLvgWtJl6G+2jU9Ii7sZr0DgGkRcVQePxTYNSJmFZa5CPgjsBswHPhyRKzSA6ukmcBMgPHjx+80Z86cUm+ulWXLljF69Og13k473PHQsw2nb7vp2IbTe6pR2TTaZ5n9re56/dVA/t5UrT+UTdX/G2ui3eUzderU+RExpbvlylQfvS4i/t9qxKAG0+oz0FrAZKCD9LjPayVtk++aXrlSxNnA2QBTpkyJjo6O1QjntTo7O+mN7bTDjGanyNM7emX7jcqm0T7L7G911+uvBvL3pmr9oWyq/t9YE/2hfMoo09D8G0l7r8a2lwITC+MTWPXhPEuBX0XEKxFxP3APKUmYmVkblH1G828k/UXSc5Kel/RcifVuBCZL2kzS2sBBwLy6ZS4CpgJI2gjYElhcPnwzM+tNZW5eG7M6G46I5ZJmAZeR2gvOjYgFkk4GboqIeXnenpIWktorPhcRf16d/ZmZ2Zpr9eS1rSLibkk7NpofETd3t/GIuAS4pG7alwrDAXwmv8zMrM1anSl8hnTFzzcazAtg90oiMrNBo9F9A75noH9r9eS1mfnv1L4Lx8zM2qnUHc2S3s2qN6/9pKKYzMysTcrc0XwesAVwKytvXgvAScHMbJApc6YwBdg6urv12czMBrwy9yncCbyh6kDMzKz9Wl2S+mtSNdEYYKGkG4C/ds2PiP2qD8/MzPpSq+qjr/dZFGZm1i+0uiT1agBJp9Z3iCfpVODqimMzM+s1ftZCOWXaFD7QYNpevR2ImZm1X6s2hU8C/whsLun2wqwxwP9WHZiZmfW9Vm0K5wO/Bf4DKD5f+fmIeKrSqMzMrC1aJYWIiCWSjqmfIWkDJwYzs8GnuzOFfUmP4Qxe+yS1ADavMC4zM2uDVlcf7Zv/btZ34ZiZWTt1e/WRpJ9IOlrSVn0RkJmZtU+ZS1JnA5sA/ynpPkkXSjqu2rDMzKwdyjyO80pJVwM7k56n/Ang7cC3Ko7NzMz6WJmus68A1gWuA64Fdo6Ix6sOzMzM+l6Z6qPbgZeBbYB3ANtIGlVpVGZm1hZlqo8+DSBpNHAE8CNSV9rrVBuamZn1tTLVR7OA9wI7AQ8A55KqkczMbJAp8+S1UcA3gfkRsbzieMzMrI3KVB+d1heBmJlZ+5VpaDYzsyHCScHMzGrKdHMxS9L6fRGMmZm1V5kzhTcAN0qaK2maJHW7hpmZDUjdJoWI+CIwGfghMAO4V9JXJW1RcWxmZtbHSrUpREQAj+bXcmB94BeSvlZhbGZm1sfK3Lx2LHA48CRwDvC5iHhF0jDgXuCEakM0M7O+UubmtY2Av4uIB4oTI2KFpH2rCcvMzNqhTPXRJUDtecySxkjaFSAi7qoqMDMz63tlksJ3gWWF8RfyNDMzG2TKJAXlhmYgVRtRrtrJzMwGmDJJYbGkYyWNyK/jgMVlNp7va7hH0iJJJ7ZY7gBJIWlK2cDNzKz3lUkKnwDeDTwELAV2BWZ2t5Kk4cCZwF7A1sDBkrZusNwY4FjgD+XDNjOzKpTpJfVx4KDV2PYuwKKIWAwgaQ6wP7Cwbrl/A74GfHY19mFmZr1IheaCxgtII4GPA28HRnZNj4gju1nvAGBaRByVxw8Fdo2IWYVldgC+GBEfkdQJfDYibmqwrZnks5Px48fvNGfOnHLvroVly5YxevToNd5OO9zx0LMNp2+76dhe2X6jsmm0zzL7W931+quB/L2pWm9+b1ZXq/+Nqv9vutPu787UqVPnR0S3VfRlGozPA+4GPgicDEwHylyK2qiPpFoGyje/nU7qOqOliDgbOBtgypQp0dHRUWL3rXV2dtIb22mHGSde3HD6kukdvbL9RmXTaJ9l9re66/VXA/l7U7Xe/N6srlb/G1X/33RnoHx3yiSFt0TEgZL2j4gfSzofuKzEekuBiYXxCcDDhfExwDZAZ+5j7w3APEn7NTpbMLPqTGr2g3nKPn0cibVbmaTwSv77jKRtSP0fTSqx3o3AZEmbkRqpDwL+vmtmRDxLulsagFbVR2ZWjn/cbU2Vufro7Pw8hS8C80gNxad2t1J+nvMs0lnFXcDciFgg6WRJ+61BzGZmVpGWZwq53v+5iHgauAbYvCcbj4hLSN1kFKd9qcmyHT3ZtpmZ9b6WZwr57uVZrZYxM7PBo0z10eWSPitpoqQNul6VR2ZmZn2uTENz1/0IxxSmBT2sSjIzs/6vzB3Nm/VFIGZm1n5lnrx2WKPpEfGT3g/HzMzaqUz10c6F4ZHA+4GbAScFM7NBpkz10aeK45LGkrq+MDOzQabM1Uf1XgQm93YgZmbWfmXaFH7Nyo7shpGejTC3yqDMzKw9yrQpfL0wvBx4ICKWVhSPmZm1UZmk8CfgkYh4CUDSKEmTImJJpZGZmVmfK9Om8HNgRWH81TzNzMwGmTJJYa2IeLlrJA+vXV1IZmbWLmWSwhPFrq4l7Q88WV1IZmbWLmXaFD4B/FTSd/L4UqDhXc5mZjawlbl57T7gnZJGA4qI56sPy8zM2qHb6iNJX5U0LiKWRcTzktaX9JW+CM7MbCCbdOLFtdcdDz3b9HGp/UmZNoW9IuKZrpH8FLa9qwvJzMzapUxSGC5pna4RSaOAdVosb2ZmA1SZhub/Aq6Q9CNSdxdH4h5SzcwGpTINzV+TdDuwByDg3yLissojMzOzPlfmTIGIuBS4FEDSbpLOjIhjulnNzMwGmFJJQdL2wMHAx4D7gV9WGZSZmbVH06QgaUvgIFIy+DPwM9J9ClP7KDYbIBpdZrfklH3aEImZralWZwp3A9cCfxsRiwAkfbpPojIzs7ZodUnqR4BHgask/UDS+0kNzWZmNkg1TQoR8d8R8TFgK6AT+DQwXtJ3Je3ZR/GZmVkf6vbmtYh4ISJ+GhH7AhOAW4ETK4/MzMz6XJk7mmsi4qmI+H5E7F5VQGZm1j49SgpmZja4OSmYmVmNk4KZmdU4KZiZWY2TgpmZ1ZTq+2h1SZoGfAsYDpwTEafUzf8McBSwHHgCODIiHqgypqGs2VOf3CWFmXWp7ExB0nDgTGAvYGvgYElb1y12CzAlIt4B/AL4WlXxmJlZ96qsPtoFWBQRiyPiZWAOsH9xgYi4KiJezKPXk26OMzOzNlFEVLNh6QBgWkQclccPBXaNiFlNlv8O8GhEfKXBvJnATIDx48fvNGfOnDWOb9myZYwePXqNt9MOdzz0bMPp2246tlfWa1Q2jdbtWm915w1E/f17U/V3o5Wefm+q0Op99MZ7XJN4xo+Cx/7Svu//1KlT50fElO6Wq7JNoVHneQ0zkKRDgCnA+xrNj4izgbMBpkyZEh0dHWscXGdnJ72xnXaY0axtYHpHr6zXqGwardu13urOG4j6+/em6u9GKz393lSh1fvojfe4JvEcv+1yvnHHWv3++19lUlgKTCyMTwAerl9I0h7AF4D3RcRfK4zHzMy6UWWbwo3AZEmbSVqb9MCeecUFJO0AfB/YLyIerzAWMzMrobKkEBHLgVnAZcBdwNyIWCDpZEn75cVOA0YDP5d0q6R5TTZnZmZ9oNL7FCLiEuCSumlfKgzvUeX+zcysZ3xHs5mZ1TgpmJlZjZOCmZnVOCmYmVmNk4KZmdU4KZiZWY2TgpmZ1TgpmJlZjZOCmZnVOCmYmVlNpd1cmA0FjR5z6kec2kDlpGA2RPgZ3VaGq4/MzKzGScHMzGqcFMzMrMZJwczMapwUzMysxknBzMxqnBTMzKzG9ylkvobbbOjy//9KPlMwM7OaIXWmUDwaOH7b5czI41UeDbgLBDMbSIZUUjAbDFzVYVVy9ZGZmdU4KZiZWY2rj8z6IVcRWbs4KfQCNyab2WDhpGADjo+izarjNgUzM6vxmYJZCV1nJ311f4sNbe08G/aZgpmZ1TgpmJlZjauPbFBxI3Tfc9Xa4OIzBTMzq3FSMDOzmkqTgqRpku6RtEjSiQ3mryPpZ3n+HyRNqjIeMzNrrbKkIGk4cCawF7A1cLCkresW+zjwdES8BTgdOLWqeMzMrHtVninsAiyKiMUR8TIwB9i/bpn9gR/n4V8A75ekCmMyM7MWFBHVbFg6AJgWEUfl8UOBXSNiVmGZO/MyS/P4fXmZJ+u2NROYmUffCtzTCyFuBDzZ7VJDk8umOZdNcy6b1tpdPm+OiI27W6jKS1IbHfHXZ6AyyxARZwNn90ZQtR1LN0XElN7c5mDhsmnOZdOcy6a1gVI+VVYfLQUmFsYnAA83W0bSWsBY4KkKYzIzsxaqTAo3ApMlbSZpbeAgYF7dMvOAw/PwAcCVUVV9lpmZdauy6qOIWC5pFnAZMBw4NyIWSDoZuCki5gE/BM6TtIh0hnBQVfE00KvVUYOMy6Y5l01zLpvWBkT5VNbQbGZmA4/vaDYzsxonBTMzqxmSSaG77jeGEknnSno83zPSNW0DSZdLujf/Xb+dMbaLpImSrpJ0l6QFko7L04d8+UgaKekGSbflsvnXPH2z3GXNvbkLm7XbHWu7SBou6RZJv8njA6JshlxSKNn9xlAyG5hWN+1E4IqImAxckceHouXA8RHxNuCdwDH5u+Lygb8Cu0fEdsD2wDRJ7yR1VXN6LpunSV3ZDFXHAXcVxgdE2Qy5pEC57jeGjIi4hlXvDSl2P/Jj4EN9GlQ/ERGPRMTNefh50j/4prh8iGRZHh2RXwHsTuqyBoZo2QBImgDsA5yTx8UAKZuhmBQ2BR4sjC/N02yl8RHxCKQfRuD1bY6n7XIPvjsAf8DlA9SqR24FHgcuB+4DnomI5XmRofy/dQZwArAij2/IACmboZgUSnWtYdZF0mjgQuCfIuK5dsfTX0TEqxGxPam3gl2AtzVarG+jaj9J+wKPR8T84uQGi/bLshmKj+Ms0/3GUPeYpE0i4hFJm5COBIckSSNICeGnEfHLPNnlUxARz0jqJLW7jJO0Vj4iHqr/W7sB+0naGxgJrEc6cxgQZTMUzxTKdL8x1BW7Hzkc+FUbY2mbXA/8Q+CuiPhmYdaQLx9JG0sal4dHAXuQ2lyuInVZA0O0bCLi8xExISImkX5froyI6QyQshmSdzTnDH4GK7vf+Pc2h9Q2ki4AOkjd+j4GnARcBMwF3gT8CTgwIoZcR4WS3gNcC9zByrrhfya1Kwzp8pH0DlJj6XDSweXciDhZ0uakizc2AG4BDomIv7Yv0vaS1AF8NiL2HShlMySTgpmZNTYUq4/MzKwJJwUzM6txUjAzsxonBTMzq3FSMDOzGicF6zWS3iBpjqT7JC2UdImkLSVNKvbC2k6Svizps720rVGSrs6dLA4Ikg7Mvb5e1YvbnCXpiN7anrWXk4L1inyj138DnRGxRURsTbqmf3x7I6vUkcAvI+LVdgfSSJNk9XHgHyNiai/u6lzg2F7cnrWRk4L1lqnAKxHxva4JEXFrRFxbXCifNVwr6eb8eneevomkayTdKulOSe/NHa7NzuN3SPp03bbGSloiaVgef52kByWNkHS0pBtzf/8XSnpdfcCSOiVNycMbSVqSh4dLOi2vf7ukf2jynqeT70pVcloh1o/l6R15P7+QdLekn+YEiqSd8pnGfEmX5S4z6mOcLel7ucz+mPvVaRpj3t9Vks4n3XRX3NaXgPcA38sGQSP9AAAD2klEQVTrNvws8rIn5Pdxm6RT8rQtJF2a471W0lb5c34RWCJplyblZANJRPjl1xq/SEeKpzeZNwm4Mw+/DhiZhycDN+Xh44Ev5OHhwBhgJ+DywnbGNdj2r4CpefhjwDl5eMPCMl8BPpWHv0y6wxSgE5iShzcCluThmcAX8/A6wE3AZnX7XRt4tDD+EVJPocNJZ0d/AjYh3S3+LKmvm2HAdaQf5hHA74GNC7Gf2+D9zQYuzetOJvXdNbJZjHl/L9THW9he8T03+yz2yrG9Lo9vkP9eAUzOw7uSum/o2u4XSM+eaPt30a81ew3FDvGsvUYA35G0PfAqsGWefiNwrlIHdBdFxK2SFgObS/pP4GLgdw229zPSD+pVpH5mzsrTt5H0FWAcMBq4rAcx7gm8Q1JXPzVjST+a9xeW2Qh4pjD+HuCCSFVJj0m6GtgZeA64ISKWAih1NT0pr7sNcHk+cRgOPNIknrkRsQK4N5fJVi1ifDnv7/7Gm3qNZp/FHsCPIp0BEBFPKfUU+27g5zleSMmoy+M5LhvgnBSstyxgZWdfrXya1MfSdqSj35cgPexH0t+QHkxynqTTIuInkrYDPggcA3yUVI9fNA/4D0kbkM4srszTZwMfiojbJM0gHUHXW87KKtSRhekinVm0SiR/abBOM8X+bV4l/d8JWBAR72qxXpf6vmiiWYxKfe28UGKb0OSzyNuu3+cw0vMAtm+yrZGkMrEBzm0K1luuBNaRdHTXBEk7S3pf3XJjgUfyke+hpCNkJL2Z1Af9D0g9k+4oaSNgWERcCPwLsGP9TiM9/esG4FvAb2Jlo+8Y4JF85jG9ScxLSIkEXpvQLgM+mddF6Qqqdev2+zQwXFJXYrgG+Fiu698Y+JscVzP3ABtLelfexwhJb2+y7IGShknaAtg8r9ttjCU0/CxIZ2RHdrXDSNog0nMk7pd0YJ6mnLC7bAn0iyvMbM04KViviIgAPgx8QOmS1AWk+vv6PuPPAg6XdD3ph6TrqLYDuFXSLaT6+W+RnkzVmatcZgOfb7L7nwGH5L9d/oXUm+nlwN1N1vs66Yf196TqoC7nAAuBm5Uupf0+jc+qf0eqNoJ05dXtwG2kBHlCRDzaZL9EehTsAcCpkm4DbiVVzzRyD3A18FvgExHxUg9ibKXhZxERl5LOwG7KZd91Ce904OM53gW89jG2uwH/v4f7t37IvaSarSZJOwCfiYhDK9zHbNIZ0C+6W7Zd+qIcrO/4TMFsNUXELcBVGkA3r1VkI9KZmQ0CPlMwM7ManymYmVmNk4KZmdU4KZiZWY2TgpmZ1TgpmJlZzf8BCLg69FSxrHIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 12.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "clf_pipe = Pipeline(\n",
    "    [('PCA',PCA(n_components=100, svd_solver='randomized')),\n",
    "     ('CLF',RandomForestClassifier(max_depth=50, n_estimators=150, n_jobs=-1))]\n",
    ")\n",
    "\n",
    "# now iterate through and get predictions, saved to the correct row in yhat\n",
    "for train, test in cv.split(X,y):\n",
    "    clf_pipe.fit(X[train],y[train])\n",
    "    yhat[test] = clf_pipe.predict(X[test])\n",
    "    \n",
    "total_accuracy = mt.accuracy_score(y, yhat)\n",
    "print ('Pipeline accuracy', total_accuracy)\n",
    "plot_class_acc(y,yhat,title=\"Random Forest + PCA\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy 0.45721173161912415\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xm8HFWd9/HPNyGQSEICBiOSaACDiqAsYXFc5gbRCcsAjjDCAwiiZFCijIIOPiI6jDPu4zLiaFQMMEhAUYmIIg+QwMy4EGQzYQsQIEBAhABBBEJ+zx/n3Kbo9O2ue3Pr1l2+79erX7eWU1W/Pt23f1Wnqk4pIjAzMwMYVXcAZmY2eDgpmJlZg5OCmZk1OCmYmVmDk4KZmTU4KZiZWYOTwggmqUvSyrrjsMFB0m8kHVl3HFYvJ4VBRtIKSU9JWiNplaT5ksbXHdeGkhSSnszva42k1QO8/V4nwLo/C0m/KNTXs5KeKYx/q8OyY3OdT+3nmI7P6z2wP9fbi+1/Nm//TYVpCyQ9Xea7lT/DNYXX05L+ODDRDw1OCoPT30bEeGBnYBfg4zXH019eHxHj82tSbxeWtFEVQXVQ22cREft21xdwLvCFQv0dP1BxNDkaeCT/HVCSXgPsC/ypxex/KfPdiohjCuXGAz8BflhRyEOSk8IgFhGrgEtJP0gASNpf0nWSHpd0r6RPF+ZNz3tRR0u6R9LDkj5RmD8u7yk9KmkZsHtxe5JeI2mRpNWSlhb3BvNy3yzsvf6PpJdK+mpe3y2SdunL+5R0nKTlkh6RtFDSywrzQtIJkm4Hbs/TXi3pslz+Vkl/Xyi/n6Rlkp6QdJ+kkyVtCvwCeFlhD/Fl6wXSRh8+i7MknZSHt87v4wN5/JU5dvWlvopy3dwh6U+SfixpSp51Vf57a36/B0vaMn9+f8zbv0jSVr3Y1vbAnsA/AAdI2qJp/qGSbsz1cbukt+bpkyWdnY+2HpV0fh/f7jeBk4C1fVz+BSRNBA4CzuqP9Q0bEeHXIHoBK4B98vBU4Cbga4X5XcBOpIT+OuBB4OA8bzoQwHeAccDrgaeB1+T5nwOuBrYApgF/AFbmeWOA5cD/BTYG9gaeAF6V588HHgZ2A8YCVwB3Ae8GRgOfAa5s874CeGWL6Xvn9e4KbAL8B3BV03KX5ZjHAZsC9wLvATbKyz0MvDaXfwB4cx7eHNi1UG8rB/CzOBb4WR7+P8AdwPmFeRf1Mpb5wGeapu0HrMrbHgvMAy7L88bmuptaKD+F9CM4DpgIXAQsKMz/DXBkmxj+tfuzAW4DPlCY92bgUWBWro+XA9vneZcD5wCT8nfrLXn6DGB1m9ffFdZ/VKH+VgFvKsxbQDp6+BOwBDiwZJ3OAW6u+39+sL1qD8Cvpg8k/RCtIf0gR/6HmtSm/FeBr+Th6S1+CH4HHJaH7wRmF+bN4fmk8Ob8zzaqMP884NN5eD7wncK8Dxb/ofKP4+o2cQbweOEf/ut5+vdIzSLd5cYDzwLTC8vtXZj/LuDqpnV/G/hUHr6HtCe7WVOZLvqWFPr6WWyX3+co4Fs5pu66Pgv4SC9jmc/6SeFc4PTC+CRgHfBSWiSFFuvcC3igMN5jUsjv4x7g+Dz+z8BvC/PPAj7bYrltgGeACRvwPzGJtMMyNY83J4XdSDsAY0hJbw0ws8R6/wc4pa9xDdeXm48Gp4MjYgLph+zVwOTuGZL2lHRlbgJ4DDi+OD9bVRj+M+mHFuBlpL3sbncXhl8G3BsR65rmb10Yf7Aw/FSL8U4nYXeNiEn59aHCdhtxRMQa0h5fcbvFmF8B7JmbuFbnk4pHkH4IAd5J2oO+W9JiSW/oEFMnffosIuIO0o/TzqSEezFwv6RXAX8NLN7AuGD9ultNSrxbtyosaYKkM3PT4uPAr1j/u9OTWaQ67m5/PxfYQ9Kr8/g00tFQs2nAQxHxRMnttPKvwLyIaHmhQERcGxGPRsSzEXER8CPgHe1WKOmVpKaw/9qAuIYlJ4VBLCIWk/YQv1SY/ANgITAtIiaS9kLLtk0/QPon7fbywvD9wDRJo5rm39fLsHvrftIPPQC5/f/FTdstduV7L7C4kFwmRTpp+H6AiLgmIg4CXgL8FLigxTp6rY+fxWLgEGDjiLgvj7+btFd7/YbEkzXX3URgM1LdtXq/p5CawXaPiM2At1P+u3M06fdiqaRVpHMWQXo/kD6X7Vosdy/wErW4akvS9k1XAjW/3pmLvhX4aD4nsQrYErhI0j/2EGuUeF9HA1f0lGhGMieFwe+rwNskdZ/gnAA8EhF/kbQHqb26rAuAj0vaXOlSxQ8W5v0WeBL4mKQxkrqAvyW111bpB8B7JO0saRPg30jNEit6KH8xsL2ko3KcYyTtrnSSfGNJR0iaGBHPkvaan8vLPQi8OP9wAo3LVHuTLHr7WSwG5vL8Sd9FpDr/74jojqv7ZHpXL+Lodh5wnKQdJY0FPk/6oVsVEU8DjwHbFspPIB05rpY0GTi1zEbyD/rfAceQjny6XycDR+Udie8C/yDpLZJGSZomafuIuCu//29Impg/o7cARMRtUbgSqMXrwhzCm0jNk93b/RPpnNJ3JG0k6R2SNpU0WtL+wKHAz9q8H5HOUcwv8/5HGieFQS4i/gicDXwyT/oAcLqkJ4DTeH5PuIx/JjU33EVqOjinsJ1ngANJl/w9TLrS490RccuGvod2IuJy0nu7kHQksx1wWJvyT5D2cA8j7SmvIv0YbpKLHAWsyM0jxwNH5uVuIf2I3pmbnV5GOmr6dS9i7e1nsZj0Q9ydFP4beFFhnJyc15BOYvdKRFwMfJZ0tHI/qXnnqEKR04Af5vd7IOkoZzLpR/W/gUtKbuoQ0mWo5+WEsyrS1VjzSEcme0fE1aT6/iYpGV1OOioBOJzU3n876fN6fy/f58NN211HSsZPko4ITs7v/1HSBQ9HR8Sv4QVHIy8prLKLdLT2k97EMVIowg/ZsZFJ0neBH0bEpTXGcCTpyqnhci+KDXFOCmZm1lBZ81G+yuEhSX/oYb4kfV3ppqUbJe1aVSxmZlZOlecU5gOz28zfl3TzygzS9fL/WWEsZmZWQmVJISKuIp2c6slBwNmR/AaY1Jtb7s3MrP/V0cFYt6154U1JK/O0B5oLSppDOppg3Lhxu02bNq25SK+tW7eOUaN88VUrrpueuW565rppr+76ue222x6OiC07laszKbS6uaTlWe+ImEe6/I2ZM2fGkiVLNnjjixYtoqura4PXMxy5bnrmuumZ66a9uutH0t2dS9V7n8JKXnh37VTStcZmZlaTOpPCQuDd+SqkvYDHImK9piMzMxs4lTUfSTqPdOfgZKUnXn2KdFcjEfEt0t2U+5F6P/wz6bZ1MzOrUWVJISIO7zA/gBOq2r6ZmfWeLxUwM7MGJwUzM2twUjAzswYnBTMza6jz5jUbQqaf8vP1pq343P41RGJmVfKRgpmZNTgpmJlZg5OCmZk1OCmYmVmDk4KZmTU4KZiZWYOTgpmZNTgpmJlZg5OCmZk1+I5mq5TvhDYbWnykYGZmDU4KZmbW4KRgZmYNTgpmZtbgpGBmZg1OCmZm1uCkYGZmDU4KZmbW4KRgZmYNTgpmZtbgpGBmZg1OCmZm1uCkYGZmDU4KZmbW4KRgZmYNTgpmZtbgpGBmZg1OCmZm1uCkYGZmDU4KZmbWUGlSkDRb0q2Slks6pcX8l0u6UtJ1km6UtF+V8ZiZWXuVJQVJo4EzgH2BHYDDJe3QVOxU4IKI2AU4DPhmVfGYmVlnVR4p7AEsj4g7I+IZYAFwUFOZADbLwxOB+yuMx8zMOlBEVLNi6RBgdkS8L48fBewZEXMLZbYCfgVsDmwK7BMR17ZY1xxgDsCUKVN2W7BgwQbHt2bNGsaPH7/B6xmOWtXNTfc9tl65nbae2HFdfV1usPL3pmeum/bqrp9Zs2ZdGxEzO5XbqMIY1GJacwY6HJgfEV+W9AbgHEk7RsS6FywUMQ+YBzBz5szo6ura4OAWLVpEf6xnOGpVN8ec8vP1yq04omu9ac36utxg5e9Nz1w37Q2V+qmy+WglMK0wPpX1m4feC1wAEBG/BsYCkyuMyczM2qgyKVwDzJC0jaSNSSeSFzaVuQd4K4Ck15CSwh8rjMnMzNqoLClExFpgLnApcDPpKqOlkk6XdGAudhJwnKQbgPOAY6KqkxxmZtZRlecUiIhLgEuapp1WGF4GvLHKGMzMrDzf0WxmZg1OCmZm1uCkYGZmDZWeUzCzoWF6i/tJAFZ8bv9+X++GrtOq5SMFMzNrcFIwM7MGNx8NQVUd6puZ9epIQdIoSZt1LmlmZkNRx6Qg6QeSNpO0KbAMuFXSR6sPzczMBlqZ5qMdIuJxSUeQ7k7+J+Ba4IuVRmZm1o/c7FpOmeajMZLGAAcDF0XEs6zfBbaZmQ0DZZLCt4EVpIfgXCXpFcDjVQZlZmb16Nh8FBFfB75emHS3pFnVhWRmZnUpc6L5xHyiWZK+J+n3wN4DEJuZmQ2wMs1Hx0bE48DbgS2B9wCfqzQqMzOrRZmk0P2s5f2A70fEDbR+/rKZmQ1xZZLCtZJ+RUoKl0qaAKyrNiwzM6tDmfsU3gvsDNwZEX+W9GJSE5KZmQ0zZa4+WifpLmB7SWMHICYzM6tJx6Qg6X3AicBU4HpgL+DX+AokM7Nhp8w5hROB3YG7I2IWsAvwx0qjMjOzWpRJCn+JiL8ASNokIm4BXlVtWGZmVocyJ5pXSpoE/BS4TNKjwP3VhmVmZnUoc6L5HXnw05KuBCYCv6w0KjMzq0WPSUHSFi0m35T/jgceqSQiMzOrTbsjhWtJXWQX717uHg9g2wrjMjOzGvSYFCJim4EMxMzM6leml9R3SJpYGJ8k6eBqwzIzszqUuST1UxHxWPdIRKwGPlVdSGZmVpcySaFVmTKXspqZ2RBTJikskfTvkraTtK2kr5BOQpuZ2TBTJil8EHgGOB+4AHgKOKHKoMzMrB5lbl57EjhlAGIxM7OalTlSMDOzEaLSpCBptqRbJS2X1PJoQ9LfS1omaamkH1QZj5mZtVfZVUSSRgNnAG8DVgLXSFoYEcsKZWYAHwfeGBGPSnpJVfGYmVlnZR6ysyVwHDC9WD4iju2w6B7A8oi4M69nAXAQsKxQ5jjgjIh4NK/zod4Eb2Zm/UsR0b6A9L/A1aTLUJ/rnh4RF3ZY7hBgdkS8L48fBewZEXMLZX4K3Aa8ERgNfDoi1uuBVdIcYA7AlClTdluwYEGpN9fOmjVrGD9+/Aavpw433fdYy+k7bT2x5fTealU3rbZZZnt9XW6wGsrfm3b64zvVn9+bKlT9f9NJ3d+dWbNmXRsRMzuVK9N89KKI+Kc+xKAW05oz0EbADKCL9LjPqyXtmO+afn6hiHnAPICZM2dGV1dXH8J5oUWLFtEf66nDMaf8vOX0FUd09cv6W9VNq22W2V5flxushvL3pp3++E715/emClX/33QyVL47ZU40Xyxpvz6seyUwrTA+lfUfzrMSuCgino2Iu4BbSUnCzMxqUPYZzRdLekrS45KekPR4ieWuAWZI2kbSxsBhwMKmMj8FZgFImgxsD9xZPnwzM+tPZW5em9CXFUfEWklzgUtJ5wvOjIilkk4HlkTEwjzv7ZKWkc5XfDQi/tSX7ZmZ2YZr9+S1V0fELZJ2bTU/In7faeURcQlwSdO00wrDAXwkv8zMrGbtjhQ+Qrri58st5gWwdyURmZlZbdo9eW1O/jtr4MIxM7M6lbqjWdJfsf7Na2dXFJOZmdWkzB3N5wDbAdfz/M1rATgpmJkNM2WOFGYCO0SnW5/NzGzIK3Ofwh+Al1YdiJmZ1a/dJak/IzUTTQCWSfod8HT3/Ig4sPrwzMxsILVrPvrSgEVhZmaDQrtLUhcDSPp8c4d4kj4PLK44NjMzG2Blzim8rcW0ffs7EDMzq1+7cwrvBz4AbCvpxsKsCcD/VB2YmZkNvHbnFH4A/AL4LFB8vvITEfFIpVGZmVkt2iWFiIgVkk5oniFpCycGM7Php9ORwgGkx3AGL3ySWgDbVhiXmZnVoN3VRwfkv9sMXDhmZlanMn0fnQ1cDVwdEbdUH5JZ303v6Tm8n9t/gCMxG5rKXJI6H9gK+A9Jd0i6UNKJ1YZlZmZ1KPM4ziskLQZ2Jz1P+XjgtcDXKo7NzMwGWJnmo8uBTYFfk5qRdo+Ih6oOzMzMBl6Z5qMbgWeAHYHXATtKGldpVGZmVosyzUcfBpA0HngP8H1SV9qbVBuamZkNtDLNR3OBNwO7AXcDZ5KakczMbJgp8+S1ccC/A9dGxNqK4zEzsxqVaT764kAEYmZm9StzotnMzEYIJwUzM2vomBQkzZW0+UAEY2Zm9SpzpPBS4BpJF0iaLUkdlzAzsyGpY1KIiFOBGcD3gGOA2yX9m6TtKo7NzMwGWKlzChERwKr8WgtsDvxI0hcqjM3MzAZYmZvXPgQcDTwMfBf4aEQ8K2kUcDvwsWpDNLP+MBK6Fe/rexwJdVNWmZvXJgN/FxF3FydGxDpJB1QTlpmZ1aFM89ElQON5zJImSNoTICJuriowMzMbeGWSwn8CawrjT+ZpZmY2zJRJCsonmoHUbES5ZiczMxtiyiSFOyV9SNKY/DoRuLPMyvN9DbdKWi7plDblDpEUkmaWDdzMzPpfmT3+44GvA6cCAVwOzOm0kKTRwBnA24CVpBvgFkbEsqZyE4APAb/tXei9V7zC4KSd1nJMHh+JVxiYmbVSppfUh4DD+rDuPYDlEXEngKQFwEHAsqZy/wJ8ATi5D9swM7N+pMLpgtYFpLHAe4HXAmO7p0fEsR2WOwSYHRHvy+NHAXtGxNxCmV2AUyPinZIWASdHxJIW65pDPjqZMmXKbgsWLCj37prcdN9jjeEp4+DBp9LwTltP7NP66lJ8H0X99T7WrFnD+PHjO26zzPb6ulxf1VE3Q0W7uumPeuvP701f9fU9Vv29gfq/O7Nmzbo2Ijo20ZdpPjoHuAX4G+B04AigzKWorfpIamSgfPPbV0hdZ7QVEfOAeQAzZ86Mrq6uEptf3zFNzUdfvim9/RVH9G19dTmmpxtt+ul9LFq0iOY6brXNMtvr63J9VUfdDBXt6qY/6q0/vzd91df3WPX3BobOd6fMieZXRsQngScj4ixgf2CnEsutBKYVxqcC9xfGJwA7AoskrQD2Ahb6ZLPZ0DL9lJ8z/ZSfc9N9jzWGbegqkxSezX9XS9oRmAhML7HcNcAMSdtI2ph0XmJh98yIeCwiJkfE9IiYDvwGOLBV85GZmQ2MMs1H8/LzFE4l/aiPBz7ZaaGIWCtpLnApMBo4MyKWSjodWBIRC9uvwWxoaLVn7CvabKhqmxRyu//jEfEocBWwbW9WHhGXkLrJKE47rYeyXb1Zt5mZ9b+2zUf57uW57cqYmdnwUeacwmWSTpY0TdIW3a/KIzMzswFX5pxC9/0IJxSmBb1sSjIzs8GvzB3N2wxEIGZmVr8yT157d6vpEXF2/4djZmZ1KtN8tHtheCzwVuD3gJOC2SDjx0rahirTfPTB4rikiaSuL8zMbJjpy8Ny/gzM6O9ArHreizSzTsqcU/gZz3dkNwrYAbigyqDMzKweZY4UvlQYXgvcHRErK4rHzMxqVCYp3AM8EBF/AZA0TtL0iFhRaWRmZkNcq6c9Dvbm2jJ3NP8QWFcYfy5PMzOzYabMkcJGEfFM90hEPJO7wjYzq4QviqhPmSOFP0o6sHtE0kHAw9WFZGZmdSlzpHA8cK6kb+TxlUDLu5zNzGxoK3Pz2h3AXpLGA4qIJ6oPy8zM6tCx+UjSv0maFBFrIuIJSZtL+sxABGdmZgOrzDmFfSNidfdIfgrbftWFZGZmdSmTFEZL2qR7RNI4YJM25c3MbIgqc6L5v4DLJX2f1N3FsbiHVCvwg+vNho8yJ5q/IOlGYB9AwL9ExKWVR2ZmZgOuVC+pEfFL4JcAkt4o6YyIOKHDYmZmNsSUSgqSdgYOB94F3AX8uMqgzMysHj0mBUnbA4eRksGfgPNJ9ynMGqDYzMxsgLU7UrgFuBr424hYDiDpwwMSlZmZ1aLdJanvBFYBV0r6jqS3kk40m5nZMNXjkUJE/AT4iaRNgYOBDwNTJP0n8JOI+NUAxWhmBe5B1KrU8ea1iHgyIs6NiAOAqcD1wCmVR2ZmZgOuzB3NDRHxSER8OyL2riogMzOrT6+SgpmZDW9OCmZm1uCkYGZmDU4KZmbW4KRgZmYNlSYFSbMl3SppuaT1LmOV9BFJyyTdKOlySa+oMh4zM2uvsqQgaTRwBrAvsANwuKQdmopdB8yMiNcBPwK+UFU8ZmbWWaleUvtoD2B5RNwJIGkBcBCwrLtARFxZKP8b4MgK46mFH0BjZkOJIqKaFUuHALMj4n15/Chgz4iY20P5bwCrIuIzLebNAeYATJkyZbcFCxb0Kaab7nusMTxlHDz4VBreaeuJfVpfb7fZbUO312qdZdZbdrk1a9Ywfvz4jst2L9fXeVXoa92U1du6qUK791jFvLLxtPqf6mvdVPEdr+r9l42nu36q/G60M2vWrGsjYmanclUmhUOBv2lKCntExAdblD0SmAv8dUQ83W69M2fOjCVLlvQppuJe+0k7reXLN6UDpSr33Ks4Uuhr3zdll1u0aBFdXV0dl+1erq/zqlB1v0C9rZsqtHuPVcwrG0+r/6m+1k0V3/Gq3n/ZeLrrp66WAkmlkkKVzUcrgWmF8anA/c2FJO0DfIISCcHMzKpV5dVH1wAzJG0jaWPSA3sWFgtI2gX4NnBgRDxUYSxmZlZCZUkhItaSmoQuBW4GLoiIpZJOl3RgLvZFYDzwQ0nXS1rYw+rMzGwAVNl8RERcAlzSNO20wvA+VW7fzMx6x3c0m5lZg5OCmZk1OCmYmVmDk4KZmTU4KZiZWYOTgpmZNTgpmJlZg5OCmZk1OCmYmVmDk4KZmTU4KZiZWYOTgpmZNTgpmJlZg5OCmZk1OCmYmVlDpc9TMKtC1c9hNhvJfKRgZmYNPlIoqdXeqfdMrRN/b2yocVIw4IU/XifttJZj8rh/wMxGFieFzO3UZmY+p2BmZgVOCmZm1uDmI7OauMnSBiMfKZiZWYOPFMzMatDuSLHOo0gnBbMSuv9JfbmuDXduPjIzswYnBTMza3DzkZl15O46Rg4nhRr5kkQzG2ycFMwGIe8wWF18TsHMzBp8pNAP3N5qZsOFjxTMzKzBScHMzBoqTQqSZku6VdJySae0mL+JpPPz/N9Kml5lPGZm1l5lSUHSaOAMYF9gB+BwSTs0FXsv8GhEvBL4CvD5quIxM7POqjxS2ANYHhF3RsQzwALgoKYyBwFn5eEfAW+VpApjMjOzNhQR1axYOgSYHRHvy+NHAXtGxNxCmT/kMivz+B25zMNN65oDzMmjrwJu7YcQJwMPdyw1Mrlueua66Znrpr266+cVEbFlp0JVXpLaao+/OQOVKUNEzAPm9UdQjQ1LSyJiZn+uc7hw3fTMddMz1017Q6V+qmw+WglMK4xPBe7vqYykjYCJwCMVxmRmZm1UmRSuAWZI2kbSxsBhwMKmMguBo/PwIcAVUVV7lpmZdVRZ81FErJU0F7gUGA2cGRFLJZ0OLImIhcD3gHMkLScdIRxWVTwt9Gtz1DDjuumZ66Znrpv2hkT9VHai2czMhh7f0WxmZg1OCmZm1jAik0Kn7jdGEklnSnoo3zPSPW0LSZdJuj3/3bzOGOsiaZqkKyXdLGmppBPz9BFfP5LGSvqdpBty3fxznr5N7rLm9tyFzcZ1x1oXSaMlXSfp4jw+JOpmxCWFkt1vjCTzgdlN004BLo+IGcDleXwkWgucFBGvAfYCTsjfFdcPPA3sHRGvB3YGZkvai9RVzVdy3TxK6spmpDoRuLkwPiTqZsQlBcp1vzFiRMRVrH9vSLH7kbOAgwc0qEEiIh6IiN/n4SdI/+Bb4/ohkjV5dEx+BbA3qcsaGKF1AyBpKrA/8N08LoZI3YzEpLA1cG9hfGWeZs+bEhEPQPphBF5Sczy1yz347gL8FtcP0GgeuR54CLgMuANYHRFrc5GR/L/1VeBjwLo8/mKGSN2MxKRQqmsNs26SxgMXAv8YEY/XHc9gERHPRcTOpN4K9gBe06rYwEZVP0kHAA9FxLXFyS2KDsq6GYmP4yzT/cZI96CkrSLiAUlbkfYERyRJY0gJ4dyI+HGe7PopiIjVkhaRzrtMkrRR3iMeqf9bbwQOlLQfMBbYjHTkMCTqZiQeKZTpfmOkK3Y/cjRwUY2x1Ca3A38PuDki/r0wa8TXj6QtJU3Kw+OAfUjnXK4kdVkDI7RuIuLjETE1IqaTfl+uiIgjGCJ1MyLvaM4Z/Ks83/3Gv9YcUm0knQd0kbr1fRD4FPBT4ALg5cA9wKERMeI6KpT0JuBq4Caebxv+v6TzCiO6fiS9jnSydDRp5/KCiDhd0rakize2AK4DjoyIp+uLtF6SuoCTI+KAoVI3IzIpmJlZayOx+cjMzHrgpGBmZg1OCmZm1uCkYGZmDU4KZmbW4KRg/UbSSyUtkHSHpGWSLpG0vaTpxV5Y6yTp05JO7qd1jZO0OHeyOCRIOjT3+nplP65zrqT39Nf6rF5OCtYv8o1ePwEWRcR2EbED6Zr+KfVGVqljgR9HxHN1B9JKD8nqvcAHImJWP27qTOBD/bg+q5GTgvWXWcCzEfGt7gkRcX1EXF0slI8arpb0+/z6qzx9K0lXSbpe0h8kvTl3uDY/j98k6cNN65ooaYWkUXn8RZLulTRG0nGSrsn9/V8o6UXNAUtaJGlmHp4saUUeHi3pi3n5GyX9Qw/v+QjyXalKvliI9V15elfezo8k3SLp3JxAkbRbPtK4VtKlucuM5hjnS/pWrrPbcr86PcaYt3elpB+Qbrorrus04E3At/KyLT+LXPZj+X3cIOlzedp2kn6Z471a0qvz5/xnYIWkPXqoJxtKIsIvvzb4RdpT/EoP86YDf8jDLwLG5uEZwJI8fBLwiTw8GpgA7AZcVljPpBbrvgiYlYffBXw3D7+4UOYzwAfz8KdJd5gCLAJm5uHJwIo8PAc4NQ9vAiwBtmna7sbAqsL4O0k9hY4mHR3dA2xFulv8MVJfN6OAX5N+mMcA/wtsWYj9zBbvbz7wy7ypZa92AAADL0lEQVTsDFLfXWN7ijFv78nmeAvrK77nnj6LfXNsL8rjW+S/lwMz8vCepO4butf7CdKzJ2r/Lvq1Ya+R2CGe1WsM8A1JOwPPAdvn6dcAZyp1QPfTiLhe0p3AtpL+A/g58KsW6zuf9IN6JamfmW/m6TtK+gwwCRgPXNqLGN8OvE5Sdz81E0k/mncVykwGVhfG3wScF6kp6UFJi4HdgceB30XESgClrqan52V3BC7LBw6jgQd6iOeCiFgH3J7r5NVtYnwmb++u1qt6gZ4+i32A70c6AiAiHlHqKfavgB/meCElo24P5bhsiHNSsP6ylOc7+2rnw6Q+ll5P2vv9C6SH/Uh6C+nBJOdI+mJEnC3p9cDfACcAf09qxy9aCHxW0hakI4sr8vT5wMERcYOkY0h70M3W8nwT6tjCdJGOLNolkqdaLNOTYv82z5H+7wQsjYg3tFmuW3NfNNFTjEp97TxZYp3Qw2eR1928zVGk5wHs3MO6xpLqxIY4n1Ow/nIFsImk47onSNpd0l83lZsIPJD3fI8i7SEj6RWkPui/Q+qZdFdJk4FREXEh8Elg1+aNRnr61++ArwEXx/MnfScAD+QjjyN6iHkFKZHACxPapcD787IoXUG1adN2HwVGS+pODFcB78pt/VsCb8lx9eRWYEtJb8jbGCPptT2UPVTSKEnbAdvmZTvGWELLz4J0RHZs93kYSVtEeo7EXZIOzdOUE3a37YFBcYWZbRgnBesXERHAO4C3KV2SupTUft/cZ/w3gaMl/Yb0Q9K9V9sFXC/pOlL7/NdIT6ZalJtc5gMf72Hz5wNH5r/dPknqzfQy4JYelvsS6Yf1f0nNQd2+CywDfq90Ke23aX1U/StSsxGkK69uBG4gJciPRcSqHrZLpEfBHgJ8XtINwPWk5plWbgUWA78Ajo+Iv/QixnZafhYR8UvSEdiSXPfdl/AeAbw3x7uUFz7G9o3A/+vl9m0Qci+pZn0kaRfgIxFxVIXbmE86AvpRp7J1GYh6sIHjIwWzPoqI64ArNYRuXqvIZNKRmQ0DPlIwM7MGHymYmVmDk4KZmTU4KZiZWYOTgpmZNTgpmJlZw/8Hb8qe2LQ43VsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 36.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "clf = RandomForestClassifier(max_depth=50, n_estimators=150, n_jobs=-1, oob_score=True)\n",
    "\n",
    "# now iterate through and get predictions, saved to the correct row in yhat\n",
    "for train, test in cv.split(X,y):\n",
    "    clf.fit(X[train],y[train])\n",
    "    yhat[test] = clf.predict(X[test])\n",
    "    \n",
    "total_accuracy = mt.accuracy_score(y, yhat)\n",
    "print ('Accuracy', total_accuracy)\n",
    "plot_class_acc(y,yhat,title=\"Random Forest, Raw\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
      "                       max_depth=50, max_features='auto', max_leaf_nodes=None,\n",
      "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
      "                       min_samples_leaf=1, min_samples_split=2,\n",
      "                       min_weight_fraction_leaf=0.0, n_estimators=150,\n",
      "                       n_jobs=-1, oob_score=True, random_state=None, verbose=0,\n",
      "                       warm_start=False)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD8CAYAAACVZ8iyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFi9JREFUeJzt3X+sZGV9x/H310W3toqIrPa6P7rQXGzApqi3YGJLqVaL1LraqIUmZQXi+gOqNjYB1FSjNcEfKEQb7EUpbCsL1J8buxbR1mATFrmLiCCiyw/luhsWXUSIydpdv/1jzupwuT/mzsyZec7M+5Vs7plnzjlzzsnZ53Oe5zlnJjITSdJ4e9ywN0CSNHyGgSTJMJAkGQaSJAwDSRKGgSQJw0CShGEgScIwkCQBhwx7A5ZyxBFH5Pr164e9GZLUGDt27PhxZq5azjLFh8H69euZmZkZ9mZIUmNExA+Wu4zdRJIkw0CSZBhIkjAMJEkYBpIkDANJEoaBJAnDQJJEAx46+/aPHmL9ef857M0Yqnsv+Ithb4KkEVd8GIwDK3tJw2YYFKDTlo+hIakujhlIkspvGfz+6qcw4xWxJNWq+DAY9wFku4YkDULxYTCODABJg7ZkGETEZcDLgD2Z+eyq7GrgWdUshwE/zczjImI9cAdwZ/Xe9sx8Q7XM84DLgScC24C3ZGYu9fl2E0lS/TppGVwOfAzYfLAgM//64HREXAg81Db/XZl53DzruQTYBGynFQYnA19a6sPHvZsIbClIqt+SYZCZ11dX/I8REQG8BnjhYuuIiAng0My8oXq9GXgFHYSBOr/1tC6GkTT6eh0z+GPg/sz8flvZkRHxTeBnwDsz8+vAamC2bZ7ZqmxJdhNJUv16DYPTgC1tr3cD6zLzJ9UYwecj4lgg5ll2wfGCiNhEq0uJFYeuGvqVcd288pY0bF2HQUQcAvwV8LyDZZm5D9hXTe+IiLuAo2m1BNa0Lb4G2LXQujNzGpgGWDkxueQgcxNY4UsqWS8tgz8DvpuZv+r+iYhVwN7MPBARRwGTwN2ZuTciHo6I5wM3AqcDH+1lw5umhNaNgSRpIZ3cWroFOAk4IiJmgXdl5ieBU3l0FxHAicB7ImI/cAB4Q2burd57I7++tfRLOHjcMyt3Sf0SHdzqP1QrJyZzYuNFw96MX7ECllS6iNiRmVPLWcYnkJepm+4eA0RS6QyDARjUeIGhI6lbxYeBzxlIUv2KDwO/jqLFq35JdSo+DMaBFb2kYSs+DOwmkqT6FR8GdhPVy1aJJGhAGIwyK2JJpSg+DOwmkqT6FR8GdhMtj60NSd0oPgy0PO3BaTBI6pRhUDArc0mDYhjUxIpcUpMYBjUZ9XEOw04aLY8b9gZIkobPlkENvGqW1DTFh4HPGUhS/YoPA58zWD5bJpKWq5PfQL4MeBmwJzOfXZW9G3gd8EA129szc1v13vnAWbR+A/nNmXltVX4ycDGwAvhEZl7Q310ZT1b8kvphyd9AjogTgUeAzXPC4JHM/NCceY8BtgDHA88EvgIcXb39PeDFwCxwE3BaZn5nqQ2cmprKmZmZZeySJI23Wn4DOTOvj4j1Ha5vA3BVZu4D7omInbSCAWBnZt5dbehV1bxLhsGodhN5RS+pJL2MGZwTEacDM8DbMvNBYDWwvW2e2aoM4L455Sf08NnFspKX1ETdhsElwHuBrP5eCJwJxDzzJvM/z7Bg/1REbAI2Aaxbt84KVpJq1lUYZOb9B6cj4lLgi9XLWWBt26xrgF3V9ELl861/GpgGWDkxmaPYTbQYw0/SoHUVBhExkZm7q5evBG6rprcCV0bEh2kNIE8C36DVYpiMiCOBHwGnAn/Ty4aPsrnhZzhIqlsnt5ZuAU4CjoiIWeBdwEkRcRytrp57gdcDZObtEXENrYHh/cDZmXmgWs85wLW0bi29LDNv7/veFMCKW1ITLXlr6bCtnJjMiY0XDXszumIwSBqGWm4tVfcWG+swKCSVxDAYkuUOihsekupkGNTMSlxSExQfBn5rqSTVr/gwGNWvo1iKLQpJg1R8GIyrXgPQMJG0HMWHgd1EklS/4sOgyd1EXp1Laoriw6DJBh1iho+kbhkGDWblL6lfDIMG60fLw0CRBIZBI1hhS6qbYVAAK3tJw2YYFKCT7h4DQ1Kd5vs5ShWoqbfXSmoGWwYN0ksg2LKQtBjDoFBW3pIGqfgw8OsoJKl+nfwG8mXAy4A9mfnsquyDwF8CvwDuAs7IzJ9GxHrgDuDOavHtmfmGapnnAZcDTwS2AW/JDn5zs8lfR9FEtkik8dRJy+By4GPA5ray64DzM3N/RLwfOB84t3rvrsw8bp71XAJsArbTCoOTgS91ud2qSZ3Ba9BI5VoyDDLz+uqKv73sy20vtwOvWmwdETEBHJqZN1SvNwOvoIMwsJtIkurXjzGDM4Gr214fGRHfBH4GvDMzvw6sBmbb5pmtyuYVEZtotSJYcegqu4kazhaBVL6ewiAi3gHsBz5VFe0G1mXmT6oxgs9HxLFAzLP4guMFmTkNTAOsnJhcclxBj2blK2m5ug6DiNhIa2D5RQcHgjNzH7Cvmt4REXcBR9NqCaxpW3wNsKvbz9biSm1JGVJSuboKg4g4mdaA8Z9k5s/bylcBezPzQEQcBUwCd2fm3oh4OCKeD9wInA58tPfNHy1WlpKGZcmvo4iILcANwLMiYjYizqJ1d9GTgesi4paI+Hg1+4nArRHxLeDTwBsyc2/13huBTwA7ad2O6p1Ec5R6RS9p9EUHt/oP1cqJyZzYeNGwN6NRbGFI4y0idmTm1HKWKf4JZC3fMFoYBpDUbMWHgc8ZSFL9ig+DUf46Cq+mJZWi+DAYZXWFnCEjabmKDwO7iSSpfsWHwSh3E4FX8ZLKUHwYlMpKXNIoKT4M7CaSpPoVHwaj3k00KLZkJC2m+DAYN1bakoZhye8m0uAYBJKGxZbBkBkAkkpgGAzZQuMhhoSkQTIMhsTKXlJJDIMhaW8RGAyShq34MPA5A0mqX/Fh4HMG87M1IamfOgqDiLgMeBmwJzOfXZUdDlwNrAfuBV6TmQ9GRAAXA6cAPwdem5k3V8tsBN5ZrfafMvOK/u3KaLPyl1Snjn72MiJOBB4BNreFwQeAvZl5QUScBzw1M8+NiFOAv6MVBicAF2fmCVV4zABTQAI7gOdl5oOLffbU1FTOzMx0v4eSNGZq+9nLzLw+ItbPKd4AnFRNXwF8DTi3Kt+crZTZHhGHRcRENe91mbm32tjrgJOBLYt99rh2E9kSkDRIvYwZPCMzdwNk5u6IeHpVvhq4r22+2apsoXJVDABJw1LHAHLMU5aLlD92BRGbgE0A69ats5KUpJr1Egb3R8RE1SqYAPZU5bPA2rb51gC7qvKT5pR/bb4VZ+Y0MA2wcmIyR6mbyGCTVKJevqhuK7Cxmt4IfKGt/PRoeT7wUNWddC3wkoh4akQ8FXhJVTY2DAJJper01tIttK7qj4iIWeBdwAXANRFxFvBD4NXV7Nto3Um0k9atpWcAZObeiHgvcFM133sODiaPi+W2cAwPSYPS0a2lw7RyYjInNl407M3omRW7pEGp7dZSLZ+Vv6QmMQz6xMpfUpP5S2eSpPJbBn5rqSTVr/gwGIevo7CLSdKwFR8GTWdFL6kJig8Du4kkqX7Fh0FTu4lsEUhqkuLDoKmGFWCGkKRuFB8GdhNJUv2KD4OmdhP1i1f6kgah+DAYVVbykkriE8iSJFsGdfHKX1KTGAY1aeo4hyEmjSfDoIGssCX1m2HQQO2tDoNBUj8UHwY+ZyBJ9es6DCLiWcDVbUVHAf8IHAa8DnigKn97Zm6rljkfOAs4ALw5M69d6nNG9TkDr+gllaQvv4EcESuAHwEnAGcAj2Tmh+bMcwywBTgeeCbwFeDozDyw2LpH5TeQm8awkpprmL+B/CLgrsz8QUQsNM8G4KrM3AfcExE7aQXDDYut2G4iSapfv8LgVFpX/QedExGnAzPA2zLzQWA1sL1tntmqbFGj2k3UNLYUpNHWcxhExBOAlwPnV0WXAO8Fsvp7IXAmMF+TYd4+qojYBGwCWHHoql43caRZSUvqh360DF4K3JyZ9wMc/AsQEZcCX6xezgJr25ZbA+yab4WZOQ1MA0xNTaXdRJJUr36EwWm0dRFFxERm7q5evhK4rZreClwZER+mNYA8CXxjqZXbTfRotgQk1aGnMIiI3wReDLy+rfgDEXEcrS6gew++l5m3R8Q1wHeA/cDZS91JpBYDQFLd+nJraZ28tfTXDAVJnRjmraUagKW6ywwLSd0yDAbEilpSyQyDARnUILihI6kbhsGIqTt0DBtpNBUfBn4dhSTVr/gw8DmDxXmlLqkfig8DLW4cgtLAk+pXfBjYTSRJ9Ss+DMaxm8grYUmDVnwYjAsDQNIwFR8GdhNJUv2KD4Nx6yayhSBpGIoPg3FTV/AZMpIWYxiMiU5CxsCQxpdhoF9pQnecgSXVwzAomBWfpEExDArWhCv1+RhiUvM8btgboNHT1BCTxlnPLYOIuBd4GDgA7M/MqYg4HLgaWE/rd5Bfk5kPRkQAFwOnAD8HXpuZNy+2fp8zkKT69aub6E8z88dtr88DvpqZF0TEedXrc4GXApPVvxOAS6q/Cxq35wyWyy4ZSf1Q15jBBuCkavoK4Gu0wmADsDkzE9geEYdFxERm7q5pO0beYkFpUEjqVD/CIIEvR0QC/5KZ08AzDlbwmbk7Ip5ezbsauK9t2dmq7FFhEBGbgE0A69ats1KTpJr1IwxekJm7qgr/uoj47iLzxjxl+ZiCVqBMA6ycmEy7ifrHYJU0n57DIDN3VX/3RMTngOOB+w92/0TEBLCnmn0WWNu2+BpgV6/bMI6s1CX1U0+3lkbEb0XEkw9OAy8BbgO2Ahur2TYCX6imtwKnR8vzgYccL+iOrSVJ/dRry+AZwOdad4xyCHBlZv5XRNwEXBMRZwE/BF5dzb+N1m2lO2ndWnpGj58/1koIBFso0mjoKQwy827gD+Yp/wnwonnKEzi7l88cZVaskobFr6MoSAlX+gsxqKTRZhg0lJWzpH4yDBqqm1aEASJpIYZBQ1iRS6qTYdAQgxxPMHik8VN8GPitpZJUv+LDoLRvLfWqWdIoKj4MStOvYDJUJJWk+DCwm0iS6ld8GJTWTbRctgAkNUHxYdB0nQSZgSFp2AyDAvS75WO4SFouw2AEldCtZiBJzWIYDJAVpKRSGQYD1OkVu6EhadB6+qUzSdJoKL5l4HMGklS/rsMgItYCm4HfBn4JTGfmxRHxbuB1wAPVrG/PzG3VMucDZwEHgDdn5rVLfU7TnjOwi0dSE/XSMtgPvC0zb46IJwM7IuK66r2PZOaH2meOiGOAU4FjgWcCX4mIozPzQA/bUByfK5DURF2HQWbuBnZX0w9HxB3A6kUW2QBclZn7gHsiYidwPHDDYp9jN5Ek1a8vYwYRsR54DnAj8ALgnIg4HZih1Xp4kFZQbG9bbJbFwwNoXjfRuLB1I42WnsMgIp4EfAZ4a2b+LCIuAd4LZPX3QuBMIOZZPBdY5yZgE8CKQ1f1uol9YeUnaZT1FAYR8XhaQfCpzPwsQGbe3/b+pcAXq5ezwNq2xdcAu+Zbb2ZOA9MAU1NTaTeRJNWrl7uJAvgkcEdmfritfKIaTwB4JXBbNb0VuDIiPkxrAHkS+MZSnzPq3US2OCSVoJeWwQuAvwW+HRG3VGVvB06LiONodQHdC7weIDNvj4hrgO/QuhPp7FG7k6gbpQSdoSSNt17uJvpf5h8H2LbIMu8D3tftZzaRlaykJij+CeSmMgQkNYlhUBN/o0BSkxgGDVHK2MJ8DCqp+QyDAbHClFSy4sPAr6OQpPoVHwaj8pyBLQNJJSs+DEbFKARaPxmOUlmKDwO7iSSpfsWHQandRF7ZSholxYdBqUoMqE4ZZJLmKj4M7CaSpPoVHwaldhPNxytuSU1VfBg0iV9BIampDIOCDbJFZPBI480wGGFW8JI6ZRj0mRWwpCYyDLpghS9p1BgGXWjK3U3tDDBJixl4GETEycDFwArgE5l5wWLz+5yBJNVvoGEQESuAfwZeDMwCN0XE1sz8zkLLNOk5g2Hz6l9StwbdMjge2JmZdwNExFXABmDBMFDn5oam4SCpU4MOg9XAfW2vZ4ETFlvAbiJJqt/jBvx5MU9ZPmamiE0RMRMRMw888MAANkuSxtugw2AWWNv2eg2wa+5MmTmdmVOZObVq1aqBbZwkjatBh8FNwGREHBkRTwBOBbYOeBskSXMMdMwgM/dHxDnAtbRuLb0sM28f5DZIkh5r4M8ZZOY2YNugP1eStLBBdxNJkgpkGEiSDANJkmEgScIwkCQBkfmYB4CLEhEPA3cOezuG6Ajgx8PeiCEb92Mw7vsPHgNY3jH4ncxc1hO7Tfg9gzszc2rYGzEsETEzzvsPHoNx33/wGED9x8BuIkmSYSBJakYYTA97A4Zs3PcfPAbjvv/gMYCaj0HxA8iSpPo1oWUgSapZ7WEQESdHxJ0RsTMizpvn/ZURcXX1/o0Rsb7tvfOr8jsj4s+XWmf11dg3RsT3q3U+oe7968SAj8HlEXFPRNxS/Tuu7v1bSk37f1lE7ImI2+as6/CIuK46B66LiKfWuW+dGvAxeHdE/KjtHDilzn3rRL/3PyLWRsT/RMQdEXF7RLylbf6xOAeWOAbLPwcys7Z/tL6m+i7gKOAJwLeAY+bM8ybg49X0qcDV1fQx1fwrgSOr9axYbJ3ANcCp1fTHgTfWuX+FHoPLgVcNe7/r3P/qvROB5wK3zVnXB4DzqunzgPeP4TF4N/APw97vmv8PTADPreZ5MvC9tv8DY3EOLHEMln0O1N0yOB7YmZl3Z+YvgKuADXPm2QBcUU1/GnhRRERVflVm7svMe4Cd1frmXWe1zAurdVCt8xU17lunBnYMBrAv3ahj/8nM64G983xe+7pG+RxY7BiUpu/7n5m7M/NmgMx8GLiD1m+sz13XyJ4DSxyDZas7DFYD97W9nuWxG/ureTJzP/AQ8LRFll2o/GnAT6t1LPRZwzDIY3DQ+yLi1oj4SESs7MdO9KCO/V/MMzJzd7Wu3cDTu97y/hn0MQA4pzoHLiugm6TW/a+6U54D3FgVjd05MM8xgGWeA3WHQcxTNvf2pYXm6Vf5sA3yGACcD/we8IfA4cC5nW1mberY/6YZ9DG4BPhd4DhgN3DhUhtYs9r2PyKeBHwGeGtm/qzrLazfoI/Bss+BusNgFljb9noNsGuheSLiEOAptJq+Cy27UPmPgcOqdSz0WcMwyGNA1XTMzNwH/CtVl8IQ1bH/i7k/IiaqdU0Ae7re8v4Z6DHIzPsz80Bm/hK4lBE9ByLi8bQqwU9l5mfb5hmbc2ChY9DVOVDzoMkhwN20Bj0ODpocO2ees3n0oMk11fSxPHrQ5G5agyYLrhP4Dx49gPymOvev0GMwUf0N4CLgglHb/7bl1vPYwdMP8ujBww+M4jmwxDGYaJv+e1r9zSO1/9X5vRm4aJ7PG4tzYIljsOxzYBAH4RRao9x3Ae+oyt4DvLya/g1alfhO4BvAUW3LvqNa7k7gpYutsyo/qlrHzmqdK4d9EgzhGPw38G3gNuDfgSeN6P5vodX8/T9aV05nVeVPA74KfL/6e/iw938Ix+DfqnPgVmBre8UwKvsP/BGtrpJbgVuqf6eM0zmwxDFY9jngE8iSJJ9AliQZBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJKA/wfqfWXwHgXW1wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generalization score estimate from training data 0.44193262411347517\n"
     ]
    }
   ],
   "source": [
    "# now lets get access to the different properties of our RF\n",
    "\n",
    "print (clf)\n",
    "\n",
    "plt.barh(range(len(clf.feature_importances_)), clf.feature_importances_)\n",
    "plt.show()\n",
    "\n",
    "print ('Generalization score estimate from training data', clf.oob_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMYAAAD8CAYAAAAsetuWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHfJJREFUeJztnX2sXlWVxp/V2y8UagtU+nFbSj8AG6XFFELCmBAUg85kMMZJdMiESUj4ZybRjJMRZ5LJmEwi/iP+M5E0g7ExKn5gAjFGQhBiJpLWlgpDW+gnbW9bKViqKErLvWv+uG+b9zz76T27t+25b53nlzT37nPPxz7nfVfPevZae+3ITBhjmkyb6g4YM4jYMIwR2DCMEdgwjBHYMIwR2DCMEdgwjBHYMIwRnJNhRMSdEfFyROyOiPvPV6eMmWpispHviBgCsBPAHQBGAPwSwGcyc/uZjpkzZ07Onz//dHvatNIujx071mifPHmy0VbH8D1ExIRtAPjTn/7UaA8NDRX7jI6OTngdBV+rrW8AMDY2NuE+77zzTut1p0+f3mhz39U+J06caLRnzJhRHNPWF/V58Da+P3Utfv7cVwCYOXPmhOddsmRJcQx/n/bt2/d6Zs4vdiTKq9dzM4DdmbkXACLiEQB3ATijYcyfPx8PPPDA6fall15a7POtb32r0X799dcb7UsuuaQ4hr/ks2fPbrTVl37nzp2NturLb3/720a75gvK1+a+zZo1qzjm97///YTneO2114pj+As7b968Rvt3v/tdcczll1/eaB86dKjRfu9731scw1/g3/zmN432u9/97uIY7v9bb71V7LNw4cJG+z3veU+jfcUVVxTHLFu2rNH+4x//2Gg/+OCDxTHf+c53Gu277757f7GT4FxcqcUADva1R3rbjLnoORfDKH0CoPA1IuK+iNgcEZvV/2LGDCLn4kqNAOh36oYBHOadMnM9gPUAcP311+dVV111+m+/+MUv2jtIvuaBAweKfdiNYL/y+PHjxTHsX/NrGSjdCNY7fA6gdE/mzJnTaL/99tvFMbyNr6PcR742H6M0xuHDzY+HtYByFfk/s3e9613FPgzrKtUXdlN5H+6r2mf16tWN9kMPPVQcs2bNmok7ewbO5Y3xSwCrIuKaiJgJ4NMAHj+H8xkzMEz6jZGZ70TEPwJ4AsAQgG9k5rbz1jNjppBzcaWQmT8B8JPz1BdjBgZHvo0RnNMb42x56623sHXr1tPtV199tdiHhfLzzz/faKsgzr59+xptHkdXo2GXXXZZ6z48/j537txGm+MPah8OSqnA1Ztvvtloc4xCBchYbP/6179utJVgZ1gksyBW+/D9qIAlf4YcP1H78Gd23XXXFcf0B4eBUowvXlxGC44ePVpsq8FvDGMENgxjBDYMYwSdaozR0dGGP62CXVdeeWWjvWrVqkZbJfJxXg3nV9X4uCqBjgNgrClUsIv9dO6vygFqC7QtWLCgOGbPnj2NNuc5KV3CeVscJFT5YqyzWA8pzcT65w9/+EOxDz8H1nhKv3HwdNeuXRO2AWDFihXFthr8xjBGYMMwRmDDMEbQqcaYOXMmrr766tNtlVzGfiLrEPaTAWBkZKTRZp9cHcO+stILPEbP+kH58awXeC4Ij9cDpZ/+xhtvNNpqPgbHS7hvPOavrsPHqOfEvj7Pm6iZqKSSLXkbP/+a+TEct+j/bk20rQa/MYwR2DCMEdgwjBHYMIwRdCq+Z8yY0RDGasYVT67nYJFKCmMhykEpJZJZRPKsP6AU3ywQ+RxAKWC5zcFHoAwusnhVwUcW0pzcd+TIkdZj+H6USGYRzIMJ6hnws1y6dGmxDw+qsMjfuHFja194AEU9W/7+1OI3hjECG4YxAhuGMYJONcbJkycbvi9X0ADK5D7WFMrf5n1Yp/CkJHUdBfePi6WpyiJ8bU4IVJOB+iunqHOo67BuUsHStr6xj66SLXkC1KJFixptFXxk7cLXAUrfnz8jlWDK98zXUcHGT37yk432l7/85WIfhd8YxghsGMYIbBjGCGwYxgg6Fd8nTpxolNhUApgzZTlApmZ2sXjlwJwK/PA+PHMQKDNhWTCqChk8W42DZhyMBEpRybP+VOlM7hsLdDXgwAK9JlOWr8Ofmap2zv1XQprLqnLFGDXr74Ybbmi0+Z6vv/764phXXnml2FaD3xjGCGwYxghsGMYIOk8i7NcDagUfrgTBSWuqyoZKfutH+cFty3sBpZ/LgSp1Xq4AyH69qqrBgUPum9ILDPv1Si/weVkvqGNYh7DmU8+Nn5Oatcjwd4FnMQKlVuTzKj3x05/+tPXaCr8xjBHYMIwR2DCMEXRe7fyFF1443VbxBZ7kwtXN1VJjnPzGPrvycWvGt1XCYhvsx/O1lR5iXdJWmQMofXBeBVUl97Fe4woavCIrUMZQ+P6UZuIKJSoOw1qSJyGpOAbrmRrtos5Tg98YxghsGMYIWg0jIr4REUcj4sW+bZdHxJMRsav3c95E5zDmYqPmjfFNAHfStvsBPJWZqwA81Wsb82dDq/jOzJ9HxDLafBeA23q/bwDwDIAvtJ3rxIkT2L9//+n27t27i33aksvUrD8WWCzS1Ow2DpopgciCkPdRFSj42ioAxrAQ5euoAB8HOllsq2RFfg5cTUWJfB4IUKUzGU7uY9EPlKKdj6kp68nn4CXnAD0rsYbJaoyrMvMIAPR+liFsYy5iLrj4joj7ImJzRGxuS90wZlCYrGG8GhELAaD384xLY2bm+sxcl5nruCiYMYPKZAN8jwO4B8ADvZ+P1Rw0OjraWFKqpoQ8+9eqqiAH4tivV8uTcYUJFSxqm2SkdAn7vXwOFURbuXJlo80JdSpYx8l8H/jAByb8OwAcPny40Wa9oJ4BB+t40pFaBpoDrEpnsb7hiWJKG3CFFdZ4fE5AJ6rWUDNc+10AzwK4LiJGIuJejBvEHRGxC8AdvbYxfzbUjEp95gx/+vB57osxA4Mj38YIOk0iBJr+vho3Z7+Rx96VX8/j5Kw5uFgC9wMoYwlA6V9zf3mpZQDYuXNno80+uVoCjP1pvkc1OYvjO6y9VGEAPg9fRy0HPDw83GhzFXWVRMjPSVWob5vApbRY26imivfwc6rFbwxjBDYMYwQ2DGMENgxjBKGCXxeK2bNnZ/+sMRWsa1vzWiXHsfjmIKESciyClYjkINmKFSsabQ5kAe2VRVjMAuVsQhaRSrzy7DsOfql1y/ke+xM6AeDaa68tjuH74eev+saCl2cXAmXCH8/cVEFBDkjywIaqJsmf0ZYtW7Zk5rpiR8JvDGMENgxjBDYMYwSdB/j6fUc1gYV9Qva31bJbvA8Hgm655ZbiGK7Cx/oBKJP3uJqHCjayf837vPjii2BYA3GFQ6UDORDK11HLGXPgbe3atY02V2QByiRBfm7vf//7i2NYy6jAIesD/i6o7wZ/rpz0qJakXr58ebGtBr8xjBHYMIwR2DCMEXSqMaZPn94YB+elcoFy8j3712oWIPvB7EurRDJOAFQJaBzH+OhHP9poP/vss8UxnMDIyYmqMh5rF/al1aQdju9wgqBaQviDH/xgo12ToMmaj+MnO3bsKI5hHaIKWPC1+H7URCt+TjUzQnlyVi1+YxgjsGEYI7BhGCOwYRgj6FR8j46ONsSnSiLkAB4HcVRVQRa8LOCVEOWgGgtTxRNPPNFoL1q0qNiH+8/BOrW0LwfEuK0S9ViQHzx4sNFmoQoAP/vZzxptThpU98MDJHfccUejrZbyYuHMCY5A+Tny/aiBmbaERiXYVXJoDX5jGCOwYRgjsGEYI+g8wNcfiFJL1nKgjX1yVW2btQprDFVdgifCKD9+wYIFE17nueeeK47h5D72i1W1Pw5UcfBLJRHyZCx+LqoqH/vknNynqoXz/fCz5bbqi6qMwgmarAXUMm98z/yclJ5QS07X4DeGMQIbhjECG4Yxgk41xrRp0xo+tiomsHTp0kZ7+/btjbaKA7BPy4UAVGEA9rfVBCIuUsD95WQ5oByf5xWilE/O98R+sXpOnDjZtkIRUC4fXVPxkH19jpeoVaX4vHwMUOoq1m9qghEXb2DNoZbHVjGUGvzGMEZgwzBGYMMwRmDDMEbQaSXCGTNmZL8YVTO7GK4up8Q3i+AaIcpJdqoqBQf4uPKGChy2LZWmBDvvw8EvVWGPA4csglUSIQtcfv4q4MrBOW6rxD0+Dw+oAOWgCn9masCEnwvfjwoK8vd7//79rkRozGSxYRgjqFmccklEPB0ROyJiW0R8trf98oh4MiJ29X7OazuXMRcLNQG+dwB8PjOfi4jLAGyJiCcB/D2ApzLzgYi4H8D9AL4w4cWmT28s+6UmELGGYN9TaQGe5MLnuO2224pj2A9WPi1PluGlcdVyzFyVgn1/pRcY1gdqSTYO8LVVAgfKxELWSGqiEmsI9uPV53HTTTc12i+//HKxDydTst5RyySzXmDtqPSy0lo1tL4xMvNIZj7X+/1NADsALAZwF4ANvd02APjEpHpgzAByVhojIpYBuBHARgBXZeYRYNx4AExupXFjBpBqw4iISwE8CuBzmVm+58583H0RsTkiNqv52sYMIlWGEREzMG4U387MH/U2vxoRC3t/XwignOkDIDPXZ+a6zFyn/FFjBpFW8R3jU90eBrAjM7/a96fHAdwD4IHez8fazpWZDcGk3iActOHsSA66AWXFDxZ2IyMjxTEcrFPCkwU6Z8YqUcyClsWrEuxts+RU1ihnp9bMdOQsXR78UMFHfpb8+aj/7B599NHW83KgkGdQqlKmPPjBz1oNOKgAZA01o1K3Avg7AP8bEb/qbftXjBvE9yPiXgAHAPzNpHpgzADSahiZ+T8AypUCx/nw+e2OMYOBI9/GCDpfaqwNDtJwNQ9VZYODTuxrvvTSS8UxixcvbrRVhQwOLvKsP5WcyIEpDuipaiR8z+y3q2XDOCDG+kHpEr4O3496titXrmy0eTaempHIn4cKsrEuZC2gAq6sO1hjqKUc1AzDGvzGMEZgwzBGYMMwRtCpxsjMRuyC9cOpffrhihM1y5Ox/62qqrdVzgbKWAFPrlFJa3xt7pvqC/vgnOynYh8cA+K2qkTIz5u1jKqowfpMxRcYvsea5dX4HlVMom2pMXXPFyyJ0Jj/j9gwjBHYMIwR2DCMEXQqvsfGxhqBHCUqOajGyXA1gSsWYao8PO+jKmSwGGWBXlP+kYNQSrCzcOYAmQpSsThlgauuw0l4nNynAny8RnpNWX3um6rewc+fBwZUgiYfwwMzajBHzQSswW8MYwQ2DGMENgxjBJ1qjKGhoYZmUAl17DfWBLs4oMRtNSGK/VHlB/M+rClU2XzWFKwPVHIc6wFOCFT6p7/airouP0cAOHbsWKPNSZAqGMbb+LzqOnxe1jZAqV1YvyldyJ8jByjV58wVZGo1h98YxghsGMYIbBjGCDrVGKOjo42kOpVQx74m++hqrJr9YJ7kr6qSMyppjRMCOVagJulwLID1grpnta0fVb2QdQm3a5Zw5vtTz5ZjHTXVGPlZKo3EcQqeuKR0CetA1kwLFy4sjnEcw5jziA3DGIENwxiBDcMYQecz+PqTCJXYY4HLSYQquYxFMAfRVLCoZpZZm8BV5+WAFw8MqHtmWBTXVPLbs2dPo82z24AyIMbtmooa3BcV4ONnq9Yp58+RP0M1eMDie3h4uNFWIn+y+I1hjMCGYYzAhmGMoFONMW3atMZEFzWZhv1I9jVVUI39dtYYvPwXUFatU9XBOTjESWoquMX954CfSiJkrcLBLqVleMIWaw71nLi/rAWUXuhffhrQ1ReZmsAhazrWO6pKOT+7moolPLGqVof4jWGMwIZhjMCGYYyg82rn/f6mStzjsfS2SUhA6Z+yn8+TVYBSL6gxfNYhPIavfGfWKuyTq+uwpuBxf6Ux2Fdm/aD6xpO+2P9WE6/Y1+dnqXQiaxWlF1gDcf/Vc+LnwOetWUWqFr8xjBHYMIwRtBpGRMyOiE0R8XxEbIuIL/W2XxMRGyNiV0R8LyLKHARjLlJq3hhvA7g9M9cAWAvgzoi4BcBXADyYmasAvAHg3gvXTWO6pWZxygRwSinN6P1LALcD+Nve9g0A/gPA1yc619jYWEN01QTVGCUqOZmMK1CoGXA8+0uVkGcRz+KUrwOUAwEs4FX/eRCiRojyIATvs3Tp0uKYvXv3NtocvFP3w9fhgQI1sDGZ9dx5IKAmOZE/H3WMqhxSQ5XGiIih3lLGRwE8CWAPgOOZeaonIwAWn+l4Yy42qgwjM0czcy2AYQA3A3if2k0dGxH3RcTmiNjcNrfZmEHhrEalMvM4gGcA3AJgbkSccsWGAZQJSePHrM/MdZm5TuUWGTOItGqMiJgP4GRmHo+ISwB8BOPC+2kAnwLwCIB7ADzWdi5OIlSwb8xLiynjYt+YtYvyPfk8qsIEB5D4mGuvvbY4hgNvbRX3gDIIxb6/Wl6tLajJWgAo9Q5rJlUZkp8LJy8qL4CDfirxkK/N+6jzcoBvMlqslprI90IAGyJiCONvmO9n5o8jYjuARyLiPwFsBfDwpHpgzABSMyr1AoAbxfa9GNcbxvzZYaffGIENwxhB51VC+oUwi0GgFJosuGrWcmaxrQKJLCrVPq+99lqjzdm1SlTyjEMO6KlsT3VP/Sxbtuysj1GBt4MHDzbaHFi89dZbi2M2bdrUaPPyAxwoBUrBq/rCAxs1y7ixuObPTC3loAZeavAbwxiBDcMYgQ3DGEHnM/j6UdXmuEIdV4JQs9n4POx/qxlkrDFU8iL7p4sXN9PBDh06VBzTtgyA8nlvvLE5Gs6Jbyq5j/fhoCAvhQBoTdePCgouWLCg0d61a1ej/aEPfag4hnWiev6sB/hzVTMD22Z3qu+TZ/AZcx6xYRgjsGEYI+hcY/T7jqoqHFebY79YVdhjf5X1A1fpA4Dt27c32qtXry72eemllxptTqBTvjP7yjWTaTie0FbxHQAOHDjQaLdNkALKOAwndHLFdKCcBMbPVh3DsQ01aYr1Gk+iUjEijgmxjlKV8FU8pAa/MYwR2DCMEdgwjBHYMIwRdL4MQH8ATAksFqssnlSAj4NzXBVk//79rdfhBDugDA5xgEkt58WCnEUxBwCBMmjGwS1O3APalzBTQUG+Nt8PB/NUX1jU1wRcFTyQwZ+hWue7bc13lVipPqMa/MYwRmDDMEZgwzBGMKUTlWoqEapJLgzrBQ4wqcocNX5w2xIEqsod+/EcrFM+L2si9pVVgiM/Fw4cqmAX++g7d+5stJXGYB3I91xTzUMFQlk78udRU1WwRj9Mpioi4DeGMRIbhjECG4Yxgk41RkQ0/E+lMTjRrW1yClD6kawpVOLhkSNHGm2VaNgWH1E6hTUG+8qqkiL72xyjUAUHuC+ckKliROz7czVAVfGQaYslqL4p+DPhz1Dpt7Ylj2sKZdTiN4YxAhuGMQIbhjECG4YxgikN8ClRzOKVBZZaRoAFLSfQqUoRPFNQBYK4AiDPmqsJHLL4VjPKOEDJ96wCbzy7kIW16htXAeG+qsocLIL5+SuR3LZ8AlDeMw8WqBmIbWuZty0xcTb4jWGMwIZhjMCGYYyg84lK/b4jB9mAMiDDAb+apDD2nWv8VRUIGhkZabTZn1ZLW7HvzJUslL/Neof1AvcDKO+Jr6MmZ/E9ciBOVS/k4GKbn18L3zMHKJUW4/63BQnPpX9+YxgjsGEYI6g2jIgYioitEfHjXvuaiNgYEbsi4nsRMbnJtcYMIGejMT4LYAeAOb32VwA8mJmPRMRDAO4F8PWJTjA2NtbwY9VkFJ58UlOVnP1tTk5UyWWsF5Te4f5xoqEqBMCrMHFfVOyGfee2VZmA8p5qJu1wPIf7qiaFta08xVUUgVIXKjhmwtdWMQlebpk/d/V9uqArKkXEMIC/BPDfvXYAuB3AD3u7bADwiUn1wJgBpNaV+hqAfwFwKuf7CgDHM/OUOY4AWKwOjIj7ImJzRGye7GLkxnRNq2FExF8BOJqZW/o3i13LfAIAmbk+M9dl5jo1VGnMIFKjMW4F8NcR8XEAszGuMb4GYG5ETO+9NYYBHL5w3TSmW1oNIzO/COCLABARtwH458y8OyJ+AOBTAB4BcA+Ax8724irwxoEdDpgpscfb+Lxqph2Lv5pZZ3yMEt8cuOIqISq5j8UqJw2uXLmyOIaFMycIquqF/GzZtVVv9LbkPhUYZcGrlhnmwQMeGKgJ8HH/1SDFVFQJ+QKAf4qI3RjXHA+fw7mMGSjOKiUkM58B8Ezv970Abj7/XTJm6rEaNkbQaRLh2NhYw+dWVUJ4W00lP/ZPuVK2GibmSn3btm0r9mHdwctjKZ+c/WCetKMSD9kP5iCg0mIcEONAnApq8nPhCV1qSTA+DycVquXi+NkqvcDX2r17d6PNWg0otUuNRlITqWrwG8MYgQ3DGIENwxhBqAnwF4qhoaHs9x2VXmAfnWMFynfmAgq8D1fcU8co/5S1CycRsv4ByvgCxy14JSGg9KdrMgTaJlop/fbKK6802qxdVCLlkiVLGm3+PFgbAKUWWLNmTbEPazo+r9IuvA/rB1Xhnfty6NChLZm5rtiR8BvDGIENwxiBDcMYgQ3DGEHnywD0J5Qp8d0mnFVSGAs1FmFq1t+cOXMabSX2+NosNFXiIYt6FrQ15fn37ds3YT/UdTjwpgYpOLjIz0WJV14aoC3oCZRJnerZ8sBFTVVBTgblwRE1W0897xr8xjBGYMMwRmDDMEbQaYBv+vTp2R/wUtdmv5H9YtYGQOl71ixbxdtU0tq8efOKbf2oYB1PEGJfWgXROBmOA4lKL/AxrNdUhXdOaOSgoAostgVY1WQt/sxUEiF/Zqxd1DF8T3xtpTH4no8dO+YAnzGTxYZhjMCGYYzAhmGMoPNlAPpFohLFLMhZIKpgHZevHx4envCc6tpK7HGgigUjz4gDyqxWDm4pQc8VPliwq1l//FyUQGfaBhPUMgAsyFn0qwEIDtZxMBIony2fVwX4+B5rlgFQgyo1+I1hjMCGYYzAhmGMoFONMTo62vBjFy1a1HoM+57Kl+aKGawXlL/KfrAKVLGmYL9YlfTnbdxWlTj4HrkvSlctX7680W7TWUAZONy0aVOjzZVGgDKJsG15Y9UXpV24L1yNUS1JwHqHtZeqBuMqIcacR2wYxghsGMYIOtUYs2bNwooVK063efweKH1/rpStJjfxZBSecKPGt9n3VBqDE9BqlvNiuFL54cPlagnsK/N11IQoPuamm25qtNVyXzwBip+BWjaZdRZrPJV4yPEdVW2er82fu5pgxLEbjk/VVHmsxW8MYwQ2DGMENgxjBDYMYwSdiu/MbIguJSpZ8LLYU8tJ8Xk40KPKVbLwV8EhFuTcVqKSBeLBgwcb7ZqEOg6aqSob3N+tW7dO2A8A2LFjR6PNM+JUwJL34aQ8lZjI96OCbDyowuVCVeInBwr5HtV3Qw2q1OA3hjECG4YxAhuGMYJOq4RExGsA9gO4EkA5w2UwuZj6Clxc/Z2Kvl6dmfPbdurUME5fNGJzTQmTQeBi6itwcfV3kPtqV8oYgQ3DGMFUGcb6KbruZLiY+gpcXP0d2L5OicYwZtCxK2WMoFPDiIg7I+LliNgdEfd3ee0aIuIbEXE0Il7s23Z5RDwZEbt6PycuztQREbEkIp6OiB0RsS0iPtvbPqj9nR0RmyLi+V5/v9Tbfk1EbOz193sRcfaTXi4AnRlGRAwB+C8AHwOwGsBnImJ1V9ev5JsA7qRt9wN4KjNXAXiq1x4E3gHw+cx8H4BbAPxD73kOan/fBnB7Zq4BsBbAnRFxC4CvAHiw1983ANw7hX08TZdvjJsB7M7MvZl5AsAjAO7q8PqtZObPARyjzXcB2ND7fQOAT3TaqTOQmUcy87ne728C2AFgMQa3v5mZp7IUZ/T+JYDbAfywt31g+tulYSwG0J9qOtLbNuhclZlHgPEvI4CyLucUExHLANwIYCMGuL8RMRQRvwJwFMCTAPYAOJ6Zpxa2GJjvRJeGUeYEj/+PYc6BiLgUwKMAPpeZZQGqASIzRzNzLYBhjHsQ71O7ddsrTZeGMQJgSV97GEBZGWDweDUiFgJA7+fRKe7PaSJiBsaN4tuZ+aPe5oHt7yky8ziAZzCujeZGxKlJEwPznejSMH4JYFVvFGImgE8DeLzD60+WxwHc0/v9HgCPTWFfThPjs3IeBrAjM7/a96dB7e/8iJjb+/0SAB/BuC56GsCnersNTH+RmZ39A/BxADsx7lv+W5fXruzfdwEcAXAS42+4ewFcgfHRnV29n5dPdT97ff0LjLsdLwD4Ve/fxwe4vzcA2Nrr74sA/r23fTmATQB2A/gBgFlT3dfMdOTbGIUj38YIbBjGCGwYxghsGMYIbBjGCGwYxghsGMYIbBjGCP4PG92+eBcwAS8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "h = 50\n",
    "w = 37\n",
    "plt.imshow(clf.feature_importances_.reshape((h, w)), cmap=plt.cm.gray)\n",
    "plt.grid(False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tree Ensemble Comparisons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stump 0.21418537287929862\n",
      "Tree 0.23501519619900021\n",
      "Random Trees 0.4248248605998759\n",
      "Extra Random Trees 0.43485982327407063\n",
      "Boosted Tree 0.3686687431461075\n",
      "Wall time: 31min 10s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "num_estimators = 50\n",
    "# lets train some trees\n",
    "clf_array = [\n",
    "    ('Stump',              DecisionTreeClassifier(max_depth=1, min_samples_leaf=1)),\n",
    "    ('Tree',               DecisionTreeClassifier()),\n",
    "    ('Random Trees',       RandomForestClassifier(max_depth=50, n_estimators=num_estimators)),\n",
    "    ('Extra Random Trees', ExtraTreesClassifier(n_estimators=num_estimators,min_samples_split=2)),\n",
    "    ('Boosted Tree',       GradientBoostingClassifier(n_estimators=num_estimators)), #takes a long time\n",
    "    ]\n",
    "\n",
    "for clf in clf_array:\n",
    "    acc = cross_val_score(clf[1],X,y,cv=3)\n",
    "    print (clf[0], acc.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline accuracy 0.5942145439935718\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xm8HFWd9/HPNyGQSCAIwYBsAQxiBAUJuzo3ghqQB1xAQWRRMA+OEUQYRQdFcRkdBhVHHEREFIGAohiVYRG4kOcZUBL2BJAAASO7QEJQlJDf/HHO7RSd7r51l7p9l+/79erXrb1+fbpv/6rOqTqliMDMzAxgVLsDMDOzwcNJwczMapwUzMysxknBzMxqnBTMzKzGScHMzGqcFEYASZ+TdE6747DBRdK2kla0Ow4bXJwUhgBJiyU9LmntwrSjJXWWWT8ivhYRR1cQV6ekFyQtl7RU0g2Stu/v/Qw0SUdKeim/r2WSbpO0X2H+upK+LenhvMyiPD6xbjudkp6RtFbJ/W6et9f1CknPF8bf0s36x0j6Xe/edcvt3pS/f2v097Zb7PPrkl6sK49XF+a/T9LCPH2upG1KbPNV+fPo9zIaTpwUho41gOPaHUQDsyJiPLAB0Amc395wmsvJdXLJxW/M72s94IfAJZLWl7QmcA3wemAGsC6wB/AXYJfCviYDbwEC2L/MDiPi4YgY3/XKk99YmDa3ZOz9RtJrgZ2BNYF9Bnj3Py6WR0Q8kmN6PXAu8GHS53MtcJmk7n7PTgfurDTiYcBJYeg4DThR0nqNZko6Q9Kf8pHt/OJRpaQvSvppHr5C0qy6dW+X9N48vK2kqyU9LeleSe8vE1xErABmA1ML291F0o2SnpX0qKTv5h9VJJ0p6fS6OH4t6ZN5+NWSLpX0pKQHJR1bt915+b0+LumbZWLsjYhYSfoBGgdsBRwObA68JyIWRsTKiHgiIr4cEZcXVj0cuAk4Dziiv+LJienCQrl8WsmOwLeBjnz0/Fhe/j35810m6SFJn+vhLo8gJfuL6t+HpLUlfSd/75ZKur7rbEJSRz7DWJrPqD7Y5ze/yj7A7yLi9/l791VgG2D3ZitI6iB9bhf2YxzDkpPC0DGP9M95YpP5NwM7AOuTvvg/kzS2wXIXAod0jUiaCmwB/FapeurqvMyr8nLfy0dmLeUf+0NJP4RdXgKOByaS/mH3Av45z/sxcEjX0V2uetkLuChP+zVwO7BJnv5JSe/M654BnBER6wJbA5d0F19v5R+5o4HlwH3A3sAVEbG8m1UPBy7Ir3dKmtRPIZ0FjAG2BN4OfAz4YETcCnwS6MxH1Rvl5ZcBHyQdUb+HdGAxo8yO8ufwocL72E/SKwuLfAfYlnQmsT5wMhCSXgP8hnQgswGwE7Agb/OUfJDQ6PVYXQgH5oOTOyUVqz+VX/W2a/I+xuRYZzWab3Uiwq9B/gIWk36MtgOWAhuSfqg6W6zzDKnqAeCLwE/z8DrA88AWefyrwLl5+APA3LrtfB84pck+OoG/As8C/8ix7dUipk8CvyyM3w28PQ/PAi7Pw7sCD9et+1ngR3n4BuBLwMRelOPkEssdCazI7+spUqLbO8+7Gvh6N+u/GXixKz7gHuD4XnzuAbymML4WKdFuVZh2HClJARxDOoJutc2zgH/Lw9sCK1osuzfwAjCB9CP8IPCxPG9Mfo+vbbDel4CL+vid3w7YCBgNvBV4knR2BvAGUpLek1St9RVgZbMyzt+db5Uto5H+8pnCEBIRd5GOwE6qnyfpBEl359P1Z0n/yBMbbOM54LfAwXnSwaSjQEhnDLsWj95IR/8b1W+n4NiIWA8YC+wH/FzSG3JM20j6jaTHJC0DvlYX049JR6Lkv13tEVsAr66L43NA19H2UaTqgnsk3axCI3BdmWxet43NgTsK01pVadwUEetFxMSI2C0iuhon/wJs3GI9SNUsV0XEU3n8QvqnCmkj0tn9w4VpD5HOphqStGeu1nlS0lJSwlvte9HEEcBvI2JppF/UYhXSxqR2rgcarLcZcH/JfTQUEXdFxGMR8VJE3ACcCRyY590BfBT4AfAIKVneDyyp346kLUgHUKf0JZ6RZMCuJrB+cwpwC6nRDIDcfvAZUjXLgohYKekZGp9iQ/rnPkXSDaS68uvy9D8B10fE23saVKS697mSFgHvAO4A/gu4FTgkIp7L7QUHFlb7KXCXpDcCrwMuK8TxYERMabKv+1hV9fReUiLaICKer1vuYVK1CZAamoGOiFjc0/dX8DvgK5LWrt9f3sc44P3A6EJ1yFrAepLeGBG392Hfj5GOiDdn1Y/x5sCf83CjLo8vYdXZ4AuSzqLE/72k8aSyXdngfbw2738FqZ3l3rrV/0RK2o22+yXghCa7fSEimiWsoPB9joiLSN/jrqrHjwPzG6y3O/Bq4I+SIH3fx0paEhGbNtnXiOYzhSEmIhYBFwPHFiavQ/oHfRJYQ9IXSFfFNHM56Wj8VODi/IMO6SxkG0mHSRqTXztLel2Z2CTtTmpoXlCIaxmwXNK2pPrv4ntZQmoLOR+4NCL+lmf9AVgm6TOSxkkaLWk7STvn/XxI0oY57mfzOi+VibEfnE/60btUqVF+lKQNlO4F2Rd4d45lKqmNZwdSwptLamfouuR1cU93HBF/B34JfC038m5Nqj76aV7kcWCzXIeO0q/geOAvOSHsARxUcncHkapotq17H38ADo+IF4GfAGdImpQ/ozdLGp2n76fUyD1a0oZdZ48RcUq8/Iqi4quWEPK6E5TsTvrR/1Vh/rRc9pOAc4DZEdHorOUyUvtL13v4KvB7UjuHNdLu+iu/un+R2xQK45uR6no78/ho0mWTy4BHgU8X16HQplDYxg9JR187101/Lal66UlSVcm1wA5N4urMcSzPr0UU6nVJdcH35HlzSUno/9Vt40M5jul1019NOhJ8jNQ+UqzX/ynwRN7uAuDdPSjHySWWO7I+zrr5E0hX+vwpx3A/8E1So+oVwOkN1nl/fi9rAJ8HLigRx8vaFPK0DUhXeT1Fqjr6LKA8byxwZS6vJXnaITnO50g/kGcB5+R5TdsU8mf71QbTDydVX40C1iZV6zxCSs7XAWvk5d5GSvjLcpyH9PA7/4v8PpaT2p4+Vjf/5jzvqRzDuMK8o4D5TbbrNoVuXl1fJrO2kPRW0o/85Fh1xjKsSboKOC4i7m53LGb1nBSsbXI1x2zg9og4td3xmFmFbQqSzpX0hKS7msyX0o0viyTdIelNVcVig09up3iWdBXLt9scjpllVTY0n0fqBqCZfYAp+TWTdKWKjRARcXdErB0Re0TEsnbHY2ZJZUkh0rXFT7dY5ADgJ5HcRLrUrbvrv83MrELtvE9hE9JVEV2W5GmP1i8oaSbpbIJx48bttNlmm/V55ytXrmTUKF+R24jLpjmXTXMum9baXT5//OMfn4qIDbtbrp1JodGNVQ1bvSPibOBsgGnTpsW8efP6vPPOzk46Ojr6vJ3hyGXTnMumOZdNa+0uH0kPlVmunWl9Cel6+y6bkq53NjOzNmlnUpgDHJ6vQtoNWBoRq1UdmZnZwKms+kjSRUAHMFHSElKfPWMAIuIsUlcL+5Lugv0r6YEZZmbWRpUlhYg4pJv5QerPxMzMBglfKmBmZjVOCmZmVuOkYGZmNU4KZmZW46RgZmY1TgpmZlbjpGBmZjVOCmZmVuOkYGZmNU4KZmZW46RgZmY1TgpmZlbjpGBmZjVOCmZmVuOkYGZmNU4KZmZW46RgZmY1TgpmZlbjpGBmZjVOCmZmVuOkYGZmNU4KZmZW46RgZmY1TgpmZlbjpGBmZjVOCmZmVuOkYGZmNU4KZmZW46RgZmY1TgpmZlbjpGBmZjVOCmZmVuOkYGZmNU4KZmZW46RgZmY1lSYFSTMk3StpkaSTGszfXNJ1km6VdIekfauMx8zMWqssKUgaDZwJ7ANMBQ6RNLVusZOBSyJiR+Bg4HtVxWNmZt1bo8Jt7wIsiogHACTNBg4AFhaWCWDdPDwBeKTCeKwPJp/029WmLf76u9oQiZlVSRFRzYalA4EZEXF0Hj8M2DUiZhWW2Ri4CnglsDawd0TMb7CtmcBMgEmTJu00e/bsPse3fPlyxo8f3+ftDEeNyubOPy9dbbntN5kwUCENGv7eNOeyaa3d5TN9+vT5ETGtu+WqPFNQg2n1GegQ4LyIOF3S7sD5kraLiJUvWynibOBsgGnTpkVHR0efg+vs7KQ/tjMcNSqbIxudKRzasdq0esPtDMPfm+ZcNq0NlfKpsqF5CbBZYXxTVq8eOgq4BCAibgTGAhMrjMnMzFqoMincDEyRtKWkNUkNyXPqlnkY2AtA0utISeHJCmMyM7MWKqs+iogVkmYBVwKjgXMjYoGkU4F5ETEHOAH4gaTjSVVLR0ZVjRxm1lSjaj4Y2lV91jtVtikQEZcDl9dN+0JheCGwZ5UxmJlZeb6j2czMapwUzMysxknBzMxqKm1TMLORbbjdpzIS+EzBzMxqnBTMzKzGScHMzGp6lBQkjZK0bvdLmpnZUNRtQ7OkC4FjgJeA+cAESd+MiNOqDs4a892nZlaVMlcfTY2IZZIOJd2d/BlScnBSMLMhwwdT5ZSpPhojaQzwbuBXEfEiq3eBbWZmw0CZpPB9YDHpITg3SNoCWFZlUGZm1h7dVh9FxHeA7xQmPSRpenUhmZlZu3R7piDpOEnrKvmhpFuAtw1AbGZmNsDKVB99JCKWAe8ANgQ+DHy90qjMzKwtyiSFrmct7wv8KCJup/Hzl83MbIgrkxTmS7qKlBSulLQOsLLasMzMrB3K3KdwFLAD8EBE/FXSBqQqJDOzQcX3IvRdmauPVkp6ENhG0tgBiMnMzNqkTDcXRwPHAZsCtwG7ATfiK5DMzIadMm0KxwE7Aw9FxHRgR+DJSqMyM7O2KJMUXoiIFwAkrRUR9wCvrTYsMzNrhzINzUskrQdcBlwt6RngkWrDMjOzdijT0PyePPhFSdcBE4ArKo3KzHrFV99YXzVNCpLWbzD5zvx3PPB0JRGZmVnbtDpTmE/qIrt493LXeABbVRiXmfUzn0VYGU2TQkRsOZCBmJlZ+5XpJfU9kiYUxteT9O5qwzIzs3Yoc0nqKRGxtGskIp4FTqkuJDMza5cySaHRMmUuZTUzsyGmTFKYJ+mbkraWtJWkb5Eaoc3MbJgpc8T/CeDzwMV5/Crg5MoiahNfmWFmVu7mteeBkwYgFjMza7My1UdmZjZCVJoUJM2QdK+kRZIanm1Ier+khZIWSLqwynjMzKy1yq4ikjQaOBN4O7AEuFnSnIhYWFhmCvBZYM+IeEbSq6qKx8xsoBXbKk/YfgVHnvTbQd9OWeYhOxsCHwUmF5ePiI90s+ouwKKIeCBvZzZwALCwsMxHgTMj4pm8zSd6ErxZvZFwwcBIeI/WPoqI1gtI/wPMJV2G+lLX9Ii4tJv1DgRmRMTRefwwYNeImFVY5jLgj8CewGjgixGxWg+skmYCMwEmTZq00+zZs0u9uVaWL1/O+PHja+N3/nlpw+W232RCw+ntVHWs9WXTbJ9l9tfb9XqrHWUz0Fq9xyrmldWf35veqvo99iWeSePg8b+17zdl+vTp8yNiWnfLlak+ekVEfKYXMajBtPoMtAYwBeggPe5zrqTt8l3Tq1aKOBs4G2DatGnR0dHRi3BerrOzk+J2jmx29HVo3/fV33oba9kjzPqyabbPrv012m7XNlutV4WqP8dGZTPQWr3HKuaV1dPvTRWqfo99ieeE7Vdw+p1rDMrflKIyDc2/kbRvL7a9BNisML4pqz+cZwnwq4h4MSIeBO4lJQkzM2uDss9o/o2kv0laJuk5SctKrHczMEXSlpLWBA4G5tQtcxkwHUDSRGAb4IHy4ZuZWX8qc/PaOr3ZcESskDQLuJLUXnBuRCyQdCowLyLm5HnvkLSQ1F7xLxHxl97sz8zM+q7Vk9e2jYh7JL2p0fyIuKW7jUfE5cDlddO+UBgO4FP5ZWZmbdbqTOFTpCt+Tm8wL4C3VRKRmZm1Tasnr83Mf6cPXDhmZtZOpe5olrQHq9+89pOKYjJraSTcvDUS3uNIN1g/4zJ3NJ8PbA3cxqqb1wJwUjAzG2bKnClMA6ZGd7c+j2Ctbt4yMxtKytyncBewUdWBmJlZ+7W6JPXXpGqidYCFkv4A/L1rfkTsX314ZmY2kFpVH/3HgEVhZmaDQqtLUq8HkPSN+g7xJH0DuL7i2MzMbICVaVN4e4Np+/R3IGZm1n6t2hQ+BvwzsJWkOwqz1gH+f9WBmZnZwGvVpnAh8N/AvwHF5ys/FxFPVxqVmZm1RaukEBGxWNLH62dIWt+Jwcxs+OnuTGE/0mM4g5c/SS2ArSqMy8zM2qDV1Uf75b9bDlw4ZkOP72i34aTbq48k/UTSRyVtOxABmZlZ+5Tp++g84M3Af0raitQx3g0RcUaVgZnZ4DHQZ0ODtQfRkaDM4zivlXQ9sDPpecrHAK8HnBTMzIaZMl1nXwOsDdwIzAV2jognqg7MzMwGXpnqozuAnYDtgKXAs5JujIi/VRpZBYqnpCdsv4Ij87hPSc3MkjLVR8cDSBoPfBj4Eakr7bWqDc3MzAZameqjWcBbSGcLDwHnkqqRRhRfdmhmI0GZ6qNxwDeB+RGxouJ4zMysjcpUH502EIGYmVn7lTlTMBv2fF28WeKk0Eb+ITKzwaZsQ/MFEfHMAMRjbeLLdc0Myj15bSPgZkmXSJohSd2uYWZmQ1K3SSEiTgamAD8EjgTuk/Q1SVtXHJuZmQ2wUm0KERGSHgMeA1YArwR+LunqiPh0lQGaDQZd1WuuWrPhrkybwrHAEcBTwDnAv0TEi5JGAfcBTgpmNqT5oo9VypwpTATeGxEPFSdGxEpJ+1UTlpmZtUOZhubLgdrzmCWtI2lXgIi4u6rAzMxs4JVJCv8FLC+MP5+nmZnZMFOm+kgREV0judrIN72V5I70zGwoKXOm8ICkYyWNya/jgAfKbDzf13CvpEWSTmqx3IGSQtK0soGbmVn/K5MUjgH2AP4MLAF2BWZ2t5Kk0cCZwD7AVOAQSVMbLLcOcCzw+/Jhm5lZFcr0kvoEcHAvtr0LsCgiHgCQNBs4AFhYt9yXgX8HTuzFPszMrB+p0FzQeAFpLHAU8HpgbNf0iPhIN+sdCMyIiKPz+GHArhExq7DMjsDJEfE+SZ3AiRExr8G2ZpLPTiZNmrTT7Nmzy727Onf+eWlteNI4eDw/UHT7TSa8bF7R9ptMWG3dKucV5zdTxXqtyqbZun2d11tl30dP9ln28+9p2fRWb99jFfOaxVP/HpcvX8748eNfNq23ZVP1d7wn8/qi0f9VX///e2v69OnzI6LbKvoyDcbnA/cA7wROBQ4FylyK2qiPpFoGyje/fYvUdUZLEXE2cDbAtGnToqOjo8TuV3dkXadvp9+Z3v7iQzteNq9o8aEdq61b5bzi/GaqWK9V2TRbt6/zeqvs++jJPst+/j0tm97q7XusYl6zeOrfY2dnJ/X/m70tm6q/4z2Z1xeN/q/6+v9ftTJtCq+JiM8Dz0fEj4F3AduXWG8JsFlhfFPgkcL4OsB2QKekxcBuwBw3NpuZtU+ZpPBi/vuspO2ACcDkEuvdDEyRtKWkNUntEnO6ZkbE0oiYGBGTI2IycBOwf6PqIzMzGxhlksLZkl4JnEz6UV8IfKO7lfLznGcBV5Kqmy6JiAWSTpW0fx9iNjOzirRsU8j1/svyA3ZuALbqycYj4nJSNxnFaV9osmxHT7ZtZmb9r+WZQkSsJB3tm5nZCFDm6qOrJZ0IXEzq9wiAiHi6+SpmNlL4WRPDS5mk0HU/wscL04IeViWZmdngV+aO5i0HIhAzM2u/Mk9eO7zR9Ij4Sf+HY2Zm7VSm+mjnwvBYYC/gFsBJwcxsmClTffSJ4rikCaSuL8zMbJgpc/Navb8CU/o7EDMza78ybQq/ZlVHdqNIz0a4pMqgzIaLVk/eazSvON+sHcq0KfxHYXgF8FBELKkoHjMza6MySeFh4NGIeAFA0jhJkyNicaWRmZmNUO08iyzTpvAzYGVh/KU8zczMhpkyZwprRMQ/ukYi4h+5K2yzyri+3aw9ypwpPFns6lrSAcBT1YVkZmbtUuZM4RjgAknfzeNLgIZ3OZuZ2dBW5ua1+4HdJI0HFBHPVR+WmZm1Q7fVR5K+Jmm9iFgeEc9JeqWkrwxEcGZmNrDKtCnsExHPdo3kp7DtW11IZmbWLmWSwmhJa3WNSBoHrNVieTMzG6LKNDT/FLhG0o9I3V18BPeQamY2LJVpaP53SXcAewMCvhwRV1YemZmZDbgyZwpExBXAFQCS9pR0ZkR8vJvVrA9885bZ0NCq08OhqFRSkLQDcAjwAeBB4BdVBmVmZu3RNClI2gY4mJQM/gJcTLpPYfoAxWZmZgOs1ZnCPcBc4P9ExCIASccPSFRmZtYWrS5JfR/wGHCdpB9I2ovU0GxmZsNU06QQEb+MiA8A2wKdwPHAJEn/JekdAxSfmZkNoG5vXouI5yPigojYD9gUuA04qfLIzMxswJW5o7kmIp6OiO9HxNuqCsjMzNqn1CWpZmYj2XC7F6GVHp0pmJnZ8OakYGZmNU4KZmZW46RgZmY1lSYFSTMk3StpkaTVLmOV9ClJCyXdIekaSVtUGY+ZmbVWWVKQNBo4E9gHmAocImlq3WK3AtMi4g3Az4F/ryoeMzPrXpVnCrsAiyLigYj4BzAbOKC4QERcFxF/zaM3kW6OMzOzNlFEVLNh6UBgRkQcnccPA3aNiFlNlv8u8FhEfKXBvJnATIBJkybtNHv27F7FdOefl9aGJ42Dx/+WhrffZMLL5hVtv8mE1datcl6ZeLqLtZmy26wvm2br9nVef8Tak3m93V9xftVl0zV/MM1rFmtfy6aVqr/jPZnXbLs9nddVPn39/Htr+vTp8yNiWnfLVZkUDgLeWZcUdomITzRY9kPALOCfIuLvrbY7bdq0mDdvXq9iKt6AcsL2Kzj9znTv3uKvv6vbh9q0unmlP+eViae3D+Apu836smm2bl/n9UesPZnX2/0V51ddNl3zB9O8ZrH2tWxaqfo73pN5zbbb03ld5dPXz7+3JJVKClXe0bwE2KwwvinwSP1CkvYG/pUSCcG65ye2mVlfVNmmcDMwRdKWktYkPbBnTnEBSTsC3wf2j4gnKozFzMxKqCwpRMQKUpXQlcDdwCURsUDSqZL2z4udBowHfibpNklzmmzOzMwGQKUd4kXE5cDlddO+UBjeu8r9m5lZz/iOZjMzq3FSMDOzGicFMzOrcVIwM7MaP3nN2sb3VJgNPk4KZtYWPigYnFx9ZGZmNU4KZmZW46RgZmY1TgpmZlbjpGBmZjVOCmZmVuOkYGZmNU4KZmZW46RgZmY1TgpmZlbjbi5sxHC3Cmbd85mCmZnVOCmYmVmNk4KZmdU4KZiZWY0bms1sSPEFA9XymYKZmdU4KZiZWY2TgpmZ1TgpmJlZjZOCmZnVOCmYmVmNk4KZmdU4KZiZWY2TgpmZ1TgpmJlZjZOCmZnVOCmYmVmNk4KZmdVUmhQkzZB0r6RFkk5qMH8tSRfn+b+XNLnKeMzMrLXKkoKk0cCZwD7AVOAQSVPrFjsKeCYiXgN8C/hGVfGYmVn3qjxT2AVYFBEPRMQ/gNnAAXXLHAD8OA//HNhLkiqMyczMWlBEVLNh6UBgRkQcnccPA3aNiFmFZe7KyyzJ4/fnZZ6q29ZMYGYefS1wbz+EOBF4qtulRiaXTXMum+ZcNq21u3y2iIgNu1uoyievNTrir89AZZYhIs4Gzu6PoGo7luZFxLT+3OZw4bJpzmXTnMumtaFSPlVWHy0BNiuMbwo80mwZSWsAE4CnK4zJzMxaqDIp3AxMkbSlpDWBg4E5dcvMAY7IwwcC10ZV9VlmZtatyqqPImKFpFnAlcBo4NyIWCDpVGBeRMwBfgicL2kR6Qzh4KriaaBfq6OGGZdNcy6b5lw2rQ2J8qmsodnMzIYe39FsZmY1TgpmZlYzIpNCd91vjCSSzpX0RL5npGva+pKulnRf/vvKdsbYLpI2k3SdpLslLZB0XJ4+4stH0lhJf5B0ey6bL+XpW+Yua+7LXdis2e5Y20XSaEm3SvpNHh8SZTPikkLJ7jdGkvOAGXXTTgKuiYgpwDV5fCRaAZwQEa8DdgM+nr8rLh/4O/C2iHgjsAMwQ9JupK5qvpXL5hlSVzYj1XHA3YXxIVE2Iy4pUK77jREjIm5g9XtDit2P/Bh494AGNUhExKMRcUsefo70D74JLh8iWZ5Hx+RXAG8jdVkDI7RsACRtCrwLOCePiyFSNiMxKWwC/KkwviRPs1UmRcSjkH4YgVe1OZ62yz347gj8HpcPUKseuQ14ArgauB94NiJW5EVG8v/Wt4FPAyvz+AYMkbIZiUmhVNcaZl0kjQcuBT4ZEcvaHc9gEREvRcQOpN4KdgFe12ixgY2q/STtBzwREfOLkxssOijLpsq+jwarMt1vjHSPS9o4Ih6VtDHpSHBEkjSGlBAuiIhf5Mkun4KIeFZSJ6ndZT1Ja+Qj4pH6v7UnsL+kfYGxwLqkM4chUTYj8UyhTPcbI12x+5EjgF+1MZa2yfXAPwTujohvFmaN+PKRtKGk9fLwOGBvUpvLdaQua2CElk1EfDYiNo2IyaTfl2sj4lCGSNmMyDuacwb/Nqu63/hqm0NqG0kXAR2kbn0fB04BLgMuATYHHgYOiogR11GhpDcDc4E7WVU3/DlSu8KILh9JbyA1lo4mHVxeEhGnStqKdPHG+sCtwIci4u/ti7S9JHUAJ0bEfkOlbEZkUjAzs8ZGYvWRmZk14aRgZmY1TgpmZlbjpGBmZjVOCmZmVuOkYP1G0kaSZku6X9JCSZdL2kbS5GIvrO0k6YuSTuynbY2TdH3uZHFIkHRQ7vX1un7c5ixJH+6v7Vl7OSlYv8g3ev0S6IyIrSNiKuma/kntjaxSHwF+EREvtTuQRpokq6OAf46I6f24q3OBY/txe9ZGTgrWX6YDL0bEWV0TIuK2iJhbXCifNcyVdEt+7ZGnbyzpBkm3SbpL0ltyh2vn5fE7JR1ft60JkhZLGpXHXyHpT5LGSPqopJtzf/+XSnpFfcCSOiWxGQe9AAAEMUlEQVRNy8MTJS3Ow6MlnZbXv0PS/23yng8l35Wq5LRCrB/I0zvyfn4u6R5JF+QEiqSd8pnGfElX5i4z6mM8T9JZucz+mPvVaRpj3t91ki4k3XRX3NYXgDcDZ+V1G34WedlP5/dxu6Sv52lbS7oixztX0rb5c/4rsFjSLk3KyYaSiPDLrz6/SEeK32oybzJwVx5+BTA2D08B5uXhE4B/zcOjgXWAnYCrC9tZr8G2fwVMz8MfAM7JwxsUlvkK8Ik8/EXSHaYAncC0PDwRWJyHZwIn5+G1gHnAlnX7XRN4rDD+PlJPoaNJZ0cPAxuT7hZfSurrZhRwI+mHeQzwP8CGhdjPbfD+zgOuyOtOIfXdNbZZjHl/z9fHW9he8T03+yz2ybG9Io+vn/9eA0zJw7uSum/o2u6/kp490fbvol99e43EDvGsvcYA35W0A/ASsE2efjNwrlIHdJdFxG2SHgC2kvSfwG+Bqxps72LSD+p1pH5mvpenbyfpK8B6wHjgyh7E+A7gDZK6+qmZQPrRfLCwzETg2cL4m4GLIlUlPS7pemBnYBnwh4hYAqDU1fTkvO52wNX5xGE08GiTeC6JiJXAfblMtm0R4z/y/h5svKmXafZZ7A38KNIZABHxtFJPsXsAP8vxQkpGXZ7IcdkQ56Rg/WUBqzr7auV4Uh9LbyQd/b4A6WE/kt5KejDJ+ZJOi4ifSHoj8E7g48D7SfX4RXOAf5O0PunM4to8/Tzg3RFxu6QjSUfQ9Vawqgp1bGG6SGcWrRLJ3xqs00yxf5uXSP93AhZExO4t1utS3xdNNItRqa+d50tsE5p8Fnnb9fscRXoewA5NtjWWVCY2xLlNwfrLtcBakj7aNUHSzpL+qW65CcCj+cj3MNIRMpK2IPVB/wNSz6RvkjQRGBURlwKfB95Uv9NIT//6A3AG8JtY1ei7DvBoPvM4tEnMi0mJBF6e0K4EPpbXRekKqrXr9vsMMFpSV2K4AfhAruvfEHhrjquZe4ENJe2e9zFG0uubLHuQpFGStga2yut2G2MJDT8L0hnZR7raYSStH+k5Eg9KOihPU07YXbYBBsUVZtY3TgrWLyIigPcAb1e6JHUBqf6+vs/47wFHSLqJ9EPSdVTbAdwm6VZS/fwZpCdTdeYql/OAzzbZ/cXAh/LfLp8n9WZ6NXBPk/X+g/TD+j+k6qAu5wALgVuULqX9Po3Pqq8iVRtBuvLqDuB2UoL8dEQ81mS/RHoU7IHANyTdDtxGqp5p5F7geuC/gWMi4oUexNhKw88iIq4gnYHNy2XfdQnvocBROd4FvPwxtnsCv+vh/m0Qci+pZr0kaUfgUxFxWIX7OI90BvTz7pZtl4EoBxs4PlMw66WIuBW4TkPo5rWKTCSdmdkw4DMFMzOr8ZmCmZnVOCmYmVmNk4KZmdU4KZiZWY2TgpmZ1fwvZIgUIvwxSK4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# from sklearn.svm import SVC\n",
    "# from sklearn.linear_model import LogisticRegression\n",
    "# from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "\n",
    "\n",
    "# setup pipeline to take PCA, then fit a different classifier\n",
    "clf_pipe = Pipeline(\n",
    "    [('PCA',PCA(n_components=100,svd_solver='randomized')),\n",
    "     ('CLF',GaussianNB())]\n",
    ")\n",
    "\n",
    "# now iterate through and get predictions, saved to the correct row in yhat\n",
    "for train, test in cv.split(X,y):\n",
    "    clf_pipe.fit(X[train],y[train])\n",
    "    yhat[test] = clf_pipe.predict(X[test])\n",
    "\n",
    "total_accuracy = mt.accuracy_score(y, yhat)\n",
    "print ('Pipeline accuracy', total_accuracy)\n",
    "plot_class_acc(y,yhat,title=\"Naive Bayes + PCA\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Accuracy: 0.5942145439935718\n",
      "Number of infrequent faces: 26 with average recall of: 0.4198880585448181\n",
      "Number of frequent faces: 17 with average precision of: 0.6183521795874749\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics as mt\n",
    "\n",
    "freq_infreq_threshold = 40\n",
    "\n",
    "# get various measures of performance\n",
    "total_accuracy = mt.accuracy_score(y, yhat)\n",
    "per_class_acc_list = per_class_accuracy(y,yhat)\n",
    "\n",
    "prec_for_freq_classes = []\n",
    "recall_for_infreq_classes = []\n",
    "rec_tot = []\n",
    "prec_tot = []\n",
    "\n",
    "for cls in np.unique(y):\n",
    "    idx = (y==cls) # get classes\n",
    "    ytmp_actual = np.zeros(y.shape) # make binary class problem\n",
    "    ytmp_actual[idx] = 1 # set the instances for this specific class\n",
    "    \n",
    "    ytmp_predicted = np.zeros(y.shape) # binary prediction array\n",
    "    ytmp_predicted[yhat==cls] = 1\n",
    "    \n",
    "    num_in_class = sum(idx)\n",
    "    \n",
    "    rec = mt.recall_score(ytmp_actual, ytmp_predicted)\n",
    "    prec = mt.precision_score(ytmp_actual, ytmp_predicted)\n",
    "    rec_tot.append(rec)\n",
    "    prec_tot.append(prec)\n",
    "    \n",
    "    if num_in_class < freq_infreq_threshold:\n",
    "        recall_for_infreq_classes.append(rec)\n",
    "    elif num_in_class >= freq_infreq_threshold:\n",
    "        prec_for_freq_classes.append(prec)\n",
    "        \n",
    "print ('Total Accuracy:',total_accuracy)\n",
    "print ('Number of infrequent faces:',len(recall_for_infreq_classes), \n",
    "       'with average recall of:', np.mean(recall_for_infreq_classes))\n",
    "print ('Number of frequent faces:',len(prec_for_freq_classes), \n",
    "       'with average precision of:',np.mean(prec_for_freq_classes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA20AAAEzCAYAAACi6etkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGtxJREFUeJzt3X20ZWddH/Dvz5mEQJjECRlayAtJJZjELAoyRlpQZ2pkBaqJL6jJ0mrawFSXCbWKLV2xKUajBax2AaESG0WxDgKiGTGaKCZCkNBMCIG8GBwjmDF0EcxIoQRJ4tM/zp5wcrkz98y95+Y8957PZ62zZp+9n7v375w5z9n7u99OtdYCAABAn75i1gUAAABwYEIbAABAx4Q2AACAjgltAAAAHRPaAAAAOia0AQAAdGzJ0FZVv1JVn6qq2w8wvarq9VW1p6o+UlVfO/0yAQAA5tMkR9rekuTsg0x/cZJThseOJP9j5WUBAACQTBDaWmvvTfLAQZqcm+TX28hNSb6yqp42rQIBAADm2TSuaTsuyb1jz/cO4wAAAFihjVOYRy0yri3asGpHRqdQ5sgjj3zeqaeeOoXFAyzPLbfcMusSVuR5z3verEsAAFbglltu+XRrbctS7aYR2vYmOWHs+fFJ7lusYWvtyiRXJsnWrVvb7t27p7B4gD5VVVpbdB8WAECq6hOTtJvG6ZG7kvzAcBfJ5yf5TGvtk1OYLwAAwNxb8khbVe1Msi3JsVW1N8l/SXJYkrTWfinJNUlekmRPks8n+derVSwAAMC8WTK0tdbOX2J6S/IjU6sIAACAR03j9EgAAABWidAGAADQsWncPRIAAFiGqsV+PWvtcJfkx4fQBgAAM7KaocdPz6wfTo8EAADomNAGAADQMaENAACgY0IbAABAx4Q2AACAjgltAAAAHRPaAAAAOia0AQAAdExoAwAA6JjQBgAA0DGhDQAAoGNCGwAAQMeENgAAgI4JbQAAAB0T2gAAADomtAEAAHRMaAMAAOiY0AYAANAxoQ0AAKBjQhsAAEDHhDYAAICOCW0AAAAdE9oAAAA6JrQBAAB0TGgDAADomNAGAADQMaENAACgY0IbAABAx4Q2AACAjgltAAAAHRPaAAAAOia0AQAAdExoAwAA6JjQBgAA0DGhDQAAoGNCGwAAQMeENgAAgI4JbQAAAB0T2gAAADomtAEAAHRMaAMAAOiY0AYAANCxiUJbVZ1dVXdX1Z6qetUi00+squur6taq+khVvWT6pQIAAMyfJUNbVW1IckWSFyc5Pcn5VXX6gmY/meTtrbXnJjkvyZumXSgAAMA8muRI25lJ9rTW7mmtfTHJ25Kcu6BNS3LUMHx0kvumVyIAAMD82jhBm+OS3Dv2fG+Sr1/Q5tVJrquqi5McmeSsxWZUVTuS7EiSE0888VBrZYGqmnUJK9Jam3UJAADQvUmOtC2WDBZubZ+f5C2tteOTvCTJW6vqy+bdWruytba1tbZ1y5Yth14tj9FaW9XHai8DAABY2iShbW+SE8aeH58vP/3xwiRvT5LW2geSHJHk2GkUCAAAMM8mCW03Jzmlqk6uqsMzutHIrgVt/jrJNydJVZ2WUWi7f5qFAgAAzKMlQ1tr7eEkFyW5NsldGd0l8o6quqyqzhma/XiSl1fVbUl2JrmgOf8NAABgxSa5EUlaa9ckuWbBuEvHhu9M8oLplgYAAMBEP64NAADAbAhtAAAAHRPaAAAAOia0AQAAdExoAwAA6JjQBgAA0LGJbvkPALCYqpp1CSviZ2WBtUBoA7p2zDHHZN++fbMuY9nW6gbt5s2b88ADD8y6DNaA1Qw9VSVUAURoAzq3b98+G20zsFbDJgCsR65pAwAA6JjQBgAA0DGhDQAAoGNCGwAAQMeENgAAgI4JbQAAAB1zy38AADgAvxc6O34z9EuENgAAOAC/Fzo7azlwTpvTIwEAADomtAEAAHRMaAMAAOiY0AYAANAxoQ0AAKBjQhsAAEDHhDYAAICOCW0AAAAdE9oAAAA6JrQBAAB0TGgDAADomNAGAADQMaENAACgY0IbAABAx4Q2AACAjgltAAAAHds46wIADubqbZuya/tRsy5j7ly9bdOsSwAABkIb0LVzb/hsWmuzLmPuVFW86wDQB6dHAgAAdExoAwAA6JjQBgAA0DGhDQAAoGNCGwAAQMeENgAAgI4JbQAAAB3zO20AAHAAV2/blF3bj5p1GXPp6m2bZl1CN4Q2AAA4gHNv+Gxaa7MuYy5VVbzzI06PBAAA6NhEoa2qzq6qu6tqT1W96gBtvqeq7qyqO6rqN6dbJgAAwHxa8vTIqtqQ5Iok35Jkb5Kbq2pXa+3OsTanJPlPSV7QWttXVU9drYLXmmOOOSb79u2bdRnLVlWzLmFZNm/enAceeGDWZQAAwIpNck3bmUn2tNbuSZKqeluSc5PcOdbm5UmuaK3tS5LW2qemXehatW/fPudBz8BaDZsAALDQJKdHHpfk3rHne4dx456V5FlV9f6quqmqzp5WgQAAAPNskiNtix2yWHjoaGOSU5JsS3J8kvdV1Rmttb97zIyqdiTZkSQnnnjiIRcLAAAwbyY50rY3yQljz49Pct8iba5urT3UWvurJHdnFOIeo7V2ZWtta2tt65YtW5ZbMwAAwNyYJLTdnOSUqjq5qg5Pcl6SXQva/G6S7UlSVcdmdLrkPdMsFAAAYB4tGdpaaw8nuSjJtUnuSvL21todVXVZVZ0zNLs2yd9W1Z1Jrk/yE621v12togEAAOZFzerOhlu3bm27d++eybIfT1Xl7pEz4H1fP/xfzob3nR74HNIDn8PZmYf3vqpuaa1tXardRD+uDQAAwGwIbQAAAB0T2gAAADomtAEAAHRMaAMAAOjYxlkXAACsnmOOOSb79u2bdRnLVlWzLmFZNm/enAceeGDWZQDrhNAGAOvYvn371v0ts3u0VsMm0CenRwIAAHRMaAMAAOiY0AYAANAxoQ0AAKBjQhsAAEDH3D0SAAAOwt1AZ2Pz5s2zLqEbQhsAABzAWv7JjKpa0/XzJU6PBAAA6JjQBgAA0DGhDQAAoGNCGwAAQMeENgAAgI4JbQAAAB0T2gAAADomtAEAAHTMj2uvsqu3bcqu7UfNuoy5c/W2TbMuAQAApkJoW2Xn3vBZv0Q/A1UV7zoAAOuB0yMBAAA6JrQBAAB0TGgDAADomNAGAADQMaENAACgY0IbAABAx4Q2AACAjgltAAAAHfPj2gCwjl29bVN2bT9q1mXMnau3bZp1CcA6IrQBwDp27g2fTWtt1mXMnaqKdx2YFqdHAgAAdExoAwAA6JjQBgAA0DGhDQAAoGNCGwAAQMeENgAAgI4JbQAAAB0T2gAAADomtAEAAHRMaAMAAOiY0AYAANCxiUJbVZ1dVXdX1Z6qetVB2r20qlpVbZ1eiQAAAPNrydBWVRuSXJHkxUlOT3J+VZ2+SLtNSV6R5IPTLhIAAGBebZygzZlJ9rTW7kmSqnpbknOT3Lmg3U8neW2SV061QmDuVdWsS5g7mzdvnnUJAMBgktB2XJJ7x57vTfL14w2q6rlJTmitvbuqhDZgalprsy5h2apqTdcPAPRhkmvaFtvF/ehWSFV9RZJfTPLjS86oakdV7a6q3ffff//kVQIAAMypSULb3iQnjD0/Psl9Y883JTkjyQ1V9fEkz0+ya7GbkbTWrmytbW2tbd2yZcvyqwYAAJgTk4S2m5OcUlUnV9XhSc5Lsmv/xNbaZ1prx7bWTmqtnZTkpiTntNZ2r0rFAAAAc2TJ0NZaezjJRUmuTXJXkre31u6oqsuq6pzVLhAAAGCeTXIjkrTWrklyzYJxlx6g7baVlwUAAEAy4Y9rAwAAMBtCGwAAQMeENgAAgI4JbQAAAB0T2gAAADomtAEAAHRMaAMAAOiY0AYAANAxoQ0AAKBjQhsAAEDHhDYAAICOCW0AAAAdE9oAAAA6tnHWBcyDqpp1CXNn8+bNsy4BAACmQmhbZa21WZewbFW1pusHAID1wOmRAAAAHRPaAAAAOia0AQAAdExoAwAA6JjQBgAA0DGhDQAAoGNCGwAAQMeENgAAgI4JbQAAAB0T2gAAADq2cdYFAADAvKqqNT3/1tqqzp8RoQ0AAGZE6GESTo8EAADomNAGAADQMadHAsA6t9rXtPDlNm/ePOsSgHVEaAOAdWwtXy9TVWu6foBpcXokAABAx4Q2AACAjgltAAAAHRPaAABgHdm5c2fOOOOMbNiwIWeccUZ27tw565JYITciAQCAdWLnzp255JJLctVVV+WFL3xhbrzxxlx44YVJkvPPP3/G1bFcjrQBAMA6cfnll+eqq67K9u3bc9hhh2X79u256qqrcvnll8+6NFagZnUr3a1bt7bdu3fPZNlMxq2WYWX0IVgZfQgO3YYNG/KFL3whhx122KPjHnrooRxxxBF55JFHZlgZi6mqW1prW5dq50gbAACsE6eddlpuvPHGx4y78cYbc9ppp82oIqZBaAMAgHXikksuyYUXXpjrr78+Dz30UK6//vpceOGFueSSS2ZdGivgRiQAALBO7L/ZyMUXX5y77rorp512Wi6//HI3IVnjXNPGAbmWAFZGH4KV0YeA9c41bQAAAOuA0AYAANAxoQ0AAKBjQhsAAEDHJgptVXV2Vd1dVXuq6lWLTP+xqrqzqj5SVe+pqmdMv1QAAID5s2Roq6oNSa5I8uIkpyc5v6pOX9Ds1iRbW2vPTvLOJK+ddqEAAADzaJIjbWcm2dNau6e19sUkb0ty7niD1tr1rbXPD09vSnL8dMsEAACYT5OEtuOS3Dv2fO8w7kAuTPIHi02oqh1Vtbuqdt9///2TVwkAADCnJglttci4RX/psqq+P8nWJK9bbHpr7crW2tbW2tYtW7ZMXiUAAMCc2jhBm71JThh7fnyS+xY2qqqzklyS5Jtaa38/nfIAAADm2yRH2m5OckpVnVxVhyc5L8mu8QZV9dwkb05yTmvtU9MvEwAAYD4tGdpaaw8nuSjJtUnuSvL21todVXVZVZ0zNHtdkicneUdVfbiqdh1gdgAAAByCSU6PTGvtmiTXLBh36djwWVOuCwAAgEz449oAAADMhtAGAADQMaENAACgY0IbAABAx4Q2AACAjgltAAAAHRPaAAAAOia0AQAAdExoAwAA6JjQBgAA0LGNsy4AYFaqak0vo7W2avMGAPohtAFzS+gBANYCp0cCAAB0TGgDAADomNAGAADQMaENAACgY0IbAABAx4Q2AACAjgltAAAAHRPaAAAAOia0AQAAdExoAwAA6JjQBgAA0DGhDQAAoGNCGwAAQMeENgAAgI4JbQAAAB0T2gAAADq2cdYFsHxVtaaX0VpbtXkD8PhY7XXRas/fughYC4S2NcyKBoBZsy4CWH1OjwQAAOiY0AYAANAxoQ0AAKBjQhsAAEDHhDYAAICOCW0AAAAdE9oAAAA6JrQBAAB0TGgDAADomNAGAADQMaENAACgY0IbAABAx4Q2AACAjgltAAAAHRPaAAAAOjZRaKuqs6vq7qraU1WvWmT6E6rqt4bpH6yqk6ZdKAAAwDxaMrRV1YYkVyR5cZLTk5xfVacvaHZhkn2ttWcm+cUkr5l2oQAAAPNokiNtZybZ01q7p7X2xSRvS3LugjbnJvm1YfidSb65qmp6ZQIAAMynSULbcUnuHXu+dxi3aJvW2sNJPpPkKdMoEAAAYJ5tnKDNYkfM2jLapKp2JNkxPP1cVd09wfKZnWOTfHrWRcAapg/ByuhDsDL6UP+eMUmjSULb3iQnjD0/Psl9B2izt6o2Jjk6yQMLZ9RauzLJlZMUxuxV1e7W2tZZ1wFrlT4EK6MPwcroQ+vHJKdH3pzklKo6uaoOT3Jekl0L2uxK8oPD8EuT/Elr7cuOtAEAAHBoljzS1lp7uKouSnJtkg1JfqW1dkdVXZZkd2ttV5Krkry1qvZkdITtvNUsGgAAYF5McnpkWmvXJLlmwbhLx4a/kOS7p1saHXAqK6yMPgQrow/ByuhD60Q5ixEAAKBfk1zTBgAAwIwIbR2oqkeq6sNVdXtV/V5VfeWU539BVb1xGH51Vb1ymfPZVlXvXjjPBW2eUFV/PLye711Z5TCyoI+8o6qeNIV5bq2q1x9k+tOr6p0rXc4h1PNo36yqt1TVSxdpc+rwPtxaVV/1eNXG2tbDOmbSdU9V7ayqj1TVv59mjYssR19ialZjHbVg/idV1YPDMvY/Dp/mMhZZpu25zghtfXiwtfac1toZGd3I5UdmXdAKPDfJYcPr+a1ZF8O6Md5Hvpjkh8Yn1sghfZ+11na31l5xkOn3tda+LDjN2Lcnubq19tzW2l/OuhjWjDWxjqmqf5zkn7fWnt1a+8UF0ya6Bv8Q6EtM09TXUYv4y2EZ+x9fXLCMafcR23OdEdr684Ekx+1/UlU/UVU3D3sef2ps/A8M426rqrcO476tqj447Dn846r6R8spoKrOrKo/G+bzZ1X11RP+3VOT/EaS5wx7Zr6qqi4d6r+9qq6sqhraPnOo8baq+tD+PZ2Lvd6qOrKqfn9oe7s9PnPvfUmeOex5vKuq3pTkQ0lOqKoXVdUHhs/UO6rqyUlSVV83fJZvq6r/XVWbFhw5/qaxvZe3DtNPqqrbh+lHVNWvVtVHh+nbh/EXVNW7quoPq+ovquq1SxVfVS8fPuO3VdVvT7pHtqpekuRHk7ysqq4fxv1uVd1SVXdU1Y6xtmcP78FtVfWeYdyRVfUrw7Jvrapzh/FfM7wnHx763SkT/0+wFvWwjrmhql4zfO4+VlXfMEy6LslTh8/iNwztfraq/jTJv6uqLUOfuXl4vGCY31Oq6rqhrjdX1Seq6tiDLF9fYjVNZR01yYJqdAT7yqq6LsmvV9WGqnrdWJ/+t0O7qqo3VtWdNdqeuqYWOZtjbL6253rUWvOY8SPJ54Z/NyR5R5Kzh+cvyuiuP5VRwH53km9M8jVJ7k5y7NDumOHfzfnSzWVeluS/DcMXJHnjMPzqJK9cop6jkmwchs9K8tvD8LYk7144zwV/+2ib8dqG4bcm+bZh+INJvmMYPiLJkw7yer8ryS+PzefoWf+feTy+j7E+sjHJ1Ul+OMlJSf4hyfOHaccmeW+SI4fn/zHJpUkOT3JPkq8bxh81zGf88/x7SV4wDD95mH5SktuHcT+e5FeH4VOT/PXwub1gmPfRw/NPJDlhidfylLHhn0ly8TD8aN9M8pYkL13kbx/Tf8f6/hOT3J7kKUm2JLk3yckL2vxsku8fhr8yyceSHJnkDUm+bxh/eJInzvr/22PV+s/M1jELPt83jP3tS5L88TD8aJ8ba/emsee/meSFw/CJSe4ahl+f5NJh+F8maftrP8h7oi95TO2RVVhHLZj/SUkeTPLh4XHFMP7VSW7Z/1lLsiPJTw7DT0iyO8nJSb4zyR9l9B3w9CR/l0XWMQuWuS2257p6TPtQKsvzxKr6cEad8paMOlYy+tC/KMmtw/MnJzklyT9N8s7W2qeTpLX2wDD9+CS/VVVPy+hL4K+WWc/RSX5t2EvYkhy2zPkkyfaq+g8ZdeJjktxRVTckOa619jtD/V9Ikqo60Ot9X5Kfr6rXZPQF8r4V1MPatL+PJKPPw1UZrXg+0Vq7aRj//CSnJ3n/sAPw8IyOKnx1kk+21m5Oktba/02Soc1+70/yC1X1v5K8q7W2d8H0F2a0QZbW2p9X1SeSPGuY9p7W2meGed6Z5BkZbegdyBlV9TMZbew9OaPfwFyuV1TVdwzDJ2TUX7YkeW9r7a+Gevd/P7woyTn1peuKjshow/cDSS6pquMzeu1/sYJ66FNv65gkedfw7y1DXQcyflrWWUlOH+ubRw1HJL4xo43StNZ+v6r2LaMefYmVmPo6ahF/2Vp7ziLjd7XWHhyGX5Tk2WNH0Y7O6LP8jUl2ttYeSXJfVf3JMl6j7bkZE9r68GBr7TlVdXRGeyN+JKM9h5Xk51prbx5vXFWvyChMLfSGJL/QWttVVdsy2gOzHD+d5PrW2ndU1UkZ7e08ZFV1RJI3JdnaWru3ql6d0cqtDvQnWeT1DvN6XkZ7ZH+uqq5rrV22nJpYsx5cuLIaVnr/b3xUkj9qrZ2/oN2zs3h/eVRr7b9W1e9n9Bm7qarOSvKFBfM+kL8fG34kS3+vviXJt7fWbquqCzLam3nIhj5+VpJ/1lr7/LDy3N+/Fnu9leS7Wmt3Lxh/V1V9MKMjFNdW1ctaa8tZodOv3tYxyZf6zVJ9ZryPf0VGn/cHxxsM3wXL/v0ifYkpWNV11BIWLuPi1tpjdgbW6JTglfQR23MdcE1bR4a99a9I8sqqOiyjPfD/Zuyc5+OG84zfk+R7quopw/hjhlkcneRvhuEfXGp5VXVRVV20yKTx+VywzJeTjDp0knx6eA0vTR7di7S3qr59qOMJNbquZ9HXW1VPT/L51tpvJPn5JF+7gppYv25K8oKqemaSVNWTqupZSf48ydOr6uuG8ZtqwQXbVfVVrbWPttZek9HpJKcumPd7k3zf0PZZGe1VX7jB9hhV9etVdeYikzYl+eTQx7/vUF/kmKOT7Bs2Mk/NaC9uMtpz+01VdfJQx/7vh2uTXDx2HcJzh3//SZJ7WmuvT7IrybNXUBMde7zXMavguiSPrrOqav9G8nj/fHFGp3EeCn2Jx8Oy11GH4NokPzz071TVs6rqyIz6yHk1uubtaUm2H+J8bc91wJG2zrTWbq2q25Kc11p7a1WdluQDw7rhcxmdR39HVV2e5E+r6pGMDj9fkNFez3dU1d9k9OVw8hKLOzWj08IWem1Gp0f+WJJl7yVsrf1dVf1yko8m+XiSm8cm/6skb66qy5I8lOS7W2vXLfZ6kzwzyeuq6h+Gtj+83JpYv1pr9w9HrnZW1ROG0T/ZWvtYjS52fkNVPTGj6wLOWvDnP1qjm4s8kuTOJH+Q5Glj09+U5Jeq6qNJHk5yQWvt76sOdgAuz07yyUXG/+eMrgH4REZ9Y6ILzhfxh0l+qKo+klGAvCl59H3YkeRdNbpb2aeSfEtGR9D/e5KPDBubH0/yrUm+N8n3V9VDSf5PEns917HHeR0zba9IcsXwmd+Y0YboDyX5qYz6/YeS/GlG15wmSarqmiQva63dd5D56kusumWsoz63jMX8z4xON/7Q8Nm8P6M7pf5Okn+R0TrnYxn1kyTJsB22u7W26yC1257rwP4LiplDNbpz3ne2BbeNBVamqo5KclVr7btnXQvMm6r6eEancX161rVAj6rqLRldU/a4/RYpK+dI2xxrrX3rrGuA9Wg4ZURgAwCmwpE2AACAjrkRCQAAQMeENgAAgI4JbQAAAB0T2gAAADomtAEAAHRMaAMAAOjY/weiQ0Bg66I/UQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# But we can really summarize this data much better than this. \n",
    "# How about looking at more statistics of the precision and recall for each class?\n",
    "\n",
    "import seaborn as sns\n",
    "sns.set_palette(\"dark\")\n",
    "plt.figure(figsize=(15,5))\n",
    "plt.boxplot([ rec_tot, prec_tot, recall_for_infreq_classes,prec_for_freq_classes],\n",
    "            labels=['Recall, all faces','Precision, all faces','Recall Infreq. faces','Prec Freq. faces'])\n",
    "plt.ylim([0,1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFlBJREFUeJzt3X+MXWWdx/HPp9Mp5Ue1tB3ZhnYXMGbVGLeQsTFh13XxRxCNYMImoDFsQlLdSBaju1LcZBcTTWCjYjYmmLogdVdF1h+BEN21EVhCsgEGKKVYlR+iFrr0B1aKQtuZ+e4f93Sdrfecc+eec+4588z7lUzuvefc557vPPfO9z5zvuc5xxEhAMDCt6TtAAAA9SChA0AiSOgAkAgSOgAkgoQOAIkgoQNAIkjoAJAIEjoAJKJSQrd9vu2f2H7C9ua6ggIAzJ+HnSlqe0zSTyW9Q9JuSQ9IujQifpTXZs3qVXHG+vX5L7qEfxgA4HgPPrx9f0RMlD1vaYVtbJT0REQ8JUm2b5F0oaTchH7G+vV64M7v576gl59cIZx2xexs7jo3/EXV5rYXq7KBkO0RRYLFwCev/Pkgz6vy1366pF/Oebw7WwYAaEGVhN5vCPJ7wxbbm2xP2Z7ad+BAhc0BAIpUSei7Jc3dIb5O0rPHPykitkTEZERMTqxeXWFzAIAiVfahPyDpNbbPlPSMpEskvb+wxZIlhfvJ46VDhc194or8ti3v02x3XzWnQB419pGji4ZO6BExbfsKSf8paUzSTRHxWG2RAQDmpcoIXRHxPUnfqykWAEAFHNMGAIkgoQNAIirtcqlbUdFTkmL6SP7KsfGaowHytV2EB/phhA4AiSChA0AiSOgAkAgSOgAkolNF0TJeuix3Xbz8YnHj5afUHE2HmO/lUVvIRc8uF3S7HFuZothHFTeZAAASQUIHgESQ0AEgESR0AEjEgiqKFlp2YuHqOPR84XqvWFVp8zEznf/aYw138+xM7qpYMlbYdCEXwNosoA17Ld5j2uz3Kv3a+GmoK7x+2wXVJmMfFCN0AEgECR0AEkFCB4BEkNABIBGVqnW2n5Z0SNKMpOmImKwjqKFiKSn+qaToWXhqXhXPUpWKC59NF9AaL7o2pGqRqsuFRYzeQn5P6oq9jkzwFxGxv4bXAQBUwC4XAEhE1YQekn5g+0Hbm/o9wfYm21O2p/btP1BxcwCAPFUT+rkRcY6kd0n6iO23HP+EiNgSEZMRMTmxZnXFzQEA8lTahx4Rz2a3e21/V9JGSffUEdiolRU9i2aCSsWFyaaLNTF9NH/bS7nWKuapqIi/gAuPi8HQI3TbJ9tecey+pHdK2llXYACA+akyQj9N0nez0edSSV+PiP+oJSoAwLwNndAj4ilJf1JjLACACjhsEQASsTCnGPYRBaeQ7SmZbbmk+LutbDbm9D9flbtu6d9cV9i28mk/Z/KLoupwUTRmZwvXl70nRcVgqdmCcOns3yj73UpmNreorN+7qu3T5xZ9nks/y5w+FwAwFwkdABJBQgeARJDQASARJHQASEQyR7m0fdRA0ZEspUfguOL3aoePmChS9WiKNk9rUHrEhBfme7KQtX4+9Arbryt2RugAkAgSOgAkgoQOAIkgoQNAIpIpilYtPFYtShRN3S0r2M4+tb1wvc/aULLx/CnHbU+HblLV363wPVvA/YKFp+w0GINihA4AiSChA0AiSOgAkIjShG77Jtt7be+cs2yV7W22H89uT202TABAmUGKojdL+qKkr85ZtlnSDyPiWtubs8f5JwQfgbZnilYpoi0pKXrG9JHiFxhfPvS2F7KqhUsKn+iKus5BX/oqEXGPpOePW3yhpK3Z/a2SLqolGgDA0Ib9WjgtIvZIUnb7qrwn2t5ke8r21L79B4bcHACgTONF0YjYEhGTETE5sWZ105sDgEVr2IT+nO21kpTd7q0vJADAMIadKXq7pMskXZvd3lZbREOKIy8XP2H8hMLVVWYVSpIKZqqWXWC69GLJS5cVrv/wyety133pN7sL25ap6+K1jSgpFrvkPa8yU7TsAtUqec+7XJCNmencdeWf5eIZ22UHL1Rp3/SF4hs/DXYNBjls8RuS/lvSH9vebfty9RL5O2w/Lukd2WMAQItKR+gRcWnOqrfVHAsAoIL2/0cAANSChA4AiUjm9LlVi55lSotkFWaqVp0ldsPBJ3PXzf5iV2HbJX/4usL1TRbvSovBZf1SVvSs+vpFSoqDC1qlz3K1GdtV2jc9W7xSQXdE15hlhA4AiSChA0AiSOgAkAgSOgAkIpnKTtsz79rcftGMSJcUPeOF/cWv/Yo1Q8XUBXWdkrTva3d4pmdVKf9uTWr7FN4SI3QASAYJHQASQUIHgESQ0AEgEckURdtW5VSslbddcCpXLx0vbFtW9GxytmWTRUup/NS/FP9QpzZzwDGM0AEgESR0AEgECR0AEjHIFYtusr3X9s45y66x/Yzt7dnPBc2GCQAoM0hR9GZJX5T01eOWXx8Rn609ogWq1QJbgzPUSq+zWHD9San8GpRVUPREl3Th81Y6Qo+IeyQ9P4JYAAAVVNmHfoXtHdkumVNriwgAMJRhE/oNkl4taYOkPZI+l/dE25tsT9me2rf/wJCbAwCUGSqhR8RzETETEbOSvixpY8Fzt0TEZERMTqxZPWycAIASQ1WsbK+NiD3Zw/dJ2ln0/C7ocgGty7GVKSt6xkuH8tueuKLativ2Sxdm9gFSeQ4YVGlCt/0NSW+VtMb2bkn/KOmttjdICklPS/pQLdEAAIZWmtAj4tI+i29sIBYAQAXMFAWARJDQASARi+b0uY2fwrZCga16bPnbbrvgWlT4jF/9T3HbU/+gcH3V343CZ1ra/qy3mwN6GKEDQCJI6ACQCBI6ACSChA4AiVg0RVF0T9NFTywubX8e2t6+xAgdAJJBQgeARJDQASARJHQASEQyRdHS00+WFdhKrp1ZptWCiPNj70KhZlhlscfRw8Xtx0+oM5x5oaCLNjBCB4BEkNABIBEkdABIRGlCt73e9l22d9l+zPaV2fJVtrfZfjy7PbX5cAEAeQYpik5L+nhEPGR7haQHbW+T9FeSfhgR19reLGmzpKuaC7VEw0XP0s3PzhRse6zRbWv6SO6qWLqssOlCLs6VFT2LrmcqVb+maeFrL+B+xXC6cI3a0iwXEXsi4qHs/iFJuySdLulCSVuzp22VdFFTQQIAys1r2Gr7DElnS7pP0mkRsUfqJX1Jr6o7OADA4AZO6LZPkfRtSR+NiBfm0W6T7SnbU/v2HxgmRgDAAAZK6LbH1UvmX4uI72SLn7O9Nlu/VtLefm0jYktETEbE5MSa1XXEDADoo7Qo6t7e/Bsl7YqIz89ZdbukyyRdm93e1kiEg2rxeoFStcJn5VmFY81N+O3yjMeYnS1cX1b0jN8czG978sqhYsLiVeVvoeyzPKhBMsG5kj4o6VHb27Nln1Qvkd9q+3JJv5D0l7VEBAAYSmlCj4h7JeV99byt3nAAAMNipigAJIKEDgCJIKEDQCKSOR9600dbNPn6lV+7aMrxkmqv3eUp7JXPYV9wJEvMTBe3bfDIIiw+dZ2ahBE6ACSChA4AiSChA0AiSOgAkAgqOymIeqYN43fKip5x5OXi9suW1xnOSHXhvN55uhxbFzBCB4BEkNABIBEkdABIBAkdABJBUbQmrV4k2nwvj1pZ0TNe/FVx+1NOrTOcRaPLhc8uFGzJBACQCBI6ACSChA4AiShN6LbX277L9i7bj9m+Mlt+je1nbG/Pfi5oPlwAQJ5BiqLTkj4eEQ/ZXiHpQdvbsnXXR8Rnmwtv4Wi88FmkzW2jr7KiZxz+bXH7E06qMxyMQBcKtoNcU3SPpD3Z/UO2d0k6venAAADzM6996LbPkHS2pPuyRVfY3mH7Jtt9hyS2N9mesj21b/+BSsECAPINnNBtnyLp25I+GhEvSLpB0qslbVBvBP+5fu0iYktETEbE5MSa1TWEDADoZ6CEbntcvWT+tYj4jiRFxHMRMRMRs5K+LGljc2ECAMqU7kN3b0//jZJ2RcTn5yxfm+1fl6T3SdrZTIgLQ6uzxIpOn+t0C6ZFfS51o0iVa9mJhavj4HO567zytLqj+f+v3+V+67AuzBQd5CiXcyV9UNKjtrdnyz4p6VLbGySFpKclfaiRCAEAAxnkKJd7JfX7evle/eEAAIbFTFEASAQJHQASwelzE9DqLFUMpbRIVlD4jJdfLH7t5acME9KC13aRvAvFZEboAJAIEjoAJIKEDgCJIKEDQCIoitakzYJIq9czbVGrfd5iAa6s6BmHni9uv2JVneF0RheKkm1jhA4AiSChA0AiSOgAkAgSOgAkIpmiaMwWnEJWkpc0+93V7qkzKQaNWpcLcGVFz5g+Utx+6bI6w8EIMUIHgESQ0AEgESR0AEhEaUK3vdz2/bYfsf2Y7U9ly8+0fZ/tx21/0zY73gCgRYMURQ9LOi8iXswuFn2v7e9L+pik6yPiFttfknS5pBsajLVYh4tUjZs5mr9uyQmji2MRaftUrVWUFT3jyEv5bUuuhdplC/k9G1TpCD16jp2AeTz7CUnnSfpWtnyrpIsaiRAAMJCB9qHbHssuEL1X0jZJT0o6GBHT2VN2Szq9mRABAIMYKKFHxExEbJC0TtJGSa/r97R+bW1vsj1le2rf/gPDRwoAKDSvo1wi4qCkuyW9WdJK28f2wa+T9GxOmy0RMRkRkxNrVleJFQBQoLQoantC0tGIOGj7RElvl3SdpLskXSzpFkmXSbqtyUDLtF3QaHP7Hh++8LkYCkVNSLlfigqf8dtfF7c96ZV1h1OblN+zYwY5ymWtpK22x9Qb0d8aEXfY/pGkW2x/WtLDkm5sME4AQInShB4ROySd3Wf5U+rtTwcAdAAzRQEgESR0AEhEMqfPbVubp8+NmencdR4rfosXQ6EI9Skrei7mInu7p9DuYYQOAIkgoQNAIkjoAJAIEjoAJKJTRdEqBZW2izGtzhQtKXyifm1/3rqq7PdO+XqmXXjPGaEDQCJI6ACQCBI6ACSChA4AiehUNa1SUaGkSJXyNUfj6OHcdVVOrYt8XSiALUSl1zOl2FwJI3QASAQJHQASQUIHgESUJnTby23fb/sR24/Z/lS2/GbbP7O9PfvZ0Hy4AIA8gxRFD0s6LyJetD0u6V7b38/W/V1EfKu58AbnJYv3nw0Kn0hF6UzTIy/lty24Fqq0OAqug1yCLiS9mD0cz35KDikBAIzaQMNa22O2t0vaK2lbRNyXrfqM7R22r7fNMBEAWjRQQo+ImYjYIGmdpI223yDpakmvlfQmSaskXdWvre1NtqdsT+3bf6CmsAEAx5vXjueIOCjpbknnR8Se6Dks6SuSNua02RIRkxExObFmdeWAAQD9DXKUy4Ttldn9EyW9XdKPba/NllnSRZJ2NhkoAKDYIEe5rJW01faYel8At0bEHbbvtD0hyZK2S/pwg3GiQBcuTot0dPnzVHQkS/x6X3HbV07UHU7nDHKUyw5JZ/dZfl4jEQEAhrJ4D94GgMSQ0AEgESR0AEhEp86HvpC1WUhqu1CFtCzUz1NZ0TPlC1QfwwgdABJBQgeARJDQASARJHQASARF0Zq0WUgqKvakUOjJsxjOb436lF6genamuP2SseL2HZhhywgdABJBQgeARJDQASARJHQASARF0RQUFGNSLhwu5NhTfl8WqtKi5/TR4vZLx/Pbjuj9ZoQOAIkgoQNAIgZO6LbHbD9s+47s8Zm277P9uO1v2k73gGcAWADmM0K/UtKuOY+vk3R9RLxG0q8kXV5nYACA+RmoKGp7naR3S/qMpI9lF4Y+T9L7s6dslXSNpBsaiHEgVWd5Vd5+m7PEjh7OX7eIZ4qWafJ9oeiZnqKipyTN7Piv3HVjb/zzwrZVP8vHDDpC/4KkT0iazR6vlnQwIqazx7slnV5LRACAoZQmdNvvkbQ3Ih6cu7jPU/t+xdjeZHvK9tS+/QeGDBMAUGaQEfq5kt5r+2lJt6i3q+ULklbaPrbLZp2kZ/s1jogtETEZEZMTa1bXEDIAoJ/ShB4RV0fEuog4Q9Ilku6MiA9IukvSxdnTLpN0W2NRAgBKVZkpepWkW2x/WtLDkm6sJ6ThNF30LN1+QZGr6QKZT3pFpfYLVaunLKboieMsecOf5a6Lw78tbOsTTqolhnkl9Ii4W9Ld2f2nJG2sJQoAQGXMFAWARJDQASARJHQASEQyp8/tcpGq6W0XFVzqKrZ0UZszRSl6Yj7K/g7jyEu1bIcROgAkgoQOAIkgoQNAIkjoAJAI13XaxoE2Zu+T9PM5i9ZI2j+yAOaH2IZDbPPX1bgkYhtW3bH9UURMlD1ppAn99zZuT0XEZGsBFCC24RDb/HU1LonYhtVWbOxyAYBEkNABIBFtJ/QtLW+/CLENh9jmr6txScQ2rFZia3UfOgCgPm2P0AEANWklods+3/ZPbD9he3MbMeSx/bTtR21vtz3VgXhusr3X9s45y1bZ3mb78ez21I7EdY3tZ7K+2277glHHlcWx3vZdtnfZfsz2ldnyLvRbXmyt953t5bbvt/1IFtunsuVn2r4v67dv2l7Wodhutv2zOf22YdSxZXGM2X7Y9h3Z43b6LCJG+iNpTNKTks6StEzSI5JeP+o4CuJ7WtKatuOYE89bJJ0jaeecZf8kaXN2f7Ok6zoS1zWS/rYDfbZW0jnZ/RWSfirp9R3pt7zYWu879S7+fkp2f1zSfZLeLOlWSZdky78k6a87FNvNki7uwGfuY5K+LumO7HErfdbGCH2jpCci4qmIOKLehacvbCGOBSEi7pH0/HGLL5S0Nbu/VdJFIw1KuXF1QkTsiYiHsvuHJO2SdLq60W95sbUuel7MHo5nP6HeheG/lS1vq9/yYmud7XWS3i3pX7LHVkt91kZCP13SL+c83q2OfKAzIekHth+0vantYHKcFhF7pF6CkPSqluOZ6wrbO7JdMiPfpXE822dIOlu9EV2n+u242KQO9F2262C7pL2Stqn33/TBiJjOntLa3+vxsUXEsX77TNZv19s+oYXQviDpE5Jms8er1VKftZHQ+51IuhPftJlzI+IcSe+S9BHbb2k7oAXkBkmvlrRB0h5Jn2szGNunSPq2pI9GxAttxnK8PrF1ou8iYiYiNkhap95/06/r97TRRpVt9LjYbL9B0tWSXivpTZJWqXfx+pGx/R5JeyPiwbmL+zx1JH3WRkLfLWn9nMfrJD3bQhx9RcSz2e1eSd9VNy+E/ZzttZKU3e5tOR5JUkQ8l/3RzUr6slrsO9vj6iXMr0XEd7LFnei3frF1qe+yeA6qd0H4N0taafvYxXBa/3udE9v52S6siIjDkr6i0ffbuZLea/tp9XYfn6feiL2VPmsjoT8g6TVZFXiZpEsk3d5CHL/H9sm2Vxy7L+mdknYWt2rF7ZIuy+5fJum2FmP5P8eSZeZ9aqnvsn2YN0raFRGfn7Oq9X7Li60LfWd7wvbK7P6Jkt6u3j7+uyRdnD2trX7rF9uP53xBW7391CPtt4i4OiLWRcQZ6uWyOyPiA2qrz1qqCF+gXnX/SUl/30YMOXGdpd5RN49IeqwLsUn6hnr/gh9V77+by9XbR/dDSY9nt6s6Ete/SnpU0g71kufalvrsT9X7F3eHpO3ZzwUd6be82FrvO0lvlPRwFsNOSf+QLT9L0v2SnpD075JO6FBsd2b9tlPSvyk7Eqalz91b9bujXFrpM2aKAkAimCkKAIkgoQNAIkjoAJAIEjoAJIKEDgCJIKEDQCJI6ACQCBI6ACTifwHwvoztiGYTawAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# how about plotting a confusion matrix?\n",
    "cm = mt.confusion_matrix(y, yhat)\n",
    "plt.imshow(cm,cmap=plt.get_cmap('Reds'),aspect='auto')\n",
    "plt.grid(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What is the problem with this graph??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHb9JREFUeJzt3X2QXWV9B/Dvd9/YJBvcJLtgSoIQZACLGuI2YulQxZciWsEOdYiOgy3T+Npq0UJsR4tjaUVF1LHiBEFiKyC+FUtRYQSkWBtcQhJCAw1ihEjMCxBIAnnZ3V//uGfbbdzz/M7e5557zj5+PzOZ3b3Pfc757XPP/e3J+d3nPDQziIjI9NdRdQAiItIaSugiIolQQhcRSYQSuohIIpTQRUQSoYQuIpIIJXQRkUQooYuIJCIqoZM8k+RDJB8muaJVQYmIyNSx2ZmiJDsB/DeA1wLYAuCnAJaZ2X/l9ZnX02VH9/bkbrPzhSc0FQsAwMbC7Sz5PyMjB/LbuvJ/52m/7xixr1mZ/b2+Y6Ph9o7O5vddZP9l8mKLEft7xbxmZSsxtnvvW7vTzAa953VF7GMpgIfN7BEAIHkDgLMB5Cb0o3t7cOdQftJ+3s13Nh2MHdgXbGdPb9PbLmJs+y9y2zqOeEGy+44R+5qV2d/tu3dXeN+z+pved5H9l8mLLUbs7xXzmpWtzNg4qz//TT5BzJ+NowA8NuHnLdljIiJSgZiEzkke+7XrNySXkxwmOfzEgZGI3YmISEhMQt8CYOGEnxcAePzQJ5nZSjMbMrOheT0xV3hERCQkJsP+FMDxJI8F8EsA5wF4a6hD5wtPCF4n/9ExLwru8Pc3516eBw48F+yLkq+vVXqt+umduU3Wf0SwK3tmtDqawuKvecbd+jlq/92HVbfv0uWPa5XHS2P/+eNmTg4oP/bqb0XedEI3sxGS7wPwAwCdAK4xswdaFpmIiExJ1DUQM7sFwC0tikVERCJopqiISCKU0EVEElGrj50Ei54AnjnnjNy22V9eFd5435xmQpoWbNO63LaO419W7r6ffTrYzpnPK6UvUHGB7pn8QjQAYGBhuL3Gqi58NqvquKveP6AzdBGRZCihi4gkQgldRCQRSugiIomoVVHUc/i/3J7b9vCScPHvuP/8cbC93jP3wjpO/6PctrLv6ucVLsvq2wpRd8c73L2TaW3ZHudOkX3hO0WWyZ7cGmzn3Pn5fSPvEhl9J8jAuHpj2qo7XOoMXUQkEUroIiKJUEIXEUmEErqISCJqVRS10fACGOzMD/e4O28N9j34obcH23s+/41guxcbnvpVbhMHFkRtO/R7A+Hl0Njz/GBfT3Rsof7emDrbxkGn4NvbF7f9CHVeYi6mQFd23KGip2vMOVa94yFSlcXkcTpDFxFJhBK6iEgilNBFRBKhhC4ikoioqhDJzQB2AxgFMGJmQ1HbiylSdfeEmy+7Jti+d9mZwfZZ138/vP9A4dOb/eZyCkU8fF5+W+xM0MjCYbB/5LbtQOSaojH7Hz0Y3vaM2c1vu2p7nspviylals2qX9Ozac7xVFQryvyvMjPn5tAiIlI2XXIREUlEbEI3ALeSvJfk8smeQHI5yWGSwzt2PhG5OxERyROb0E8zsyUAXg/gvSRPP/QJZrbSzIbMbGhwIP9ar4iIxIm6hm5mj2dft5P8DoClAO5qRWBTxri/TV7Rc/S6y4PtHX/0rty2qNlvBYyu+mRuW+c7VgT7sndWi6Npo8jX3Pbtzd+0My7TuujpsCcDs55rXBQt+zUJHS+edh1PTb8jSM4iOXv8ewCvA7ChJVGJiMiUxZyhHwngOyTHt3OdmTmf7RMRkbI0ndDN7BEAL21hLCIiEkEfWxQRSUStbp8bpaMzrt3b/LILg+2fGzwut+0DOx8J9rWxsWA7O8J/dzuX/UV+38iipzmz77JLbs31927N29UdbG98arZ5MWPjvWYYORDed43XsOVxi6sOoSmx7yOPd7zYSPOzPd3bcxekM3QRkUQooYuIJEIJXUQkEUroIiKJUEIXEUlEOp9yMafC3RX5qYIDzwWbQ59kGV39b8G+HSedGt534H7nADB62Ydy27r+flV42w7vUyxR/Z1Psdj+Z52NV3c+4n5iosJPsXjjxsNmhjcQOta9vpFiYo/9FIvHPR53B24+OLAw2DV23YFxOkMXEUmEErqISCKU0EVEEqGELiKSiHSKomOjwWZvWq43zdwrJIW23/nyNwT7rjn+JcH2Ux5aG2zv+LMP5sd1cH+wL7sPC7aXOZ3a3XbEmBfaf2BsYsfFU2YBzy16erqbL+jGHi/uax4x7tFT/73YIm4v4hZcC9IZuohIIpTQRUQSoYQuIpIIN6GTvIbkdpIbJjw2l+RtJDdlX+eUG6aIiHiKFEWvBfAFAF+d8NgKAD80s0+QXJH9fLG7pbFR2LPP5DZz5uEFwsnh3bfbvbd2nJjtL9m0Pti+922vD7bP+Pg/5LbZYxuDfbkofO/rUmff7dsTbneOh9jX1Ct8BjmxRx3LFYt5zcuerVn29qM8vSO/bWBBsGt0ITvjjo6Z3QXgyUMePhvA+JzyVQDOaUk0IiLStGb/3B1pZlsBIPt6RN4TSS4nOUxyeMcTh/5dEBGRVin9/y9mttLMhsxsaHDe3LJ3JyLyG6vZhL6N5HwAyL5ub11IIiLSjGZnin4XwPkAPpF9valQr47O8opFXT3B5lIXOwaCM1W9W2N6s99mfe17wfZ3zcovuHxp75ZgX4/7e8eYMTtu395CzN5sz8D23dsG98xoetuFtl+h0ILF/rEcnrFNZzZlTH+vLxAec6/g6m5/3lHO/stX5GOL1wP4CYATSG4heQEaify1JDcBeG32s4iIVMg9QzezZTlNr25xLCIiEqHGH+oUEZGpUEIXEUlEOrfP9W5d6RXYnCJVTNE09larni8+/KPctntf+OJg35c9fH+wvczine3bG36CU3h0i57O9tk7K7z/ELcA5yhx5nL0LY+jCuGRa9BG3II2pm8rth+aBW+9fa0OZ1I6QxcRSYQSuohIIpTQRUQSoYQuIpKIZIqiZd8et877Z+DWnF7R872zFgbb/3HvY03FVMiep4LNHAgXLd31UmOKnh6viF7l8RC7dmZE7HW+vW3s+rquQP92jUt9R19ERKZECV1EJBFK6CIiiVBCFxFJRDJF0bKZt/5lYFYi+49scTT/3+gVF+W2dV30uWBfr+g5uvEnwfbOk14RbA+NW6iYW4Q7U3TXtvAGAkVTOjP7ogtoEdxj0eH9bnUWPJ4iXzNvXN1x27srv63kHDBOZ+giIolQQhcRSYQSuohIIoqsWHQNye0kN0x47BKSvyS5Nvt3VrlhioiIp0hR9FoAXwDw1UMev8LMPt3yiEoSurUlAHetU7cgEmiP3beHZ5T399Qreo588W+C7V3vuTS3LXZcbKezXmrsaxqhzNfci9vb93RW5msWPa7O7Z6jtl2Qe4ZuZncBeLIlexMRkdLEXEN/H8n12SWZOS2LSEREmtJsQr8SwHEAFgPYCuDyvCeSXE5ymOTwjp1PNLk7ERHxNJXQzWybmY2a2RiAqwAsDTx3pZkNmdnQ4MC8ZuMUERFHUzNFSc43s63Zj28GsCH0/HawA/vCTzjwXLg9sjAZ2r9b3AvNMAPAWf3h9qNPDLaXKVT0BIAHX7okt+3EdWui9h070zQ07t6Ye8dbbKE7RpX7Tlmp4+rlp4LchE7yegCvBDBAcguAvwXwSpKLARiAzQDe2ZJoRESkaW5CN7Nlkzx8dQmxiIhIBM0UFRFJhBK6iEgi2nz7XIONjjTdm52BcENtADC75E/YjB7MbbLRcGxeAc7Dw2bm79srFjvjFhzzAkKFz9WLTg72ffkj4Vq7W5js6Q23x4y7My7ecR47rtNVzPsfCI9b2ce6G3ug3T0WW3R7XZ2hi4gkQgldRCQRSugiIolQQhcRSUSbKzMMFzWefTrcfebz8tv256/pWUho2wXYoxtz2zpOyL0zQqOv83vTiW3sm1fm7/tN7whve/DoYHuZ3KLnjkfDG5jlvGZOISqGW0CLOZYTVmYx2Cs8Rm/fiz3id3OPl4J0hi4ikggldBGRRCihi4gkQgldRCQRtZqu5hX/gg4eCDbb9nCBjce/LNx/X7joyhf8drA92DeyQNZx7rvzt/28wahtV8kr2O55y+uC7X033trKcKbEngkv5hL7mpcpdKyzd1YbI0mHmz9adDzoDF1EJBFK6CIiiVBCFxFJhJvQSS4keQfJjSQfIPn+7PG5JG8juSn7Oqf8cEVEJE+RougIgA+a2RqSswHcS/I2AO8A8EMz+wTJFQBWALi4vFAdNhZs7nCKnh6vGGQ7H8tv9PqOhAu67OoJto+t/n5uW8fpbw5vu7cv2B4r9Lt5v5fHK3o++yd/GGyfcdW3cttiY+t4/qJge+xrXiYVPptj+/bktpX9PhvnnqGb2VYzW5N9vxvARgBHATgbwKrsaasAnFNWkCIi4pvSNXSSxwA4BcBqAEea2VagkfQBHNHq4EREpLjCCZ1kH4BvAfiAmT0zhX7LSQ6THN6xM/zZXBERaV6hhE6yG41k/jUz+3b28DaS87P2+QC2T9bXzFaa2ZCZDQ0OlLwMnIjIbzC3KEqSAK4GsNHMPjOh6bsAzgfwiezrTaVEWJRTdLCR/DU/AYBd3eH+3kyvgYXB9qB9z4bb+8IFso4lrwq0curxTOD+3k4BLVTc814T156ngs0zv/KvwfYnXv27uW3zfvgfTYVUVJVFTylHTOHTdk16PjxlRT7lchqAtwO4n+Ta7LG/RiOR30jyAgCPAvjjlkQkIiJNcRO6md2N/NO8V7c2HBERaZZmioqIJEIJXUQkEUroIiKJaO/90MdGg4uhRt0TeGwk2MyuuOnMZU6HZl9/VP/Rz300t63r49dEbbvU39v5ZJHHeg6L6h/6JMvIFR8K9u1850eC7XW+33mZYhc8L1OdY2N/a+Zl6gxdRCQRSugiIolQQhcRSYQSuohIItpbFO3oLK/w0N0bbLaD+6M2z+5wAS5m+9623f6LlzTd14s7NrYylVnE6vrLTwfbvaKp17/O7OkduW3uouPO+zBWsLDp7TsyttgcEtKq95nO0EVEEqGELiKSCCV0EZFEKKGLiCSivUXRMjF83+/S7z/9dOB+xv1HBrvGLhjMJac3v+0aFz09VS607BU9H3xpuFB94ro1rQyntToj0oLzPoxV6WxO74MR02GRaBERmR6U0EVEEqGELiKSCDehk1xI8g6SG0k+QPL92eOXkPwlybXZv7PKD1dERPIUqX6MAPigma0hORvAvSRvy9quMLNaTImretHdqEWiY/c9d35+W8qLEXuLTJf4u3uLZ5+w+t+D7asXnRxsf/kjG6YcU8uMhm9FHVTj18Tj3SraXzC9PYXPkCJrim4FsDX7fjfJjQCOKjswERGZmildQyd5DIBTAKzOHnofyfUkryE5J6fPcpLDJId37HwiKlgREclXOKGT7APwLQAfMLNnAFwJ4DgAi9E4g798sn5mttLMhsxsaHBgXgtCFhGRyRRK6CS70UjmXzOzbwOAmW0zs1EzGwNwFYCl5YUpIiIe9xo6SQK4GsBGM/vMhMfnZ9fXAeDNACqs4vjMKdZEr29Z1lqpRRx4rtzt11XJsxKDu45ca3Xphh8H2/914Qm5bX/42ENR+/a4t8gN9S1xDVog/D52i5pewdbhbr/KHJAp8imX0wC8HcD9JNdmj/01gGUkFwMwAJsBvLOUCEVEpJAin3K5G8Bkp0K3tD4cERFplmaKiogkQgldRCQRydw+15zCIHtmlBvA6Gi52w/g4NGV7bvScT+wL9x+2Mzy9u1wx8UpkoUKn7csPDHY96zHHgy2T2tj+bNYbee2YFcOLGh1NP9/+xXe2necztBFRBKhhC4ikggldBGRRCihi4gkor1FUTPYwf25zTHrW5Ze9PT2P3tuZfseCxTBOpwCWqwqx73KMQ/NCgTKLZB5Rc8Pzg4XyS/f/Wgrw2mr0PFmETNcU6EzdBGRRCihi4gkQgldRCQRSugiIolob1GUjCp8howO/yDY3jn0B6Xsd5zt3JLbVvYMNdv2i/zGkouiv6nqMCswj1f03PW63wu29996dyvDaZ/ABy4AACXlnjrRGbqISCKU0EVEEqGELiKSCDehk+wleQ/JdSQfIPmx7PFjSa4muYnk10n2lB+uiIjkKVIU3Q/gDDPbky0WfTfJ7wG4EMAVZnYDyS8BuADAlSXGGtTxolOD7aEZqkW4xdyZs5ved2yh2O65K7+x5GJwlcoe1+C+K5wpGssreo58/F25bV0f+VKrw2kd5/Wezq9ZUe4ZujXsyX7szv4ZgDMAfDN7fBWAc0qJUERECil0DZ1kZ7ZA9HYAtwH4GYBdZjZ+t/ktAI4qJ0QRESmiUEI3s1EzWwxgAYClAE6a7GmT9SW5nOQwyeEdO59oPlIREQma0qdczGwXgDsBnAqgn+T4NfgFAB7P6bPSzIbMbGhwYF5MrCIiEuAWRUkOAjhoZrtIzgDwGgCXAbgDwLkAbgBwPoCb3L3ZmLvWYjCW0K1au8IfsimzQAZUW1Dpes+luW3ueO931r6s8Ba1nlKLnt64OcfbdBYqfH58zjHBvh95anNrg5kC93j4DZgpWuRTLvMBrCLZicYZ/Y1mdjPJ/wJwA8m/A3AfgKtLjFNERBxuQjez9QBOmeTxR9C4ni4iIjWgmaIiIolQQhcRSUSbb5/bUd4alCMHwu0lr3059tA9uW0dJ4SvTMXOYBv5q7fmtnV+LDx5t85FzypVvUZtXXlFz9D7APDfC9OZ7ci/bTEHw+u8ejmgKJ2hi4gkQgldRCQRSugiIolQQhcRSUR7i6KO0LqcQHhtTvvVz8N9Fy1uKqaiYoo9sbNMuz51XVR/mbqYY7Xu7Nlncts48/BgX+99UOZ6pqG4i/B+N9es/oh9t2amuc7QRUQSoYQuIpIIJXQRkUQooYuIJKJWRVH0NV9UMOc2sG7/fXuC7ezta7q/1zfWnvPy1w3tu+EHpe67SlW+ZnUuesaOS3RxMMAretqubcF29h+Z3+bE7Y1LrJhxa1VsOkMXEUmEErqISCKU0EVEEuEmdJK9JO8huY7kAyQ/lj1+Lcmfk1yb/St35o6IiAQVKYruB3CGme0h2Q3gbpLfy9r+ysy+WXhvY6Ow53bnNnPG7MKbOlTnSa9oui8QX7gsu/AZknLhM2Q6v2aeqPeJWYujaZ9Q0RMArv+t43PbzvvZmvDGA2MKACjxeAi9nkBc7puoyBJ0BmC8BNud/Zu+R4yISKIKXUMn2UlyLYDtAG4zs9VZ06Uk15O8gmT6S2qLiNRYoYRuZqNmthjAAgBLSZ4M4MMATgTwOwDmArh4sr4kl5McJjm844knWxS2iIgcakqfcjGzXQDuBHCmmW21hv0AvgJg0tusmdlKMxsys6HBeVruTESkLEU+5TJIsj/7fgaA1wB4kOT87DECOAfAhjIDFRGRsCKfcpkPYBXJTjT+ANxoZjeTvJ3kIAACWAvgXe6W2AF0l3Op3ZxFotnVU8p+i+y/9H1XeNuBGFW/ZjHKjj30qQd3384nJrz+2Lc3f9t9c8J9I3mxLXt8U27b5wcXBfv++dYHm4qpFVr1KRZPkU+5rAdwyiSPn1FKRCIi0hTNFBURSYQSuohIIpTQRUQS0d77oZPlFboO7g+3O/uNnZprWx7K73vMi0vd93Sd6l3noqenyti9fXvHk6uzwmUSIt7Hf/7oumDX/e9+S7C996p/Ce/bUdZtTaZCZ+giIolQQhcRSYQSuohIIpTQRUQSUa9FoiPEFh2i+z//2Mr2PfrRd+a2dX3quqht15k99atgO+c8v7x9Ry7qW+YM3nYV4MrgfvggNO5ksK9X9By95Zpge+dZfxpsD82whYqiIiIyFUroIiKJUEIXEUmEErqISCKSKYpWLabIZQeeC2+7Z0Z4A4cfnr/t3eFVoji7ukVHYn/v2KJnaP/umHd0xu27xq9LnZVZTO54zbJg++gNnwm2d553YW5bu15vnaGLiCRCCV1EJBGFEzrJTpL3kbw5+/lYkqtJbiL5dZLT905LIiIJmMoZ+vsBbJzw82UArjCz4wE8BeCCVgYmIiJTU6goSnIBgDcAuBTAhdnC0GcAeGv2lFUALgFwpbctGxvL309H81eALDRLCwB7ZzW9bSAcNwBgbCS/rcMZ5q64dVZ5dqCYM6O+a4rG/t7u2ph0jqeY/Xt9Q8cD6l30jHmPeu+TmPe4J3bfXiE8VPQEgJ8ce3Ju2yt+viHY1z2WCyo6up8FcBGA8RGbB2CXmY0ftVsAHNWSiEREpCluQif5RgDbzezeiQ9P8tRJV1kguZzkMMnhHTt3NhmmiIh4ipyhnwbgTSQ3A7gBjUstnwXQT3L8WsICAI9P1tnMVprZkJkNDQ4MtCBkERGZjJvQzezDZrbAzI4BcB6A283sbQDuAHBu9rTzAdxUWpQiIuKKmSl6MYAbSP4dgPsAXF2kU1lFkdiip7t9L+6O/E9t2s4t4W0PLGgmpP/b9YlL87dd53U7R5z1I72ZopG/W3CmqjdTdO+uYPO0noEb8R4ts+hZ9r5jx+3lP/pGbttDi5cE+56wdk2wvagpJXQzuxPAndn3jwDIzyQiItJWmikqIpIIJXQRkUQooYuIJCKZ2+fac7vDT+gOz+yLLrCFZnr1HxG1bc/oZ1fktnVd9LlS9x3DvUWtw7slKbq6w+3OMRFS55meseMaw5vxWOcifey4sa8/t80reo58/uKofY/TGbqISCKU0EVEEqGELiKSCCV0EZFE0GzSe2qVszNyB4BfTHhoAEBd79il2Jqj2KaurnEBiq1ZrY7tBWY26D2prQn913ZODpvZUGUBBCi25ii2qatrXIBia1ZVsemSi4hIIpTQRUQSUXVCX1nx/kMUW3MU29TVNS5AsTWrktgqvYYuIiKtU/UZuoiItEglCZ3kmSQfIvkwyfwbkVSA5GaS95NcS3K4BvFcQ3I7yQ0THptL8jaSm7Kvc2oS1yUkf5mN3VqSZ7U7riyOhSTvILmR5AMk3589Xodxy4ut8rEj2UvyHpLrstg+lj1+LMnV2bh9nWTbb8gSiO1akj+fMG6L2x1bFkcnyftI3pz9XM2YmVlb/wHoBPAzAIsA9ABYB+BF7Y4jEN9mAANVxzEhntMBLAGwYcJjnwSwIvt+BYDLahLXJQA+VIMxmw9gSfb9bAD/DeBFNRm3vNgqHzs0Fn/vy77vBrAawKkAbgRwXvb4lwC8u0axXQvg3BoccxcCuA7AzdnPlYxZFWfoSwE8bGaPmNkBNBaePruCOKYFM7sLwKG3FTwbwKrs+1UAzmlrUMiNqxbMbKuZrcm+3w1gI4CjUI9xy4utctawJ/uxO/tnaCwM/83s8arGLS+2ypFcAOANAL6c/UxUNGZVJPSjADw24ectqMkBnTEAt5K8l+TyqoPJcaSZbQUaCQJAuffnnZr3kVyfXZJp+yWNQ5E8BsApaJzR1WrcDokNqMHYZZcO1gLYDuA2NP43vcvMRrKnVPZ+PTQ2Mxsft0uzcbuCZPP3RG7eZwFcBGAs+3keKhqzKhI6J3msFn9pM6eZ2RIArwfwXpKnVx3QNHIlgOMALAawFcDlVQZDsg/AtwB8wMyeqTKWQ00SWy3GzsxGzWwxgAVo/G/6pMme1t6osp0eEhvJkwF8GMCJAH4HwFw0Fq9vG5JvBLDdzO6d+PAkT23LmFWR0LcAWDjh5wUAHq8gjkmZ2ePZ1+0AvoN6LoS9jeR8AMi+bq84HgCAmW3L3nRjAK5ChWNHshuNhPk1M/t29nAtxm2y2Oo0dlk8u9BYEP5UAP0kxxfDqfz9OiG2M7NLWGZm+wF8Be0ft9MAvInkZjQuH5+Bxhl7JWNWRUL/KYDjsypwD4DzAHy3gjh+DclZJGePfw/gdQA2hHtV4rsAzs++Px/ATRXG8r/Gk2Xmzaho7LJrmFcD2Ghmn5nQVPm45cVWh7EjOUiyP/t+BoDXoHGN/w4A52ZPq2rcJovtwQl/oInGdeq2jpuZfdjMFpjZMWjkstvN7G2oaswqqgifhUZ1/2cA/qaKGHLiWoTGp27WAXigDrEBuB6N/4IfRON/NxegcY3uhwA2ZV/n1iSufwJwP4D1aCTP+RWN2e+h8V/c9QDWZv/Oqsm45cVW+dgBeAmA+7IYNgD4aPb4IgD3AHgYwDcAHFaj2G7Pxm0DgH9G9kmYio67V+L/PuVSyZhppqiISCI0U1REJBFK6CIiiVBCFxFJhBK6iEgilNBFRBKhhC4ikggldBGRRCihi4gk4n8AAPHNq3F8zV8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# %load -r 32-34 statcompare.py\n",
    "cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "plt.imshow(cm_normalized,cmap=plt.get_cmap('Reds'),aspect='auto')\n",
    "plt.grid(False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline accuracy 0.5881880273202089\n"
     ]
    }
   ],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "\n",
    "\n",
    "# setup pipeline to take PCA, then fit a different classifier\n",
    "clf_pipe = Pipeline(\n",
    "    [('PCA',PCA(n_components=100,svd_solver='randomized')),\n",
    "     ('CLF',GaussianNB())]\n",
    ")\n",
    "\n",
    "yhat_score = np.zeros((y.shape[0],len(lfw_people.target_names)))\n",
    "\n",
    "# now iterate through and get predictions, saved to the correct row in yhat\n",
    "for train, test in cv.split(X,y):\n",
    "    clf_pipe.fit(X[train],y[train])\n",
    "    yhat[test] = clf_pipe.predict(X[test])\n",
    "    yhat_score[test] = clf_pipe.predict_proba(X[test])\n",
    "\n",
    "total_accuracy = mt.accuracy_score(y, yhat)\n",
    "print ('Pipeline accuracy', total_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the ROC Curve\n",
    "In the below example we can see how to graph the false negatives and false positives in an ROC curve for a given classifier. Please note that the \"scores\" from the classifier have already been populated using the code above. The scores can be interpretted as the the probability that a given class should be designated as positive. These scores are needed so that the ROC can change thresholds deciding if theclass is positive or negative. Once we have the scores, we can send each column of scores (i.e., the probability for that class being positive) into the ROC curve generator and it will give use the arrays of false positives and negatives for that class as the threshold is increased. \n",
    "\n",
    "We save the outputs into a dictionary of fpr and tpr (false positive and true positive rates). The keys to the dictionary are the class value. We can also compute the ROC treshold for all the classes by placing their probabilities into one giant vector and whether they should or should not be a value of one. Please note that this method of combining all classes into a single ROC is not considered a standard method--it has limited utility outside classifier comparison. If you understand the limitations it can be an effective tool. If not, then I would not recommend using it to compare models right away. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XlclOX+//HXBai4o6Lmrrmzu2+heQyXTFQOpeYpl8wKt6wsz7fyqOEvj1qZWcc0XFJcUlOp3HJX3ABZVNx3FBUVTRRku35/jHMHzADDMsMg1/Px4JEzc899X8zE/bmX63pfQkqJoiiKogDYFHUDFEVRFOuhioKiKIqiUUVBURRF0aiioCiKomhUUVAURVE0qigoiqIoGlUUFEVRFI0qCoqiKIpGFQVFURRFY1fUDcgrR0dH2bBhw6JuhqIoSrESFhZ2R0pZPbflil1RaNiwIaGhoUXdDEVRlGJFCHHFlOXU5SNFURRFo4qCoiiKolFFQVEURdGooqAoiqJoVFFQFEVRNGYrCkKIxUKI20KIE9m8LoQQ84QQ54UQUUKI1uZqi6IoimIac54pLAV65/B6H6Dp05/RwP/M2BZFURTFBGYbpyCl3CeEaJjDIv2Bn6VuPtDDQggHIUQtKWWsudqkWJeFgaGs3Hi8qJuhFCP3yodyv1yU0dfKpz+gXHqCyeuqfK0BlWLrFVbTzMKWy9hxDZs0iU062Ah7lp84bdZtFuU9hTrAtQyPY54+Z0AIMVoIESqECI2Li7NI4xTzW7nxOBHRN4u6GUoxcr9cFImljf8/Uy49gVI8MXldlWLrUeZh5cJqWqGy5TJl2E8ZIrDlLjbpIKS0yLaLckSzMPKc0d9aSrkQWAjQtm1by3wyikV4OD3HnrUjiroZSjHh/f1m4DmCxqw1eC34/ZcB6DJ3s0nr+mLQ9wB8vmZMobUvL3auW8LBLYa/B8CpsAgAWrZ5gc59XuXOwmUADNq+x+ztKsqiEANkPHerC9woorYoiqKYRXY7/1NhBwDdjj8rfTHo4as7YFrztChYQlEWhSBgrBBiNdABeKDuJ1iHZYcCWX9so9m3c7G67jKA7uhPUXJ3/Ho0rnWcimz7OR3dZye7nX/WHb+1MFtREEKsAl4EHIUQMcB/gFIAUsoFwGbgZeA88Biwrk/GAqz1RuvF6otJLH2TssnPmXU7CY+TqVCutFm3oTxbXOs48c/WA8y+nfwc3WfHWnf+2TFn76MhubwugaK5mGcl9DdaPZzMu/PNj7LJz/F83Eizb+d1L1dGD21r9u0UFztXHuLgpmNF3QyrdnXffb6Y+73B8w/O1wdg1yDD14y5En2dBk6GfVt2rltCgP8EoPgc3RemYhed/ayxxhut+ss5QWOsq10lwcFNx7LdWZUUSXdu8uR+3nsZpiYmYFe2gsnLV6p9nfjH+/jirT8yPa8/G3jrs2+f6Z1/dlRRKObMcf2/qK/blnQNnOoUWY8YaxD8/ss8OH+cyk1cc1325I2bnL11GwC7ClCmSg2ocDXH9yTcvMHj27e5m6Ib05ByM3MhqVaqAnXtq3Jn4TKL3uDNSVxUBNXdPCyyLVUUirn1xzYW+k7cUtdtFSU7lZu4GnQtNXad/9TZ80DervE/vn2blEcJVCuv2/k3LOtY8AabWXU3D1oOet0i21JF4amiuOlbWPcTXOs4Ge23rSjFWdYiYOwmb36u8a/p+SJUsUyf/+JIFYWniuKmr4fTc7w+IPdTZEV5Vhk7+n9wXhdjcWNP5iJQEm7yWgNVFDIw103fnK77/35/M7+b1lnCKHX9X7F2OY/cNX0Al2IZqihYgDmu++sV9fV/1YWycFlTz6P8DNQyJq87/rzGVSiFSxUFC0i6e5P6iWl8cu6ReTZwLpDgNYHmWXcutuyvz50HZXCsbHoQmZI9h9JQM+kSwe//kfvChSxjTx6AGw/+AqB25UoFWm/typVoVrMGzhWzWeDAWoIP/F18TO15pJiHKgr5lJeuoKfjr1IvMR3szdyoIuJY+QkDPXPuBqj87Vb0Ve6czyHRJR5OmrHPw41S6dwqlW7w/AM7XdZk5VRdVmVlBDVTbKj9V2LBN3rtCie5YvLicbEXien5YsG3a2zdFuzeWRypopBPebkk1IiydLV3eCZPh/WjR7vMXVDELSk+1vR8keTHMWbZMV1OvENM0r0cl9H3z69WKkv/fCg2XTQLwpLdO4sjVRQKwNSuoPprpMqzKa/X3m/Hn4V65YivV67Q23IqTHfGllO//RqgbuAq2VJFQVHyIWMhyE9ImrmoHjtKQamiYAJj9w+epa6gBelBZE29ZfKqIL1rMhaCvO6I1zy9Vj4o4Nm7nKgUfyW2KGQdwZzTwDVj9w+KuitoYSpICFsDpzp07t/aDK3Kn7zs6AtyhK+OyJVnVYktCllHMOc2utiU+weXf1vC9Z2GyxSHLnbFKYQtv4OhslI7dkUxVGKLAhT+CObrO9caLQCVm7hSp8erhbadksjUa/hqR68oBVOii0JW2Y09yMv9A2Ppjkr2TL3cU5Br+AUVFbCQU2tWFtr6VD95xZqpopBBdmMPnqX7B9ZCXwxMvdxTlGcAp9asLNQdueonr1gzVRSyyG8M9c6Vh9iyP2/TAVoLS/Ugyu4SUHG43FPdzUNFLSslgioKheTgpmPFNgOooD2IisMlIEVRTKOKAn/fSyjo2AN9BtCzGvmQ3c6/OFwCUhTFNKookPleQk73DrLrcgrw4Hx9UhMTzNXEImNKrx+1s1eUZ0eJKwr6QWtZB6uZci8huy6nenZlK1CnR49Cba+l5TQFotr5K8qzr8QVhYwFIT9TYWbX5VR/c7lhv+K3w8zpbEAVAkUpWUpcUYDMg9aWHQok+MJhujTuaPL7jWUFFZcMIGP3BYrj2UBhjx3IiRpXoJQkJbIoZKQfrJaXcQjGsoKsLQPImJ3rlhDgPwHIfF+guBSCjAp77EBO1LgCpSQp8UUBoEvjjgzrNDRP7ykOWUHZ3R9467Nvi1UByI4aO6Aohc+mqBtgSQsDQ9l7+O8pAfWXjp5F+rMCfSEA3RnBs1IQFEUxjxJ1pqCPytbfYM7PpSNrlNN9AlUEFEXJixJVFAC6dWzA6KFttcc5XTrKOi7BGiKwc7tRrFcc7xMoilL0zFoUhBC9gW8BW+AnKeXMLK/XB5YBDk+XmSyltJqIUf24hBjbFzgbUxmoT5nE6ty6bfmeRjkFyKkCoChKYTFbURBC2ALfA15ADBAihAiSUkZnWOwz4Bcp5f+EEE7AZqChudqUH5WbuBIW+wL3k/8uBA0csUhPI0sEyFmya2dhUt1EFcU8zHmm0B44L6W8CCCEWA30BzIWBQlUevrvysANM7anQMzd26ioxg9YsmtnYVLdRBXFPMxZFOoA1zI8jgE6ZFlmKrBdCDEOKA+8ZMb2WJ3ccoUsdVlIde1UFEXPnEVBGHlOZnk8BFgqpfxKCNEJWC6EcJFSpmdakRCjgdEA9evXN0tjLSW7QqDuCyiKYg3MWRRigHoZHtfF8PLQW0BvACnlISGEPeAI3M64kJRyIbAQoG3btlkLS7Fg7EaxKgSKolgbcxaFEKCpEKIRcB0YDGS9CHwV6AEsFUK0BOyBODO2yST6rqj6nkenIi7QsmPjfK8va7yEKgSKolgrsxUFKWWqEGIssA1dd9PFUsqTQojpQKiUMgj4EFgkhJiI7tLScCllkZ8JZIzIDj7eBHiU795GGQuCGkimKIq1M+s4hadjDjZneW5Khn9HA13M2Yb80kdk7xr0PS0docfrnfK8DlUQFEUpbkrciGZLyHr/oKgLQk5jEYpjd1RFUcxHFYVCZo33D3Iai6D6+yuKkpEqCoXImi8XqbEIiqKYokRFZ5uTNRcERVEUU6kzhaf0U2wm3blJQkxlSlWozK5B35s0zaYqCIqiPCtUUXhKP8WmQ+l7AJRxqA7kPM2mtd1QVhRFKShVFDJo4FSHf9SKAKDL3C+yXc7Y6GRruKGsKIpSUCW2KOin4uzSuGOe33twy1qunDluNcUgt/hr1e1UURRTldiikN+pOHeuW8KpsAO0bPMCnwdYx3xAucVfq26niqKYqsQWBch5Kk5jMt5Q7tznVXM1K19Ul1NFUQqDSV1ShRClhRBNzN2YorJz5SFOHb6Q63L6yGt1Q1lRlGdVrkVBCNEXOA78+fSxhxBig7kbZglJd28S/P7LbPlqGQA1kw7w4Pxxo8tmvGykCoKiKM8qU84UpqObMe0+gJQyAngmzhqexMdpRaC242OcGz2gchNX6vTIfGnImi8bKYqiFCZT7imkSCnvC5FpIrUij7cuLJWbuFK5vCsAXeYuMLqMumykKEpJYUpROCWEeA2weTphzgTgsHmbZX3UZSNFUUoCU4rCWGAKkA78im7SnH+bs1HWJOO9hPzKbRxBQalxCIqiFBZT7in0klJ+IqVs9fRnMtDH3A0zp50rD1F2VXUc93Vgw/76XIm+bny5QrqXoB9HYC5qHIKiKIXFlDOFz9CdIWT0qZHnio2Dm45hc7s0VHwEZJ9vVJj3EtQ4AkVRioNsi4IQohfQG6gjhPg6w0uV0F1KKpZckw7y4IYtyRUTudluFx/Yu2Z7gxnUvQRFUUqWnC4f3QZOAEnAyQw/2ynGl49aJIWRmpgAgF3Z8gbdT/X09xIURVFKkmzPFKSU4UC4ECJQSplkwTaZnV3ZCtiVtaFyE1ca9jM8C1DjEhRFKalMuadQRwgxA3AC7PVPSimbma1VRUhNmKMoSklmSu+jpcASQKC7bPQLsNqMbSoyqiAoilLSmXKmUE5KuU0IMUdKeQH4TAix39wNKwqm9jbK67gDNY5AUZTiwpQzhSdCl3FxQQjxrhCiH1DDzO0qMqb0NsrruAM1jkBRlOLClDOFiUAFYDwwA6gMjDRno8zFNekg9VLPc4n2Bq/ldeSyGnegKMqzKNeiIKU88vSfD4E3AIQQdc3ZKHNpkRQGwF/l7Hnw+A6lM7ymv3SkehspilKS5Xj5SAjRTggxQAjh+PSxsxDiZ4pxIN41uyY8kMmA4VScaqCaoiglXbZFQQjxJRAIDAW2CiE+BXYDkUCx745auWylPE3FqSiKUhLkdPmoP+AupUwUQlQFbjx9fMYyTVMURVEsLafLR0lSykQAKeU94PSzUBBCqyTyIPGvom6GoiiKVcqpKDwvhPj16c8GoGGGxyYlpAohegshzgghzgshJmezzGtCiGghxEkhhNkmHVgYGMr9h0lEOTwBwLGCo/aayjlSFEXRyeny0T+zPJ6flxULIWyB7wEvIAYIEUIESSmjMyzTFN2EPV2klPFCCLONf1i58TheQOlSNlQuW4nnKtfUXlM9jxRFUXRyCsTbWcB1twfOSykvAgghVqO7TxGdYZm3ge+llPFPt3m7gNvMkUNFe0qXNp76rXoeKYqimDaiOb/qANcyPI55+lxGzYBmQohgIcRhIURvYysSQowWQoQKIULj4uLM1FxFURTFnEVBGHlOZnlsBzQFXgSGAD8JIRwM3iTlQillWyll2+rVqxeoUeUu1cP2mn3uCyqKopRAJhcFIUSZPK47BqiX4XFddN1asy6zSUqZIqW8BJxBVyTMpty12gDa9JvqJrOiKMrfci0KQoj2QojjwLmnj92FEN+ZsO4QoKkQopEQojQwGAjKssxGoPvT9Tqiu5x0MQ/tz5e0ekn0eL2TmkxHURQlC1POFOYBrwB3AaSUkTzdkedESpkKjAW2AaeAX6SUJ4UQ04UQ3k8X2wbcFUJEoxstPUlKeTfvv0b+mBqVrSiKUlKYkpJqI6W8okvP1qSZsnIp5WZgc5bnpmT4twQ+ePpTJFSvI0VRlL+ZUhSuCSHaA/Lp2INxwFnzNktRFEUpCqZcPnoP3ZF8feAW0PHpc4qiKMozxpQzhVQp5WCzt0RRFEUpcqacKYQIITYLIYYJISqavUUWorqiKoqiGMq1KEgpGwP+QBvguBBioxCi2J85qLwjRVEUQyYNXpNSHpRSjgdaA3+hm3yn2MtPz6OogIXE7N9rphYpiqIULVMGr1UQQgwVQvwGHAXigM5mb5mVOrVGl+7dctDrRdwSRVGUwmfKjeYTwG/ALCnlfjO3x6xu3q9FmTvVSKuXVKD11PXshttbowupVYqiKNbDlKLwvJTSeN50MRP3UDeHQkrLx/BM/EaKoiiFK9uiIIT4Skr5IbBeCJE13RQppY9ZW2YmTxzvkurxGI4VdUsURVGsT05nCmue/jdPM64piqIoxVdOM68dffrPllLKTIVBCDEWKOjMbIqiKIqVMaVL6kgjz71V2A2xJLtT19TANUVRFCNyuqcwCN0cCI2EEL9meKkicN/cDTOnUhdiAdMHrkUFLNS6osZFRVDdzcNsbVMURSlKOd1TOIpuDoW6wPcZnn8IhJuzUZaQl4Frp9as1IpBdTcPNUZBUZRnVk73FC4Bl4AdlmuO9aru5sGg7XuKuhmKoihmldPlo71Sym5CiHggY5dUgW5+nKpmb52iKIpiUTldPtJPueloiYZYwkO7NB6RRuWiboiiKIqVyrb3UYZRzPUAWyllGtAJeAcob4G2FbpHdroTHscKz0ydUxRFKVSmdEndiG4qzsbAz0BLYKVZW2VG5bHluUo1i7oZiqIoVsmUopAupUwBfIC5UspxQB3zNqvwuSYdxIa0om6GoiiKVTOlKKQKIV4F3gB+f/pcKfM1yTxaJIUBYGNnetOjAhaypueLxEVFmKtZiqIoVsXUEc3d0UVnXxRCNAJWmbdZ5pGOLTalSpu8fMbxCWpsgqIoJUGu0dlSyhNCiPFAEyFEC+C8lHKG+ZtmHiLlHKfCjtCyzQsmLa/GJyiKUpLkWhSEEJ7AcuA6ujEKzwkh3pBSBpu7ceYgUi4Bam5mRVEUY0yZZOcb4GUpZTSAEKIluiLR1pwNM6f8zM38rEtJSSEmJoakpILNSqcoStGyt7enbt26lCqVv1u/phSF0vqCACClPCWEMP3CvFIsxMTEULFiRRo2bIgQoqiboyhKPkgpuXv3LjExMTRq1Chf6zDlRvMxIcSPQogXnv78j2cgEE/JLCkpiWrVqqmCoCjFmBCCatWqFeiM35QzhXeB8cDH6O4p7AO+y/cWi5AtlxHpt4FmBq9ljMfWK2kx2aogKErxV9C/4xyLghDCFWgMbJBSzirQlqyAHdcA4zeZM3Y/1VNdURVFKWmyvXwkhPg/dBEXQ4E/hRDGZmDLkRCitxDijBDivBBicg7L+QohpBDC7DevpU2NbG8y67ufZvxxe2u0uZuk5GDq1KnMmTOn0Nf7+PFj+vbtS4sWLXB2dmby5L//97x69Srdu3enVatWuLm5sXnz5nxto3PnzgBcvnyZlSv/PgtdunQpY8eONXk948aNo0KFCtrjffv20bp1a+zs7Fi3bl2u28+rjRs3Eh0dnfuCFhQbG8srr7xS1M3I0bJly2jatClNmzZl2bJlRpeJjIykU6dOuLq60q9fP/766y8AkpOTGTFiBK6urri7u7Nnzx7tPS+99BLx8fGW+BWAnO8pDAXcpJSvAu2A9/KyYiGELbrJefoATsAQIYSTkeUqors8dSQv61eUgvroo484ffo04eHhBAcHs2XLFgD8/f157bXXCA8PZ/Xq1fj5+eVr/QcPHgQMi0JehIaGcv9+5okO69evz9KlS3n99ZzPYvXbzytrLApff/01b7/9tsnLp6VZNtLm3r17TJs2jSNHjnD06FGmTZtmdEc+atQoZs6cyfHjxxk4cCCzZ88GYNGiRQAcP36cP//8kw8//JD0dF0m6RtvvMEPP/xgsd8lp8tHT6SUjwCklHFCCFNuSmfUHt1At4sAQojVQH8g6/9tXwCzgI/yuH7FTN6fuoWIkzcLdZ0ezs8xd2qfHJf5+eefmTNnDkII3NzcWL58eabXFy1axMKFC0lOTqZJkyYsX76ccuXKsXbtWqZNm4atrS2VK1dm3759nDx5khEjRpCcnEx6ejrr16+nadOm2rrKlStH9+66dPjSpUvTunVrYmJiAN01Wf0R3IMHD6hdu7ZBW/38/Ojduzfe3t4MHDiQKlWqsHjxYgICArh06RL+/v5UqFCBhIQEJk+ezKlTp/Dw8GDYsGFUqVKFGzdu0Lt3by5cuMDAgQOZNcvw6mxaWhqTJk1i5cqVbNiwQXu+YcOGANjY5Pwnqd/+nj17mDp1Ko6Ojpw4cYI2bdqwYsUKhBBMnjyZoKAg7Ozs6NmzJz4+PgQFBbF37178/f1Zv349u3btMvq5Dx8+nEqVKhEaGsrNmzeZNWsWvr6+AMyaNYvly5djY2NDnz59mDlzJhcuXGDMmDHExcVRrlw5Fi1aRIsWLYx+f1mtX78ef39/QFdk33jjDR49egTA/Pnz6dy5M3v27GHatGnUqlWLiIgIoqOjWbFiBfPmzSM5OZkOHTrwww8/YGtry3vvvUdISAiJiYn4+voybdq0HD/L3Gzbtg0vLy+qVtVNM+Pl5cXWrVsZMmRIpuXOnDlD165dtWV69erFF198QXR0ND169ACgRo0aODg4EBoaSvv27fH29sbT05NPP/20QG00VU5F4fkMczMLoHHGuZqllD65rLsOPL2IrxMDdMi4gBCiFVBPSvm7EEIVhRLs5MmTzJgxg+DgYBwdHbl3757BMj4+PtrR4meffUZAQADjxo1j+vTpbNu2jTp16mhH1QsWLGDChAkMHTqU5OTkHI8c79+/z2+//caECRMA3SWrnj178t133/Ho0SN27DCcfLBr167s378fb29vrl+/Tmysbt7vAwcOMHjw4EzLzpw5kzlz5vD777rosKVLlxIREUF4eDhlypShefPmjBs3jnr16mV63/z58/H29qZWrVqmfozZCg8P5+TJk9SuXZsuXboQHByMk5MTGzZs4PTp0wghuH//Pg4ODnh7e/PKK69oO3gHBwejnzvoLuscOHCA06dP4+3tja+vL1u2bGHjxo0cOXKEcuXKad/l6NGjWbBgAU2bNuXIkSP4+fmxa9cuo99fRpcuXaJKlSqUKVMG0O00//zzT+zt7Tl37hxDhgwhNDQUgKNHj3LixAkaNWrEqVOnWLNmDcHBwZQqVQo/Pz8CAwN58803mTFjBlWrViUtLY0ePXoQFRWFm5tbpu3Onj2bwMBAg/Z07dqVefPmZXru+vXrmb6/unXrcv36dYP3uri4EBQURP/+/Vm7di3Xrul2ke7u7mzatInBgwdz7do1wsLCuHbtGu3bt6dKlSo8efKEu3fvUq1aNRO+7YLJqSj8M8vj+Xlct7Fb4NoMbk/PPL4Bhue6IiFGA6NBd+qsmFduR/TmsGvXLnx9fXF01M11oT/iyujEiRN89tln3L9/n4SEBHr16gVAly5dGD58OK+99ho+PrpjlU6dOjFjxgxiYmLw8fHJdJaQUWpqKkOGDGH8+PE8//zzAKxatYrhw4fz4YcfcujQId544w1OnDiR6cjc09OTuXPnEh0djZOTE/Hx8cTGxnLo0CGDHYYxPXr0oHJl3XRPTk5OXLlyJdNO5caNG6xduzbTteWCaN++PXXr1gXAw8ODy5cv07FjR+zt7Rk1ahR9+/bN9pp9dp87wIABA7CxscHJyYlbt24BsGPHDkaMGEG5cuUA3XeZkJDAwYMHefXVvzt5PHnyBDD+/WUUGxtL9erVtccpKSmMHTuWiIgIbG1tOXv2bKbfU98/f+fOnYSFhdGuXTsAEhMTqVGjBgC//PILCxcuJDU1ldjYWKKjow2KwqRJk5g0aZIpHy9SSoPnjPUCWrx4MePHj2f69Ol4e3tTurRuyNfIkSM5deoUbdu2pUGDBnTu3Bk7u793zzVq1ODGjRtFWxSklDsLuO4YdBP06NUFbmR4XBFwAfY8/fCeA4KEEN5SytAsbVkILARo27at4aevFHtSyly70g0fPpyNGzfi7u7O0qVLtR3mggULOHLkCH/88QceHh5ERETw+uuv06FDB/744w969erFTz/9xD/+8Q+DdY4ePZqmTZvy/vvva88FBASwdetWQFdckpKSuHPnjrZDAahTpw7x8fFs3bqVrl27cu/ePX755RcqVKhAxYoVc/199Ue9ALa2tqSmpmZ6PTw8nPPnz9OkSRNAd2O8SZMmnD9/Ptd1m7o9Ozs7jh49ys6dO1m9ejXz589n165dBu/N7nPPul79jtHYd5meno6DgwMREYaJw8a+v4w7v7Jly2bqd//NN99Qs2ZNIiMjSU9Px97eXnutfPm/5/+SUjJs2DC+/PLLTNu7dOkSc+bMISQkhCpVqjB8+HCj/frzcqZQt27dTJ9LTEwML774osF7W7Rowfbt2wE4e/Ysf/zxBwB2dnZ888032nKdO3fOdCCTlJRE2bJlDdZnDnm9T5AXIUBTIUSjpyOgBwNB+hellA+klI5SyoZSyobAYcCgIFhCVMBCYvbvtfRmlQx69OjBL7/8wt27dwGMXj56+PAhtWrVIiUlJdMf64ULF+jQoQPTp0/H0dGRa9eucfHiRZ5//nnGjx+Pt7c3UVFRBuv77LPPePDgAXPnzs30fP369dm5U3dMdOrUKZKSkjIdqep16tSJuXPn0rVrVzw9PZkzZw6enp4Gy1WsWJGHDx/m6fPo27cvN2/e5PLly1y+fJly5crluyBkJyEhgQcPHvDyyy8zd+5cbYedtb3Zfe7Z6dmzJ4sXL+bx48eA7rusVKkSjRo1Yu3atYBuhx0ZGQkY//4yatasGZcvX9YeP3jwgFq1amFjY8Py5cuzvTTYo0cP1q1bx+3bt7V2XLlyhb/++ovy5ctTuXJlbt26pXUwyGrSpElEREQY/Bg7E+zVqxfbt28nPj6e+Ph4tm/fnumMSk/flvT0dPz9/Xn33XcBXdHX3yP5888/sbOzw8nJSfusbt68qd1LMjezFQUpZSowFtgGnAJ+kVKeFEJMF0J4m2u7+aEftKbGJBQdZ2dnPv30U7p164a7uzsffPCBwTJffPEFHTp0wMvLixYtWmjPT5o0CVdXV1xcXOjatSvu7u6sWbMGFxcXPDw8OH36NG+++WamdcXExDBjxgyio6Np3bo1Hh4e/PTTTwB89dVXLFq0CHd3d4YMGcLxE903AAAgAElEQVTSpUuNnsV4enqSmppKkyZNaN26Nffu3TNaFNzc3LCzs8Pd3T3T0WB+hYSEULduXdauXcs777yDs7Nzvtbz8OFDXnnlFdzc3OjWrZvWtsGDBzN79mxatWrFhQsXsv3cs6O/Ad+2bVs8PDy0LsWBgYEEBATg7u6Os7MzmzZtAox/fxmVL1+exo0ba0XRz8+PZcuW0bFjR86ePZvp7CAjJycn/P396dmzJ25ubnh5eREbG4u7uzutWrXC2dmZkSNH0qVLl3x9fhlVrVqVzz//nHbt2tGuXTumTJmiXQIdNWqUds9j1apVNGvWjBYtWlC7dm1GjNB1j799+zatW7emZcuW/Pe//83UySIsLIyOHTtmupxkTsLYtTCjCwpRRkr5xMztyVXbtm2l/gPOi++9WrMvLh4bG1tWHct8xLWm54sAJToi+9SpU7Rs2bKom6EoRm3YsIGwsDCtB1JJMmHCBLy9vbXeSaYw9vcshAiTUuY6FizXMwUhRHshxHHg3NPH7kKIYhlzoShK8TRw4ECLXT6xNi4uLnkqCAVlyuWjecArwF0AKWUkupnYFEVRLGbUqFFF3YQikZdBe4XBlKJgI6W8kuU5yw4XVBRFUSzClDsX14QQ7QH5NLpiHHA2l/coiqIoxZApZwrvAR8A9YFbQEfymINkraICFrKm54vERRn2nVYURSmJcj1TkFLeRjfG4JmTMS5bdUdVFEUxrffRIiHEwqw/lmicJejjslVEtnUqiujspUuXUr16dTw8PDKNX8irl19+mfv373P//v1MKZd79uwxKQZ60qRJtGjRAjc3NwYOHKjlAqWkpDBs2DBcXV1p2bKlwYjdrNvPqz179uQ7YdVcEhMT6datm8XTT/Ni69atNG/enCZNmjBz5kyjy1y5coUePXrg5ubGiy++qIUwXrlyhTZt2uDh4YGzszMLFizQ3mNN0dl6O4CdT3+CgRpAkY9XUJSCyi46G2DQoEHaCNb89nrZvHkzDg4OBkXBVF5eXpw4cYKoqCiaNWum7fzXrl3LkydPOH78OGFhYfz444+ZRvxm3X5eWWNRWLx4MT4+Ptja2pq0vJRSi562hLS0NMaMGcOWLVuIjo5m1apVRuPHP/roI958802ioqKYMmUK//73vwGoVasWBw8eJCIigiNHjjBz5kxu3NClAllTdDYAUso1GR8LIZYDf5qtRUqROzH/Ex6cP16o66zcxBWXsf/NcRlric42xaxZs7C3t2f8+PFMnDiRyMhIdu3axc6dO1myZAkrVqygYcOGhIaGMnnyZC5cuICHhwdeXl707duXhIQEfH19DaKsM+rZs6f2744dO2oT6gghePToEampqSQmJlK6dGkqVapk0Eb99hMSEujTpw8vvPACBw8epE6dOmzatImyZcsyb948FixYoMUqzJw5kwULFmBra8uKFSv47rvvuH//Pv7+/iQnJ1OtWjUCAwOpWbMmU6dO5erVq1y8eJGrV6/y/vvvM378+Gy/y7i4ON59912uXr0KwNy5c+nSpQt79+7VEmqFEOzbt88gPyowMFCbkyIhIYH+/fsTHx9PSkoK/v7+9O/fn8uXL9OnTx+6d+/OoUOH2LhxI2fOnOE///kPT548oXHjxixZsoQKFSowffp0fvvtNxITE+ncuTM//vhjgaaxPHr0KE2aNNFCFQcPHsymTZu0qAq96OhobeR49+7dGTBgAIAWjAe6oMCMBc3S0dn5ibloBDQo7IYoJZs+OnvXrl1ERkby7bffGizj4+NDSEgIkZGRtGzZkoCAAAAtejkyMpKgIF28lj46OyIigtDQUC0h1Bh9dHbGAULr16/Hzc0NX19fgywe+Ds6G9B2vCkpKRw4cMAg6mLmzJk0btyYiIgIbVKV8PBwLWX14sWLBAcH5/j5LF68mD59dOm1vr6+lC9fnlq1alG/fn0++ugjo6myGZ07d44xY8Zw8uRJHBwcWL9+vda28PBwoqKiWLBgAQ0bNuTdd99l4sSJRERE4OnpyQsvvMDhw4cJDw9n8ODBmeZ+OH36NNu2bdMmlklJScn2u5wwYQITJ04kJCSE9evXa2dgc+bM4fvvvyciIoL9+/cbBL8lJydz8eJFbfCavb09GzZs4NixY+zevZsPP/xQC+M7c+YMb775JuHh4ZQvXx5/f3927NjBsWPHaNu2LV9//TUAY8eOJSQkhBMnTpCYmKjFmmcUGBioXULM+KOPFM/I1Ohsd3d37bPfsGEDDx8+1PK+rl27hpubG/Xq1eOTTz7R5vHIGJ1tCbmeKQgh4vk78toGuAdkO7WmUvzldkRvDtYUnd2vXz+GDBlCmTJlWLBgAcOGDTNID23Tpg1hYWE8fPiQMmXK0Lp1a0JDQ9m/f79J0dnGoqxfeOEFo8vOmDEDOzs7hg4dCuiOSm1tbblx4wbx8fF4enry0ksvae03plGjRnh4eGht119ucnNzY+jQoQwYMEA7as0qJiaGQYMGERsbS3JyshZNDbrgvjJlylCmTBlq1KjBrVu3sv0ud+zYkemSyl9//cXDhw/p0qULH3zwAUOHDsXHx8eggN+5cyfTZTApJf/3f//Hvn37sLGx4fr161psd4MGDejYsSMAhw8fJjo6Wss2Sk5OplOnTgDs3r2bWbNm8fjxY+7du4ezszP9+vXLtN2hQ4dqn3luTI3OnjNnDmPHjmXp0qV07dqVOnXqaJlG9erVIyoqihs3bjBgwAB8fX2pWbMmYNno7BzPFITut3IHqj/9qSKlfF5K+YvZW6aUKKZGZ8+fP5/jx4/zn//8R4s7XrBgAf7+/ly7dg0PDw/u3r3L66+/TlBQEGXLlqVXr15GI6HBeHR2tWrVtEjot99+m7CwMIP3lSpVioYNG7JkyRI6d+6Mp6cnu3fv5sKFCyZlSOUWna23bNkyfv/9dwIDA7XPZ+XKlfTu3ZtSpUpRo0YNunTpQm55YNlt748//mDMmDGEhYXRpk0bo+0YN24cY8eO5fjx4/z444+ZYqaNrTe77zI9PZ1Dhw5p92quX79OxYoVmTx5Mj/99BOJiYl07NiR06dPZ3pf1ujswMBA4uLiCAsLIyIigpo1a2qvZ43O9vLy0rYXHR1NQEAASUlJ+Pn5sW7dOo4fP87bb79tNDo7L2cKdevWzXRGGRMTY3TGvtq1a/Prr78SHh7OjBkzALR5NTIu4+zsrJ2JghVFZ0td+dsgpUx7+vPMzGWg4rKtizVFZ+tnUQMICgrKdifftWtX5syZo0VnL1iwAA8PD4MdYn6is0HXm+W///0vQUFB2oQ1oIv23rVrF1JKHj16xOHDh01KL80qPT2da9eu0b17d2bNmqWdgWVt74MHD6hTpw5AthPSZ5Tdd9mzZ0/mz/97ri59VPeFCxdwdXXlk08+oW3btgZFoUqVKqSlpWk77gcPHlCjRg1KlSrF7t27uXIla+CCTseOHQkODtbSVR8/fszZs2e19Tg6OpKQkKDdq8lq6NChRqOzjS3frl07zp07x6VLl0hOTmb16tV4exuGQd+5c0e7X/Dll18ycuRIQFdEEhMTAYiPjyc4OJjmzZsD1hmdfVQI0drsLbEwFZdtXawpOnvevHk4Ozvj7u7OvHnzWLp0qdE2e3p6EhsbS6dOnahZsyb29vZGo7OrVatGly5dcHFxMXkmL9Bd93748CFeXl54eHho2ftjxowhISEBFxcX2rVrx4gRIwxmDTNFWloa//rXv3B1daVVq1ZMnDgRBwcH+vXrx4YNG/Dw8GD//v1MnTqVV199FU9PT+2SUE6y+y7nzZtHaGgobm5uODk5ad0u586di4uLC+7u7pQtW1a7d5JRz549OXDgAKDbWYeGhtK2bVsCAwOzLYjVq1dn6dKlDBkyBDc3N+0sRD+9qKurKwMGDNBmZisIOzs75s+fT69evWjZsiWvvfaaFmk+ZcoU7V7Xnj17aN68Oc2aNePWrVvazeNTp07RoUMH3N3d6datGx999BGurq6AFUVnCyHspJSpTxNSWwIXgEfoptmUUsoiKRSFFZ2t4rIzU9HZijULDw/n66+/NuiRVhJYOjo7p9JzFGgNGL/7pCiKYiGtWrWie/fupKWlmTxW4Vlh6ejsnIqCAJBSXrBQWxRFUbKlv/5e0lg6OjunolBdCGF4YfcpKeXXZmiPoiiKUoRyKgq2QAWenjEoiqIoz76cikKslHK6xVqiKIqiFLmcuqQ+k2cIIi1dzaGgKIqSjZyKguVud1uQSEtXcygUI+aKzgbo3bs37u7uODs78+6772aKZf7uu+9o3rw5zs7OfPzxx/laf+fOnQG4fPmyFuYGumjusWPH5vr+4cOHa/EUHh4e2mCvwMBA3NzccHNzo3PnzkRGRhp9v4rOtqyCRGfv3r0706hpe3t7Nm7cCOjC9c6dO2ex3wMpZbH6adOmjcyP+S+1kq+5N5SvOzeQq7265Wsdz7Lo6OiiboJR//nPf+Ts2bPNsu4HDx5IKaVMT0+XPj4+ctWqVVJKKXft2iV79Oghk5KSpJRS3rp1q0Db2b17t+zbt6/2eMmSJXLMmDG5vm/YsGFy7dq1Bs8HBwfLe/fuSSml3Lx5s2zfvn2B2peVOT/z/Jo/f76cO3euycunp6fLtLQ0M7Yos9TUVPn888/LCxcuyCdPnkg3Nzd58uRJg+V8fX3l0qVLpZRS7ty5U/7rX/8yWObu3buySpUq8tGjR1JKKffs2SNHjRqVp/YY+3sGQqUJ+1jLDJGzAjfv18I2JQFEclE3xer934apnLhxslDX6VLbmf83cGqOy1gyOhvQ4qZTU1NJTk7W4in+97//MXnyZC3Xp0aNGgZt9fPzo3fv3nh7ezNw4ECqVKnC4sWLCQgI4NKlS/j7+1OhQgUSEhKYPHkyp06dwsPDg2HDhlGlShVu3LhB7969uXDhAgMHDsyUPJob/RkI6KIcsov8VtHZxSc6O6N169bRp08fLdrE09OT4cOHk5qaapFRzfmJzi6W4h7q0gbthGHwlVL0iio6u1evXtSoUYOKFStqQWdnz55l//79dOjQgW7duhESEmLwvozR2devX9fSP7OLzvb09CQiIoKJEycCutyfNWvWcPz4cdasWWM0nhvg008/xc3NjYkTJ/LkieHcVgEBAUZjIbJS0dnWH52tt3r1aoYMGaI9trGxoUmTJtleJixsJeZMASCtVAr26YlF3Qyrl9sRvTkUVXT2tm3bSEpKYujQoezatQsvLy9SU1OJj4/n8OHDhISE8Nprr3Hx4sVMR5Kenp7afAhOTk7Ex8cTGxvLoUOHTIrO7tGjh5aO6eTkxJUrVzLtVEAXmPbcc8+RnJzM6NGj+e9//8uUKVO013fv3k1AQICWCZQTFZ1t/dHZoAtjPH78uPb/tp4+OrtNmzYmtacgSsyZgmLdZBFFZ4PuyNPb25tNmzYBuqM8Hx8fhBC0b98eGxsb7ty5k+k9derUIT4+nq1bt2opqb/88gsVKlQwuPRhjCnR2bVq1UIIQZkyZRgxYgRHjx7VXouKimLUqFFs2rTJpIx9FZ1dPKKzf/nlFwYOHEipUqUyvc9qorMVxVIsHZ2dkJCgRWSnpqayefNmLW1zwIABWhE5e/YsycnJRtNBO3XqxNy5c7WiMGfOHKMpqfmNzta3T0rJxo0bcXFxAeDq1av4+PiwfPlymjVrluf16qnobOuJztZbtWpVpktHemfPntVSV81NFQXFKlg6OvvRo0d4e3vj5uaGu7s7NWrU0KKpR44cycWLF3FxcWHw4MEsW7bM6JGvp6cnqampNGnShNatW3Pv3j2jRcHNzQ07Ozvc3d21m4ymGDp0KK6urri6unLnzh0+++wzQHcP5e7du/j5+eHh4UHbtrkGXxqlorOtJzobdF2Xr127Rrdu3TKt+9atW5QtW5ZatWoVuJ2myDY621rlNzr73RYjibffTfn0e/R6rpWKzM5CRWcr1qwkR2d/8803VKpUibfeesvk9xQkOludKSiKYvUyRmeXNA4ODgwbNsxi2zNrURBC9BZCnBFCnBdCTDby+gdCiGghRJQQYqcQooE526MoSvE1cuTIEjeXAsCIESMsNusamLEoCCFsge+BPoATMEQI4ZRlsXCgrZTSDVgHmD6CR1EURSl05jxTaA+cl1JelFImA6uB/hkXkFLullI+fvrwMGB8hJGiKIpiEeYsCnWAjMM0Y54+l523gC3GXhBCjBZChAohQuPi4gqxiYqiKEpG5iwKxkYiGe3qJIT4F9AWmG3sdSnlQillWyll2+rVq+e7QTZpEruklHy/X1EU5VlnzqIQA2Qct18XuJF1ISHES8CngLeU0jDcpRDZ6MaMqMjsYsSc0dmffvop9erVo0KFCkZfX7duHUII8tMFGgoenT106FCaN2+Oi4sLI0eOJCXl7wOaPXv24OHhgbOzs0G/dj0VnW1ZpkRnX716le7du9OqVSvc3NzYvHkzoAvU04+Ydnd3Z8OGDYAumqNr165GR5qbizmLQgjQVAjRSAhRGhgMBGVcQAjRCvgRXUG4bca2aFLtS+H21mhLbEqxcv369csUHZHRw4cPmTdvHh06dMj3+vU71qxFwVRDhw7l9OnTHD9+nMTERH766ScA7t+/j5+fH0FBQZw8eZK1a9caff/mzZszZQaZyhqLwuLFi/Hx8TG595GUUhs5bAlpaWmMGTOGLVu2EB0dzapVqzLlPOn5+/vz2muvER4ezurVq/Hz8wPAxcWF0NBQIiIi2Lp1K++88w6pqamULl2aHj16sGbNGov9Lmbr5ySlTBVCjAW2oZvvebGU8qQQYjq6XO8gdJeLKgBrn44YvSqlNBwbrljU7o/e53Yhz0xXw82D7nPm5riMpaOz9cFpxnz++ed8/PHH2Z6lWCI6++WXX9b+3b59ey0ie+XKlfj4+FC/fn3dZ2sk2htUdLY1RmcLIfjrr78AXVyHPh9JH5MNupyjjG0ZMGAA//73v00O5yswUyZdsKaf/E6y807zEfJ1pwZyhGulfL3/WZdxUo5dH06Qq726FerPrg8n5Lj9EydOyGbNmsm4uDgppW6iESkzT/hy584dbflPP/1Uzps3T0oppYuLi4yJiZFSShkfHy+llHLs2LFyxYoVUkopnzx5Ih8/fpzttsuXL5/p8bFjx6SPj4+UUspu3brJkJAQg/esWrVKfvTRR1JKKdu1ayc7dOggpZRy+PDhcuvWrZnWa2ySnUaNGsn79+/LxMREWb9+fXn16tVs25ecnCxbtWol9+3bJ6WUcsKECdLPz09269ZNtm7dWi5btszo+xo0aCDj4uLkpUuXpK2trQwPD5dSSvnqq6/K5cuXSymlrFWrljaZkP6zyzrJzr1792R6erqUUspFixbJDz74QFuuU6dOMikpScbFxcmqVavK5OTkbL/LIUOGyP3790sppbxy5Yps0aKFlFLKV155RR44cEBKKeXDhw9lSkpKpt/jyZMnsmbNmtrjlJQUbYKkuLg42bhxY5meni4vXbokhRDy0KFD2muenp4yISFBSinlzJkz5bRp0zK1SUop//Wvf8mgoCCDz2/FihXS3d3d4Oef//ynwbJr166Vb731lvb4559/NjqR0o0bN6SLi4usU6eOdHBwkKGhodprhw8flk5OTrJ8+fLy119/1Z5PTU2Vjo6OBuvKiZpkxwTJtheRpe5B+jM59XShyu2I3hyKKjo7q/T0dCZOnMjSpUtzXM4S0dl6fn5+Wuge6AL8wsLC2LlzJ4mJiXTq1ImOHTvmGI6norOtIzp71apVDB8+nA8//JBDhw7xxhtvcOLECWxsbOjQoQMnT57k1KlTDBs2jD59+mBvb4+trS2lS5fm4cOHJiXwFlSJiblIsdWdslZML3kjIosDWYTR2Rk9fPiQEydO8OKLL9KwYUMOHz6Mt7e3wc1mS0RnA0ybNo24uDhtchjQxTT37t2b8uXL4+joSNeuXXOdgEVFZ1tHdHZAQACvvfYaoDtwSUpKMohlb9myJeXLl+fEiRPac0+ePMHe3t5gfeZQYooCgEipSiWpioI1snR0dnYqV67MnTt3uHz5MpcvX6Zjx44EBQUZTSI1d3T2Tz/9xLZt21i1ahU2Nn//qfbv35/9+/eTmprK48ePOXLkSL7CDFV0tuWjs+vXr8/OnTsBXWhdUlIS1atX59KlS1pBvnLlCmfOnNFmmrt79y7Vq1c3mGPBXEpUUVCsl6WjswE+/vhj6taty+PHj6lbty5Tp07NU5vNHZ397rvvcuvWLTp16oSHhwfTp08HdEeSvXv3xs3Njfbt2zNq1ChtroW8UNHZlo/O/uqrr1i0aBHu7u4MGTKEpUuXIoTgwIEDuLu74+HhwcCBA/nhhx+0z3r37t2ZOh2YW4mJzn7D7XnS09J5nkd8cVKNis5KRWcr1qwkR2f7+Pjw5Zdf0rx5c5Pfo6KzFUV5ppXU6Ozk5GQGDBiQp4JQUCWm95GiKMVb1qkrS4LSpUsbvfRpTupMQVEURdGooqAoiqJoVFFQFEVRNKooKIqiKBpVFBSrZs7obD1vb+9M/fzXrl2Ls7MzNjY2+Y7NhoJHZ7/11lu4u7vj5uaGr68vCQkJAHz99dc4OTnh5uZGjx49sh28pd9+Xm3cuNFowmdRio2N5ZVXXinqZuRo2bJlNG3alKZNm2Y7yC8yMpJOnTrh6upKv379tHC8khKdrShW79dffzWYT8HFxYVff/2Vrl27FmjdBY3O/uabb4iMjCQqKor69etro4FbtWpFaGgoUVFR+Pr68vHHH+e4/byyxqLw9ddf8/bbb5u8vKW7rt67d49p06Zx5MgRjh49yrRp04iPjzdYbtSoUcycOZPjx48zcOBAZs/WzStWIqKzleLr52kbuBJtMB9SgTRwqs2b/xmY83YtHJ2dkJDA119/zcKFC7U8GsCkQXyWiM6uVKkSoMvwSUxM1PKEunfvri3TsWNHVqxYYbSN+u3v2bOHqVOn4ujoyIkTJ2jTpg0rVqxACMHkyZMJCgrCzs6Onj174uPjQ1BQEHv37sXf35/169eza9cuo5/78OHDqVSpEqGhody8eZNZs2ZpuUCzZs1i+fLl2NjY0KdPH2bOnMmFCxcYM2YMcXFxlCtXjkWLFtGiRQuj319W69evx9/fH9AV2TfeeINHjx4BMH/+fDp37syePXuYNm0atWrV0rKOVqxYwbx580hOTqZDhw788MMP2Nra8t577xESEkJiYiK+vr5MmzYt1+88J9u2bcPLy0sL//Py8mLr1q0MGTIk03JnzpzRDja8vLzo1asXX3zxhVVFZ6uioFiFkydPMmPGDIKDg3F0dDSafeTj46MdLX722WcEBAQwbtw4pk+fzrZt26hTp44209iCBQuYMGECQ4cOJTk52eiR4+eff86HH36Y6Q/SVF27dmX//v14e3tz/fp1YmNjAThw4ACDBw/OtOzMmTOZM2cOv//+O6C7fBQREUF4eDhlypShefPmjBs3zmhK6ogRI9i8eTNOTk589dVXBq8HBAQYjYXIKjw8nJMnT1K7dm26dOlCcHAwTk5ObNiwgdOnTyOE4P79+zg4OODt7c0rr7yi7eD1sRCQ+XMH3WWdAwcOcPr0aby9vfH19WXLli1s3LiRI0eOUK5cOe27HD16NAsWLKBp06YcOXIEPz8/du3aZfT7y+jSpUtUqVJFC9+rUaMGf/75J/b29pw7d44hQ4Zol/mOHj3KiRMnaNSoEadOnWLNmjUEBwdTqlQp/Pz8CAwM5M0332TGjBlUrVqVtLQ0evToQVRUFG5ubpm2O3v27EwZW3pdu3Y1SMK9fv16pu+vbt26XL9+3eC9Li4uBAUF0b9/f9auXZspRO/IkSOMHDmSK1eusHz5cuzs7LT3hISE5PT1FipVFBQDuR3Rm4Olo7MjIiI4f/4833zzjRYjnReWis5esmQJaWlpjBs3jjVr1jBixAjttRUrVhAaGsrevXtz3V779u21SGoPDw8t7M/e3p5Ro0bRt2/fbK/ZZ/e5g+4o1sbGBicnJy2+eseOHYwYMUIrtlWrViUhIYGDBw/y6quvau998kQ3+66x7y+j2NhYMs7NnpKSwtixY4mIiMDW1pazZ89m+j310d47d+4kLCxMyzZKTEzUJiT65ZdfWLhwIampqcTGxhIdHW1QFCZNmsSkSZNy/WzB9OjsxYsXM378eKZPn463tzelS5fWXrOW6GxVFBSrYGp09saNG3F3d2fp0qXs2bMH0J0VHDlyhD/++AMPDw8iIiJ4/fXX6dChA3/88Qe9evXip59+4h//+Ie2rkOHDhEWFkbDhg1JTU3l9u3bvPjii9o6c5M1OvvevXtmic7Wvz5o0CBmz56tFYUdO3YwY8YM9u7dm2ldedmenZ0dR48eZefOnaxevZr58+cbjRjP7nPPul79jtHYd5meno6Dg4OWjJqRse+vWrVq2utZo7O/+eYbatasSWRkJOnp6ZkipbNGZw8bNowvv/wy0/YuXbrEnDlzCAkJoUqVKgwfPtxodHZezhTq1q2b6XOJiYnhxRdfNHhvixYt2L59OwBnz57ljz/+MFgmY3S2Pp1XRWcrJY6lo7Pfe+89bty4weXLlzlw4ADNmjUzuSDomTM6W0qpRT5LKfntt9+0NNDw8HDeeecdgoKCsp2K0xQJCQk8ePCAl19+mblz52o77Kztze5zz07Pnj1ZvHgxjx8/BnTfZaVKlWjUqJE2n7SUUpsDwtj3l1GzZs0ync09ePCAWrVqYWNjw/Lly7O9qdyjRw/WrVvH7du3tXZcuXKFv/76i/Lly1O5cmVu3brFli1bjL5/0qRJRqOzjZ0J9urVi+3btxMfH098fDzbt2/PdEalp29Leno6/v7+vPvuuwAqOrsopAkoXnmwJUtRRGdnZ8OGDdStW5dDhw7Rt29fo3/cYN7obP1RrqurK66ursTGxjJlyiYuPiEAABPjSURBVBTt901ISODVV1/Fw8PDaG6/KR4+fMgrr7yCm5sb3bp109o2ePBgZs+eTatWrbhw4UK2n3t29Dfg27Zti4eHh9alODAwkICAANzd3XF2dmbTpk3a75P1+8uofPnyNG7cWCuSfn5+LFu2jI4dO3L27NlMZwcZOTk54e/vT8+ePXFzc8PLy4vY2Fjc3d1p1aoVzs7OjBw5UpuZrSCqVq3K559/Trt27WjXrh1TpkzRLoGOGjVKu+exatUqmjVrRosWLahdu7Z25qeiswsgv9HZgzwaYZsi6VCxLBMOnzJDy4o3FZ2tWLMNGzYQFham9UAqSSwdnV2i7ikI4LlKNYu6GYqi5NHAgQO1S4slSVFEZ5eYy0eKohRvo0aNKuomWJyKzlYURVGKlCoKiqIoikYVBUVRFEWjioKiKIqiUUVBsWrmjM7+9NNPqVevnkFK6r59+2jdujV2dnasW7cu3+t/+eWXuX//Pvfv3+eHH37Qnt+zZ49JMdBTp06lTp06WqTy5s2bgexjlgG2bt1K8+bNadKkCTNnzjS63ilTprBjx448/z75TXs1N19fXy5evFjUzcjWpUuX6NChA02bNmXQoEEkJycbLJOSkqKNS2nZsmWmUdjffvstLi4uODs7M3fuXO35jz76yOgI9IJSRUEpsfr168fRo0cNnq9fvz5Lly7l9ddfL9D6N2/ejIODg0FRyIuJEydqI2n1A5iyi1lOS0tjzJgxbNmyhejoaFatWmU0Anv69Om89NJLeW6LNRaFkydPkpaWxvPPP2/yeywdq/3JJ58wceJEzp07R5UqVQgICDBYZu3atTx58oTjx48TFhbGjz/+yOXLlzlx4gSLFi3i6NGjREZG8vvvv3Pu3DkAxo0bl23hL4gSNU5BMc3Psz7hypnjhbrOBs1defPj/+a8XQtHZ3fs2NFoO/TxAjY22R8zzZo1C3t7e8aPH8/EiROJjIxk165d7Ny5kyVLlrBixQoaNmxIaGgokydP5sKFC3h4eODl5UXfvn1JSEjA19fXIMraFNnFLB89epQmTZpoO8jBgwezadMmnJycMr1/+PDhWgpqw4YNGTZsGL/99hspKSmsXbuWFi1asHfvXiZMmADogt327dtnEAE+cODAbCOss4vqDgkJYcKECTx69IgyZcqwc+dOypUrx+TJk9mzZw9PnjxhzJgxvPPOO8TGxjJo0CD++usvUlNT+d///mcwYjwwMJD+/ftrj7OLxG7YsCEjR45k+/btjB07lnbt2hmN8f7tt9/w9/cnOTmZatWqERgYSM2a+R/bJKVk165dWjEdNmwYU6dO5b333su0nBCCR48ekZqaSmJiIqVLl6ZSpUqEhITQsWNH7Tvv1q0bGzZs4OOPP6ZBgwbcvXuXmzdv8txzz+W7jVmpMwXFKuijs3ft2kVkZCTffvutwTI+Pj6EhIQQGRlJy5YttSMuffRyZGQkQUFBwN/R2REREYSGhmoJoYVFH50NEBoaSkJCAikpKRw4cMBgxzVz5kwaN25MRESENqlKeHi4lrJ68eJFgoODjW5n/vz5uLm5MXLkyEyTthw5cgRnZ2dcXV1ZsGABdnZ2Jsc3Z+Xo6MixY8d47733tEt1c+bM4fvvvyciIoL9+/dTtmxZZs6ciaenJxEREUycOFGLsD527Bhr1qxh/Pjx2jqN/X7JyckMGjSIb7/9lsjISHbs2EHZsmUJCAigcuXKhISEEBISwqJFi7h06RIrV66kV69eREREEBkZiYeHh0Hbg4ODadOmjfZ4xowZ2gREe/fuzZR5ZW9vr0Wbjx49mu+++46wsDDmzJmDn58fAC+88AKHDx8mPDycwYMHG53n4syZM9rlu6w/WaO/7969i4ODgxaDnd134uvrS/ny5alVqxb169fno48+omrVqri4uLBv3z7u3r3L48eP2bx5c6ZsqNatW2f7/05+mfVMQQjRG/gWsAV+klLOzPJ6GeBnoA1wFxgkpbxszjYpucvtiN4cLB2dXVBt2rQhLCyMhw8fUqZMGVq3bk1oaCj79+83KTrbWJT1Cy+8kGmZ9957j88//xwhhDb3w+LFiwHjMcumxjdnpf/M2rRpw6+//groPtMPPviAoUOH4uPjY7So5hZhnfX3q1y5MrVq1dKirPWTCG3fvp2oqCjt/s2DBw84d+4c7dq1Y+TIkaSkpDBgwACjRSFrrHZOkdiDBg0CyDHGOyYmhkGDBhEbG0tycrIWw51R8+bNjaa9GmPqd3L06FFsbW25ceMG/7+98w+uqjzz+OdrgCYurQosTCGN2MGWhEAiBQdlQpflR0BGbSsC/qDiBLSusNOqFXfcWbH6R22tdRVqYKkKnVJYFHYzXTtuB0LpMMSSoRWEUvklGGFWNoOMQliIPvvHOTm9JDfkJLn3hps8n5k7c8973nPe5znn3vOc933P+T4nT56krKyMyZMnU1hYyOLFi5kyZQp9+/alpKQkCjAQ5JY4diy1CbHS1lOQlAMsA6YDRcAdkoqaVasATprZMOCnQOavRs4lQVzp7KVLl7J7926eeOKJSO64srKSp59+mvfff5/S0lLq6+u58847qaqqIi8vj/Ly8pRPyPXu3ZuhQ4fyyiuvcOONN1JWVkZ1dTUHDx6MpSEVRzp70KBB5OTkcNlll7FgwYKk8x+JMsv5+fkX3EXW1dUxePDg2LYk2vHYY4+xcuVKGhoaGDduHPv27WuxXaKEdW1t7QUTqMn8a+0cmxkvvvhiNHdy+PBhpk6dyoQJE9i6dStDhgxh7ty5rF69usW2ibLaTZLYmzZtYteuXcyYMeMCSewm4bxEGe+mz5//HOihLVq0iIULF7J7926WL1+eVFK7PT2FAQMG8NFHH0XHtbVzsmbNGqZNm0bv3r0ZOHAg48ePj0T0Kioq2LlzJ1u3bqVfv34X3OCcPXuWvLy8FvvrDOkcProeOGBmh8zsHLAWuLVZnVuBpgzXrwGTFHdg1elWZFo6OxVMmDCBZ599NpLOrqyspLS0tMWFryPS2UCUzQ0CQbji4mKgdZnlsWPHsn//fg4fPsy5c+dYu3ZthxVUDx48yMiRI1m8eDFjxoxh3759LfyIK2HdxPDhwzl27FiURezjjz+msbGR8vJyXnrpJc6fPw8EeQZOnz7NkSNHGDhwIAsWLIgujM0pLCyM1FPjSmJfTMb71KlTDBkyBIBVq1Yl3b6pp5Dsc+WVV15QVxITJ06MekGrVq26YA6kiYKCAjZv3oyZcfr0aWpqaiJF2ia57aNHj7Jhw4YLUny+++670e8iVaQzKAwBEoXR68KypHXMrBE4BfQnDeQ0GsoyRdieRFdIZz/66KPk5+dz5swZ8vPzWbJkCQA7duwgPz+f9evXc//99zNixIikNpeVlXH8+HFuuOEGBg0aRG5ublLp7P79+zN+/HiKi4tjZ/Jqsm/kyJGMGjWK6urqSNq6NZnlXr16sXTpUsrLyyksLGTWrFmt2t4Wzz//PMXFxZSUlJCXl8f06dNbSIDHlbBuok+fPqxbt45FixZRUlLClClTOHv2LPPnz6eoqIjRo0dTXFwcPU21ZcsWSktLue6663j99dejie9EZsyYEeXBaI8kdmsy3kuWLOH222+nrKwsGsrsLM888wzPPfccw4YNo76+noqKCgCqqqoiOfQHH3yQTz75hOLiYsaOHcu9994bDXvddtttFBUVcfPNN7Ns2TKuuuoqIBi+O3DgQJSIJ1WkTTpb0u1AuZnND5fnAteb2aKEOnvCOnXh8sGwTn2zfd0H3AdQUFDwtSNHjrTbnrnFw8lp/IyHvv8Ioyru66hb3RaXznaykYaGBiZOnMi2bdvIycnpanMyysaNG9m5cydPPfVUi3WXqnR2HZCYdDYfaD4j0lSnTlIv4AqgxbiBma0AVkCQT6EjxvzinZZjoo7jZDd5eXk8+eSTfPDBBxQUFHS1ORmlsbGRhx9+OOX7TWdQ2AFcK+ka4ANgDtD8baAq4B5gOzAT2GzZlvXHcZwupbXMeN2dxKenUknagoKZNUpaCLxJ8Ejqy2a2R9IPgFozqwJ+DvxC0gGCHsKcdNnjtE2cJ4Acx7m06ex9dVrfUzCzN4A3mpX9S8L3s0B6wp3TLnJzc6mvr6d///4eGBwnSzEz6uvryc3N7fA+XObCAYI3Levq6jhx4kRXm+I4TifIzc3t1Bv8HhQcIHgZK9nbm47j9Cxc+8hxHMeJ8KDgOI7jRHhQcBzHcSLS9kZzupB0Amj/K80BA4D/TaE52YD73DNwn3sGnfH5ajP727YqZV1Q6AySauO85t2dcJ97Bu5zzyATPvvwkeM4jhPhQcFxHMeJ6GlBYUVXG9AFuM89A/e5Z5B2n3vUnILjOI5zcXpaT8FxHMe5CN0yKEiaJukvkg5IeizJ+s9JWheuf0vS0MxbmVpi+PyQpL2SdknaJOnqrrAzlbTlc0K9mZJMUtY/qRLHZ0mzwnO9R9KaTNuYamL8tgskVUv6Y/j7vqkr7EwVkl6W9KGkd1pZL0kvhMdjl6TRKTXAzLrVh0Cm+yDwZaAP8DZQ1KzOPwCV4fc5wLqutjsDPk8ELg+/P9ATfA7rfR7YCtQAY7ra7gyc52uBPwJXhcsDu9ruDPi8Angg/F4EvNfVdnfS5wnAaOCdVtbfBPwGEDAOeCuV7XfHnsL1wAEzO2Rm54C1QPNM2bcCTVm5XwMmKbv1otv02cyqzexMuFhDkAkvm4lzngGeAn4EnM2kcWkijs8LgGVmdhLAzD7MsI2pJo7PBnwh/H4FLTM8ZhVmtpUkGSgTuBVYbQE1wJWSvpiq9rtjUBgCvJ+wXBeWJa1jZo3AKaB/RqxLD3F8TqSC4E4jm2nTZ0nXAV8ys19n0rA0Euc8fwX4iqRtkmokTcuYdekhjs9LgLsl1RHkb1lE96a9//d20R2ls5Pd8Td/xCpOnWwitj+S7gbGAF9Pq0Xp56I+S7oM+CkwL1MGZYA457kXwRDS3xH0Bn8vqdjMPkqzbekijs93AK+a2U8k3UCQzbHYzD5Lv3ldQlqvX92xp1AHfClhOZ+W3cmojqReBF3Oi3XXLnXi+IykycDjwC1m9n8Zsi1dtOXz54FiYIuk9wjGXquyfLI57m/7P83svJkdBv5CECSylTg+VwD/DmBm24FcAo2g7kqs/3tH6Y5BYQdwraRrJPUhmEiualanCrgn/D4T2GzhDE6W0qbP4VDKcoKAkO3jzNCGz2Z2yswGmNlQMxtKMI9yi5nVdo25KSHOb/s/CB4qQNIAguGkQxm1MrXE8fkoMAlAUiFBUOjOKQSrgG+HTyGNA06Z2fFU7bzbDR+ZWaOkhcCbBE8uvGxmeyT9AKg1syrg5wRdzAMEPYQ5XWdx54np84+BvsD6cE79qJnd0mVGd5KYPncrYvr8JjBV0l7gU+D7ZlbfdVZ3jpg+Pwz8m6TvEQyjzMvmmzxJvyIY/hsQzpM8AfQGMLNKgnmTm4ADwBng3pS2n8XHznEcx0kx3XH4yHEcx+kgHhQcx3GcCA8KjuM4ToQHBcdxHCfCg4LjOI4T4UHBueSQ9KmkPyV8hl6k7tDW1CTb2eaWUInz7VAi4qsd2Md3JH07/D5P0uCEdSslFaXYzh2SSmNs811Jl3e2badn4EHBuRRpMLPShM97GWr3LjMrIRBL/HF7NzazSjNbHS7OAwYnrJtvZntTYuVf7fwZ8ez8LuBBwYmFBwUnKwh7BL+XtDP83JikzghJfwh7F7skXRuW351QvlxSThvNbQWGhdtOCnX6d4c6958Ly3+ov+aneDYsWyLpEUkzCfSlfhm2mRfe4Y+R9ICkHyXYPE/Six20czsJQmiSXpJUqyCPwpNh2T8SBKdqSdVh2VRJ28PjuF5S3zbacXoQHhScS5G8hKGjjWHZh8AUMxsNzAZeSLLdd4B/NbNSgotyXSh7MBsYH5Z/CtzVRvs3A7sl5QKvArPNbCSBAsADkvoB3wRGmNko4OnEjc3sNaCW4I6+1MwaEla/BnwrYXk2sK6Ddk4jkLVo4nEzGwOMAr4uaZSZvUCgizPRzCaG0hf/DEwOj2Ut8FAb7Tg9iG4nc+F0CxrCC2MivYGl4Rj6pwSaPs3ZDjwuKR/YYGb7JU0CvgbsCOU98ggCTDJ+KakBeI9AfvmrwGEzezdcvwp4EFhKkJ9hpaT/AmJLc5vZCUmHQs2a/WEb28L9tsfOvyGQfUjMujVL0n0E/+svEiSc2dVs23Fh+bawnT4Ex81xAA8KTvbwPeB/gBKCHm6LpDlmtkbSW8AM4E1J8wlkhleZ2T/FaOOuRME8SUlzbIR6PNcTiLDNARYCf98OX9YBs4B9wEYzMwVX6Nh2EmQg+yGwDPiWpGuAR4CxZnZS0qsEwnDNEfBbM7ujHfY6PQgfPnKyhSuA46FG/lyCu+QLkPRl4FA4ZFJFMIyyCZgpaWBYp5/i56feBwyVNCxcngv8LhyDv8LM3iCYxE32BNDHBPLdydgAfIMgD8C6sKxddprZeYJhoHHh0NMXgNPAKUmDgOmt2FIDjG/ySdLlkpL1upweigcFJ1v4GXCPpBqCoaPTSerMBt6R9CdgOEHKwr0EF8//lrQL+C3B0EqbmNlZAgXK9ZJ2A58BlQQX2F+H+/sdQS+mOa8ClU0Tzc32exLYC1xtZn8Iy9ptZzhX8RPgETN7myA38x7gZYIhqSZWAL+RVG1mJwiejPpV2E4NwbFyHMBVUh3HcZwEvKfgOI7jRHhQcBzHcSI8KDiO4zgRHhQcx3GcCA8KjuM4ToQHBcdxHCfCg4LjOI4T4UHBcRzHifh/NZqFRlVHgJIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "sns.set_palette(\"dark\")\n",
    "# code manipulated from http://scikit-learn.org/stable/auto_examples/plot_roc.html\n",
    "from sklearn.preprocessing import label_binarize\n",
    "\n",
    "# Compute ROC curve for a subset of interesting classes\n",
    "fpr = dict()\n",
    "tpr = dict()\n",
    "roc_auc = dict()\n",
    "for i in np.unique(y):\n",
    "    fpr[i], tpr[i], _ = mt.roc_curve(y, yhat_score[:, i], pos_label=i)\n",
    "    roc_auc[i] = mt.auc(fpr[i], tpr[i])\n",
    "\n",
    "for i in np.random.permutation(43)[0:6]:\n",
    "    plt.plot(fpr[i], tpr[i], label='class {0} with {1} instances (area = {2:0.2f})'\n",
    "                                   ''.format(i, sum(y==i), roc_auc[i]))\n",
    "\n",
    "plt.legend(loc=\"lower right\")  \n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2078200f978>]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsYAAAD8CAYAAAB0FmJXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl0XOd55/nvU4WNAAiCJMB9lURKpGRtpmQtdluO5W5ZdqTTJ44jJZm2PZ5oOsdLJs6kR5lO2z3yZDK2JyczsSUncuJI9ulEUbtzYnaHbuUkdrxqoyRrIbVRlLhTBEEQJLHW8s4fBYAgCJIlsYACWd/POTiqe+9bVc89r0D++N73vjdSSkiSJEm1LlPtAiRJkqSZwGAsSZIkYTCWJEmSAIOxJEmSBBiMJUmSJMBgLEmSJAEGY0mSJAkwGEuSJEmAwViSJEkCoK5aX9zR0ZFWrVpVra+XJElSjXjqqacOppQ6z9SuasF41apVbN68uVpfL0mSpBoRETvKaedUCkmSJAmDsSRJkgQYjCVJkiTAYCxJkiQBBmNJkiQJKCMYR8Q3I+JARLxwiuMREX8SEdsi4rmIuLryZUqSJElTq5wR4weAW05z/IPAmpGfu4Cvn31ZkiRJ0vQ64zrGKaUfRcSq0zS5HfhWSikBj0VEe0QsTintq1CNkiRJmsFyuQJ9/cP09A5y8FA/+7uO8ezW/QwNF4gotYkI/o/feV91Cz2DSjzgYymwa9z27pF9JwXjiLiL0qgyK1asqMBXS5IkqZJSShQKRXK5IvlCkVyuwFPP7+Op5/fy2NO72bP/KAe6+8jlCuQLRfoHcvT15075eaPBOJOpjWAck+xLkzVMKd0P3A+wYcOGSdtIkiSpMgYGcrz02kH6+od5Y/dhenoHeWV7N02NdQznCuzYfZi+gRzPbt1Pa0sDO/f0UiicPqLNndPE+rWdrLuok/q6DHV1GfL5InPnzGL5kjYigjWr5zFndhOXru2kqal+ms727FUiGO8Glo/bXgbsrcDnSpIknVNSSnR199E/kGNouMDOPb1kMkFKiWJx5Gf865Ht7Tt6mNVUP+mx8e/d9sYhmhrryOWLvPp6NymVRmTzhSL5fJEtr3Qxq6mOlKCnd+C0IXdOWyPZTIbDRwa55oolDOcK/MINq+npHeTydQtpbMhSX5elri5DSokb3rmcd1yykFmzzp2g+1ZVIhhvBD4dEQ8B7wJ6nV8sSZKmy+il/0IhUSiW/tvTO0Bff47hXIFjfcPsffMoKSW2vtpFU2Md+XyRQjGRzxfZtbd3bC7s9p09DA0XaKjPHg+kKZESk4bVl1/rHguio+F0Oixe0Ep9fZau7j7e+Y4lNDbU0TIrw3uvW0lXdz8bLl9CQ32WQrFI57wW3nn5YtpaG5nXPouVy9ppaW6YljrPNWcMxhHx18BNQEdE7Aa+ANQDpJT+FNgE3ApsA/qBT0xVsZIk6dyWyxX4+Zb9vPp691iQzBeKvLTtIM2z6sfCaqFQ2v/0C/vomNvMwZ5+tu/oYXZrI/l8kcNHBjl0eOCs68lkgmKxNKq6ekU7zU31HOju4/J1C2moz5LJBBFBJoJMJka2S+/LRHDVpYvp6R3g8nULx0ZXjx4b4h2XlEZch3MFVi+fSzZ7/P3jP2v0B6BzXgsNDdmx42PfM+E92WyGiMlmsupslbMqxZ1nOJ6AT1WsIkmSNKnh4Ty9R4coFo+PWKbE2OX38ZfcE3Do8ADDwwVSOt5u/PvGXsPY+0dHRnft66W+LjsWXPP5Iltf7aKttZFCsUixmMZGaIvFRC5f5Odb9rOos3Vs1LZQKI3KvrHr8NhNWsf6hk97jg0NWbIj4S+bDQqFRARccmEH8+c2s2hBKws7Wqivy9I/kGNhZwvz5zaTiSCbHXlfJhgYzLNscRtz2hppqM8SEaxe3k5jQx2LF7ZSX5cdCas+60zHVWIqhSRJ57w0YS7naEA8dHiAgcHcWDgcHMqzZ/9RstkYC43DwwVeeu3gSGgcGfEsFjlwsI/unoGROaEFunsG2Lm3l7aRUc98ocjLrx0km81QX5cZC6jHazhxPuqZboqaLu1zmsiOG73Mjox45gtFunv6WbmsfSzcNjQEl128gAPdfVx75VIyESxb3Mbl6xZy6dpO6uoy1GVLN3B1zGs2qKqqDMaSpGmXyxUYzhUYGgmZw7kC/QM53th9mLpsZmyJqO07e2hoyJ54M9LIHM9CYSQsFksjlQs7jo9UjobVn2/dz8KOlrGwms8X2b6zZyTQxQnzRqdSx7xm2lobqavL0Nc/THNTPQs6WqjLZrhgxVz2dx3j6ssWj10yP+ES+rjL6KOju8sXtx2/xD96eR/GXeo/fnk+E8GCjpbjl+ajdHz0O0bbBhx/HaXXc2Y3Mru1cSy4ZrNB8yznpur8ZTCWpBowOJijfyBHoXj8JqXiyA1LBw/1k8sXTwqeE7df23Fo7Kal0dHT5186QHtb09iyTwNDeRobsieE1tHPyBeKbH5uL0NDhbd1DpOFxWy2dLd8vlAkE8HqFXOpG7kEn81kWLygtRQ+V47uz3D9O5dRKCRWL28/IVieKpT29A6y7qIOGuqzY6ObhWJi1bL2UljMBHV1GRob6uic30w2czxEOvopnVsMxpI0xYrFUvjc+moXw8OlBfEnzr+sr8+csG80vL702kFmtzYwNFTgmS37mDtn1gltSuEz8eKrXcxqqh+bkzk6closJvoHTr3wfqXMa59FNht0dfePLfOUzWZKI7MjAbOhPsv1Vy+nvi5D5/wWrrx0EQ31WQaH8qxf0zl2o9OKpXPGRijr6zLMnTOLpqY6bzaSNOUMxpJqQkrHL6XnR0ZJd+7pHbtk3314gCNHh8hkYix4FovHg+pL2w4yZ3bjSYG0UCjSe3SInXt6aZvdSC5XYGAwz2s7DnGsf5hcrliRy/TLl7TRUJ9lx55erly/iIaG0ojo6J3uyxa18ebB45fjR+d9jobTvv5hFnW2jswNPf6+bCZDoVhkzuwm2tuaTnvXfEqJJQtnnzByWl+fddknSecNg7GkGWdoKM/hI4Ps2F26k31ouMAr27vZvrOHN7uOcaC7r3S3/Mjl/OFcgadf2MeiztbjNymNGzHd++bRitU2uiD++GAJ0DdQCp4LO1qZ3drADe9czkWr5tE8q576+ixDQ3kuXDWPlUvnMLu18YS77rOZDO1tTbQ015+w7/jyTl6Ol6TpYDCWNOUGB3N09wyQyxcYGi7w4qtdPPr0bo4cHWLLKweor88yPFyaKlAoJAaH8mf8zIsvnM/slsaxkctL13ZSLCYuWDH3hEv4o68PHR7ginULRy7PZ2loyLJqWTvtbU1jl+ybGutGpgQcD6ejN2llM5nz+mlPkiSDsaS3KaVE75FBhnMFcrki3Yf7OXJ0iO6eAfa+eZSuQ31s39HDQ//1hdPebLWws4Wjx4Z511VLec+1KznWN8y6NR1csGIuzbPqueTCjpF1TUt37y9a0Eo26wiqJKnyDMZSDRsczNF1qJ9crkAuX2TP/iPsO3CMN3Ydpn8gRwRjN4rl80V27DlMd88AQ8N5Hn9mzxk/v6W5ngtXziObCf71LetYvbyd+pHHrG64fAkXrZpHXV12Gs5UkqQzMxhL56F8vsDBQ/3sO3CMw0cGeWV7N89u3U9DQ5ahoQKvvN7NCy8f4MDBvtN+TmZkGarRFQIa6rOsXDqH+XObue0DF5MvFLn1fWuory9NT6ivz7ByaTutLQ2sWDqHuXOanB8rSTpnGIylKhoezjM4lCc3slrCkWNDJ0xP2LHnMAcO9vHEs3uYM7tpbEWF3iODbNtxiPa2JnK5Ii+9dnBsMf/RVRFOpXN+Mw31WRZ1tnL91cu4dO0C1l4wn/q60goDC+a3cOGqucxvb3ZOrSSpphiMpQpJqbRe7PYdPeza18vrOw8zOJTnyWf3sGvfEXbu6aVQOP5ghJ7ewbf8HaPru2azwcBgnob6LEsXtbF00Wy6DvWz4fIlNNRnxx4te8mFHTTPqmfF0jmsXDqHRQtmV/7EJUk6TxiMpTIc7h3gQHcfb3b18fOt+0thd+8RmmfVM5wr8NRze08bdJctbuO6q5cxd2QN2dHpCb1HB7lw5TxmtzZQX1daG7ZQKLJ6+dyx6QnLFrexctkcpyRIkjTFDMbSJFJKPPb0br789Z/yd4+8dMp2K5bOYdniNq5Yv4jhXIHLL1nIquXtXLFuIevXdtLe1kRTYx0NDf6qSZI00/m3tWpWSonXd/bwo8d3MDCY5/mX3qShIcuTz+7lZ5t3jbVraa7nF2++mF+4cTWNDVnWrJ7PpWs7aZvdVMXqJUlSpRmMdd4pFovsffMor75+iL7+YXbtPVL6774j/PCxN2ieVc/zLx3gWN/wpO9f0NHCgo4WrrtqGf/ht/4FG65YOs1nIEmSqsFgrHNCsVjk6ef30Xt0iKHhPD99cheHj5Tm9A7nCmx+bi+vbO9mOFcgny+e9rOWLprNTdevon8gx5XrF3HtlUu58ZrlzG5ppG12IxExHackSZJmGIOxZpS9+4/wz4++weEjg/zwsR0c6C7d7Hb02BCFQjqpfWtLA7NbGqiry9De1sS6NR2859qVDAzmuOrSxXTOb6ZzXgtz25tYML+F+vqswVeSJE3KYKwpl88X6B/Isb/rGEePDdN7dJBtbxziBz97g1df76Z5Vj2HDg+w5ZWuSd9/yUUdbLh8CddesZR3X7uCee2zaGzIsu6iThob/V9YkiRVhqlCZ61YLPLci2/y8H/dwuBQnh8/sZM9+49QX5/lwME+BodO/bCJiFLwXb54DksWzubiCzt43w2ruPqyxXTMa6a1pXEaz0SSJNUyg7HK1tc/zBu7DvPK9m5++PgbHOsb5m+/9+JJ6/d2zGumqbGOee2zuOWmixgYzLFm1XyaZ9WzZOFs5s+dRVNjHZdevID5c5urdDaSJEknMhjrJPl8ga2vdvH08/t49OndfP+nr7PtjUOTtr1w5Vzmtc/if/ilK3jvdSu54Z3LXbNXkiSdk0wwOsGffvtJfvN///uT9rfNbuRf/6tLuPLSRaxePpdLLupg7QXzvZFNkiSdNwzGNS6lxFf+9Kf85cM/51jfMLv3HQHgc79xPe+9biXXXrmURQtmV7lKSZKkqWcwrjGFQpHnX3qTb/7NM/zFQ8/QP5AbO3bp2k7+zS+9h//pzqtZvWJuFauUJEmafgbjGpDPF/iDr/6YL3/9pycEYYBbf2EN1121jM/ddT0tzQ1VqlCSJKn6DMbnoZQSO3Yf5s/+01P86PEd/GzzrrFjN2xYzq3vW8MNG5Zz/dXLaGqqr2KlkiRJM4fB+DwwNJTnoY0v8NyLb/JX332e/QeOnXD8hg3LueaKJfz+Z/8FHfNaqlSlJEnSzGYwPof19Q9z9x/+I1974Imxfe1zmljQ0cKv/OKlvO/61Xzo/WtcPk2SJKkMJqZz1D/9ZDs33/mtse27P/Vu/rffvJH2ObOqWJUkSdK5q6xgHBG3AP8fkAX+PKX0f084vgJ4EGgfaXN3SmlThWsVpUD8qd//e15+rRuAL/z2e/l3v3kjzbO8cU6SJOlsnDEYR0QWuBf4ALAbeDIiNqaUto5r9vvAwymlr0fEemATsGoK6q1JP3liB5+75xGefHbv2L5LLupg04O/5rJqkiRJFVLOiPG1wLaU0naAiHgIuB0YH4wT0Dbyeg6wF5213ft6+cTvfJd//PH2sX0f/fClfOG338v6tQuqWJkkSdL5p5xgvBTYNW57N/CuCW3+I/APEfEZoAW4uSLV1aCUEl/9y8f5X//PfyCXKwKwbHEb3/+bj7Fm9fwqVydJknT+KicYxyT70oTtO4EHUkp/FBHXA9+OiMtSSsUTPijiLuAugBUrVrydes9rm5/dwwd+7dsc7h0EYM3qedz3Bx/i5vdcWOXKJEmSzn/lBOPdwPJx28s4earEJ4FbAFJKj0ZEE9ABHBjfKKV0P3A/wIYNGyaG65rU1z/MQxtf4P/66o/ZvrMHgA/fvJZv/j+30znfNYclSZKmSznB+ElgTUSsBvYAdwC/OqHNTuD9wAMRsQ5oAroqWej55mebd3LP//tDHvnha2P7Ll3byZ9/5Tauu3r5ad4pSZKkqXDGYJxSykfEp4FHKC3F9s2U0paIuAfYnFLaCPwO8I2I+G1K0yw+nlJyRHgSz7ywj1/9zH/hpW0HAbjmiiV86mPX8qH3r/GpdJIkSVVU1jrGI2sSb5qw7/PjXm8FbqxsaeefL933E+7+w38E4N3XrOC+P/gQ71i3sMpVSZIkCXzy3ZQrFovc++CTfO2BJ3hle+mhHN/9izu47V9eUuXKJEmSNJ7BeIoUi0U+8x++x33fenJs303Xr+Jv7/8V5rb72GZJkqSZxmA8BY4cHeSSm77GvgPHaGzM8uH3r+UbX7rNQCxJkjSDGYwrKJ8v8NfffYHP/9EP2HfgGB/50HoeuvcjZLOZapcmSZKkMzAYV8h9Dz7B733pnzhydAiAr37xg3z64xMfEChJkqSZymBcAX/8jUf53D2PAPC7//YGPvOJd7F8yZwqVyVJkqS3wmB8ln62eedYKH71R5/hotXzq1yRJEmS3g4nv56Fnzyxgw/86rdpaMiy/ae/ZSiWJEk6hxmM36Yv3fcT3vNLf0n/QI5ND/4aq1fMrXZJkiRJOgtOpXgb/uSbj409we6Z//4/c+Wli6tckSRJks6Wwfgtuu/BJ/itL/x32mY3su3Hn6Vzfku1S5IkSVIFGIzfgpe2dfHZL3yPRQtaefVHn6G1pbHaJUmSJKlCnGNcpl17e3nPL/0lhULi7x/4VUOxJEnSecZgXIZisci7fvEbHDzUz9e+eCtXv2NJtUuSJElShRmMy3D3H/4j+w4c45N3XMWnPn5ttcuRJEnSFDAYn8GxviH+6P5HAfjaF2+tcjWSJEmaKgbjM7jjU9+hWEz8yT0fpKmpvtrlSJIkaYoYjE+jq7uPTd9/lXddtZTPfOJd1S5HkiRJU8hgfBqf/fz3SAm+eo9TKCRJks53BuNT+OGjb/DQxhf45Q+v55orl1a7HEmSJE0xg/EpfPtvnwW84U6SJKlWGIwncfBQHw9+51k+fPNaFnS0VrscSZIkTQOD8SS+8/dbyeeL/Lt/e2O1S5EkSdI0MRhPkFLivm89ydoL5vPua1dUuxxJkiRNE4PxBI89vZvnXzrApz52DRFR7XIkSZI0TQzGE/z1d58nkwk++uFLq12KJEmSppHBeIJ/+NFrXL5uIYsWzK52KZIkSZpGBuNxCoUi3T0DLOp0JQpJkqRaYzAe5wc/e52Dh/qdRiFJklSDDMbjfOu/PEt9fYZf/vD6apciSZKkaWYwHmfT91/lmiuW0trSWO1SJEmSNM3KCsYRcUtEvBwR2yLi7lO0+WhEbI2ILRHxV5Utc+pte72b7p4B3n2NaxdLkiTVorozNYiILHAv8AFgN/BkRGxMKW0d12YN8HvAjSmlnohYMFUFT5VNP3gVgP/xV66qciWSJEmqhnJGjK8FtqWUtqeUhoGHgNsntPkN4N6UUg9ASulAZcucet/4q6dZvKCViy/sqHYpkiRJqoJygvFSYNe47d0j+8ZbC6yNiJ9GxGMRcctkHxQRd0XE5ojY3NXV9fYqngKDgzleePkAv3Dj6mqXIkmSpCopJxhP9lzkNGG7DlgD3ATcCfx5RLSf9KaU7k8pbUgpbejs7HyrtU6ZR5/eDcCt71tT5UokSZJULeUE493A8nHby4C9k7T5bkopl1J6HXiZUlA+J3ztgScAeP+7L6hyJZIkSaqWcoLxk8CaiFgdEQ3AHcDGCW3+DngfQER0UJpasb2ShU6V4eE8f/u9F7nmiiUs9Il3kiRJNeuMwTillAc+DTwCvAg8nFLaEhH3RMRtI80eAbojYivwA+B3U0rdU1V0Jf18y34APvIhH+ohSZJUy864XBtASmkTsGnCvs+Pe52Az438nFO2vXEIgFtuuqjKlUiSJKmaav7Jdz97qrTgxgUr5la5EkmSJFVTzQfj0RHjluaGKlciSZKkaqr5YPzU8/u46rJFREy2Kp0kSZJqRU0H45QSBw/1s3LpSUsuS5IkqcbUdDB+9fXSwhk3u36xJElSzavpYLxn/1EALr5wfpUrkSRJUrXVeDA+AsCyxW1VrkSSJEnVVtPB+MdP7ARg+ZI5Va5EkiRJ1VbTwfjZraWn3rlUmyRJkmo6GANctGpetUuQJEnSDFDTwfjxZ/Zw3dXLql2GJEmSZoCaDca9RwYB6JjbXOVKJEmSNBPUbDA+0N0HwNXvWFzlSiRJkjQT1Gww7hoJxh3zHDGWJElSDQfjn28prUixZOHsKlciSZKkmaBmg/Gx/mEAVi51DWNJkiTVcDDes/8orS0NzGlrqnYpkiRJmgFqNhjvO3CUxQtaiYhqlyJJkqQZoGaD8ZPP7mW+S7VJkiRpRM0G46bGOjIZR4slSZJUUrPB+OChfi67eEG1y5AkSdIMUZPBeHAwx8FD/Sxf3FbtUiRJkjRD1GQw3nfgGACLFrRWuRJJkiTNFDUZjA8e6gcgm6nJ05ckSdIkajIZHu0bAmDV8vYqVyJJkqSZoiaD8ZtdfQDU19Xk6UuSJGkSNZkMU0oAdMxzHWNJkiSV1GQw7hvIAdDS3FDlSiRJkjRT1GQw7u4p3Xw3r31WlSuRJEnSTFGTwXjLK10ANM+qr3IlkiRJminKCsYRcUtEvBwR2yLi7tO0+0hEpIjYULkSK6+luRSII3wktCRJkkrOGIwjIgvcC3wQWA/cGRHrJ2k3G/gs8Hili6y0l7Yd5IIVc6tdhiRJkmaQckaMrwW2pZS2p5SGgYeA2ydp90Xgy8BgBeubEm2tjfQenfFlSpIkaRqVE4yXArvGbe8e2TcmIq4ClqeU/lsFa5syu/YdYd1FndUuQ5IkSTNIXRltJpuIm8YORmSAPwY+fsYPirgLuAtgxYoV5VU4BXqPDpLNOL9YkiRJx5UzYrwbWD5uexmwd9z2bOAy4J8j4g3gOmDjZDfgpZTuTyltSClt6Oys3ohtJoIVS+dU7fslSZI085QTjJ8E1kTE6ohoAO4ANo4eTCn1ppQ6UkqrUkqrgMeA21JKm6ek4gp4bUcP8+f61DtJkiQdd8ZgnFLKA58GHgFeBB5OKW2JiHsi4rapLnAqNM+q5/ARb76TJEnSceXMMSaltAnYNGHf50/R9qazL2tq5fIF1qyaV+0yJEmSNIPU3JPvCoUiuVyRpqay/k0gSZKkGlFzwXhgMAfAkaNDVa5EkiRJM0nNBePBoTwAq5a3V7kSSZIkzSQ1F4yHhwsANNRnq1yJJEmSZpKaC8ZDI8G4scE5xpIkSTqu5oLx6DJt/QO5KlciSZKkmaTmgnGhUARg2eK2KlciSZKkmaTmgvHxqRTOMZYkSdJxNReMR1elcI6xJEmSxqu5YHysbxiAlub6KlciSZKkmaTmgnF+ZI5xg1MpJEmSNE7tBeN8KRjX1xmMJUmSdFztBeOREeO6upo7dUmSJJ1GzaXD0RHjumzNnbokSZJOo+bSoSPGkiRJmkzNpcN9bx4FIBNR5UokSZI0k9RcMG6f0wS4KoUkSZJOVHPB2DnGkiRJmkzNpcNCIQGQzTqVQpIkScfVXjAulkaMs44YS5IkaZyaS4dOpZAkSdJkai4dOpVCkiRJk6m9YOxUCkmSJE2i5tJh/0AOgHAdY0mSJI1Tc8G4p3ew2iVIkiRpBqq5YNze1lTtEiRJkjQD1VwwzueLtM1urHYZkiRJmmFqLxgXii7VJkmSpJPUXELM54vU1dXcaUuSJOkMai4hOmIsSZKkydRcQnTEWJIkSZMpKyFGxC0R8XJEbIuIuyc5/rmI2BoRz0XEP0XEysqXWhmv7+ohk3ENY0mSJJ3ojME4IrLAvcAHgfXAnRGxfkKzZ4ANKaXLge8AX650oZWyqLOV/QeOVbsMSZIkzTDljBhfC2xLKW1PKQ0DDwG3j2+QUvpBSql/ZPMxYFlly6ycQjFxwcq51S5DkiRJM0w5wXgpsGvc9u6RfafySeB7kx2IiLsiYnNEbO7q6iq/ygrK54tknUohSZKkCcoJxpOlyDRpw4hfBzYAX5nseErp/pTShpTShs7OzvKrrKB8wZvvJEmSdLK6MtrsBpaP214G7J3YKCJuBv498N6U0lBlyqu8gsu1SZIkaRLlJMQngTURsToiGoA7gI3jG0TEVcCfAbellA5UvszKyeeLZA3GkiRJmuCMCTGllAc+DTwCvAg8nFLaEhH3RMRtI82+ArQC/zkifh4RG0/xcVXnAz4kSZI0mXKmUpBS2gRsmrDv8+Ne31zhuqaMD/iQJEnSZGouIebyRRobstUuQ5IkSTNMzQVj5xhLkiRpMjWXEJ1jLEmSpMnUXEJ0jrEkSZImU3MJsVAsks365DtJkiSdqOaCcT7vVApJkiSdrOYS4rG+YW++kyRJ0klqLiF2Heqnr3+42mVIkiRphqm5YNze1kRTY1nPNZEkSVINqblgDDB3zqxqlyBJkqQZpuaCcaFQJJNxVQpJkiSdqPaCcTG5XJskSZJOUnvBuFAkm6m505YkSdIZ1FxCLBQcMZYkSdLJai4YF1NyjrEkSZJOUnPB2KkUkiRJmkzNJUSnUkiSJGkyNRWM8/kCAHV1NXXakiRJKkNNJcR8vghAQ322ypVIkiRppqmtYFwoBeNstqZOW5IkSWWoqYQ4OmJcZzCWJEnSBDWVEAvFBDjHWJIkSSerqYToiLEkSZJOpaYS4lgwdsRYkiRJE9RUQhwazle7BEmSJM1QNRWMiyNzjAeHDMiSJEk6UU0F49Gb7+a3z6pyJZIkSZppaisYu46xJEmSTqGmEmKhUBoxzmajypVIkiRppqmtYFwcGTHO1NRpS5IkqQxlJcSIuCUiXo6IbRFx9yTHGyPib0aOPx4RqypdaCU4YixJkqRTOWMwjogscC/wQWA9cGdErJ/Q7JNAT0rpIuCPgS9VutBKcI6xJEmSTqWchHgtsC2ltD2lNAw8BNw+oc3twIMjr78DvD8iZtyw7OiqFNnMjCtNkiRJVVZOMF4K7Bq3vXtk36RtUkp5oBeYX4kCK8kQQrEEAAAFaUlEQVQRY0mSJJ1KOQlxsuHV9DbaEBF3RcTmiNjc1dVVTn0VNa99Fr/84fUs6myd9u+WJEnSzFZOMN4NLB+3vQzYe6o2EVEHzAEOTfyglNL9KaUNKaUNnZ2db6/is7B+7QIe/vpHece6hdP+3ZIkSZrZygnGTwJrImJ1RDQAdwAbJ7TZCHxs5PVHgO+nlE4aMZYkSZJmqrozNUgp5SPi08AjQBb4ZkppS0TcA2xOKW0E/gL4dkRsozRSfMdUFi1JkiRV2hmDMUBKaROwacK+z497PQj8cmVLkyRJkqaPyzNIkiRJGIwlSZIkwGAsSZIkAQZjSZIkCTAYS5IkSQBEtZYbjoguYEdVvhw6gINV+m5ND/u4NtjPtcF+Pv/Zx7Whmv28MqV0xqfLVS0YV1NEbE4pbah2HZo69nFtsJ9rg/18/rOPa8O50M9OpZAkSZIwGEuSJElA7Qbj+6tdgKacfVwb7OfaYD+f/+zj2jDj+7km5xhLkiRJE9XqiLEkSZJ0gvM2GEfELRHxckRsi4i7JzneGBF/M3L88YhYNf1V6myV0c+fi4itEfFcRPxTRKysRp06O2fq53HtPhIRKSJm9F3POlk5fRwRHx35fd4SEX813TXq7JXxZ/aKiPhBRDwz8uf2rdWoU29fRHwzIg5ExAunOB4R8Scj/w88FxFXT3eNp3NeBuOIyAL3Ah8E1gN3RsT6Cc0+CfSklC4C/hj40vRWqbNVZj8/A2xIKV0OfAf48vRWqbNVZj8TEbOBzwKPT2+FOlvl9HFErAF+D7gxpXQp8L9Me6E6K2X+Lv8+8HBK6SrgDuC+6a1SFfAAcMtpjn8QWDPycxfw9WmoqWznZTAGrgW2pZS2p5SGgYeA2ye0uR14cOT1d4D3R0RMY406e2fs55TSD1JK/SObjwHLprlGnb1yfp8BvkjpHz6D01mcKqKcPv4N4N6UUg9ASunANNeos1dOPyegbeT1HGDvNNanCkgp/Qg4dJomtwPfSiWPAe0RsXh6qjuz8zUYLwV2jdvePbJv0jYppTzQC8yflupUKeX083ifBL43pRVpKpyxnyPiKmB5Sum/TWdhqphyfpfXAmsj4qcR8VhEnG5ESjNTOf38H4Ffj4jdwCbgM9NTmqbRW/27e1rVVbuAKTLZyO/E5TfKaaOZrew+jIhfBzYA753SijQVTtvPEZGhNB3q49NVkCqunN/lOkqXXm+idOXnxxFxWUrp8BTXpsopp5/vBB5IKf1RRFwPfHukn4tTX56myYzOX+friPFuYPm47WWcfDlmrE1E1FG6ZHO6oX/NPOX0MxFxM/DvgdtSSkPTVJsq50z9PBu4DPjniHgDuA7Y6A1455Ry/8z+bkopl1J6HXiZUlDWuaOcfv4k8DBASulRoAnomJbqNF3K+ru7Ws7XYPwksCYiVkdEA6UJ/BsntNkIfGzk9UeA7ycXdT7XnLGfRy6x/xmlUOycxHPTafs5pdSbUupIKa1KKa2iNJf8tpTS5uqUq7ehnD+z/w54H0BEdFCaWrF9WqvU2Sqnn3cC7weIiHWUgnHXtFapqbYR+Dcjq1NcB/SmlPZVu6hR5+VUipRSPiI+DTwCZIFvppS2RMQ9wOaU0kbgLyhdotlGaaT4jupVrLejzH7+CtAK/OeReyt3ppRuq1rResvK7Gedw8rs40eAfxkRW4EC8Lsppe7qVa23qsx+/h3gGxHx25Qur3/cQatzS0T8NaUpTx0jc8W/ANQDpJT+lNLc8VuBbUA/8InqVDo5n3wnSZIkcf5OpZAkSZLeEoOxJEmShMFYkiRJAgzGkiRJEmAwliRJkgCDsSRJkgQYjCVJkiTAYCxJkiQB8P8DM4E3WOfcJY0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# combined ROC over all classes\n",
    "one_hot_class_encoding = label_binarize(y,np.unique(y))\n",
    "fpr[\"micro\"], tpr[\"micro\"], _ = mt.roc_curve(one_hot_class_encoding.ravel(), yhat_score.ravel())\n",
    "roc_auc[\"micro\"] = mt.auc(fpr[\"micro\"], tpr[\"micro\"])\n",
    "\n",
    "plt.figure(figsize=(12,4))\n",
    "plt.plot(fpr[\"micro\"], tpr[\"micro\"],\n",
    "         label='micro-average ROC curve (area = {0:0.2f})'\n",
    "               ''.format(roc_auc[\"micro\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Statistical Comparison \n",
    "- how might you compare two classifiers trained on the exact same datasets?\n",
    "\n",
    "Note: if these cells are not loaded, you might need to run the `%load ` magic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load -r 1-15 statcompare.py\n",
    "clf1 = Pipeline(\n",
    "    [('PCA',PCA(n_components=100,svd_solver='randomized')),\n",
    "     ('CLF',GaussianNB())]\n",
    ")\n",
    "clf2 = Pipeline(\n",
    "    [('PCA',PCA(n_components=500,svd_solver='randomized')),\n",
    "     ('CLF',GaussianNB())]\n",
    ")\n",
    "\n",
    "\n",
    "from sklearn.model_selection import cross_val_score\n",
    "# is clf1 better or worse than clf2?\n",
    "cv=StratifiedKFold(n_splits=10)\n",
    "acc1 = cross_val_score(clf1, X, y=y, cv=cv)\n",
    "acc2 = cross_val_score(clf2, X, y=y, cv=cv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Range of: -0.21181925495033532 -0.18278243001595096\n",
      "0.5865462642242995 0.3892454217411563\n"
     ]
    }
   ],
   "source": [
    "# %load -r 19-28 statcompare.py\n",
    "t = 2.26 / np.sqrt(10)\n",
    "\n",
    "e = (1-acc1)-(1-acc2)\n",
    "# std1 = np.std(acc1)\n",
    "# std2 = np.std(acc2)\n",
    "stdtot = np.std(e)\n",
    "\n",
    "dbar = np.mean(e)\n",
    "print ('Range of:', dbar-t*stdtot,dbar+t*stdtot )\n",
    "print (np.mean(acc1), np.mean(acc2))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}