week02_classification/homework_part1.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Homework part I: Prohibited Comment Classification (3 points)\n",
    "\n",
    "![img](https://github.com/yandexdataschool/nlp_course/raw/master/resources/banhammer.jpg)\n",
    "\n",
    "__In this notebook__ you will build an algorithm that classifies social media comments into normal or toxic.\n",
    "Like in many real-world cases, you only have a small (10^3) dataset of hand-labeled examples to work with. We'll tackle this problem using both classical nlp methods and embedding-based approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>should_ban</th>\n",
       "      <th>comment_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>0</td>\n",
       "      <td>\"Those who're in advantageous positions are th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>250</th>\n",
       "      <td>1</td>\n",
       "      <td>Fartsalot56 says f**k you motherclucker!!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>450</th>\n",
       "      <td>1</td>\n",
       "      <td>Are you a fool? \\n\\nI am sorry, but you seem t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>650</th>\n",
       "      <td>1</td>\n",
       "      <td>I AM NOT A VANDAL!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>850</th>\n",
       "      <td>0</td>\n",
       "      <td>Citing sources\\n\\nCheck out the Wikipedia:Citi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     should_ban                                       comment_text\n",
       "50            0  \"Those who're in advantageous positions are th...\n",
       "250           1          Fartsalot56 says f**k you motherclucker!!\n",
       "450           1  Are you a fool? \\n\\nI am sorry, but you seem t...\n",
       "650           1    I AM NOT A VANDAL!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n",
       "850           0  Citing sources\\n\\nCheck out the Wikipedia:Citi..."
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "data = pd.read_csv(\"comments.tsv\", sep='\\t')\n",
    "\n",
    "texts = data['comment_text'].values\n",
    "target = data['should_ban'].values\n",
    "data[50::200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "texts_train, texts_test, y_train, y_test = train_test_split(texts, target, test_size=0.5, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Note:__ it is generally a good idea to split data into train/test before anything is done to them.\n",
    "\n",
    "It guards you against possible data leakage in the preprocessing stage. For example, should you decide to select words present in obscene tweets as features, you should only count those words over the training set. Otherwise your algoritm can cheat evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing and tokenization\n",
    "\n",
    "Comments contain raw text with punctuation, upper/lowercase letters and even newline symbols.\n",
    "\n",
    "To simplify all further steps, we'll split text into space-separated tokens using one of nltk tokenizers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before: How to be a grown-up at work: replace \"fuck you\" with \"Ok, great!\".\n",
      "after: how to be a grown-up at work : replace \" fuck you \" with \" ok , great ! \" .\n"
     ]
    }
   ],
   "source": [
    "from nltk.tokenize import TweetTokenizer\n",
    "tokenizer = TweetTokenizer()\n",
    "preprocess = lambda text: ' '.join(tokenizer.tokenize(text.lower()))\n",
    "\n",
    "text = 'How to be a grown-up at work: replace \"fuck you\" with \"Ok, great!\".'\n",
    "print(\"before:\", text,)\n",
    "print(\"after:\", preprocess(text),)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# task: preprocess each comment in train and test\n",
    "\n",
    "texts_train = <YOUR CODE>\n",
    "texts_test = <YOUR CODE>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "assert texts_train[5] ==  'who cares anymore . they attack with impunity .'\n",
    "assert texts_test[89] == 'hey todds ! quick q ? why are you so gay'\n",
    "assert len(texts_test) == len(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solving it: bag of words\n",
    "\n",
    "![img](http://www.novuslight.com/uploads/n/BagofWords.jpg)\n",
    "\n",
    "One traditional approach to such problem is to use bag of words features:\n",
    "1. build a vocabulary of frequent words (use train data only)\n",
    "2. for each training sample, count the number of times a word occurs in it (for each word in vocabulary).\n",
    "3. consider this count a feature for some classifier\n",
    "\n",
    "__Note:__ in practice, you can compute such features using sklearn. Please don't do that in the current assignment, though.\n",
    "* `from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "example features: ['!', '12:20', '300', '_', 'adorned', 'alternative', 'archive', 'average', 'benkner', 'bout', 'came', 'chest', 'combined', 'consumers', 'cricket', 'decisions', 'dickheads', 'domestic', 'eductaion', 'essentially', 'faggot', 'firms', 'frustrated', 'goal', 'hanibal', 'hip-hop', 'identified', 'infoboxes', 'issue', 'kindergarten', 'lets', 'lot', \"mclaren's\", 'moderator', 'naturally', 'noticeable', 'opposing', 'pdf', 'plant', 'pretoria', 'punctuation', 'rebels', 'repetative', 'riadh', 'schulz', 'shes', 'slit', 'spoof', 'stupid', 't', 'theoretical', 'topic', 'uglyness', 'userspace', 'wanted', 'wikieditor', 'year', 'ீ']\n"
     ]
    }
   ],
   "source": [
    "# task: find up to k most frequent tokens in texts_train,\n",
    "# sort them by number of occurences (highest first)\n",
    "k = 10000\n",
    "\n",
    "<YOUR CODE>\n",
    "\n",
    "bow_vocabulary = <YOUR CODE>\n",
    "\n",
    "print('example features:', sorted(bow_vocabulary)[::100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def text_to_bow(text):\n",
    "    \"\"\" convert text string to an array of token counts. Use bow_vocabulary. \"\"\"\n",
    "    <YOUR CODE>\n",
    "    \n",
    "    return np.array(<...>, 'float32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train_bow = np.stack(list(map(text_to_bow, texts_train)))\n",
    "X_test_bow = np.stack(list(map(text_to_bow, texts_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "k_max = len(set(' '.join(texts_train).split()))\n",
    "assert X_train_bow.shape == (len(texts_train), min(k, k_max))\n",
    "assert X_test_bow.shape == (len(texts_test), min(k, k_max))\n",
    "assert np.all(X_train_bow[5:10].sum(-1) == np.array([len(s.split()) for s in  texts_train[5:10]]))\n",
    "assert len(bow_vocabulary) <= min(k, k_max)\n",
    "assert X_train_bow[6, bow_vocabulary.index('.')] == texts_train[6].split().count('.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Machine learning stuff: fit, predict, evaluate. You know the drill."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "bow_model = LogisticRegression().fit(X_train_bow, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlYlFX/x/H3cXBHkEBxQ1BzwzUlzR2z3NLMpzRzKYvc\nK1tNK62sTE3N7HFfCjUtHyvTX6ataJmapmKCG+GGG2oC4oIwnN8fowSKMsDM3LN8X9fF5dzrfA4D\nX2/u5RyltUYIIYR7KWJ0ACGEELYnxV0IIdyQFHchhHBDUtyFEMINSXEXQgg3JMVdCCHckBR3IYRw\nQ1LchRDCDUlxF0IIN+Rl1BsHBATokJCQAm178eJFSpcubdtATk7a7BmkzZ6hMG3+888/z2qty+W1\nnmHFPSQkhO3btxdo26ioKMLDw20byMlJmz2DtNkzFKbNSqkj1qwnp2WEEMINSXEXQgg3JMVdCCHc\nkBR3IYRwQ1LchRDCDeVZ3JVSi5RSiUqpPbdYrpRSM5RScUqp3UqpJraPKYQQIj+sOXL/FOh8m+Vd\ngJrXvgYDswsfSwghRGHkeZ+71nqjUirkNqv0ABZry3h9W5RSZZVSFbXWJ22UMYdlW48SufUys/dv\ntsfunVZSkrTZE0ib3Zvp8jlaJy7nfIWWdr+33xYPMVUGjmWbTrg276birpQajOXonsDAQKKiovL9\nZpFbL3M0xQwkFSSryzKbzSQlSZvdnbTZcYrqdBS2H0N6ZPp8KmWeApVz/pb4FMavPsS+4pk8OzKg\nQPUvP2xR3FUu83L9jmmt5wHzAMLCwnRB/uey/A+fxPpXu+R7W1cmT/F5BmmzHVw8Cwnbcs6L+wm2\nzbffewJUaw9A0sV0XvkihgUbjnBnYGk+jLibkg3uppkLHLknAEHZpqsAJ2ywXyGEByh6NQm+fwPM\n6fZ5g61zbr2s9YtQwse276dM0LA3lKmA2WymZYMG7N9/jFGjRvHWW29RsmRJux+1g22K+2rgGaXU\n50BzINle59uFEE7qSjKkX779OquGQ3ICqJx/7Lc6s+/fiRK+ts9WtDQEhkLXD3LOLxUAZYNy36aQ\nzp07xx1aYzKZeO+99wgKCiIsLMwu73UreRZ3pdRyIBwIUEolAG8CRQG01nOAtUBXIA64BDxpr7BC\nCCeUdAxmNIbMDOvWD+2RYzKROyhfrT50ngRFXPvRG601n332GSNHjmTixIkMGjSInj17GpLFmrtl\nHstjuQZG2CyREMJ1nI6Bn9+1FPawCKhQ/9brqiJQ+wHwztlbbWxUFOXd4DrDsWPHGDp0KGvXruWe\ne+6hVatWhuYxrMtfIYQbiPka9q+FcnWgxQjwr2F0IkMsX76cIUOGYDabmT59Os888wwmk8nQTFLc\nhRAFk/AnHN9hOSIfsdXoNIby8/OjefPmzJs3j2rVqhkdB5DiLoQoqDUj4fRf4FvV6CQOl5GRwYcf\nfsjVq1d5/fXX6dy5M506dUKp3O4MN4YUdyFczT+HLPduO0rGZVj6MGgNpmL/zk+/ZDmH3usTx2Vx\nAtHR0URERPDnn3/Su3dvtNYopZyqsIMUdyHsI+UkRC+DzMx8bRZ8+BBs2HbrFdJS4PcZhQxXQD5V\noN5DOefV+w94FTcmj4OlpaXx7rvvMnHiRO644w7+97//8fDDDztdUb9OirsQ9rDrM/j5nXxvVg3g\nsBUr3jMcanTI9/4LzFQUglta/vVQBw8eZNKkSfTt25dp06bh7+9vdKTbkuIuhD3oa0fsbyRanli0\n0oYNG2jXrt3tV1IKihh7J4anSE1N5ZtvvqFfv37Ur1+fffv2Ub16daNjWcW1nxgQwtkV8QKT9V+6\niCnv9aSwO8QPP/xAgwYNGDBgAHv37gVwmcIOcuQuhPWupFj3FObGKfDX/+yfR9jF+fPnefnll1m0\naBG1atViw4YN1K1b1+hY+SbFXYjsTkbDubib5x/5HbYtyN++Wj4nR9kuxmw206pVKw4cOMCYMWMY\nN24cJUqUMDpWgUhxF+JkNGxbCGjYsfj267Z5CbwD895n9fZQrpZN4gn7O3v2LHfccQcmk4kJEyZQ\ntWpVmjRx7RFDpbgLz/bNM7BzieV1mYpQyh/CnoIGvW5et4QvlKng2HzCrrTWLFmyhOeff56JEycy\nePBgHnroobw3dAFS3IVrOX8E/v7Jdvvbuxr8qsE9w6D5ENvtVzi9I0eOMGTIENavX0/Lli1p27at\n0ZFsSoq7cG6HN1lGzNHXBveKXWX792g2WAq7h1m6dCnDhg1Da83HH3/M8OHDKeLi3Q3fSIq7cA7/\nHLIM+JDN3X+MgKgEy0RA7Wv/1oLKTeG+t2z0xgq8y9toX8JVlCtXjlatWjF37lyCg4ONjmMXUtyF\nsVJOwq9Tcr0TpfT1F/+Zbxm2TIgCSk9PZ+rUqaSnpzN27Fg6depEx44dnbbrAFuQ4i6MtWPxv4W9\n3atQsXHWor/27KFBtyH2GXpNeIydO3cSERHBzp076dOnj9N29GVrUtyF46SegcTYnPP++dvy7xuJ\nN3VAde5UKSnsosCuXLnC+PHjmTx5MgEBAXz55Zf85z//MTqWw0hxF47z1dMQH3Xz/OI+HtOzoHCc\nuLg4pkyZwuOPP87UqVPx8/MzOpJDSXEX9rNtAexa9u904j6odBd0fC/nej6VHJtLuK3U1FS+/vpr\nBgwYQP369dm/f7/TjIzkaFLchf3sXQNn4yDobst0cAto2AdCjB04WLin9evXM3jwYI4dO0ZYWBh1\n69b12MIOUtyFLR3eBDFf/Tt95gCUrwP9vzQuk3B7586d48UXX2Tx4sXUqVOHX3/91SU7+rI1Ke7C\nelpDyglA3zz/y6fh2BbLdKlsgxgENXNYPOF5rnf0FRcXx+uvv84bb7zhsh192ZoUd2G9DZMg6v3b\nr/PQbGjc1zF5hMc6c+YM/v7+mEwmJk2aRHBwMI0bN857Qw8ixV1YL/U0FCsDnSfcvEyZoHYXKHWH\n43MJj6G15tNPP+XFF19k4sSJDBkyhB49ehgdyylJcRe3lmmGw7/C/wYCyjLafXEfaPK40cmEBzp8\n+DCDBw/mhx9+oE2bNrRv397oSE5NirvI3ZHN8GUEpBy3TJcPhaDmln5dhHCwJUuWMGzYMJRSzJo1\niyFDhrhdR1+2JsVd5JR+BdY8B7u/+Hdevy+hRnsZVUgYJjAwkLZt2zJnzhyqVq1qdByXIMVd/Cvm\n62unYK55YCo0eQJMRQ2LJDxTeno6kydPxmw2M27cODp27EjHjh2NjuVSpLgLi/gN8Me1DrzCx8A9\nw6GEj7GZhEfasWMHTz31FNHR0fTt2zeroy+RP1adtFJKdVZK7VdKxSmlRueyvKpS6hel1E6l1G6l\nVFfbRxV2kZYKyx6FxQ/Ckd8s/aW3e1UKu3C4y5cvM3r0aJo1a8bp06f5+uuv+eyzz6SwF1CeR+5K\nKRMwE7gfSAC2KaVWa62zd+/3BrBCaz1bKRUKrAVC7JBX2FLsalgx4N/phxdCg0eMyyM8Wnx8PNOm\nTWPgwIF88MEHHtfRl61Zc1qmGRCntY4HUEp9DvQAshd3DVw/1PMFTtgypLCBw7/Bvm9zzjuxy/Jv\nhzfh7qflaF04XEpKCuvWrSM8PJx69epx8OBBtx0ZydGsKe6VgWPZphOA5jes8xbwvVLqWSwD6Nxn\nk3TCdn77EP7+GYp555wf2ABaPgcmufwiHGvt2rUMHTqU48eP88QTT1C3bl0p7DZkzW90bie8buhc\nhMeAT7XWU5VSLYAlSqn6WuvMHDtSajAwGCy3NkVFReU7cFLSZcxmc4G2dWWpqamFanPDc+fw8r6T\nHU0/uHnhr78VPJgdFbbNrsgT2pycnMzMmTP54YcfCA4OZtKkSZw+fZrTp08bHc1hHPE5W1PcE4Cg\nbNNVuPm0SwTQGUBrvVkpVQIIABKzr6S1ngfMAwgLC9Ph4eH5Djx7/2aSkpIoyLauLCoqKv9tvnAa\nNkyEjDTIOAU+lV3q+1agNrs4d2+z2WwmNDSU+Ph4xo0bx2uvvcbmzZvdus25ccTnbE1x3wbUVEpV\nA44DfYAbe4Y6CnQAPlVK1QVKAGdsGVTk05bZsO76jU0KfKtAtbaGRhKe6/Tp05QrVw6TycSUKVMI\nDg6mYcOGRsdya3neCqm1zgCeAdYDe7HcFROjlBqvlHrw2movAYOUUtHAcmCg1vrGUzfCkY78bhl/\ntNVIGHcOXtgD971pdCrhYbTWLFy4kNq1azNv3jwAunfvLoXdAay6iqa1Xovl9sbs88Zlex0LyPA6\nRtn4wb93vlyXsB3KVIL7xxuTSXi8+Ph4Bg0axM8//0y7du247z65z8KR5BYJd7BpBhTxgjIV/51X\nyt/SBa8QBoiMjGT48OGYTCbmzJnDoEGDpKMvB5Pi7i4a9YHOeQykIYSDVKpUiXvvvZfZs2dTpUoV\no+N4JCnuriz1DKx+Bq6mGp1EeLirV68yceJEMjMzeeutt7j//vu5//77jY7l0eTvJFe1dS5MuRMO\nrIOipaCm/CIJY2zbto2mTZvy5ptvEh8fj9xL4RykuLuqswfBqyR0mgCjDkGNe41OJDzMpUuXePnl\nl7nnnns4f/48q1evZvHixdLRl5OQ4u6K1r0GMV9B0ZLQYgR4FTM6kfBAhw4d4uOPP2bQoEHExMTQ\nvXt3oyOJbOScu6vZtgC2zLS8vvcNY7MIj5OcnMxXX33Fk08+Sb169YiLiyMoKCjvDYXDSXF3ZuYM\n+OVduHyeWidOQvwkOPo7BNSG/iuhrAw3Jhzn22+/ZciQIZw8eZIWLVpQp04dKexOTE7LOLN//rb0\n5rjna/zP/QFnD4BfCDy2XAq7cJgzZ87Qr18/unXrhp+fH5s3b6ZOnTpGxxJ5kCN3Z3b9roPu09l8\n9g6P61xJGM9sNtO6dWsOHTrE22+/zejRoylWTK7xuAIp7s7qdCz8+JbRKYSHOnXqFOXLl8dkMjF1\n6lRCQkKoX7++0bFEPshpGWd0JQW2L4SD66FCA8uXEA6QmZnJ3LlzqVWrFnPnzgWgW7duUthdkBy5\nO6Mv+sGhjWAqDoM3QBETlt6WhbCfuLg4Bg0aRFRUFPfeey+dOnUyOpIoBCnuRsrMhP8bCck3FO7j\nO6FiY+j+0bXCLoR9ffLJJwwfPpxixYoxf/58IiIi5GEkFyfF3SiZZoj/BXYsBt8g8A78d1m52tB0\nIFRqbFg84VmqVq1Kp06dmDlzJpUrVzY6jrABKe5GSPgTVg213NoIED4a7upvbCbhUdLS0nj//ffJ\nzMxk/PjxdOjQgQ4dOhgdS9iQFHdH0BrSL1n+/WYExK76d1n/r2T4O+FQW7duJSIigpiYGJ544gm0\n1nIKxg1Jcbe3U3vg66Fw+q+c8x+YBndHGJNJeKSLFy8yduxYpk+fTuXKlfm///s/HnjgAaNjCTuR\n4m4vV1Lg1ymw6aN/590/3jJiUoPe4F3OuGzCIx05coRZs2YxdOhQJk6ciI+Pj9GRhB1JcbeXo1ss\nhd1UDFo8YxmoumRZo1MJD5OUlMTKlSt5+umnCQ0NJS4uTkZG8hBS3O3hwimIj7K8fmodVG5qaBzh\nmb755huGDRtGYmIirVu3pk6dOlLYPYg8oWoPGyZf65ZXQck7jE4jPExiYiJ9+vThoYceoly5cmzZ\nskU6+vJAcuRuD+Y0y33rw36H0gFGpxEexGw206pVK44ePcq7777LqFGjKFq0qNGxhAGkuNtLES8p\n7MJhTpw4QYUKFTCZTHz00UeEhIQQGhpqdCxhIDktY2sbPoC/o4xOITxEZmYms2fPpk6dOsyZMweA\nrl27SmEXUtxtRms4HWO5QybjMtTraXQi4eYOHDhA+/btGT58OM2bN6dLly5GRxJORIq7rZzaDbNb\nwtUL0OQJ6PSe0YmEG1u4cCGNGjVi9+7dLFq0iO+//55q1aoZHUs4ETnnbgtxP8EPb1pe3/8OhD1l\nbB7h9kJCQujSpQszZ86kYsWKRscRTkiKe2H9Ew/bFkBiLNTuCo37QXFvo1MJN5OWlsY777wDwLvv\nvisdfYk8yWmZwlr2KOxfCz6VLANXl/Y3OpFwM7///juNGzfmvffe4+TJk+jrY+sKcRtS3Avr6iWo\n1Rme/snoJMLNpKamMnLkSFq3bs2lS5dYt24dCxculB4chVWsKu5Kqc5Kqf1KqTil1OhbrNNbKRWr\nlIpRSi2zbUwnVyoAygTmvZ4Q+XD06FHmzp3LiBEj2LNnjwx7J/Ilz3PuSikTMBO4H0gAtimlVmut\nY7OtUxMYA7TSWp9XSpW3V2Ah3NmFCxeYN28egwcPJjQ0lPj4eCpVqmR0LOGCrLmg2gyI01rHAyil\nPgd6ALHZ1hkEzNRanwfQWifaOqgQ7u7rr7/m6aefJjk5mXbt2lG7dm0p7KLArCnulYFj2aYTgOY3\nrFMLQCm1CTABb2mt1924I6XUYGAwQGBgIFFRUfkOnJR0GbPZXKBt7eGetCucP3WS/XbOk5qa6jRt\ndhRPafM///zDjBkz2LBhA9WrV+f999/n5MmTnDx50uhoDuEpn3N2jmizNcU9t6s3N16u9wJqAuFA\nFeBXpVR9rXVSjo20ngfMAwgLC9Ph4eH5zcvs/ZtJSkqiINvaxY4SVKxQkYp2zhMVFeU8bXYQT2iz\n2WymTp06HDt2jAkTJnD33Xdz3333GR3LoTzhc76RI9pszQXVBCAo23QV4EQu63yjtU7XWh8C9mMp\n9u4r9Qx82ABSEnL/70+I20hISCAzMxOTycSMGTPYtWsXY8aMwctLHj0RtmFNcd8G1FRKVVNKFQP6\nAKtvWGcV0B5AKRWA5TRNvC2DOp2UBEg+anlw6e5BRqcRLiIzM5OPP/6YOnXqMHv2bAC6dOki/a0L\nm8uzuGutM4BngPXAXmCF1jpGKTVeKfXgtdXWA+eUUrHAL8ArWutz9grtVJo8DpUaG51CuIB9+/bR\ntm1bnnvuOVq3bk23bt2MjiTcmFV/A2qt1wJrb5g3LttrDbx47UsIcYMFCxbwzDPPUKpUKSIjIxkw\nYIA8jCTsSk7wCeEANWrUoHv37vz3v/8lMFAeeBP2J8VdCDu4cuUK48ePB2DChAm0b9+e9u3bG5xK\neBLpW0YIG9u0aRONGzfm/fff58yZM9LRlzCEFHchbOTChQs8++yztGnThrS0NNavX8/8+fPl3Low\nhBR3IWwkISGBBQsW8Oyzz/LXX3/RsWNHoyMJDybn3IUohHPnzrFixQqGDRtG3bp1iY+Pl5GRhFOQ\nI3chCkBrzcqVKwkNDeW5555j//79AFLYhdOQ4l4QsathSc9rE3I+1dOcPHmShx9+mF69ehEUFMT2\n7dupXbu20bGEyEFOyxTEyWi4fB5aPgvBLYxOIxzIbDbTpk0bjh8/zuTJk3nhhRekPxjhlOSnMr+2\nzIE9K0GZoOO7RqcRDnLs2DEqV66MyWRi5syZVKtWjVq1ahkdS4hbktMy+bV9EVxOgsaPGZ1EOIDZ\nbGbGjBk5Ovrq1KmTFHbh9OTIvSCqh0OPmUanEHa2d+9eIiIi2Lx5M126dKF79+5GRxLCanLknh/m\ndNCZRqcQDjBv3jwaN27MgQMHWLJkCd9++y1Vq1Y1OpYQVpPibo30yxD9BbwTAOcOQhH5g8fd1axZ\nk549exIbG0v//v3lKVPhcqRKWePPSFj3quV1SBtoN8rYPMLmLl++zFtvvYVSiokTJ0pHX8LlyZG7\nNdIvWf4dshGeWAPl5J5md7Jx40YaNWrE5MmTSU5Olo6+hFuQ4p6XMwcs97UDBNQG+fPcbaSkpDB8\n+HDatWuH2Wzmp59+Yvbs2XIKRrgFOS1zK5mZ8NPbsGm6ZbqEr5xrdzMnTpzg008/5cUXX2T8+PGU\nLl3a6EhC2IxUq1tJPWUp7CXKQv3/QIc3wSTfLld39uxZVqxYwfDhw6lTpw6HDh2SkZGEW5LTMnm5\n/23o9iGULGt0ElEIWmu++OILQkNDef755zlw4ACAFHbhtqS4C7d34sQJHnroIfr06UNwcDB//vmn\nPGEq3J6cZxBuzWw207ZtW44fP86UKVMYOXKkdPQlPIL8lAu3dOTIEapUqYLJZGLWrFlUr16dO++8\n0+hYQjiMnJYRbsVsNjNt2jTq1q2b1dFXx44dpbALjyNH7sJt7Nmzh4iICP744w+6devGQw89ZHQk\nIQwjxf1Gh36FA+vgaqrRSUQ+zJkzh+eeew5fX1+WLVtGnz595GEk4dGkuN/o1ykQvwGKlrLc4x4g\nd1U4M601Sinq1q1Lr169mD59OuXKlTM6lhCGk+IOkJEGB38AcxpcOA1V74Gn1hmdStzGpUuXGDdu\nHCaTiUmTJtGuXTvatWtndCwhnIZcUAXYuwa+6Acrn4Ize6HkHUYnErcRFRVFw4YNmTp1KqmpqdLR\nlxC5kCN3sBy5AwxYBT6VoKwMyuCMkpOTGTVqFPPmzaNGjRr8/PPP0i2vELdg1ZG7UqqzUmq/UipO\nKTX6Nus9opTSSqkw20V0oDuqW7rzLVrS6CQiFydPnmTp0qW8/PLL7N69Wwq7ELeR55G7UsoEzATu\nBxKAbUqp1Vrr2BvWKwM8B2y1R1Dhmc6cOcPnn3/Os88+S506dTh8+LBcMBXCCtYcuTcD4rTW8Vrr\nq8DnQI9c1nsHmAxcsWE++7t6CZKOGp1C3EBrzY8//kjdunV56aWXsjr6ksIuhHWsKe6VgWPZphOu\nzcuilLoLCNJa/58NsznGty/ChomW13I6xikcO3aM7t27895773HnnXeyc+dO6ehLiHyy5oJqbk+C\nZN2eoJQqAnwIDMxzR0oNBgaDpavVqKgoq0Jml5R0GbPZXKBtc1M/4W9Klwhkb92XSNkeC8TmuY0R\nUlNTbdZmZ2Y2m3n88cf5559/ePrpp+nTpw9nzpzxiLaD53zO2Umb7cOa4p4ABGWbrgKcyDZdBqgP\nRF17IrACsFop9aDWenv2HWmt5wHzAMLCwnR4eHi+A8/ev5mkpCQKsm2uTs6Fopdp0mOIbfZnJ1FR\nUbZrsxM6fPgwQUFBmEwmIiMjqV69OkePHnXrNufG3T/n3Eib7cOa0zLbgJpKqWpKqWJAH2D19YVa\n62StdYDWOkRrHQJsAW4q7ELkJiMjgylTplC3bl1mzZoFwH333Uf16tUNTiaEa8vzyF1rnaGUegZY\nD5iARVrrGKXUeGC71nr17fcgRO52795NREQE27dvp0ePHjz88MNGRxLCbVj1EJPWei2w9oZ5426x\nbnjhYznIiV1w4ZTRKTzSrFmzGDlyJH5+fnzxxRf06tVLOvoSwoY8t/uBzExY2BFO7LB0ECYc4npX\nAfXr16dPnz7ExsbSu3dvKexC2JgHdz+gLR2FhUVYBsEWdnXx4kXeeOMNvLy8+OCDD2jbti1t27Y1\nOpYQbsszj9wz0mDPV5bXZSpA8TLG5nFzP/30Ew0aNGD69OmkpaVJR19COIBnFve/f4avnra8Li1P\nPNpLUlISTz/9NPfddx9eXl5s3LiRGTNmyCkYIRzAM4v79V4gH/8Gmg40NIo7O336NJ9//jmvvvoq\n0dHRtGnTxuhIQngMDz7nDpQuD3IUaVPXC/rIkSOpXbs2hw8fJiAgwOhYQngczzxyFzantWbp0qWE\nhoYyatQoDh48CCCFXQiDeF5x/+1D+DLC8lp5XvPt4ejRozzwwAMMGDCA2rVrs2vXLmrWrGl0LCE8\nmuedljkdA8VKQ+sXZfBrG8jIyCA8PJzExERmzJjB8OHDMZlMRscSwuN5XnEHKOUPrZ83OoVLi4+P\nJzg4GC8vL+bPn0+NGjUICQkxOpYQ4hrPOC+RmQmHNsL+dZByIu/1xS1lZGQwadIkQkNDmTlzJgAd\nOnSQwi6Ek/GMI/ejmyGy+7/Tle4yLosL27VrFxEREezYsYOePXvSq1cvoyMJIW7BvYv7mufh2FZI\nS7VM95gJ5UPBL8TQWK7ov//9Ly+88AL+/v6sXLlSenAUwsm5d3HfuxqKeUOlRlCtLdTrabmYKqym\ntUYpRcOGDenXrx/Tpk3jjjvuMDqWECIP7l3cAWreDw9MNTqFy0lNTeX111+naNGiTJkyRTr6EsLF\neMYFVZEv33//PfXr1+fjjz8mPT1dOvoSwgVJcRdZzp8/z5NPPkmnTp0oUaIEGzdu5KOPPpKOvoRw\nQVLcRZbExERWrlzJmDFj2LVrF61btzY6khCigNz3nPuZ/WDOMDqF0zt16hTLly/nhRdeyOroy9/f\n3+hYQohCcs8j99MxMLMZpCVD0VJGp3FKWmsiIyMJDQ1lzJgxWR19SWEXwj24Z3G/nGT5t8M4aPuK\nsVmc0OHDh+ncuTMDBw4kNDRUOvoSwg2512mZpKOw/nXL/e0AVe6GEj7GZnIyGRkZtG/fnrNnzzJz\n5kyGDh1KkSLu+X+8EJ7MvYr7skchMdbyuuN7UKWZsXmcSFxcHNWqVcPLy4tFixZRvXp1goODjY4l\nhLAT9zpku3oRanSAF2Kh5TNQtITRiQyXnp7OhAkTqFevXlZHX+3bt5fCLoSbc68jd7AMeO1b2egU\nTmHHjh1ERESwa9cuevXqxaOPPmp0JCGEg7jXkbvIMmPGDJo1a8apU6f46quvWLFiBYGBgUbHEkI4\niBR3N3O9q4C77rqLxx9/nNjYWHr27GlwKiGEo7nfaRkPdeHCBcaMGUPx4sWZOnUqbdq0oU2bNkbH\nEkIYRI7c3cC6deuoX78+s2bNQmstHX0JIaS4u7Jz587xxBNP0KVLF0qXLs2mTZuYNm2adPQlhJDi\n7srOnTvH119/zdixY9m5cyctWrQwOpIQwklYVdyVUp2VUvuVUnFKqdG5LH9RKRWrlNqtlPpJKSU3\nUdvJyZMnmTJlClpratWqxZEjRxg/fjzFixc3OpoQwonkWdyVUiZgJtAFCAUeU0qF3rDaTiBMa90Q\nWAlMtnVQT6e1ZtGiRdStW5exY8cSFxcHgJ+fn8HJhBDOyJoj92ZAnNY6Xmt9Ffgc6JF9Ba31L1rr\nS9cmtwBVbBvTsx06dIhXXnmFiIgIGjVqRHR0tHT0JYS4LWtuhawMHMs2nQA0v836EcB3uS1QSg0G\nBgMEBgYWYdt2AAAZBElEQVQSFRVlXcpskpIuYzabc922+ZUrJJ8+xb4C7NdZmc1m+vfvT3JyMi+8\n8ALdunXjxIkTnDhxwuhodpeamlqgnxFXJm32DI5oszXFPbdbL3K9104p1R8IA9rltlxrPQ+YBxAW\nFqbDw8OtS5nN7P2bSUpKItdtd5WgZGAFKhRgv87m4MGDVK9eHZPJxPLly0lMTKR3795Gx3KoqKio\n3D9nNyZt9gyOaLM1xT0BCMo2XQW46bBRKXUf8DrQTmudZpt4VjKnw4bJcOkfh76tPaSnpzNp0iTe\neecdJk+ezMiRIwkPD/e4IxtXkJmZSUJCAhcvXrTZPn19fdm7d6/N9ucKpM05FS1alPLly+PjU7ju\nyq0p7tuAmkqpasBxoA/QN/sKSqm7gLlAZ611YqESFcSZfbBxMhTzhspNHf72trJ9+3YiIiLYvXs3\nffr04bHHHjM6kriNs2fPopSidu3aNusT/8KFC5QpU8Ym+3IV0uZ/aa25fPkyx48fByhUgc/zJ1Jr\nnQE8A6wH9gIrtNYxSqnxSqkHr632AeAN/E8ptUsptbrAiQri+hOZPedC88EOfWtb+eijj2jevDln\nz57lm2++Yfny5ZQvX97oWOI2kpKSCAwMlMFOhM0opShVqhSVK1cmMbFwx8lW9S2jtV4LrL1h3rhs\nr+8rVAoPprVGKUVYWBgRERFMnjyZsmXLGh1LWMFsNlO0aFGjYwg3VLJkSdLT0wu1D+k4zCApKSm8\n+uqrlChRgg8//JBWrVrRqlUro2OJfJKuHoQ92OLnSv6eNMDatWupV68e8+bNw8vLSzr6Ek5p6NCh\nvPPOO0bHEAUkxd2Bzp49S//+/XnggQfw9fXl999/54MPPpCjP2FzISEh/Pjjj4Xax5w5cxg7dmyh\n9jFw4EC8vLxuei5j4MCBvPHGGznmHT58GKUUGRkZWfOWLVtGWFgY3t7eVKxYkS5duvDbb7/lK8Oe\nPXvo1KkTAQEBVv2u7dq1i6ZNm1KqVCmaNm3Krl27spZprXn11Vfx9/fH39+fUaNG5Tg4s3bb4ODg\nm7a1NSnuDnT+/HnWrFnDm2++yY4dO2je/HbPgglhP9kLqL1cvHiRL7/8El9fXz777LN8bz9t2jSe\nf/55XnvtNU6fPs3Ro0cZPnw433zzTb72U7RoUXr37s3ChQvzXPfq1av06NGD/v37c/78eZ544gl6\n9OjB1atXAZg3bx6rVq0iOjqa3bt383//93/MnTs339tu3rw5x7Z2cb3/b0d/NW3aVBdE7zm/644T\n1+aceSJa6zd9tI5dU6B92lNCQoKeNGmSzszM1Fprff78+QLt55dffrFhKtfg7G2OjY21+T5TUlIK\nvY/+/ftrpZQuUaKELl26tJ40aZI+dOiQBvSCBQt0UFCQbtOmjdZa60ceeUQHBgZqHx8f3aZNG71n\nz56s/TzxxBP69ddf11pbPovKlSvrKVOm6HLlyukKFSroRYsW3TZHZGSkrlKlip4+fbquV69ejmXZ\n9329zdczpqen66SkJF26dGm9YsWKQn8/rjt48KC2lLxbW79+va5UqVLW76vWWgcFBenvvvtOa611\nixYt9Ny5c7OWLViwQDdv3jzf26akpOTYNje3+vkCtmsraqzrH7kf2QxrnjM6xU201syfP5/Q0FDe\neust/v77bwC5E0bY3ZIlS6hatSpr1qwhNTWVUaNGZS3bsGEDe/fuZf369QB06dKFgwcPkpiYSJMm\nTejXr98t93vq1CmSk5M5fvw4CxcuZMSIEZw/f/6W60dGRvLYY4/Rp08f9u3bx44dO6xuw+bNm7ly\n5cpth4hctmwZZcuWveXX0aNHrX6/62JiYmjYsGGO0zcNGzYkJiYma3mjRo2yljVq1CjHsoJuaw+u\nfbdM0jHYtgBO7ISanaDSXUYnAuDvv/9m0KBB/PLLL4SHhzN//nzuvPNOo2MJO3t7TQyxJ1IKtQ+z\n2YzJZLrl8tBKPrzZvV6B9//WW29RunTprOmnnnoqxzI/Pz+Sk5Px9fW9aduiRYsybtw4vLy86Nq1\nK97e3uzfv5977rnnpnWPHj3KL7/8wtSpUwkMDKRDhw5ERkbSpEkTq3KeO3eOgIAAvLxuXaL69u1L\n3759b7m8IFJTU29qu6+vLxcuXMh1ua+vL6mpqWitC7WtPa67ufaR+8onYc9KKO4L/VaAb2WjE5GR\nkUGHDh3Yvn07c+fO5aeffpLCLpxGUNC/PYmYzWZGjx5NjRo18PHxISQkBLBc+M+Nv79/jmJbqlQp\nUlNTc113yZIl1K1bl8aNGwPQr18/li1blnXvtpeX1033caenp1OkSBGKFCmCv78/Z8+edci1gey8\nvb1JScn5H3RKSkrW06Q3Lk9JScHb2xulVKG2tQfXPnK/egmCW8N/5hmdhP3791OjRg28vLyIjIyk\nRo0aVKkiPR97ksIcUV9nq0fxb1Uwss9ftmwZ33zzDT/++CMhISEkJyfj5+dnkzs4Fi9ezNGjR6lQ\noQJgOeg5d+4c3333HQ8++CBVq1a96ZTEoUOHCAoKokiRIrRo0YISJUqwatUqHnnkkVzf47PPPmPI\nkCG3zBAbG0vVqlXzlbtevXpMnTo1x9H07t27GTFiRNby6OhomjVrBkB0dDT16tUr9Lb24NpH7gAl\nyxp6xH716lXefvttGjRowMyZMwFo166dFHZhqMDAQOLj42+7zoULFyhevDj+/v5cunSJ1157zSbv\nvXnzZv7++2/++OMPdu3axa5du9izZw99+/YlMjISgIcffphvv/2W77//HrPZzIkTJ3j33Xfp06cP\nYDllMX78eEaMGMGqVau4dOkS6enpfPfdd1nXEPr160dqauotv64Xdq01V65cybpr5cqVK6Sl5d63\nYXh4OCaTiRkzZpCWlsZ///tfAO69914AHn/8caZNm8bx48c5ceIEU6dOZeDAgfne9uTJkzm2tQtr\nrrra48smd8vMbKH18r4F2o8tbN26VdevX18Dum/fvvrMmTN2ey9nv3PEHpy9zc56t4zWWq9atUoH\nBQVpX19f/cEHH+S4E+W6Cxcu6AcffFB7e3vrqlWr6sjISA3ogwcPaq1zv1smu+DgYP3DDz/c9N5D\nhgzR//nPf26av3XrVl2sWDF97tw5rbXWq1ev1k2aNNE+Pj66atWq+uWXX9aXLl3Ksc3SpUt106ZN\ndalSpXRgYKDu2rWr3rRpU76+F9fbnv0rODg4a3nnzp31e++9lzW9Y8cO3aRJE12iRAl911136R07\ndmQty8zM1K+88or28/PTfn5++pVXXslxd4y125YtW/ambW9U2LtlpLgX0IcffqiLFCmiK1eurNes\nsf8tmM5e6OzB2dvszMXdlUibcye3QjqYvnY+slmzZgwaNIiYmBi6detmcCohhMjJtS+oOlBycjKj\nRo2iZMmSTJ8+nZYtW9KyZUujYwkhRK7kyN0Ka9asITQ0lAULFlC8eHHp6EsI4fSkuN/GmTNn6Nu3\nLw8++CD+/v5s2bKFSZMmSUdfQginJ8X9NpKTk1m7di1vv/0227dv5+677zY6khBCWMUlz7mX1hdh\ncnW4dA4CbPv057Fjx1i6dCmjR4/mzjvv5MiRI7k+ii2EEM7MJY/cffUFS2Gv0w3avGSTfWZmZjJn\nzhzq1avHu+++m9XRlxR2IYQrcsninqXug1CxUd7r5eHgwYPce++9DBs2jGbNmvHXX39JfzBCCJfm\nkqdlbCkjI4P777+fpKQkFi5cyJNPPikXTIUQLs+1j9wLYe/evWRkZODl5cWSJUuIjY3lqaeeksIu\n3IIthtkD+PTTT2ndunWBtg0PD8fPz++mflzCw8NZsGBBjnlRUVE5+mPSWjNjxgzq169P6dKlqVKl\nCr169eKvv/7KVwatbz8s3o0+/vhjqlWrho+PD2FhYTmG9Js+fTrVq1fHx8eHSpUq8cILL+TotTIk\nJISSJUvi7e2Nt7c3HTt2zFoWGRlJ06ZN8fHxoUqVKowdO9buPV56XHFPS0vjzTffpGHDhlkd+7Rp\n04ZKlSoZnEwI93H48GF+/fVXlFKsXr0639uPHDmSjz76iBkzZvDPP/9w4MABHnroIb799tt87ed2\nw+LdaOvWrYwePZqVK1eSnJxMREQEPXv2xGw2A9C9e3d27NhBSkoKe/bsITo6mhkzZuTYx/UBUlJT\nU/n++++z5l+6dInp06dz9uxZtm7dyoYNG5gyZUo+vyv541HFfcuWLTRp0oTx48fz2GOPMWDAAKMj\nCWFzAwYM4OjRo3Tv3h1vb28mT54MWH7+W7ZsSdmyZWnUqBFRUVFZ23z66adUr16dMmXKUK1aNT77\n7DP27t3L0KFD2bx5M97e3vkaRWzx4sXcc889DBw4MKsnSGsdPHiQmTNnsnz5cu69916KFy9OqVKl\n6NevH6NHj87XviIjI3nppZeoUqUKlStX5qWXXuLTTz/Ndd3Dhw9Tr149mjZtilKKxx9/nLNnz5KY\nmAhAjRo1sr4HWmuKFClCXFycVTmGDRtGmzZtKFasGJUrV6Z3795s2rQpX23JL48p7lOnTqVly5Zc\nuHCBtWvXsnjxYvz9/Y2OJYTN5TbM3vHjx3nggQd44403+Oeff5gyZQoPP/wwZ86c4eLFizz33HN8\n9913XLhwgd9//53GjRtTt25d5syZQ4sWLUhNTSUpKcnqDIsXL6Zfv37069eP9evXc/r0aau3/emn\nn6hSpUpWv+e5mThx4m2H2LsuP0PbdenSBbPZzNatWzGbzSxatIjGjRtn9UkPlj7wfXx8CAgIIDo6\n+qb+5Pv160e5cuXo2LEj0dHRt8y/adMmu/blDh5wQTUzMzOr8/+hQ4cyceJEfHx8jI4l3NF3o+FU\n/s4J36ikOQNMt/m1rNAAukzM936XLl1K165d6dq1KwD3338/YWFhrF27lkceeYQiRYqwZ88eqlat\nSsWKFalYsWJBm8Bvv/3GkSNH6N27NwEBAdSoUYNly5bxwgsvWLX9uXPn8nz/0aNHW3UUn5+h7cqU\nKcPDDz9M69at0VpTtmxZvvvuuxzrXR/a7+DBgyxevJjAwMCsZZ999hlNmjRBa81HH31Ep06d2Ldv\n301/8XzyySfs3Lnzln9B2IrbHrknJSURERHByJEjAWjZsiWzZs2Swi480pEjR/jf//6X4+j2t99+\n4+TJk5QuXZovvviCOXPmULFiRR544AH27dtX4PeKjIykY8eOBAQEAOQYpANuPcRe0aJFActwfidP\nnizw+2eXn6HtFixYwKJFi4iJieHq1assXbqUbt26ceLEiZvWrVmzJvXq1WP48OFZ81q1akXJkiUp\nVaoUY8aMoWzZsvz66685tlu1ahWjR4/myy+/zPr+2ItbHrmvWrWK4cOHk5iYmHV1XO6CEXZXgCPq\nG1220zB7QUFBDBgwgPnz5+e6fqdOnejUqROXL1/mjTfeYNCgQVkXRPPj8uXLrFixArPZnHU6Iy0t\njaSkJKKjo2nUqBFVq1bl8OHDObY7dOgQwcHBAHTo0IERI0awfft2wsLCcn2fCRMmMGHChFvmuD62\na36GtouOjqZ79+7UqlULgM6dO1OxYkV+//33XIf6y8jIyHrYMTdKqRx35qxbt45Bgwbx7bffUrdu\n3VtuZytudeSemJhI79696dmzJ4GBgfzxxx9MmDBBCrvwODcOs9e/f3/WrFnD+vXrMZvNXLlyhaio\nKBISEjh9+jSrV6/m4sWLFC9eHG9vb0wmU9Z+EhISsoaoy8uqVaswmUzExsZmDbG3d+9e2rRpw+LF\niwF49NFH+eSTT/jjjz/QWnPw4EE+/PDDrCH2atasyfDhw3nssceIiori6tWrXLlyhc8//5yJEy3/\ngb722mu3HWLvutsNi3eju+++m2+//Zb4+Hi01vzwww8cOHCA+vXrA5Yj++sXV2NjY3n//ffp0KED\nAEePHmXTpk1ZWT/44APOnj1Lq1atAPj555/p168fX3755W2vJdiUNSN62OOrMCMxDZywSOs3fbTe\n9XmOZQcPHtRly5bV7733nr569WqB9u+snH1UIntw9jY780hMNw6zp7XWW7Zs0W3bttV+fn46ICBA\nd+3aVR85ckSfOHFCt23bVvv4+GhfX1/drl07HRMTo7XWOi0tTXft2lX7+flpf3//PN+3U6dO+sUX\nX7xp/hdffKEDAwOzhvlbuHChDg0N1WXKlNHVqlXT77//vjabzVnrZ2Zm6unTp+vQ0FBdsmRJXalS\nJd27d2+9Z8+efH0f8hoWr3Tp0nrjxo1Z644dO1YHBQVpb29vXadOHb148eKsdQcOHKjLly+vS5Uq\npYODg/XLL7+sL1++rLXWes+ePbpBgwa6VKlS+o477tD33nuv3rZtW9a24eHh2mQy6dKlS2d9de7c\n+bbZCzsSk9JW9E2ulOoMfASYgAVa64k3LC8OLAaaAueAR7XWh2+3z7CwML19+/Z8/2f06NzNlPpn\nH5+kPQ8953HUrwVLlizhtddeQylls9HjnU1UVBTh4eFGx3AoZ2/z3r17bf7ntbv+/N6OtDl3t/r5\nUkr9qbXO/XxVNnmellFKmYCZQBcgFHhMKRV6w2oRwHmt9Z3Ah8CkvPZbUA0rFCe8+D4ytWbWF99T\nr149JkyYkHXuy9N+SIQQIjfWnHNvBsRpreO11leBz4EeN6zTA7h+OXwl0EHZ6UT36xX/5J74mYR/\neokR7y+iRYsWxMTESEdfQgiRjTXFvTJwLNt0wrV5ua6jtc4AkgG7PCGUkZ5Op6WX+Cu5FJ8sWsT6\n9esJCQmxx1sJIYTLsuZWyNyOwG88UW/NOiilBgODwXIVPvvjz9YLZdiY97nzzjvx9/dnw4YNBdiH\n60lNTS3g98t1OXubfX19uXDhgk33aTabbb5PZydtzt31O5oKypringAEZZuuAtx4V//1dRKUUl6A\nL/DPjTvSWs8D5oHlgmphLpY584U2e3D2i4v24Oxt3rt37y0fiCkoubjoGfJqs9aaEiVKcNdddxX4\nPaw5LbMNqKmUqqaUKgb0AW7s5m018MS1148AP2trbsMRwoWZTKabnrQUwhYuX76c9cRuQeVZ3K+d\nQ38GWA/sBVZorWOUUuOVUg9eW20h4K+UigNeBPLXdZsQLqhs2bKcPn2azMxMo6MIN6G15tKlSxw/\nfpzy5csXal9WdT+gtV4LrL1h3rhsr68AvQqVRAgXExAQQEJCAvv377fZPq9cuUKJEiVstj9XIG3O\nqWjRogQGBha6Hyy37FtGCEcoUqQIVatWtek+o6KiCnWe1RVJm+3DrfqWEUIIYSHFXQgh3JAUdyGE\ncENS3IUQwg1Z1SukXd5YqTPAkQJuHgCctWEcVyBt9gzSZs9QmDYHa63L5bWSYcW9MJRS263p8tKd\nSJs9g7TZMziizXJaRggh3JAUdyGEcEOuWtznGR3AANJmzyBt9gx2b7NLnnMXQghxe6565C6EEOI2\nnLq4K6U6K6X2K6XilFI39TSplCqulPri2vKtSqkQx6e0LSva/KJSKlYptVsp9ZNSKtiInLaUV5uz\nrfeIUkorpVz+zgpr2qyU6n3ts45RSi1zdEZbs+Jnu6pS6hel1M5rP99djchpK0qpRUqpRKXUnlss\nV0qpGde+H7uVUk1sGkBr7ZRfgAn4G6gOFAOigdAb1hkOzLn2ug/whdG5HdDm9kCpa6+HeUKbr61X\nBtgIbAHCjM7tgM+5JrAT8Ls2Xd7o3A5o8zxg2LXXocBho3MXss1tgSbAnlss7wp8h2Uku3uArbZ8\nf2c+cneqgbkdJM82a61/0Vpfuja5BcvIWK7Mms8Z4B1gMnDFkeHsxJo2DwJmaq3PA2itEx2c0das\nabMGrvdz68vNI765FK31RnIZkS6bHsBibbEFKKuUqmir93fm4u5UA3M7iDVtzi4Cy//8rizPNiul\n7gKCtNb/58hgdmTN51wLqKWU2qSU2qKU6uywdPZhTZvfAvorpRKwjB/xrGOiGSa/v+/54sz9udts\nYG4XYnV7lFL9gTCgnV0T2d9t26yUKgJ8CAx0VCAHsOZz9sJyaiYcy19nvyql6mutk+yczV6safNj\nwKda66lKqRbAkmttdtehruxav5z5yD0/A3Nzu4G5XYg1bUYpdR/wOvCg1jrNQdnsJa82lwHqA1FK\nqcNYzk2udvGLqtb+bH+jtU7XWh8C9mMp9q7KmjZHACsAtNabgRJY+mBxV1b9vheUMxd3TxyYO882\nXztFMRdLYXf187CQR5u11sla6wCtdYjWOgTLdYYHtdbbjYlrE9b8bK/CcvEcpVQAltM08Q5NaVvW\ntPko0AFAKVUXS3E/49CUjrUaePzaXTP3AMla65M227vRV5TzuNrcFTiA5Sr769fmjcfyyw2WD/9/\nQBzwB1Dd6MwOaPOPwGlg17Wv1UZntnebb1g3Che/W8bKz1kB04BY4C+gj9GZHdDmUGATljtpdgEd\njc5cyPYuB04C6ViO0iOAocDQbJ/xzGvfj79s/XMtT6gKIYQbcubTMkIIIQpIirsQQrghKe5CCOGG\npLgLIYQbkuIuhBBuSIq7EEK4ISnuQgjhhqS4CyGEG/p/hYbGHSdtJjQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc2db73cb38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score, roc_curve\n",
    "\n",
    "for name, X, y, model in [\n",
    "    ('train', X_train_bow, y_train, bow_model),\n",
    "    ('test ', X_test_bow, y_test, bow_model)\n",
    "]:\n",
    "    proba = model.predict_proba(X)[:, 1]\n",
    "    auc = roc_auc_score(y, proba)\n",
    "    plt.plot(*roc_curve(y, proba)[:2], label='%s AUC=%.4f' % (name, auc))\n",
    "\n",
    "plt.plot([0, 1], [0, 1], '--', color='black',)\n",
    "plt.legend(fontsize='large')\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task: implement TF-IDF features\n",
    "\n",
    "Not all words are equally useful. One can prioritize rare words and downscale words like \"and\"/\"or\" by using __tf-idf features__. This abbreviation stands for __text frequency/inverse document frequence__ and means exactly that:\n",
    "\n",
    "$$ feature_i = { Count(word_i \\in x) \\times { log {N \\over Count(word_i \\in D) + \\alpha} }} $$\n",
    "\n",
    "\n",
    ", where x is a single text, D is your dataset (a collection of texts), N is a total number of documents and $\\alpha$ is a smoothing hyperparameter (typically 1). \n",
    "And $Count(word_i \\in D)$ is the number of documents where $word_i$ appears.\n",
    "\n",
    "It may also be a good idea to normalize each data sample after computing tf-idf features.\n",
    "\n",
    "__Your task:__ implement tf-idf features, train a model and evaluate ROC curve. Compare it with basic BagOfWords model from above.\n",
    "\n",
    "Please don't use sklearn/nltk builtin tf-idf vectorizers in your solution :) You can still use 'em for debugging though."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "### Solving it better: word vectors\n",
    "\n",
    "Let's try another approach: instead of counting per-word frequencies, we shall map all words to pre-trained word vectors and average over them to get text features.\n",
    "\n",
    "This should give us two key advantages: (1) we now have 10^2 features instead of 10^4 and (2) our model can generalize to word that are not in training dataset.\n",
    "\n",
    "We begin with a standard approach with pre-trained word vectors. However, you may also try\n",
    "* training embeddings from scratch on relevant (unlabeled) data\n",
    "* multiplying word vectors by inverse word frequency in dataset (like tf-idf).\n",
    "* concatenating several embeddings\n",
    "    * call `gensim.downloader.info()['models'].keys()` to get a list of available models\n",
    "* clusterizing words by their word-vectors and try bag of cluster_ids\n",
    "\n",
    "__Note:__ loading pre-trained model may take a while. It's a perfect opportunity to refill your cup of tea/coffee and grab some extra cookies. Or binge-watch some tv series if you're slow on internet connection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import gensim.downloader \n",
    "embeddings = gensim.downloader.load(\"fasttext-wiki-news-subwords-300\")\n",
    "\n",
    "# If you're low on RAM or download speed, use \"glove-wiki-gigaword-100\" instead. Ignore all further asserts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def vectorize_sum(comment):\n",
    "    \"\"\"\n",
    "    implement a function that converts preprocessed comment to a sum of token vectors\n",
    "    \"\"\"\n",
    "    embedding_dim = embeddings.wv.vectors.shape[1]\n",
    "    features = np.zeros([embedding_dim], dtype='float32')\n",
    "    \n",
    "    <YOUR CODE>\n",
    "    \n",
    "    return features\n",
    "\n",
    "assert np.allclose(\n",
    "    vectorize_sum(\"who cares anymore . they attack with impunity .\")[::70],\n",
    "    np.array([ 0.0108616 ,  0.0261663 ,  0.13855131, -0.18510573, -0.46380025])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train_wv = np.stack([vectorize_sum(text) for text in texts_train])\n",
    "X_test_wv = np.stack([vectorize_sum(text) for text in texts_test])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XdYleUbwPHvy0j2cKEgghtxlyM3Zm4tzZngxD0yrZ9p\nWmCZoWWuXKSWaK4sc+RITdDMyokpLkQQREGR6UDG8/vjxAmUcYADBzjP57q44Jx33e8Rb97znOe9\nb0UIgSRJklS2GOg6AEmSJEn7ZHKXJEkqg2RylyRJKoNkcpckSSqDZHKXJEkqg2RylyRJKoNkcpck\nSSqDZHKXJEkqg2RylyRJKoOMdHXgihUrCmdn5wJt++jRI8zNzbUbUAknz1k/yHPWD4U557Nnzz4Q\nQlTKaz2dJXdnZ2fOnDlToG39/f1xc3PTbkAlnDxn/SDPWT8U5pwVRQnTZD05LCNJklQGyeQuSZJU\nBsnkLkmSVAbJ5C5JklQGyeQuSZJUBuWZ3BVF2aAoSrSiKJdyWK4oirJcUZRgRVEuKorysvbDlCRJ\nkvJDkyv374DuuSzvAdT592scsLrwYUmSJEmFoWjSZk9RFGdgnxCiYTbL1gL+Qoit/z6+BrgJIe7m\nts/mzZuLgsxz3/LXbTb6X8bGxkb9XKzhceIN/873vkqT1NRUjIx0dluCTshz1g9l7Zw7XoildVB8\ntsuepqXyOPkpVLfDc+vxAu1fUZSzQojmea2njVfUAQjP9Dji3+deSO6KooxDdXWPnZ0d/v7++T7Y\nqovHiLM6zwMDRf3cM+MQAF56VjPf+ysthBCkpqbqOoxiJc+5ZOl0MY42VxK1vl8BKHmuVVS030O6\nfsRTAEKrZT2rCw8fsfzqXcwNDZhbzbpA+S8/tJHcs/t3yfYVE0L4Ar6gunIvyB1aKbdWYMBdGlfN\n/CaiPD1r9mRg3YH53l9pIe/iKxtit+8gYd++HJfHxcVhY2NdjBFp7vHp6wCYtWih1f2qztkm7xUL\nKj0Fkp/7o/QkFhJyHVwouEpg5fSE+l1bARD3KIX/bb/MuvO3qW1nzroxLTBt7UnLIv7d1kZyjwAc\nMz2uBkRqYb85Mk6159vu3xblISRJY3kl7Mwenz4NaD9BFgezFi2w6t0b28GDtLrfiF9/pglnIC1F\nq/tV+2tNzsvazQATK+0eTzGExoPAsgppaWm0adSIa9fCmTlzJt7e3piamhb5VTtoJ7nvAaYoirIN\naAXE5zXeLkmlhSaJOz8JO68Eecvfnyal8d3K03hIeZL7Oj9PgvgIULK+2W97/+p/D0yK4F2LsTnY\nuULPL7I+b1YRbByz36aQYmJiKC8EhoaGfPbZZzg6OtK8eZ7D5FqVZ3JXFGUr4AZUVBQlAvACjAGE\nEGuA/UBPIBh4DIwqqmClsiWvxGkbF0fY+g3FGNGLNEncRXVFW2rEhcPyppCu4WcFrm9meRhNeSrX\naAjdF4JB6b71RgjB999/z7Rp0/Dx8WHs2LH069dPJ7HkmdyFEG/nsVwAk7UWkVQmaPuKV1f0PnHn\nJeoy/DZfldibe0KVFybU/UcxgHq9wCJrtdogf38ql8Z3K88JDw9nwoQJ7N+/n1dffZW2bdvqNJ6y\nM/9IKlZ5JW9tXPGW2iEKfXJ5F1zbD5VcoPVkqFBL1xHpxNatWxk/fjxpaWksXbqUKVOmYGhoqNOY\nZHKXslXY5C2vePVAxFm4c051RT75L11Ho1O2tra0atUKX19fatSooetwAJncpRwk7NvH06tXMXFx\nyXa5TN4Se6dB1D9gXV3XkRS71NRUlixZwrNnz5gzZw7du3enW7duKIruZuw/TyZ36QWx23fw+PRp\nzFq0wGmTn67DkZ738BY8elB8x0t9Apv7gxBg+NJ/z6c8Vo2hD9SvacmBgYF4enpy9uxZBg0ahBAC\nRVFKVGIHmdz1RkHmYlv17l2UIZVtCXchcAukp+drM6fQWxBwOucVkhPgj+WFDK6ArKpBg75Zn2vw\nFhiV0008xSw5OZn58+fj4+ND+fLl+eGHH+jfv3+JS+oZZHIvQ3JL4Nqciy1p4ML38Nun+d6sBkCo\nBiu+Oglqdc73/gvM0Bic2qi+66kbN26wcOFChg4dyldffUWFChV0HVKuZHIvI2K37+CelxeQfQKX\nCbuYiX+v2OdGq+5Y1FBAQAAdO3bMfSVFAQPdzsTQF0lJSezevRt3d3caNmzI1atXqVmzdNSwksm9\nFMt8pZ5xZV5l3jyZwEsSA6N8JWJhYAiG8r9lSXD48GHGjRtHWFgYL7/8MvXr1y81iR1kci+1nr9S\nl1fmxeBpgmZ3YR7/Ev75oejjkYpEbGws77//Phs2bKBu3boEBARQv359XYeVbzK5lzA5jZs/fyu+\nvFIvIncDISb4xefD/oDT6/K3rzbvyOGTUiYtLY22bdty/fp1Zs+ezccff4yJiYmuwyoQmdxLgOyG\nV/L64FNeqWvR3UA4vR4QcC6PqZ/t3wMLu7z3WbMTVKqrlfCkovfgwQPKly+PoaEhCxYsoHr16rz8\ncunuGCqTezHIz92eOSVteSt+Edk9Bc5vUv1sWRXMKkDz0dAom94AJtZgWaV445OKlBCCTZs28e67\n7+Lj48O4cePo27dv3huWAjK5F5H8XI3Lq/B8iA2Dm0e1t78re8C2Brw6EVqN195+pRIvLCyM8ePH\nc+jQIdq0aUOHDh10HZJWyeReRDLfvi+TdyGEnoTT36jujgQI+ln7x2g5TiZ2PbN582YmTpyIEIIV\nK1YwadIkDEp5ueHnyeRehExcXOTt+5p6eEvV8CGTFn9PBv8I1YOK9f79XhccXoHXvbV0YAUsKmtp\nX1JpUalSJdq2bcvatWtxcnLSdThFQib3IpC5NouUh4S7cOLLbGeimGf88NY3qrZlklRAKSkpLF68\nmJSUFD766CO6detG165dS2zpAG2Qyb0IZIy1y9osGjjn919i7/gBVG2qXvTPpUs06j2+aFqvSXrj\n/PnzeHp6cv78eYYMGVJiC31pm0zuRcSsRQs5xv68pPsQHZT1uYc3Vd/nRr9QgCrmnplM7FKBPX36\nlE8++YRFixZRsWJFfvzxR9566y1dh1VsZHKXis9PYyDE/8Xny1npTWVBqfgEBwfz5ZdfMnz4cBYv\nXoytra2uQypWMrlLRef0Oriw5b/H0VfBvhl0/Szrelb2xRuXVGYlJSWxa9cuhg0bRsOGDbl27VqJ\n6YxU3GRyL6TsblDKrYORXrmyFx4Eg+O/Hyw7tYbGQ8BZt42DpbLp0KFDjBs3jvDwcJo3b079+vX1\nNrGDTO6Fll07OhMXF/38MDX0JFz+6b/H969DZRfw+FF3MUllXkxMDDNmzMDPzw8XFxdOnDhRKgt9\naZtM7lqgN/PZhYCESEC8+PyPYyD8T9Vjs0xNDBxbFlt4kv7JKPQVHBzMnDlzmDt3bqkt9KVtMrnn\nQ3ZDMHo1nz1gIfh/nvs6fVdD06HFE4+kt+7fv0+FChUwNDRk4cKFODk50bRp07w31CMyuedDdkMw\nGaUF9EJSFLxkCd0XvLhMMYR6PcCsfPHHJekNIQTfffcdM2bMwMfHh/Hjx/Pmm2/qOqwSSSZ3DWRc\nsWckdr0YggFIT4PQE/DDSEBRdbsvZwUvD9d1ZJIeCg0NZdy4cRw+fJj27dvTqVMnXYdUosnknofn\nOx7pzVV62Cn40RMS7qgeV3YFx1aqui6SVMw2bdrExIkTURSFVatWMX78+DJX6EvbZHLPQ8YYu950\nPEp5CnvfgYvb/3vO/Ueo1Ul2FZJ0xs7Ojg4dOrBmzRqqV6+u63BKBZncNaA3pQQu7/p3COZfvRbD\nyyPA0FhnIUn6KSUlhUWLFpGWlsbHH39M165d6dq1q67DKlVkcpdUQgLg738LeLnNhlcngYmVbmOS\n9NK5c+cYPXo0gYGBDB06VF3oS8ofjQatFEXprijKNUVRghVFmZXN8uqKohxTFOW8oigXFUXpqf1Q\npSKRnARbBoPfGxD2u6peescPZGKXit2TJ0+YNWsWLVu2JCoqil27dvH999/LxF5AeV65K4piCKwE\nugARwGlFUfYIITKX95sL7BBCrFYUxRXYDzgXQbySNgXtgR3D/nvcfz00GqC7eCS9FhISwldffcXI\nkSP54osv9K7Ql7ZpMizTEggWQoQAKIqyDXgTyJzcBZBxqWcNRGozSEkLQn+Hq79kfS7ygup7Zy9o\nMUZerUvFLiEhgYMHD+Lm5kaDBg24ceNGme2MVNw0Se4OQHimxxFAq+fW8QZ+VRRlKqoGOq9rJTpJ\ne35fAjd/g5cssj5v1wjavAOG8uMXqXjt37+fCRMmcOfOHUaMGEH9+vVlYtciTf5HZzfg9VxxEd4G\nvhNCLFYUpTWwSVGUhkKI9Cw7UpRxwDhQTW3y9/fPd8CpqakIIQq0bX6ZnjiB1enTPKtTh1vFcLzc\nJCUlFeqcG8fEYGRRm3OvfPHiwhO/FzywIlTYcy6N9OGc4+PjWblyJYcPH8bJyYmFCxcSFRVFVFSU\nrkMrNsXx76xJco8AHDM9rsaLwy6eQHcAIcQpRVFMgIpAdOaVhBC+gC9A8+bNhZubW/4DvrWC1NRU\nCrJtfoWt38BjoLqHB02K4Xi58ff3z/85J0ZBgA+kJkPqPbByKJbXTVsKdM6lXFk/57S0NFxdXQkJ\nCeHjjz/mww8/5NSpU2X6nLNTHP/OmiT300AdRVFqAHeAIcDzlaFuA52B7xRFqQ+YAPe1GaiulNo5\n7n+uhoMZE5sUsK4GNTroNCRJf0VFRVGpUiUMDQ358ssvcXJyonHjxroOq0zLcyqkECIVmAIcAq6g\nmhVzWVGUTxRFeePf1d4DxiqKEghsBUYKIZ4fupGKU9gfqv6jbafBxzEw/RK87qXrqCQ9I4Rg/fr1\n1KtXD19fXwD69OkjE3sx0OhTNCHEflTTGzM/93Gmn4MA2V5HV45/8d/MlwwRZ8DSHrp8opuYJL0X\nEhLC2LFj+e233+jYsSOvvy7nWRQnOUWiLDi5HAyMwLLqf8+ZVVCV4JUkHdi4cSOTJk3C0NCQNWvW\nMHbsWFnoq5jJ5F5WNBkC3fNopCFJxcTe3p7XXnuN1atXU61aNV2Ho5dkci/Nku7DninwLEnXkUh6\n7tmzZ/j4+JCeno63tzddunShS5cuug5Lr8n3SaXVX2vhy9pw/SAYm0Ed+R9J0o3Tp0/zyiuv4OXl\nRUhICHIuRckgk3s2YrfvIGzYcJ5evarrUHL24AYYmUK3BTDzFtR6TdcRSXrm8ePHvP/++7z66qvE\nxsayZ88e/Pz8ZKGvEkIm92xkbqlXIjsvHfwQLv8ExqbQejIYvaTriCQ9dOvWLVasWMHYsWO5fPky\nffr00XVIUiZyzP05sdt38Pj0acxatCiZvVJPr4M/V6p+fm2ubmOR9E58fDw//fQTo0aNokGDBgQH\nB+Po6Jj3hlKxk8k9k8z9UkvEFXtaKhybD09iqRt5F0IWwu0/oGI98NgJNrLdmFR8fvnlF8aPH8/d\nu3dp3bo1Li4uMrGXYHJYJpMS1y/14U1VNcdLu6gQ8zc8uA62zvD2VpnYpWJz//593N3d6d27N7a2\ntpw6dQoXFxddhyXlQV65P6dE1ZLJmHXQZymnHpTXu+JKku6lpaXRrl07bt26xbx585g1axYvvSQ/\n4ykNZHIvqaKC4Ii3rqOQ9NS9e/eoXLkyhoaGLF68GGdnZxo2bKjrsKR8kMMyJdHTBDizHm4cgiqN\nVF+SVAzS09NZu3YtdevWZe3atQD07t1bJvZSSF65l0Tb3eHWcTAsB+MCwMAQVbVlSSo6wcHBjB07\nFn9/f1577TW6deum65CkQpDJXZfS02HfNIh/LnHfOQ9Vm0KfZf8mdkkqWt9++y2TJk3ipZde4ptv\nvsHT01PejFTKyeSuK+lpEHIMzvmBtSNY2P23rFI9eGUk2DfVWXiSfqlevTrdunVj5cqVODg46Doc\nSQtkcteFiLPw8wTV1EYAt1nQzEO3MUl6JTk5mc8//5z09HQ++eQTOnfuTOfOnXUdlqRFMrkXByEg\n5bHq++7JEPTzf8s8fpLt76Ri9ddff+Hp6cnly5cZMWIEQgg5BFMGyeSO6s7UzPVktOreJdg1AaL+\nyfp8r6+ghad2jyVJuXj06BEfffQRS5cuxcHBgX379tGrVy9dhyUVEZncKaJCYU8T4MSXcHLZf891\n+UTVManRILCopJ3jSJKGwsLCWLVqFRMmTMDHxwcrKytdhyQVIb1P7kVWKOz2n6rEbvgStJ6ialRt\naqO9/UuSBuLi4ti5cydjxozB1dWV4OBg2RlJT+h1ci+yQmGJ9yDEX/Xz6IPg8Ir29i1JGtq9ezcT\nJ04kOjqadu3a4eLiIhO7HtHrO1SLrFBYwKJ/y/IqYFpee/uVJA1ER0czZMgQ+vbtS6VKlfjzzz9l\noS89pNdX7lBEhcLSklXz1if+AeYVtbtvScpFWloabdu25fbt28yfP5+ZM2dibGys67AkHdD75F5k\nDIxkYpeKTWRkJFWqVMHQ0JBly5bh7OyMq6urrsOSdEivh2WKRMAXcNNf11FIeiI9PZ3Vq1fj4uLC\nmjVrAOjZs6dM7JJM7lojBERdVs2QSX0CDfrpOiKpjLt+/TqdOnVi0qRJtGrVih49eug6JKkEkcld\nW+5dhNVt4FkivDwCun2m64ikMmz9+vU0adKEixcvsmHDBn799Vdq1Kih67CkEkQvx9y1fkdq8FE4\nrJpSSZdPofnowu9TknLh7OxMjx49WLlyJVWrVtV1OFIJpHfJPfPcdrMWLQo/v/1hCJxeB9FBUK8n\nNHWHchZaiFSS/pOcnMynn34KwPz582WhLylPepfctT63fctgVXVHa0dV42pJ0rI//vgDT09Prl69\nyujRo2WhL0kjejnmrtW57c8eQ93uMOaodvYnSf9KSkpi2rRptGvXjsePH3Pw4EHWr18vE7ukEY2S\nu6Io3RVFuaYoSrCiKLNyWGeQoihBiqJcVhRli3bDLOHMKoKlXd7rSVI+3L59m7Vr1zJ58mQuXbok\n295J+ZJnclcUxRBYCfQAXIG3FUVxfW6dOsBsoK0QogHwbhHEWmgZRcIkqaRKTEzE19cXAFdXV0JC\nQlixYgWWlpY6jkwqbTQZc28JBAshQgAURdkGvAkEZVpnLLBSCBELIISI1nag2pAx3q7VImGSpCW7\ndu1izJgxxMfH07FjR+rVq4e9vb2uw5JKKU2SuwMQnulxBNDquXXqAiiKchIwBLyFEAef35GiKOOA\ncQB2dnb4+/vnO+DU1FSEEAXa1jYuDurUIdCuMhRg++y8mvyU2Ht3uaal/eUkKSmpQOdcmunLOT98\n+JDly5cTEBBAzZo1+fzzz7l79y53797VdWjFQl/+nTMrjnPWJLln9+mNyGY/dQA3oBpwQlGUhkKI\nuCwbCeEL+AI0b95cuLm55TdejG6tIDU1lYJsG7Z+AwBNCrBtjs6ZULVKVapqc5/Z8Pf3L9A5l2b6\ncM5paWm4uLgQHh7OggULaNGiBa+//rquwypW+vDv/LziOGdNPlCNABwzPa4GRGazzm4hRIoQ4hZw\nDVWyL7uS7sOSRpAQkf2fP0nKRUREBOnp6RgaGrJ8+XIuXLjA7NmzMTLSu9nJUhHRJLmfBuooilJD\nUZSXgCHAnufW+RnoBKAoSkVUwzQh2gy0xEmIgPjbqhuXWozVdTRSKZGens6KFStwcXFh9erVAPTo\n0UPWW5e0Ls/kLoRIBaYAh4ArwA4hxGVFUT5RFOWNf1c7BMQoihIEHAP+J4SIKaqgS5SXh4N9U11H\nIZUCV69epUOHDrzzzju0a9eO3vKDfakIafQeUAixH9j/3HMfZ/pZADP+/ZIk6Tnr1q1jypQpmJmZ\nsXHjRoYNGyZvRpKKlBzgk6RiUKtWLfr06cPXX3+NnZ284U0qejK5S1IRePr0KZ988gkACxYsoFOn\nTnTq1EnHUUn6RC9ry0hSUTp58iRNmzbl888/5/79+6hGLSWpeMnkLklakpiYyNSpU2nfvj3Jyckc\nOnSIb775Ro6tSzohk7skaUlERATr1q1j6tSp/PPPP3Tt2lXXIUl6TI65S1IhxMTEsGPHDiZOnEj9\n+vUJCQmRnZGkEkFeuUtSAQgh2LlzJ66urrzzzjtcu3YNQCZ2qcSQyb0ggvbApn7/PpDjqfrm7t27\n9O/fn4EDB+Lo6MiZM2eoV6+ersOSpCz0YlhG6w2x7wbCk1hoMxWcWhd+f1KpkZaWRvv27blz5w6L\nFi1i+vTpsh6MVCLpxW9l5sRe6Fruf66BSztBMYSu87UToFTihYeH4+DggKGhIStXrqRGjRrUrVtX\n12FJUo70ZljGxMUFp01+he+demYDPImDpm9rJzCpREtLS2P58uVZCn1169ZNJnapxNOLK3etq+kG\nb67UdRRSEbty5Qqenp6cOnWKHj160KdPH12HJEka05srd61ISwGRrusopGLg6+tL06ZNuX79Ops2\nbeKXX36hevXqug5LkjQmk7smUp5A4Hb4tCLE3AAD+YanrKtTpw79+vUjKCgIDw8PeZepVOrILKWJ\nsxvh4Aeqn53bQ8eZuo1H0ronT57g7e2Noij4+PjIQl9SqSev3DWR8lj1ffxxGLEXKsk5zWXJ8ePH\nadKkCYsWLSI+Pl4W+pLKBJnc83L/umpeO0DFeiDfnpcZCQkJTJo0iY4dO5KWlsbRo0dZvXq1HIKR\nyoQyn9xjt+/g8enT+d8wPR0Oe8HKFhD0M5hYy7H2MiYyMpLvvvuOGTNmcPHiRV577TVdhyRJWlPm\ns1XCvn0A+b95KekenFwKJjbQ8C3o7AWGZf7lKvMePHjAjh07mDRpEi4uLty6dUt2RpLKpDJ/5Q5g\n1qJFwW9e6jIPei8BUxvtBiUVKyEE27dvx9XVlXfffZfr168DyMQulVl6kdwl/RYZGUnfvn0ZMmQI\nTk5OnD17Vt5hKpV5cpxBKtPS0tLo0KEDd+7c4csvv2TatGmy0JekF8r0b3nGh6lmLVroOhSpmIWF\nhVGtWjUMDQ1ZtWoVNWvWpHbt2roOS5KKTZkelinwh6lSqZWWlsZXX31F/fr11YW+unbtKhO7pHfK\n9JU7FPLDVKlUuXTpEp6envz999/07t2bvn376jokSdKZMp/c8+3WCbh+EJ4l6ToSKR/WrFnDO++8\ng7W1NVu2bGHIkCHyZiRJr8nk/rwTX0JIABibqea4V5SzKkoyIQSKolC/fn0GDhzI0qVLqVSpkq7D\nkiSdk8kdIDUZbhyGtGRIjILqr8Log7qOSsrF48eP+fjjjzE0NGThwoV07NiRjh076josSSoxyuwH\nqvkqO3BlL2x3h52j4f4VMC1ftMFJheLv70/jxo1ZvHgxSUlJstCXJGWjzF6552umTGqy6vuwn8HK\nHmxkU4aSKD4+npkzZ+Lr60utWrX47bffZFleScqBRlfuiqJ0VxTlmqIowYqizMplvQGKoghFUZpr\nL8T8id2+g7Bhw3l69Wr+Z8qUr6kq52tsWnQBSgV29+5dNm/ezPvvv8/FixdlYpekXOR55a4oiiGw\nEugCRACnFUXZI4QIem49S+Ad4K+iCFRTCfv28fTqVUxcXOT89jLg/v37bNu2jalTp+Li4kJoaKj8\nwFSSNKDJsExLIFgIEQKgKMo24E0g6Ln1PgUWAe9rNcICMHFxwWmTn2YrP3sMcbeLNiAp34QQHDly\nhAEDBpCQkEC3bt2oW7euTOySpCFNhmUcgPBMjyP+fU5NUZRmgKMQYp8WYysev8yAAB/Vz3I4pkQI\nDw+nT58+fPbZZ9SuXZvz58/LQl+SlE+aXLlndyeIenqCoigGwBJgZJ47UpRxwDhQlVr19/fXKMjM\nUlNTEULg7++P6YkTmPyddUaMUUQEqdWqcUvDfTeMuIm5iR1X6r9HwpkgXnxDUjIkJSUV6PUqbdLS\n0hg+fDgPHz5kzJgxDBkyhPv37+vFuYP+/DtnJs+5aGiS3CMAx0yPqwGRmR5bAg0B/3/vCKwC7FEU\n5Q0hxJnMOxJC+AK+AM2bNxdubm75D/jWClJTU3FzcyNs/Qae3ruHiYvLfyvY2GDVuzdNNN333bVg\n/ISX3xyf71iKk7+/PwV5vUqL0NBQHB0dMTQ0ZOPGjdSsWZPbt2+X6XPOTln/d86OPOeioUlyPw3U\nURSlBnAHGAIMzVgohIgHKmY8VhTFH3j/+cReVPI1vi6VOKmpqSxdupSPPvqIRYsWMXXqVF5//XUA\nbt+Wn4VIUkHlmdyFEKmKokwBDgGGwAYhxGVFUT4Bzggh9hR1kFLZdPHiRTw9PTlz5gxvvvkm/fv3\nL9T+EhISiI6OJiUlRUsRFj9ra2uuXLmi6zCKlTznrIyNjalcuTJWVlaFOoZGNzEJIfYD+5977uMc\n1nUrVETFKfICJN7TdRR6adWqVUybNg1bW1u2b9/OwIEDC1XoKyEhgaioKBwcHDA1NS21RcMSExOx\ntLTUdRjFSp7zf4QQPHnyhDt37gAUKsGX2fIDeUpPh/VdIfKcqkCYVCwySgU0bNiQIUOGEBQUxKBB\ngwqdjKOjo3FwcMDMzKzUJnZJUhQFMzMzHBwciI6OLtS+ymz5gbwJVaGw5p6qJthSkXr06BFz587F\nyMiIL774gg4dOtChQwet7T8lJQVTUzmVVSobTE1NCz28qJ9X7qnJcOkn1c+WVaCcfr0lLG5Hjx6l\nUaNGLF26lOTk5CIr9CWv2KWyQhu/y/qZ3G/+Bj+NUf1sLu94LCpxcXGMGTOG119/HSMjI44fP87y\n5cv1Mgk7Oztz5MgRXYeRbxYWFoSEhOg6DKkA9DO5Z1SBHL4bXhmp01DKsqioKLZt28YHH3xAYGAg\n7du313VIesPf359q1aoVej9JSUnUrFmzwNs/evQICwsLevbs+cIyRVEIDg7O8py3tzceHh7qxwkJ\nCbz77rtUr14dCwsLateuzbvvvsuDBw/yFcfXX39N8+bNKVeuHCNHjsxz/SVLllClShWsra0ZPXo0\nycnJ6mWhoaF06tQJMzMzXFxcXvijrcm2dnZ22W6rTfqZ3DOYVwY9vIosSlFRUSxbtgyAevXqERoa\nio+Pjxy9BQTJAAAgAElEQVQPL4FSU1OL/Bg7d+6kXLly/Prrr9y9ezdf2z579ozOnTtz+fJlDh48\nSEJCAn/88QcVKlTg77//zte+7O3tmTt3LqNHj85z3UOHDuHj48PRo0cJDQ0lJCQELy8v9fK3336b\nZs2aERMTw2effcaAAQO4f/9+vrYNDQ19YVtt0+/kLmmNEILNmzfj6urKzJkzuXHjBgAVK1bMY0v9\ncfr0aVxdXbG1tWXUqFE8ffpUveybb76hdu3aVK9enTfeeIPISNVN4F5eXkydOhVQfWhsbm7OzJkz\nAXjy5AkmJibExsZmOc6jR4/o0aMHkZGRWFhYYGFhQWRkJN7e3gwYMAAPDw+srKz47rvv+Pvvv2nd\nujU2NjZUrVqVKVOm8OzZM/W+Ml9djxw5ksmTJ9OrVy8sLS1p1aoVN2/ezPWcN27cyIQJE2jcuDHf\nf/99vl4vPz8/bt++za5du3B1dcXAwIDKlSvz0UcfZftOIDdvvfUWffv2pUKFCnmuu3HjRjw9PWnQ\noAG2trZ89NFHfPfddwBcv36dc+fOMW/ePExNTenfvz+NGjXixx9/LPS22qZ/yf33JfCjp+pnRf9O\nvyjcvn2bXr16MWzYMOrVq8eFCxeoU6eOrsMqcb7//nsOHTrEzZs3uX79OvPnzwfgt99+Y/bs2ezY\nsYMbN27g5OTEkCFDAOjYsaO6Bsnp06epUqUKAQEBAJw6dYp69epha2ub5Tjm5uYcOHAAe3t7kpKS\nSEpKwt7eHoDdu3czYMAA4uLicHd3x9DQkCVLlvDgwQNOnTrF0aNHWbVqVY7nsHXrVry8vIiNjaV2\n7drMmTMnx3Vv376Nv78/7u7uuLu74+eXvzvJjxw5Qvfu3bGwsMhxnd69e2NjY5PtV+8Clvy+fPky\nTZo0UT9u0qQJUVFRxMTEcPnyZWrWrJlljnqTJk24fPlyobfVNv2bChl1GV4yh3YzZPNrLcio8xMd\nHc3y5cuZNGkShoaGug6LeXsvExSZUKTHcLW3wqtPA43XnzJlCo6OqjJNc+bMYerUqcyfP5/vv/+e\n0aNH8/LLL5OYmMjnn3+Ora0toaGhtG7dmhs3bhATE8Px48fx9PRk1apVJCUlERAQkO++sa1bt6Zv\n376AarrdK6+8ol7m7OzM+PHjCQgI4N133812+7feeouWLVsC4O7uzowZM3I8lp+fH40bN8bV1RUb\nGxtmzpzJ+fPnadasmUaxxsTEZIkvO/v2ab8QbVJSEtbW1urHGT8nJia+sCxjecZNR4XZVtv089LV\nrAK0excM9PP0tSEkJIS0tDSMjIz45ptvuHTpElOnTi0Rib2kykjsAE5OTuqhl8jISJycnNTLLCws\nqFChAnfu3MHU1JTmzZsTEBDA8ePH6dixI23atOHkyZMFSu6ZYwDVUEHv3r2pUqUKVlZWfPjhh7l+\nWFmlShX1z2ZmZiQlJeW4rp+fH+7u7oBqzLtjx45s3LhRvdzQ0PCFudwpKSkYGxsDUKFChXyP02uD\nhYUFCQn/XRhk/GxpafnCsozlGVfjhdlW2/Tjyj09HcJ+VzXmSIjMe30pR6mpqSxevBgvLy8WLVrE\nO++8Q+fOnXUd1gvyc0VdXMLD/2uLcPv2bfVQib29PWFhYepljx49IiYmBgcHVduEjh078ttvv3H+\n/HlatGhBx44dOXToEH///XeON4LlNN30+ecnTpxIs2bN2Lp1K5aWlixdupSdO3cW6jwB/vjjD27c\nuMHnn3/O4sWLAdXV6+XLl/nyyy8xMjKievXqhIaGZpnVc+vWLXXt/tdff525c+fy6NEjzM3Nsz1O\njx49OHHiRLbL2rdvz4EDB/Ide4MGDQgMDGTQIFWLzsDAQOzs7KhQoQINGjQgJCQkS/mAwMBAhg4d\nmq9tM2TeVtv049L19inY2Ae2Doawk2Binfc20gsuXLhAq1atmDVrFj179mTgwIG6DqlUWblyJRER\nETx8+JAFCxYwePBgAIYOHcq3337LhQsXSE5O5sMPP6RVq1Y4OzsDquTu5+eHq6srL730Em5ubqxb\nt44aNWrk2JnKzs6OmJgY4uPjc40pMTERKysrLCwsuHr1KqtXr9bKuW7cuJEuXboQFBTEhQsXuHDh\nApcuXeLx48fqhDt48GDmz5/PnTt3SE9P58iRI+zdu5cBAwYAMGzYMBwdHenfvz9Xr14lPT2dmJgY\nFixYwP79qlJXBw4cUH+u8PxX5sSemprK06dPSUtLIy0tjadPn+Y4W2j48OGsX7+eoKAgYmNjmT9/\nvnr6ZN26dWnatCnz5s3j6dOn7Nq1i4sXL6qL3hVmW60TQujk65VXXhEF0XLDW+Jl3zeEEEKEegwT\noR7Dcl55zzQhVr4qxFcNhfCyEuLcJiEizgrxKKZAx9alY8eO6fT4K1asEEZGRsLOzk7s3LmzWI6Z\nn3MOCgoqukC0wMnJSSxYsEDUr19fWFtbi+HDh4tHjx6pl69evVrUrFlT2NjYiF69eonw8HD1ssTE\nRGFkZCS8vb2FEEKkp6eLSpUqiQkTJuR6zFGjRony5csLa2trcefOHeHl5SXc3d2zrBMQECDq1asn\nzM3NRbt27cRHH30k2rZtq14OiBs3bgghhBgxYoSYM2eOetmxY8eEg4PDC8d98uSJsLGxEXv27Hlh\n2cSJE0X//v2FEEI8fvxYvP/++6J69erCyspKNGvWTOzevTvL+nFxcWLatGmiWrVqwtzcXNSsWVNM\nnz5dPHjwINdzf56Xl5dA1WRI/eXl5SWEECIsLEyYm5uLsLAw9fqLFy8WlStXFpaWlmLkyJHi6dOn\n6mW3bt0SHTt2FCYmJqJu3bri8OHDWY5VmG0zy+l3GlU13jxzrCKK6FbwvDRv3lycOZP/ku+tvu1P\namoqZ8fuJmzYcICc67kvqgkvWUDVxlDOGnouUn2YWgrpqqGBEAJFUTh+/DgbNmzgq6++onz58sVy\n7Pyc85UrV6hfv37RBlQMZIVE/aDJOef0O60oylkhRPO8jlH2x9zrdIFei3UdRamTlJTEnDlzMDY2\n5ssvv9R6oS9JkoqWfoy5S/ny66+/0rBhQ1asWEFKSkqRFfqSJKnoyOQuqcXGxjJq1Ci6deuGiYkJ\nx48fZ9myZXpZ6EuSSjuZ3CW16Ohodu7cyezZs7lw4QLt2rXTdUiSJBVQ2R1zv38N0oq+MFJpd+/e\nPbZu3cr06dPVhb40qb8hSVLJVjav3KMuw8qWkBwPxma6jqZEEkKwceNGXF1dmT17trrQl0zsklQ2\nlM3k/iRO9b3zx9Dhf7qNpQQKDQ2le/fujBw5EldXV1noS5LKoLI1LBN3Gw7NgSt7VI+rtQCTgncP\nL4tSU1Pp1KkTDx48YOXKlUyYMAEDWWNHksqcsvW/esvg/xJ718+gWkvdxlOCBAcHqwt9bdiwgUuX\nLjFp0iSZ2ItJaW2zJ5VeZet/9rNHUKszTA+CNlPA2ETXEelcSkoKCxYsoEGDBqxcuRKATp06ZalC\nKJU92mqzB6hr2eTXrVu3MDAwYNKkSVmeDw0NRVGUF2q7jBw5krlz56of3717F09PT6pWrYqlpSUu\nLi54eXnx6NGjfMWRV1u8zB4+fMjgwYOpWLEiFStWxN3dPUslx06dOlGpUiWsrKxo0qQJu3fvVi/z\n9/fHwMBA3SDFwsJCXQUzOTkZT09PnJycsLS0pF27dgUqapYfZSu5g6rhtbWDrqMoEc6dO0fLli2Z\nM2cOb775prpQlSQVBz8/P2xtbdm2bVuWPqKaePjwIa1bt+bJkyecOnWKxMREDh8+TFxcXJ7dn56X\nW1u8582dO5fY2FhCQkK4efMmUVFReHt7q5cvW7aMu3fvkpCQgK+vLx4eHlnKEmdukJKUlMSIESMA\n1XCoo6MjAQEBxMfHM2fOHAYNGkRoaGi+ziU/yl5ylwBYvnw5LVu25N69e/z000/s2LEDOzs7XYel\n13TdZi89PR0fHx9q1apFhQoVGDRoEA8fPgTg6dOneHh4UKFCBWxsbGjRogVRUVHMmTOHEydOMGXK\nFCwsLJgyZYrG5+vn58f8+fMxNjZm7969+XqtvvrqKywtLdm8ebO6OqajoyPLli2jcePGGu8nv63t\nbt26Rd++fbGyssLa2pp+/fpl6ZTUuHFjjIxUH1UqikJKSkqWUs45MTc3x9vbG2dnZwwMDOjRowc1\natTg7NmzGp9LfsnkXsZklApo1qwZw4cPJygoiH79+uk4Kgl032Zv+fLl/PzzzwQEBBAZGYmtrS2T\nJ08GVCV64+PjCQ8PJyYmhjVr1mBqaspnn31G+/bt+frrr0lKSuLrr7/W6FxPnDhBREQEQ4YMYdCg\nQQVqsffWW2/l+plQ48aNc2yxlzEUlN/WdpMnT2bfvn3ExsYSGxvLjz/+SI8ePbKs07t3b0xMTGjV\nqhVubm40b/5fDa/o6Gjs7OyoUaMG06dPz3EIKTo6muvXr9OgQdH1HShbs2X0WGJiIrNnz6ZcuXIs\nXryY9u3b0759e12HpTsHZsG9f4r2GFUaQQ8fjVfXdZu9tWvX8vXXX6vH4r29valevTqbNm3C2NiY\nmJgYgoODady4cZ7t7fKyceNGevToga2tLUOHDqVDhw5ER0dTuXJljbaPiYmhatWqua5z8eLFPPeT\n39Z2L7/8Ms+ePVPf79G5c+cXPjPYt28fKSkpHDlyhKtXr6r/ALm4uHDhwgVcXFwICwtjxIgRzJgx\ng7Vr12bZPiUlhTFjxjBixAhcXFzyPIeCklfuZcDBgwdp2LAhq1atUtdylkoeXbfZCwsLo1+/fuqr\n2/r162NoaEhUVBTDhg2jW7duDBkyBHt7e2bOnPlCCzxNPXnyhB9++EHdYq9169ZUr16dLVu2AKiH\nNYqjxV5+W9sNHDiQunXrkpiYSEJCArVq1cLDw+OF9YyNjenRoweHDh1izx7VDL0qVarg6uqKgYEB\nNWrUYNGiRS90tUpPT2fYsGEYGxtr/C6ooOSVeykWExPDjBkz8PPzo379+pw8eZLWrVvrOqySIR9X\n1MVF1232HB0d2bBhA23bts12Gy8vL7y8vAgNDaVnz57Uq1cPT0/PfBeO27VrFwkJCUyaNEn9eUFc\nXBx+fn68++67VK1aFWNjY0JDQ7PUK7916xZdunQBVC32du3ahZeXV45DMw0aNMjyumXm4eHBmjVr\n8myL97zAwEBWrVqlbus3YcKEXGsspaam5vgBr6IoWS60hBB4enoSFRXF9u3b1X/Iioq8ci/FYmJi\n2LVrFx999BHnz5+Xib2E03WbvQkTJjBnzhx1Qrx//756Kt+xY8f4559/SEtLw8rKCmNjY3Wzczs7\nO0JCQjQ+z40bNzJ69Gj++ecfdYu9kydPcuHCBf755x8MDQ3p378/c+bMISYmhpSUFLZu3UpQUJB6\nfHvGjBkkJCQwYsQIdbx37txhxowZ6uGYy5cv59hib82aNUD+W9u1aNGCdevW8eTJE548eYKvry9N\nmjQB4OrVqxw4cIAnT56QkpLC5s2b1e+mQDUV8vbt2wghCA8PZ9asWbz55pvqfU+cOJErV66wd+9e\nTE1NNX49C0yTdk1Ad+AaEAzMymb5DCAIuAgcBZzy2meRtNlb0kiIH8cVaL8lXUbLucjISPHFF1+I\n9PR0IYQQDx8+1GFURUu22VPRVpu9tLQ0sXjxYlG3bl1hYWEhatasKWbPni2EEGLLli2ibt26wszM\nTFSuXFlMnTpVpKSkCCGE+OOPP0SdOnWEjY2NmDp1aq7HjIiIEIaGhuLixYsvLOvRo4d47733hBCq\n31tPT09hb28vbGxsRJs2bcTvv/+eZf07d+6IUaNGCTs7O2FhYSHq1asnvL29s7xumsittd3mzZuF\nq6ur+nFISIjo3bu3KF++vLC1tRXdunUT169fF0KofsdatmwpLCwshLW1tWjevLn46aef1NsuXrxY\n2NvbC1NTU1GtWjUxZcoUkZCQIIQQIjQ0VACiXLlywtzcXP21efPmHOMubJs9TRK7IXATqAm8BAQC\nrs+t0wkw+/fnicD2vPYrk3v+/Pbbb2L9+vXC2tpamJiYqH/hyrKylNw1lZEM9Ik85+wVNrlrMizT\nEggWQoQIIZ4B24A3M68ghDgmhHj878M/Ae3cGicBqrHI//3vf3h6etKkSRMCAwNloS9JknKlyQeq\nDkDmWfoRQKtc1vcEsr2vVlGUccA4UI3jZczfzY/U1FSEEPj7+2Mbp6r+eOvf/bR6+pT4qHtcLcB+\nS6q0tDQ8PDyIj49n+vTp9O7dm8jISPVMi7IsKSlJ498Ra2trEhMTizagYpCWllYmziM/5Dln7+nT\npwXKkRk0Se7ZfVSe7Vw7RVE8gOZAtvOzhBC+gC9A8+bNhaad7TMzurWC1NRU3NzcCFu/AYAmGfu5\nYIKpXRWqFGC/Jc2NGzeoWbMmhoaGbN26lejoaAYNGqTrsIqVv78/mv6OXLlyJc9u8qVB5lkd+kKe\nc/ZMTExo1qxZgY+hybBMBOCY6XE14IXLRkVRXgfmAG8IIfJXSKKw0lLgt8/g8cNiPWxRSElJYf78\n+TRs2FA9D9bNzU3jmz8kSZJAsyv300AdRVFqAHeAIUCWSaKKojQD1gLdhRDRWo8yL/evwvFF8JIF\nOBTuzjpdOnPmDJ6enly8eJEhQ4bw9ttv6zokSZJKqTyv3IUQqcAU4BBwBdghhLisKMoniqK88e9q\nXwAWwA+KolxQFGVPkUWcfZCq7/3WQqtxxXpobVm2bBmtWrXiwYMH7N69m61bt8qrdUmSCkyjO1SF\nEPuB/c8993Gmn1/Xclx6QwiBoig0b94cT09PFi1ahI2Nja7DkiSplJPlB3QkISGBDz74ABMTE5Ys\nWULbtm1zvC1ckiQpv2T5AR3Yv38/DRo0wNfXFyMjI1noSyoxevTooe4eJJVuMrkXowcPHuDh4UGv\nXr2wtrbmjz/+4Isvvsh3YSZJyo6iKAQHBxdqHwcOHFB3DyooNzc3bG1tX+i+lF27Pn9//yxlb4UQ\nLF++nIYNG2Jubk61atUYOHAg//yTv/LNQgg++OADKlSoQIUKFZg5c2aOF1FCCD777DOqV6+OlZUV\nQ4YMyVJJ8s6dO7z55puUL1+eatWqqevWZNi7dy8NGzbEwsKCNm3aEBQUpF6WnJzM9OnTsbe3x9bW\nlkmTJhW42mZ+ldrkHrt9B49Pn9Z1GPkSGxvL3r178fLy4ty5c7Rqldu9YJKkXc/3LC0KoaGhnDhx\nAkVR1KVw82PatGksW7aM5cuX8/DhQ65fv07fvn355Zdf8rUfX19ffv75ZwIDA7l48SL79u17oa56\nBj8/PzZt2sTJkyeJjIzkyZMn6mqWoKowWaNGDaKiovjll1/48MMPOXbsGKC6H8Xd3Z01a9YQFxdH\nnz59eOONN9SvtY+PD2fOnOHSpUvqrlAZTVqKnCY1Coriq7C1ZUI9homgei7i4bbtQkQGCuFlJUTQ\n3gLtsyhFRESIhQsXqgt9xcbGFmg/+amzUlaUldoyn3/+uejfv3+W59555x11Ea64uDgxevRoUaVK\nFVG1alUxZ84ckZqaql7X19dXuLi4CAsLC1G/fn1x9uzZF47Rvn17AQgzMzNhbm4utm3bJo4dOyYc\nHByEj4+PsLOzEx4eHuLhw4eiV69eomLFitkWKevYsaP45ptvhBBCfPvtt6Jt27bivffeEzY2NsLZ\n2Vns378/13OdN2+eaNOmjZg+fbro1atXlmWZ953h2LFjwt7eXgghxPXr14WBgYH466+/8npJ89S6\ndWuxdu1a9eN169aJVq1aZbtu//79xaJFi9SPT548KcqVKycePXokEhMTBSCio6PVy8eOHSs8PDyE\nEEKsWLFC9OzZU70sLS1NmJiYiCNHjgghhHjllVfEjh071Mu///57Ua1atRJTW6bEMmvRAttXHWHv\nO7oO5QVCCL755htcXV3x9vZW13yWM2H0z9tvv83+/fvVb/XT0tLYsWOHuqb4iBEjMDIyIjg4mN9/\n/51ff/1VPXzxww8/4O3tjZ+fHwkJCezZs0fdJSiz48ePA6p65ElJSepywvfu3ePhw4eEhYXh6+tL\neno6o0aNIiwsjNu3b2NqapprX9S//vqLevXq8eDBA2bOnImnp2eunxH5+fnh7u6Ou7s7hw4dIioq\nSuPX6ejRo1SrVo2WLVvmuI6Pj0+OrfUy/9+6fPmyulQv5N5aLyMZZn6cnJzMjRs31M8/v/zSpUs5\nbpvX8oiIiCylmItK6Z4tk5oMp9dB5Hmo0w3sC36rrjbdvHmTsWPHcuzYMdzc3NTNj6Xis/DvhVx9\neLVIj+FS3oUPWn6Q53pOTk68/PLL/PzzzwwfPpzffvsNMzMzXn31VaKiojhw4ABxcXGYmppSqVIl\npk+fjq+vL+PHj2fdunXMnDmTFi1aAOT798jAwIB58+ZRrlw5AHWT6Axz5syhU6dOucY+duxYQPVH\naNKkSURFRVGlSpUX1v39998JCwtj0KBBVKxYkVq1arFlyxamT5+uUayatNabNWsWs2bNynNfz7fX\ns7a2JikpST31OLMePXqwaNEiBg0ahK2tLQsXLgTg8ePHWFpa0rZtWz799FO++OILgoKC+PHHH9V1\n9Lt06cKsWbPw9/enTZs2LFy4kGfPnvH48WP1vpctW0anTp1IS0tj+fLlgKpbVVEr1Vfu3L8Kl3ZC\nOWtw3wHWDrqOiNTUVDp37syZM2dYu3YtR48elYldYujQoWzduhWALVu2qK/aw8LCSElJoWrVqtjY\n2ODo6Mj48eOJjlbd6B0eHk6tWrUKfNxKlSphYmKifvz48WPGjx+Pk5MTVlZWdOjQgbi4ONLS0rLd\nPnMSNzMzA1SJMzsbN26ka9euVKxYUX3OmWfeGBkZFUtrPXixvV5CQgIWFhbZTl4YPXo0b7/9Nm5u\nbjRo0ED9xy6j1+z333/PrVu3cHR0ZOLEibi7u6uXubi4sHHjRqZMmULVqlV58OABrq6u6uVz5syh\nWbNmNG3alDZt2tC3b1+MjY1zbLKiTaX7yl2kg1M7eMtX15Fw7do1atWqhZGRERs3bqRWrVrqf2Cp\n+GlyRV2cBg4cyHvvvUdERAS7du3i1KlTgKr1Xbly5Xjw4AFGRkYvFJRydHTMsY2bJp5PZosXL+ba\ntWv89ddfVKlShQsXLtCsWbNCT8d98uQJO3bsIC0tTf0HITk5mbi4OAIDA2nSpAnVq1cnNDQ0y3YZ\nSRNUzagnT57MmTNnaN68ebbHWbBgAQsWLMgxjow/PA0aNCAwMFA9xBMYGEiDBg2y3Sbj3c28efMA\n+PXXX3FwcFC3OXRycmLfvn3q9YcOHZpl6GjAgAEMGDAAULUT3LBhg/qdlqmpKV9//bW6TpSvry+v\nvPKKustVUSrdV+4ApjY6vWJ/9uwZ8+bNo1GjRqxcuRJQtUWTiV3KrFKlSri5uTFq1Chq1Kih7h1a\ntWpVunbtynvvvUdCQgLp6encvHmTgIAAAMaMGcOXX37J2bNnEUIQHBycY99QTdrhJSYmYmpqio2N\nDQ8fPlQntML6+eefMTQ0JCgoSN1a78qVK7Rv3x4/Pz8ABg8ezLfffsvff/+NEILr16+zZMkS9TBR\nnTp1mDRpEm+//Tb+/v48e/aMp0+fsm3bNnx8VD1xP/zwwxxb62V+RzF8+HC++uor7ty5Q2RkJIsX\nL2bkyJHZxv7w4UNu3ryJEIKgoCBmzJjBxx9/rO7deuXKFRITE3n27BmbN2/m119/ZcaMGertz549\nS1paGvfv32f8+PH06dNHPb0z4/hCCP78808+/fRTrb3medLkU9ei+NLGbJnQ1xoKsXVogfajDX/9\n9Zdo2LChAMTQoUPF/fv3i+xYcrZM7krybJkMfn5+AsgyM0MI1WyZCRMmCAcHB2FlZSWaNm0qtm7d\nql6+evVqUbduXWFubi4aNGggzp07l+3+V69eLapUqSKsra3F9u3b1bNlMrtz547o2LGjMDc3F3Xq\n1BFr1qwRgLqlXnazZTIDxI0bN144drdu3cSMGTNeeH779u3Czs5Ovf/169cLV1dXYWlpKWrVqiU+\n//xzERcXp14/PT1dLF26VLi6ugpTU1Nhb28vBg0aJC5dupTj65qd9PR08b///U/Y2toKW1tb8b//\n/U89Y00IIczNzcXx48eFEEJcu3ZN1K1bV5iamorq1auLxYsXZ9nXkiVLRMWKFYWZmZlo27atOH36\ndJblbdu2FRYWFsLW1laMGzdOJCUlqZcFBAQIJycnYWpqKurWratuq1ccs2Vkci+gJUuWCAMDA+Hg\n4CD27i36KZgyueeuNCR3TciWc/pBToUsgcS/Y5MtW7Zk7NixXL58md69e+s4KkmSpKxK9weqxSg+\nPp6ZM2diamrK0qVLadOmDW3atNF1WJIkSdmSV+4a2Lt3L66urqxbt45y5coVemaBJElSUZPJPRf3\n799n6NChvPHGG1SoUIE///yThQsXykJfkiSVeDK55yI+Pp79+/czb948zpw5o567KkmSVNKVyjF3\nQ5EG4X+pGmMbaPdmgPDwcDZv3sysWbOoXbs2YWFhWW5jliRJKg1K5ZW7IWmqxG5WAdq/p5V9pqen\ns2bNGho0aMD8+fPVdwXKxC5JUmlUKpO7mnkFqNok7/XycOPGDV577TUmTpxIy5Yt+eeff2Q9GEmS\nSrVSOSyjTampqXTp0oW4uDjWr1/PqFGj5AemkiSVeqX7yr0Qrly5QmpqKkZGRmzatImgoCBGjx4t\nE7tUammjzR6At7c3Hh4e+d5OCEHNmjVxdXV9YZmzszNHjhzJ8tx3331Hu3bt1I+fPXuGt7c3derU\nwdzcHGdnZ0aPHv1CsbG8JCcnM3r0aKysrKhSpQpfffVVruvm1AYvOTkZT09PnJycsLS0pFmzZhw4\ncCDL9uvWraN27dpYWFjQvXt3IiMjXzjGs2fPcHFxKfZ6U3qX3JOTk/Hy8qJx48bqSm3t27fH3t5e\nxyFDs2YAAA18SURBVJFJUul2/PhxoqOjCQkJ4XQBWmAOGDCAPXv2sGXLFuLj4wkMDOSVV17h6NGj\n+dqPt7c3N27cICwsjGPHjrFo0SIOHjyY7bq5tcFLTU3F0dGRgIAA4uPj+fTTTxk0aJD6j01AQAAf\nfvghu3fv5uHDh9SoUYO33377hWN88cUXVK5cOX8vhjZoUqOgKL4KU1vm1bXdRWi7GiK0X7d8bXvq\n1Cnh6uoqADFs2DDx4MGDAsWgC7K2TO5Kcm0ZXbXZE0KIvXv3iiZNmghra2vRunVrERgYqN7Gx8dH\n2NvbCwsLC1G3bl1x5MgRceDAAWFsbCyMjIyEubm5aNy4scbnOWrUKDF06FDRr18/MXny5CzLnJyc\nxOHDh7M8l1GYLCEhQRw+fFiYmJiI27dva3y8nNjb24tDhw6pH8+dO1cMHjw423VzaoOXk0aNGomd\nO3cKIYR47733xKRJk9TL7ty5IwARHBysfi4kJES4uLiI/fv3ZyniJmvLZKPjhVhm7rjL0zjjfG23\nePFi2rRpQ2JiIvv378fPzy/bdmWSpG26arN37tw5Ro8ezdq1a4mJiWH8+PG88cYbJCcnc+3aNb7+\n+mtOnz5NYmIihw4dwtnZme7du/Phhx8yePBgkpKSCAwM1OgcHz9+zM6dO9Ut9rZt28azZ880fo2O\nHDlCy5Yt1bXdszNp0qQc2+s1btwYUDWhj4yMLFSLvZza4EVFRXH9+nV1XfjstgXULfYApk6dyoIF\nCzA1NdXkZdCqUveBauugeKrff4aJTQpWbRvluX56ejoGBga0bt2aCRMm4OPjg5WVVTFEKunSvQUL\nSL5StG32ytV3ocqHH+a5nq7a7H3zzTeMHz+eVq1aAao/IgsWLODPP//EwcGB5ORkgoKCqFSpEs7O\nzgV6DTL89NNPlCtXjq5du5KWlkZqaiq//PIL/fr102h7TVrsrVq1ilWrVuW6TkZN9+db7CUmJma7\nfk5t8B4/fpxlHykpKbi7uzNixAh1rfaePXsyePBgJkyYQJ06dfjkk09QFEXdYm/Xrl2kpqbSr18/\n/P39c38BikCpu3I3oBx3Klrg1DkG29ez79YCqo4onp6eTJs2DYA2bdqwatUqmdglndBFm72wsDAW\nL16c5Qo3PDycyMhIateuzdKlS/H29qZy5coMGTIk2w8DNbVx40YGDRqEkZER5cqV46233tJJiz0L\nCwuAF1rsZe5ulVlObfAyj5Gnp6czbNgwXnrpJfXndKDqHDVv3jz69++Pk5MTzs7OWFpaUq1aNR49\nesTMmTNZsWJFoc+pwDQZuymKr4KOue/v/KY42KGLEF5WQlzYlu06u3btElWrVhWGhoZi9uzZWYr0\nl1ZyzD13JXnMXQghoqOjhYmJiQgPDxfW1tbqeCMjI4WJiYm6mcXzY7Fdu3YVS5cu1egYPNdIY9y4\ncWL+/Pl5bhcfHy+GDBkiPDw8hBBCeHt7C3d3d42OKYQQ4eHhwsDAQFhZWQk7OzthZ2cnLC0thbGx\nsbqBTadOncSqVauybDf7/+3df2zU9R3H8ecbQZrWSgmsR7MOQStK1YSaOiEs0EWyMI34D2yamBXS\nTHQ//MN/hBjJ4hTCks2ExAlEmNmSbbCZbITYjIxZXQx1bYaIWBmdtKO1QUmhhbRXSu+1P+6obWnv\nvm3vR+/6fiSX3I/Pfe/9vu/du9fP5/v9fLZuVXV19bA+93Pnzo35Ops3b1ZBQcGol/Ly8sF2JSUl\nOnLkyODtF198ccw+95H27Nmj5cuXD96ORCLauHGjqqqq1NPTE/e5p0+fVn5+vjo7O3X8+HHNnDlz\n8P2YO3euZsyYoVAopLNnz/piHaOJV9zPnz+vDRs2CNCyZctGHXjKVl7c45vqxV2S1q5dqzVr1mjZ\nsmXD7l+3bp2effZZdXV16dKlS2publZdXZ0k6eDBgyotLVVjY6MikYjOnDmjlpaWUbcfCoWGDSQ2\nNDSotLRU9fX1ikQiunLlig4fPqzu7m59+umnOnr0qMLhsPr6+rRp0yZVV1dLiq7otHLlSg0MDATK\na/v27br77rvV0dEx7LJ48WLt2rVLkrR7924tWbJETU1NikQiamhoUCgUUm1t7WChe/TRR1VZWanG\nxkb19/eru7tbr7/+uvbt2zeu9/n555/XqlWr1NnZqaamJi1YsEC1tbWjtm1ra1N7e7sikYiOHTum\n0tLSYe/h5s2b9eCDD+ry5cs3PLe3t1cnT55UJBJRa2urVq9era1bt0qS+vv7h70Xb731lkpKStTR\n0aFr1655cR9NvOJ+5swZFRUV6ZVXXtHVq1cntP2pyot7fNlQ3NO9zJ4k1dbWqrKyUnPmzNGCBQu0\nfv16dXd368SJE3rggQcGl4d75JFH1N7eLkm6cOGCVq5cqaKiIlVUVCTM66677hos4kPt3LlT17/n\nAwMD2rFjh8rKylRYWKilS5fqjTfekPTVfyt9fX3atm2b7rjjDuXn52vhwoWqqalRa2trwhiGCofD\n2rRpkwoLC1VcXDxs2bzW1lYVFBQMbnOsZfAkqaWlRYBmz5497L+E620uXryo++67T/n5+QqFQtqy\nZcuwo5yGGrnk4ZQp7sBa4DTQDGwZ5fHZwIHY4x8AixJtM1nFvbW1VS+//PJg10uuLtnlxT2+bCju\nQeTq5zcez3l0KT8U0sxuAl4DvguUA0+Y2chT0GqAi5LKgFeBnZMaCIij4OYZFM3oISLx6wNHuOee\ne9i+ffvgRF9jDZw459x0EuRomW8CzZI+k3QV+CPw2Ig2jwHXh8b/DDxkKTqPf0lZHv3WTNWbPfx4\nx35WrFjBqVOnfKIv55wbIkhx/zpwbsjttth9o7aRdA3oAlJyhtD873+LJ//9BSe78vnN/v2DJ184\n55z7SpCTmEb7BT5yEdEgbTCzp4CnAEKh0AQP7C/nma07KCsrY968ebz77rsT2Eb2uXLlSkZOhMik\n8eQc70SVbDIwMJATeYyH5zy6cDg8qe98kOLeBgw9J7gUGHm2w/U2bWY2E5gDdI7ckKS9wF6AyspK\nVVVVTSDkqMk8NxvV1dV5znE0NTVxyy23ZP2snpcvX55240ae840kkZeXR0VFxYRfI0i3TANwp5kt\nNrObgceBQyPaHAKqY9fXA/+Ijeo6lxazZs2it7c302E4lxS9vb2DZ+9OVMLiHutD/wnwN6AJOCjp\nlJm9ZGbrYs32AfPMrBl4DtgyqaicG6fi4mLa29vp6enBf1e4bCWJnp4e2tvbJz1NcKCJwyS9Dbw9\n4r5tQ66HgQ2TisS5Sbg+Z9Dnn39+wxwm2SQcDpOXl5fpMNLKcx5u1qxZhEKhSc+DlXWzQjo3lltv\nvTXrJ4arq6ubVD9rNvKcUyPrZoV0zjmXmBd355zLQV7cnXMuB3lxd865HGSZOmzMzL4EWif49PnA\nhSSGkw085+nBc54eJpPzbZK+lqhRxor7ZJhZo6Sx19jLQZ7z9OA5Tw/pyNm7ZZxzLgd5cXfOuRyU\nrcV9b6YDyADPeXrwnKeHlOeclX3uzjnn4svWX+7OOefimNLF3czWmtlpM2s2sxtmmjSz2WZ2IPb4\nB2a2KP1RJleAnJ8zs0/M7CMzO2pmt2UizmRKlPOQduvNTGaW9UdWBMnZzL4X29enzOz36Y4x2QJ8\nthea2Ttmdjz2+X44E3Emi5ntN7MvzOzjMR43M9sVez8+MrP7kxpAkFW0M3EBbgL+C9wO3AycAMpH\ntPkRsDt2/XHgQKbjTkPO3wbyY9efmQ45x9oVAu8B9UBlpuNOw36+EzgOzI3dLs503GnIeS/wTOx6\nOdCS6bgnmfMq4H7g4zEefxioJbqS3XLgg2S+/lT+5T6lFuZOk4Q5S3pHUk/sZj3RlbGyWZD9DPBz\n4BdAOJ3BpUiQnH8IvCbpIoCkL9IcY7IFyVnA9Wk953Djim9ZRdJ7jLIi3RCPAb9VVD1QZGYlyXr9\nqVzcp9TC3GkSJOehaoj+5c9mCXM2swrgG5IOpzOwFAqyn5cAS8zsfTOrN7O1aYsuNYLk/DPgSTNr\nI7p+xE/TE1rGjPf7Pi5TeT73pC3MnUUC52NmTwKVwOqURpR6cXM2sxnAq8DGdAWUBkH280yiXTNV\nRP87+6eZ3SvpUopjS5UgOT8BvCnpl2a2AvhdLOdI6sPLiJTWr6n8y308C3MTb2HuLBIkZ8xsDfAC\nsE5SX5piS5VEORcC9wJ1ZtZCtG/yUJYPqgb9bP9VUr+ks8BposU+WwXJuQY4CCDpGJBHdA6WXBXo\n+z5RU7m4T8eFuRPmHOui2EO0sGd7PywkyFlSl6T5khZJWkR0nGGdpMbMhJsUQT7bfyE6eI6ZzSfa\nTfNZWqNMriA5/w94CMDMlhIt7l+mNcr0OgT8IHbUzHKgS1JH0rae6RHlBKPNDwP/ITrK/kLsvpeI\nfrkhuvP/BDQD/wJuz3TMacj578B54MPY5VCmY051ziPa1pHlR8sE3M8G/Ar4BDgJPJ7pmNOQcznw\nPtEjaT4EvpPpmCeZ7x+ADqCf6K/0GuBp4Okh+/i12PtxMtmfaz9D1TnnctBU7pZxzjk3QV7cnXMu\nB3lxd865HOTF3TnncpAXd+ecy0Fe3J1zLgd5cXfOuRzkxd0553LQ/wGMz66r/8+kUQAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fc2d6dba550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wv_model = LogisticRegression().fit(X_train_wv, y_train)\n",
    "\n",
    "for name, X, y, model in [\n",
    "    ('bow train', X_train_bow, y_train, bow_model),\n",
    "    ('bow test ', X_test_bow, y_test, bow_model),\n",
    "    ('vec train', X_train_wv, y_train, wv_model),\n",
    "    ('vec test ', X_test_wv, y_test, wv_model)\n",
    "]:\n",
    "    proba = model.predict_proba(X)[:, 1]\n",
    "    auc = roc_auc_score(y, proba)\n",
    "    plt.plot(*roc_curve(y, proba)[:2], label='%s AUC=%.4f' % (name, auc))\n",
    "\n",
    "plt.plot([0, 1], [0, 1], '--', color='black',)\n",
    "plt.legend(fontsize='large')\n",
    "plt.grid()\n",
    "\n",
    "assert roc_auc_score(y_test, wv_model.predict_proba(X_test_wv)[:, 1]) > 0.92, \"something's wrong with your features\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If everything went right, you've just managed to reduce misclassification rate by a factor of two.\n",
    "This trick is very useful when you're dealing with small datasets. However, if you have hundreds of thousands of samples, there's a whole different range of methods for that. We'll get there in the second part."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}