17. Tensorflow加速训练的优化器optimizer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensorflow加速训练的优化器optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实验：对比SGD和RMSProp的学习速度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "# 使用sklearn现成的波斯顿房价数据集\n",
    "from sklearn.datasets import fetch_california_housing\n",
    "\n",
    "# 做训练集测试集验证集拆分\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# 对数值数据归一化\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. 下载sklearn的波士顿房价数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用这个函数下载波斯顿房价数据集\n",
    "housing = fetch_california_housing()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 分割训练集、测试集、验证集\n",
    "X_train_full, X_test, y_train_full, y_test = train_test_split(housing.data, housing.target)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X_train_full, y_train_full)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用sklearn对数值数据做归一化\n",
    "scaler = StandardScaler()\n",
    "# 在训练集上做fit和transform\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "\n",
    "# 在测试集和验证集上做transform\n",
    "X_valid = scaler.transform(X_valid)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 把数据分割为Wide部分输入和Deep部分输入"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A是0~4共5列，B是2~7共6列\n",
    "X_train_A, X_train_B = X_train[:, :5], X_train[:, 2:]\n",
    "\n",
    "# valid和test做同样的处理\n",
    "X_valid_A, X_valid_B = X_valid[:, :5], X_valid[:, 2:]\n",
    "X_test_A, X_test_B = X_test[:, :5], X_test[:, 2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 搭建WideDeep模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 输入：wide和deep\n",
    "input_A = keras.layers.Input(shape=[5], name=\"wide_input\")\n",
    "input_B = keras.layers.Input(shape=[6], name=\"deep_input\")\n",
    "\n",
    "# 构造deep部分\n",
    "hidden1 = keras.layers.Dense(30, activation=\"relu\")(input_B)\n",
    "hidden2 = keras.layers.Dense(30, activation=\"relu\")(hidden1)\n",
    "hidden3 = keras.layers.Dense(30, activation=\"relu\")(hidden2)\n",
    "\n",
    "# 合并wide和deep\n",
    "concat = keras.layers.concatenate([input_A, hidden3])\n",
    "\n",
    "# 输出层\n",
    "output = keras.layers.Dense(1, name=\"output\")(concat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. 对参数优化的直观认识"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "给定参数w、学习率learning_rate、LOSS函数的梯度g，那么参数会这样更新：  \n",
    "w = w - learning_rate * g\n",
    "\n",
    "动量、自适应学习率等方法，都是在优化这个更新公式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.optimizer_v2.gradient_descent.SGD at 0x7f3ea051b890>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 优化器\n",
    "opt = tf.keras.optimizers.SGD(learning_rate=0.1)\n",
    "opt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=10.0>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 待优化参数\n",
    "w = tf.Variable(10.0)\n",
    "w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function __main__.<lambda>()>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 梯度：d(loss)/d(var1) = 2*var1\n",
    "# w初始化为10，所以首次梯度是20\n",
    "loss = lambda: w ** 2\n",
    "loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=() dtype=int64, numpy=1>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 执行参数计算\n",
    "opt.minimize(loss, [w])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8.0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# w = 10 - 0.1*20，第一个1是之前的数字，0.1学习率，后面的20是梯度\n",
    "w.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. 对比SGD和RMSProp的训练速度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SGD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.8352 - val_loss: 0.5722\n",
      "Epoch 2/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.5100 - val_loss: 0.5064\n",
      "Epoch 3/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.5434 - val_loss: 0.4842\n",
      "Epoch 4/50\n",
      "363/363 [==============================] - 0s 994us/step - loss: 0.4354 - val_loss: 0.4780\n",
      "Epoch 5/50\n",
      "363/363 [==============================] - 0s 980us/step - loss: 0.4393 - val_loss: 0.4967\n",
      "Epoch 6/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.4116 - val_loss: 0.4258\n",
      "Epoch 7/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4017 - val_loss: 0.4407\n",
      "Epoch 8/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3832 - val_loss: 0.4171\n",
      "Epoch 9/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3727 - val_loss: 0.3955\n",
      "Epoch 10/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3668 - val_loss: 0.3956\n",
      "Epoch 11/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3626 - val_loss: 0.3948\n",
      "Epoch 12/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3639 - val_loss: 0.3816\n",
      "Epoch 13/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3515 - val_loss: 0.3775\n",
      "Epoch 14/50\n",
      "363/363 [==============================] - 0s 996us/step - loss: 0.3476 - val_loss: 0.3783\n",
      "Epoch 15/50\n",
      "363/363 [==============================] - 0s 967us/step - loss: 0.3444 - val_loss: 0.3763\n",
      "Epoch 16/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3425 - val_loss: 0.4075\n",
      "Epoch 17/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3590 - val_loss: 0.3715\n",
      "Epoch 18/50\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.3364 - val_loss: 0.3664\n",
      "Epoch 19/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3373 - val_loss: 0.3634\n",
      "Epoch 20/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3347 - val_loss: 0.3638\n",
      "Epoch 21/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3339 - val_loss: 0.3668\n",
      "Epoch 22/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3300 - val_loss: 0.3719\n",
      "Epoch 23/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3344 - val_loss: 0.3545\n",
      "Epoch 24/50\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.3292 - val_loss: 0.3701\n",
      "Epoch 25/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3271 - val_loss: 0.3937\n",
      "Epoch 26/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3286 - val_loss: 0.3546\n",
      "Epoch 27/50\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.3267 - val_loss: 0.3552\n",
      "Epoch 28/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3226 - val_loss: 0.3478\n",
      "Epoch 29/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3204 - val_loss: 0.3521\n",
      "Epoch 30/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3214 - val_loss: 0.3511\n",
      "Epoch 31/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3203 - val_loss: 0.3751\n",
      "Epoch 32/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3217 - val_loss: 0.3667\n",
      "Epoch 33/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3194 - val_loss: 0.3569\n",
      "Epoch 34/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3202 - val_loss: 0.3491\n",
      "Epoch 35/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3156 - val_loss: 0.3470\n",
      "Epoch 36/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3188 - val_loss: 0.3465\n",
      "Epoch 37/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3143 - val_loss: 0.3563\n",
      "Epoch 38/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3190 - val_loss: 0.3940\n",
      "Epoch 39/50\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3138 - val_loss: 0.3674\n",
      "Epoch 40/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3108 - val_loss: 0.3501\n",
      "Epoch 41/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3133 - val_loss: 0.3423\n",
      "Epoch 42/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3164 - val_loss: 0.3406\n",
      "Epoch 43/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3125 - val_loss: 0.3493\n",
      "Epoch 44/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3110 - val_loss: 0.3362\n",
      "Epoch 45/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3105 - val_loss: 0.3677\n",
      "Epoch 46/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3191 - val_loss: 0.3456\n",
      "Epoch 47/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3092 - val_loss: 0.3606\n",
      "Epoch 48/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3072 - val_loss: 0.3557\n",
      "Epoch 49/50\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.3084 - val_loss: 0.3424\n",
      "Epoch 50/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3060 - val_loss: 0.3548\n"
     ]
    }
   ],
   "source": [
    "# 构造model，注意多输入，单输出\n",
    "model_sgd = keras.models.Model(inputs=[input_A, input_B], outputs=[output])\n",
    "\n",
    "# 模型编译\n",
    "model_sgd.compile(\n",
    "    loss=\"mse\", \n",
    "    optimizer=keras.optimizers.SGD()\n",
    ")\n",
    "\n",
    "# 模型训练\n",
    "history_sgd = model_sgd.fit(\n",
    "    (X_train_A, X_train_B), \n",
    "    y_train, \n",
    "    epochs=50,\n",
    "    validation_data=((X_valid_A, X_valid_B), y_valid)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": false
   },
   "source": [
    "#### RMSProp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3019 - val_loss: 0.3368\n",
      "Epoch 2/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3010 - val_loss: 0.3567\n",
      "Epoch 3/50\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.3006 - val_loss: 0.3418\n",
      "Epoch 4/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2998 - val_loss: 0.3585\n",
      "Epoch 5/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3015 - val_loss: 0.3341\n",
      "Epoch 6/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2990 - val_loss: 0.3293\n",
      "Epoch 7/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2961 - val_loss: 0.3403\n",
      "Epoch 8/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2928 - val_loss: 0.3347\n",
      "Epoch 9/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2918 - val_loss: 0.3352\n",
      "Epoch 10/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2943 - val_loss: 0.3531\n",
      "Epoch 11/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2904 - val_loss: 0.3333\n",
      "Epoch 12/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2888 - val_loss: 0.3383\n",
      "Epoch 13/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2909 - val_loss: 0.3329\n",
      "Epoch 14/50\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.2844 - val_loss: 0.3245\n",
      "Epoch 15/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.2859 - val_loss: 0.3282\n",
      "Epoch 16/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2817 - val_loss: 0.3266\n",
      "Epoch 17/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2811 - val_loss: 0.3330\n",
      "Epoch 18/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2795 - val_loss: 0.3235\n",
      "Epoch 19/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2778 - val_loss: 0.3173\n",
      "Epoch 20/50\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.2753 - val_loss: 0.3205\n",
      "Epoch 21/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2723 - val_loss: 0.3212\n",
      "Epoch 22/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2771 - val_loss: 0.3277\n",
      "Epoch 23/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2755 - val_loss: 0.3386\n",
      "Epoch 24/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2718 - val_loss: 0.3148\n",
      "Epoch 25/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.2762 - val_loss: 0.3192\n",
      "Epoch 26/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2697 - val_loss: 0.3290\n",
      "Epoch 27/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.2717 - val_loss: 0.3114\n",
      "Epoch 28/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2684 - val_loss: 0.3101\n",
      "Epoch 29/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.2672 - val_loss: 0.3230\n",
      "Epoch 30/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.2633 - val_loss: 0.3258\n",
      "Epoch 31/50\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.2652 - val_loss: 0.3110\n",
      "Epoch 32/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2672 - val_loss: 0.3261\n",
      "Epoch 33/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.2641 - val_loss: 0.3201\n",
      "Epoch 34/50\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.2634 - val_loss: 0.3067\n",
      "Epoch 35/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2624 - val_loss: 0.3084\n",
      "Epoch 36/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2592 - val_loss: 0.3182\n",
      "Epoch 37/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2684 - val_loss: 0.3131\n",
      "Epoch 38/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2682 - val_loss: 0.3147\n",
      "Epoch 39/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2588 - val_loss: 0.3029\n",
      "Epoch 40/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2580 - val_loss: 0.3507\n",
      "Epoch 41/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2588 - val_loss: 0.3115\n",
      "Epoch 42/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2562 - val_loss: 0.3000\n",
      "Epoch 43/50\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.2577 - val_loss: 0.3000\n",
      "Epoch 44/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2560 - val_loss: 0.3004\n",
      "Epoch 45/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2535 - val_loss: 0.3081\n",
      "Epoch 46/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2571 - val_loss: 0.3312\n",
      "Epoch 47/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2559 - val_loss: 0.3216\n",
      "Epoch 48/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2530 - val_loss: 0.3430\n",
      "Epoch 49/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2507 - val_loss: 0.3057\n",
      "Epoch 50/50\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2506 - val_loss: 0.3092\n"
     ]
    }
   ],
   "source": [
    "# 构造model，注意多输入，单输出\n",
    "model_rmsprop = keras.models.Model(inputs=[input_A, input_B], outputs=[output])\n",
    "\n",
    "# 模型编译\n",
    "model_rmsprop.compile(\n",
    "    loss=\"mse\", \n",
    "    optimizer=keras.optimizers.RMSprop()\n",
    ")\n",
    "\n",
    "# 模型训练\n",
    "history_rmsprop = model_rmsprop.fit(\n",
    "    (X_train_A, X_train_B), \n",
    "    y_train, \n",
    "    epochs=50,\n",
    "    validation_data=((X_valid_A, X_valid_B), y_valid)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 查看速度对比"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sgd</th>\n",
       "      <th>rmsprop</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.835152</td>\n",
       "      <td>0.301941</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.510029</td>\n",
       "      <td>0.300994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.543367</td>\n",
       "      <td>0.300553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.435356</td>\n",
       "      <td>0.299793</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.439304</td>\n",
       "      <td>0.301549</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        sgd   rmsprop\n",
       "0  0.835152  0.301941\n",
       "1  0.510029  0.300994\n",
       "2  0.543367  0.300553\n",
       "3  0.435356  0.299793\n",
       "4  0.439304  0.301549"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({\n",
    "    \"sgd\": history_sgd.history[\"loss\"],\n",
    "    \"rmsprop\": history_rmsprop.history[\"loss\"]\n",
    "})\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f3e704253d0>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsIAAAHSCAYAAADmLK3fAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXhcV33/8c+ZXdKM9s22ZMuL5H1JYpw9NjQhSQGzh5imhKaQAAmUtcCvkELaUii0KaUBGkpogAABCiWQHchKVjtxnHiRV9mSbcmy9n228/vjjhbLsizbkkbWfb+eZ557586dmSM9D+Hjo+/5HmOtFQAAAOA2nnQPAAAAAEgHgjAAAABciSAMAAAAVyIIAwAAwJUIwgAAAHAlgjAAAABcyZeuLy4sLLQVFRXp+noAAAC4xKZNm45aa4uGX09bEK6oqNDGjRvT9fUAAABwCWPM/pGuUxoBAAAAVyIIAwAAwJUIwgAAAHCltNUIAwAA4MRisZjq6urU29ub7qGcNUKhkMrKyuT3+8d0P0EYAABgCqqrq1MkElFFRYWMMekezpRnrVVTU5Pq6uo0d+7cMb2H0ggAAIApqLe3VwUFBYTgMTLGqKCg4JRm0AnCAAAAUxQh+NSc6u+LIAwAAIBxV1FRoaNHj6Z7GKMiCAMAAMCVWCwHAACAEXV1demaa65RXV2dEomEvvjFLyoSieiTn/ykCgsLde6552rv3r363e9+p6amJm3YsEGNjY1as2aNrLXpHv5JEYQBAACmuC//dqu2HWof189cMjNbf/+WpaPe89BDD2nmzJm6//77JUltbW1atmyZnnzySc2dO1cbNmwYHOOXv6xLLrlEt956q+6//37deeed4zreiUBpBAAAAEa0fPly/f73v9dnP/tZPfXUU9q3b5/mzZs30J5saBB+8skndd1110mS3vSmNykvLy8tYz4VzAgDAABMcSebuZ0oVVVV2rRpkx544AF9/vOf1xVXXDHq/WdblwtmhAEAADCiQ4cOKTMzU9ddd50+/elP65lnntHevXtVU1MjSbr33nsH7r3ssst0zz33SJIefPBBtbS0pGPIp4QZYQAAAIzo1Vdf1Wc+8xl5PB75/X595zvf0eHDh3XVVVepsLBQa9asGbj37//+77Vhwwade+65Wrt2rWbPnp3GkY8NQRgAAAAjuvLKK3XllVcec62zs1M7duyQtVY333yzVq9eLUkqKCjQI488MnDf7bffPqljPR2uKo1IJK3aumOKxpPpHgoAAMBZ6Xvf+55WrVqlpUuXqq2tTTfddFO6h3TaXBWEn9/XpJW3PaJN+6d+zQoAAMBU9IlPfEKbN2/Wtm3bdM899ygzMzPdQzptrgrCkaBfktTRG0vzSAAAAJBu7grCIackurMvnuaRAAAAIN1cFYTDqSDc0UsQBgAAcDtXBWFmhAEAANDPVUE46PMq4PWonRphAAAA13NVEJacWeFOSiMAAABOibVWyeTEtqBNJBIT+vnDuS4Ih0M+aoQBAADGoKamRosXL9ZHPvIRnXvuufJ6vfrsZz+r8847T5dffrleeOEFrVu3TvPmzdN9990nSdq6davWrFmjVatWacWKFdq1a5dqamq0aNEiXX/99VqxYoXe9a53qbu7W5JUUVGh2267TZdccol+8YtfaPPmzbrgggu0YsUKvf3tbx/YqnndunX6+Mc/rosuukjLli3TCy+8cMY/n+t2louEfNQIAwCAs8uDn5PqXx3fzyxdLl391ZPeVl1drR/84Af69re/LWOM1q1bp6997Wt6+9vfri984Qt69NFHtW3bNl1//fVav369vvvd7+pv/uZv9Bd/8ReKRqNKJBJqaGhQdXW1vv/97+viiy/WDTfcoG9/+9v69Kc/LUkKhUJ6+umnJUkrVqzQt771La1du1a33nqrvvzlL+vf//3fJUldXV165pln9OSTT+qGG27Qa6+9dka/AvfNCAd99BEGAAAYozlz5uiCCy6QJAUCAV111VWSpOXLl2vt2rXy+/1avny5ampqJEkXXnihvvKVr+hrX/ua9u/fr4yMDElSeXm5Lr74YknSddddNxB8Jek973mPJKmtrU2tra1au3atJOn666/Xk08+OXDfhg0bJEmXXXaZ2tvb1draekY/m+tmhMNBv+pautM9DAAAgLEbw8ztRMnKyho49/v9MsZIkjwej4LB4MB5PO78xf29732vzj//fN1///268sor9d///d+aN2/ewPv6DX0+9DtGM9pnnA7XzQhnUyMMAAAwYfbu3at58+bpYx/7mNavX68tW7ZIkg4cOKBnn31WkvTTn/5Ul1xyyXHvzcnJUV5enp566ilJ0o9+9KOB2WFJuvfeeyVJTz/9tHJycpSTk3NGY3XfjDA1wgAAABPm3nvv1Y9//GP5/X6Vlpbq1ltvVXt7uxYvXqy7775bN910kyorK/XhD394xPfffffd+tCHPqTu7m7NmzdPP/jBDwZey8vL00UXXaT29nbdddddZzxWY6094w85HatXr7YbN26c9O/9+sM79N0n9mr3P119xtPpAAAAE2X79u1avHhxuocxLmpqavTmN7/5jBa3rVu3Tt/4xje0evXqUe8b6fdmjNlkrT3uja4rjQgH/UokrXpik9unDgAAAFOL60ojBrZZ7o0rM+C6Hx8AAGDSVVRUnHGrs8cff3x8BjOE62aE+4NwOwvmAAAAXM21QZgFcwAAYKpL11qus9Wp/r5cF4TDQb8ksakGAACY0kKhkJqamgjDY2StVVNTk0Kh0Jjf47oi2aE1wgAAAFNVWVmZ6urq1NjYmO6hnDVCoZDKysrGfL/rgnA46PzIbKoBAACmMr/fr7lz56Z7GNPamEojjDFXGWOqjTG7jTGfG+H12caYx4wxLxtjthhj/nz8hzo+skOp0ghqhAEAAFztpEHYGOOVdIekqyUtkbTBGLNk2G1fkPRza+05kq6V9O3xHuh4yQp6JVEjDAAA4HZjmRFeI2m3tXavtTYq6WeS3jrsHispO3WeI+nQ+A1xfPm8HmUGvNQIAwAAuNxYgvAsSbVDntelrg31JUnXGWPqJD0g6aMjfZAx5kZjzEZjzMZ0Fn6Hgz5qhAEAAFxuLEHYjHBteB+PDZL+x1pbJunPJf3IGHPcZ1tr77TWrrbWri4qKjr10Y6TcMhHH2EAAACXG0sQrpNUPuR5mY4vffhrST+XJGvts5JCkgrHY4ATIRLys1gOAADA5cYShF+UVGmMmWuMCchZDHffsHsOSPozSTLGLJYThKds07tI0MdiOQAAAJc7aRC21sYl3SLpYUnb5XSH2GqMuc0Ysz5126ckfdAY84qkn0p6v53C26BEQj4WywEAALjcmDbUsNY+IGcR3NBrtw453ybp4vEd2sRhsRwAAADGtKHGdBMJ+VksBwAA4HKuDML9XSMSySlbvQEAAIAJ5sognB1yKkK6oswKAwAAuJUrg3A46ARh6oQBAADcy5VBOBLySxKdIwAAAFzMlUE4HOqfEaaXMAAAgFu5MghH+oMwnSMAAABcy51BmBphAAAA13NnEKZGGAAAwPVcGYT7a4Q7+6gRBgAAcCtXBuFMv1fGUBoBAADgZq4Mwh6PUTjoIwgDAAC4mCuDsOQsmCMIAwAAuJd7g3DIT40wAACAi7k2CIdDzAgDAAC4mWuDcCTkUycbagAAALiWa4Mwi+UAAADczbVBOBLyE4QBAABczMVB2KeOXhbLAQAAuJV7g3DQp754UtF4Mt1DAQAAQBq4Ngj3b7PcxYI5AAAAV3JtEI6E/JLYZhkAAMCtXBuEw0FnRriDTTUAAABcybVBODtVGsGMMAAAgDu5Ngj31wh3EoQBAABcyb1BmNIIAAAAV3NtEO5fLMeMMAAAgDu5OAg7M8LtBGEAAABXcm0QDvo88nuNOukjDAAA4EquDcLGGIWDbLMMAADgVq4NwpJTJ0yNMAAAgDu5Ogg7M8IEYQAAADdydRCOhHzqoEYYAADAlVwfhCmNAAAAcCeXB2E/G2oAAAC4lKuDcDjIjDAAAIBbuToIR0LOYjlrbbqHAgAAgEnm6iAcDvkUT1r1xZPpHgoAAAAmmauDcCTklyS1s6kGAACA67g7CAd9kkSdMAAAgAu5OgiHU0GYTTUAAADcx9VBOBJKzQizqQYAAIDruDoIh0P9M8LUCAMAALiNq4NwdmqxHKURAAAA7uPqIEyNMAAAgHu5OwhTIwwAAOBarg7Cfq9HIb+HIAwAAOBCrg7CkrOpBovlAAAA3IcgHPRRIwwAAOBCBOEQQRgAAMCNXB+EwyEfNcIAAAAu5PogHAlSIwwAAOBGrg/C4ZBPnZRGAAAAuA5BmMVyAAAAruT6IJwd8qkzGlcyadM9FAAAAEwi1wfhcMgna6WuKLPCAAAAbuL6IBwJ+SWxzTIAAIDbuD4Ih4M+SaJOGAAAwGXGFISNMVcZY6qNMbuNMZ8b4fXbjTGbU4+dxpjW8R/qxIiECMIAAABu5DvZDcYYr6Q7JF0hqU7Si8aY+6y12/rvsdZ+Ysj9H5V0zgSMdUL0B2FKIwAAANxlLDPCayTtttbutdZGJf1M0ltHuX+DpJ+Ox+AmQ3+NMJtqAAAAuMtYgvAsSbVDntelrh3HGDNH0lxJfzzzoU2O/hphNtUAAABwl7EEYTPCtRM13b1W0i+ttYkRP8iYG40xG40xGxsbG8c6xglFjTAAAIA7jSUI10kqH/K8TNKhE9x7rUYpi7DW3mmtXW2tXV1UVDT2UU6grIBPxkgd1AgDAAC4yliC8IuSKo0xc40xATlh977hNxljFkrKk/Ts+A5xYnk8RuGAjxphAAAAlzlpELbWxiXdIulhSdsl/dxau9UYc5sxZv2QWzdI+pm19qzbqzgc8lEjDAAA4DInbZ8mSdbaByQ9MOzarcOef2n8hjW5IiEfNcIAAAAu4/qd5SSncwR9hAEAANyFICwpHPJTIwwAAOAyBGGlSiOYEQYAAHAVgrCkSJAaYQAAALchCMuZEaZrBAAAgLsQhCWFg371xBKKJ5LpHgoAAAAmCUFYg9ss0zkCAADAPQjCcjbUkESdMAAAgIsQhCVlE4QBAABchyAsp0ZYojQCAADATQjCGqwRZlMNAAAA9yAIa7BGmBlhAAAA9yAIa3BGuJ0aYQAAANcgCEuK9NcIE4QBAABcgyAsKeT3yOcx1AgDAAC4CEFYkjFG4ZCPGmEAAAAXIQinhIM+SiMAAABchCCcEgn5WSwHAADgIgThlEjQp84+aoQBAADcgiCcEgn52GIZAADARQjCKSyWAwAAcBeCcAozwgAAAO5CEE4JB/10jQAAAHARgnBKJORTNJFUbyyR7qEAAABgEhCEUyIhnyRRJwwAAOASBOGU/iBMnTAAAIA7EIRTwkG/JFEnDAAA4BIE4ZTBGWE21QAAAHADgnBKOJgKwtQIAwAAuAJBOCU7RGkEAACAmxCEU8KURgAAALgKQTilvzSC9mkAAADuQBBOCfg8Cvo8tE8DAABwCYLwEJGQj8VyAAAALkEQHiIS8jMjDAAA4BIE4SHCQZ86WSwHAADgCgThISIhHzPCAAAALkEQHiIc9NE1AgAAwCUIwkNQIwwAAOAeBOEhnNIIaoQBAADcgCA8RCTklEZYa9M9FAAAAEwwgvAQ4aBPSSt1RxPpHgoAAAAmGEF4iEjIL4ltlgEAANyAIDxEOOSTJOqEAQAAXIAgPERkIAgzIwwAADDdEYSHiAQJwgAAAG5BEB6ivzSCGmEAAIDpjyA8RP9iOWqEAQAApj+C8BBhSiMAAABcgyA8BEEYAADAPQjCQ3g9RlkBLzXCAAAALkAQHiYS8lMjDAAA4AIE4WHCIR8zwgAAAC5AEB4mEvJRIwwAAOACBOFhwkGCMAAAgBsQhIfJDvkpjQAAAHABgvAwzowwi+UAAACmO4LwMJGQT52URgAAAEx7BOFhwiGfuqIJJZI23UMBAADABBpTEDbGXGWMqTbG7DbGfO4E91xjjNlmjNlqjPnJ+A5z8kRCfkmiThgAAGCa853sBmOMV9Idkq6QVCfpRWPMfdbabUPuqZT0eUkXW2tbjDHFEzXgiRYZ2GY5ppwMf5pHAwAAgIkylhnhNZJ2W2v3Wmujkn4m6a3D7vmgpDustS2SZK09Mr7DnDzhkBOEmREGAACY3sYShGdJqh3yvC51bagqSVXGmD8ZY54zxlw1XgOcbJFQ/4wwQRgAAGA6O2lphCQzwrXhK8l8kiolrZNUJukpY8wya23rMR9kzI2SbpSk2bNnn/JgJ0M4VRpB5wgAAIDpbSwzwnWSyoc8L5N0aIR7fmOtjVlr90mqlhOMj2GtvdNau9pau7qoqOh0xzyh+hfLtdNLGAAAYFobSxB+UVKlMWauMSYg6VpJ9w275/8kvV6SjDGFckol9o7nQCdLhBphAAAAVzhpELbWxiXdIulhSdsl/dxau9UYc5sxZn3qtoclNRljtkl6TNJnrLVNEzXoiTQQhCmNAAAAmNbGUiMsa+0Dkh4Ydu3WIedW0idTj7Naht8rr8ewWA4AAGCaY2e5YYwxCgd9lEYAAABMcwThEYSDPhbLAQAATHME4RFEQj5qhAEAAKY5gvAIIiEfNcIAAADTHEF4BJGQnxphAACAaY4gPIJw0KcOaoQBAACmNYLwCCIhukYAAABMdwThEYRDPrVTIwwAADCtEYRHEAn6FI0n1RdPpHsoAAAAmCAE4RFEQn5JbLMMAAAwnRGERxAOOjtPUycMAAAwfRGERxAJOUGYXsIAAADTF0F4BGGCMAAAwLRHEB5Bdn+NMKURAAAA0xZBeAT9NcJsqgEAADB9EYRH0F8jzIwwAADA9EUQHgE1wgAAANMfQXgEQZ9XAZ+HIAwAADCNEYRPIBL0USMMAAAwjRGETyAS8lEjDAAAMI0RhE8gHPJRGgEAADCNEYRPIBL0q5MgDAAAMG0RhE8gHPKpnRphAACAaYsgfAKRIDXCAAAA0xlB+ARYLAcAADC9EYRPoH+xnLU23UMBAADABCAIn0Ak5FciadUbS6Z7KAAAAJgABOETCAf7t1lmwRwAAMB0RBA+gUgoFYSpEwYAAJiWCMInMBCE6SUMAAAwLRGETyAS8ksSm2oAAABMUwThE6BGGAAAYHojCJ/A6dYI72ro0Bf+71XFEnSbAAAAmMoIwicQCTqlEadSI2yt1d/9+jX9+LkDqq7vmKihAQAAYBwQhE8gnJoRPpUa4Ue3NeiFmmZJIggDAABMcQThE/B6jDID3jHXCMcSSX31oR2aV5SlgNejnUcIwgAAAFMZQXgUkZBPnWOsEf7Zi7Xa29ilz1+9WPOKsrSTGWEAAIApjSA8inDQN6bFcp19cX3z9zu1Zm6+Ll9crKqSiHY2dE7CCAEAAHC6CMKjCIf8Y1os919P7NHRzqj+7s8XyxijqpKwDrb2jHk2GQAAAJOPIDyK7JBPnSepEa5v69X3ntqrt6ycqZXluZKkqpKIJKeVGgAAAKYmgvAowkHfSWeE/+3RaiWT0t9euXDgWn8Q3kkQBgAAmLIIwqM42WK5HfXt+sWmOr3vwjkqz88cuF6en6mQ30OdMAAAwBRGEB5FODh6jfA/P7BDkaBPt7xhwTHXvR6jyuIIM8IAAABTGEF4FP0zwsmkPe61p3cd1RM7G/XRN1QqNzNw3OuVJWGCMAAAwBRGEB5FpH93ueixs8LJpNVXHtiusrwMve+iOSO+d2FJRA3tfWrtjk74OAEAAHDqCMKjiJxgm+Vfv3xQ2w636zNXLlTQ5x3xvYML5qgTBgAAmIoIwqMIB/2SdEydcG8soX99pForynL0lhUzT/jeqlI6RwAAAExlBOFRDMwI9w32Er7rT/t0qK1X/+/PF8vjMSd878yckMJBH0EYAABgiiIIjyKcCsLtqRnhps4+feexPbp8cbEumFcw6nuNMSyYAwAAmMIIwqPIHlYj/B9/2KXuWEKfu3rRmN5fVRyhRhgAAGCKIgiPor9GuLMvrr2Nnbrn+QN6z+vKtaA4Mqb3V5VG1NwV1dHOvokcJgAAAE4DQXgU/TXCHb0x/ctD1Qr6PPr45ZVjfn9VSViStLOe8ggAAICphiA8isyAV8ZIj1c36qGt9bpp7XwVR0Jjfv/CEjpHAAAATFUE4VEYYxQO+vTMniYVR4L6wKVzT+n9RZGgcjL8qqZOGAAAYMohCJ9EdsipE/7UG6uUGfCd0nuNMVpYEtEuZoQBAACmHILwSRSEA1pYEtG7zis/rfdXloRV3dAha+04jwwAAABn4tSmOF3oWxvOUYbfK+8om2eMZmFpRB3Px9XQ3qfSnLHXFwMAAGBiMSN8EnMKslScffoBtjLVaq2a8ggAAIAphSA8wfpbqFEnDAAAMLWMKQgbY64yxlQbY3YbYz43wuvvN8Y0GmM2px4fGP+hnp0KwkEVhoOqppcwAADAlHLSGmFjjFfSHZKukFQn6UVjzH3W2m3Dbr3XWnvLBIzxrFdVEtbOI7RQAwAAmErGMiO8RtJua+1ea21U0s8kvXVihzW9VKVaqCWTdI4AAACYKsYShGdJqh3yvC51bbh3GmO2GGN+aYwZsdeYMeZGY8xGY8zGxsbG0xju2amqJKLuaEIHW3vSPRQAAACkjCUIj9Q3bPjU5m8lVVhrV0j6vaS7R/oga+2d1trV1trVRUVFpzbSs9jCUmfBHFstAwAATB1jCcJ1kobO8JZJOjT0Bmttk7W2L/X0e5LOG5/hTQ8LUi3UdrLVMgAAwJQxliD8oqRKY8xcY0xA0rWS7ht6gzFmxpCn6yVtH78hnv1yMvyakRNiRhgAAGAKOWnXCGtt3Bhzi6SHJXkl3WWt3WqMuU3SRmvtfZI+ZoxZLykuqVnS+ydwzGelypIILdQAAACmkDFtsWytfUDSA8Ou3Trk/POSPj++Q5teFpaE9dzeJiWS9rS3awYAAMD4YWe5SVJZElE0ntT+pq50DwUAAAAiCE+ahSUsmAMAAJhKCMKTZEExLdQAAACmEoLwJMkK+lSen0EQBgAAmCIIwpOoqjhCEAYAAJgiCMKTqKo0or2NXYrGk+keCgAAgOsRhCfRwpKI4kmrGjpHAAAApB1BeBJVlrBgDgAAYKogCE+i+UVheYy0kx3mAAAA0o4gPIlCfq8qCrLoJQwAADAFEIQnWVUJnSMAAACmAoLwJKsqCaumqUu9sUS6hwIAAOBqBOFJVlUaUdJKexopjwAAAEgngvAkqyqJSJJ2UScMAACQVgThSVZRkCW/16iaOmEAAIC0IghPsoDPo7mFWdpFEAYAAEgrgnAaVJVEmBEGAABIM4JwGlSVRFTb3KOuvni6hwIAAOBaBOE06F8wt/sIC+YAAADShSCcBlUlYUmiPAIAACCNCMJpMKcgSwGfhwVzAAAAaUQQTgOvx2hBUVjV9BIGAABIG4JwmiwsjTAjDAAAkEYE4TSpKonocFuv2npi6R4KAACAKxGE06R/wdzuI8wKAwAApANBOE36W6hV11MnDAAAkA4E4TSZlZuhzIBXO6kTBgAASAuCcJp4PEaVJRGCMAAAQJoQhNOoqjisnbRQAwAASAuCcBotLI3oaGefmrui6R4KAACA6xCE06gytWCO8ggAAIDJRxBOo4UEYQAAgLQhCKdRSXZQkZCPIAwAAJAGBOE0MsZoYUlEO+klDAAAMOkIwmlWWRLRziMdstaO6f5YIqnXDrZp66G2CR4ZAADA9OZL9wDcbmFJWD99IabGjj4VZ4eOec1aq4OtPdpc26rNB1q1ubZVrx1qU28sKb/X6LcfvUSLSrPTNHIAAICzG0E4zaoGFsx1KiPg1Za6Nm2ubdXLqeB7tLNPkhTwebRsZrbeu2aOlpdl65/u365P/+IV/fojF8vvZWIfAADgVBGE06yq1AnCH/vZy2rpjqq/QmJeYZYuqyzUqtm5WlWeq0Wl2Qr4BgNvyOfVh+95SXc+uVc3v35BOoYOAABwViMIp1lhOKi3rJypzt6YVpXnOcG3LFc5mf5R33f18hl60/IZ+ubvd+mKJSUDM8sAAAAYGzPWRVrjbfXq1Xbjxo1p+e7p4mhnn954+5Mqz8vQ/374IvkokQAAADiOMWaTtXb18Oskp7NYYTio2966VK/Utel7T+1L93AAAADOKgThs9ybls/QVUtLdfvvd2r3ETbmAAAAGCuC8FnOGKN/eNsyZQa8+vQvtiiRTE+pCwAAwNmGIDwNFEWC+vL6pdpc26rvP7033cMBAAA4KxCEp4n1K2fqiiUl+tdHdmpPI1s2AwAAnAxBeJowxuif3rZMIb9Xf/tLSiQAAABOhiA8jRRnh/Sl9Uu0aX+LfvAnukgAAACMhiA8zbxt1Sz92aJifeORau072pXu4QAAAExZBOFpxhijr7xjuQJej/72l68oSYkEAADAiAjC01BJdki3vmWpXqxp0d3P1qR7OAAAAFMSQXiaeue5s7RuYZH+5aFq7W+iRAIAAGA4gvA0ZYzRP79juXweo7/95RZKJAAAAIYhCE9jM3Iy9IU3L9bz+5r14+f3p3s4AAAAUwpBeJq7ZnW5Lq0s1Fcf3KFN+5vTPRwAAIApgyA8zRlj9NV3rlAk5NM7v/OsPvSjTew8BwAAIIKwK8zKzdAfP7VOn7i8Sk/tatQbb39Sn//VFtW39aZ7aAAAAGljrE3PIqrVq1fbjRs3puW73exoZ5/+84+7dc/z++X1GP3VxXP1obXzlZPhT/fQAAAAJoQxZpO1dvVx1wnC7lTb3K1/faRav3nlkLJDft38+vl634UVCvm96R4aAADAuDpREB5TaYQx5ipjTLUxZrcx5nOj3PcuY4w1xhz3RZhayvMz9e/XnqPfffQSrSrP1Vce2KE3fONx/WJjrRK0WgMAAC5w0iBsjPFKukPS1ZKWSNpgjFkywn0RSR+T9Px4DxITZ+nMHN19wxr95IPnqyg7pM/8couu/uaTenRbg9L11wIAAIDJMJYZ4TWSdltr91pro5J+JumtI9z3D5L+RRIrsM5CF80v1P995CJ95y/OVTxh9cEfbtRf371RvbFEuocGAAAwIcYShGdJqh3yvC51bYAx5hxJ5dba343j2DDJjDG6evkMPfyJy/TFNy/RY9VH9L3lF20AACAASURBVMEfEoYBAMD0NJYgbEa4NvA3c2OMR9Ltkj510g8y5kZjzEZjzMbGxsaxjxKTyu/16K8vmauvvXOFnt59VDf9aBNhGAAATDtjCcJ1ksqHPC+TdGjI84ikZZIeN8bUSLpA0n0jLZiz1t5prV1trV1dVFR0+qPGpLhmdbm++o7lemJnoz78403qixOGAQDA9DGWIPyipEpjzFxjTEDStZLu63/RWttmrS201lZYayskPSdpvbWW3mjTwHteN1v//I7leqy6UR/58UuEYQAAMG2cNAhba+OSbpH0sKTtkn5urd1qjLnNGLN+ogeI9NuwZrb+8W3L9IcdR3TzPS8rGk+me0gAAABnjA01MGY/erZGX/zNVr1xSYnu+Itz5feyQzcAAJj6zmhDDUCS/vLCCn15/VI9sq1BH/3Jy4olmBkGAABnL4IwTsn1F1Xo1jcv0UNb6/U3PyMMAwCAs5cv3QPA2eeGS+Yqaa3+8f7tMmazvvmeVfJRJgEAAM4yBGGclg9cOk9Ja/WVB3bIY4xuv2YlYRgAAJxVCMI4bTdeNl9JK331wR3yGOnfrlklr2ek/VcAAACmHoIwzsiH1s5XImn19Yer5TFGX33ncgV93nQPCwAA4KQIwjhjN79+gay1+sYjO7Vpf4v+358v1pVLS2QMs8MAAGDqoqgT4+KWN1Tq7hvWKOjz6EM/3qQN33tOWw+1pXtYAAAAJ0QQxrhZW1WkB//mUv3D25apur5Db/7W0/rsL7foSEdvuocGAABwHIIwxpXP69FfXjBHj3/m9frri+fqVy/X6fVff1x3PLZbvbFEuocHAAAwgCCMCZGT4dcX3rxEj3xirS5aUKivP1yty//tCd2/5bDSta03AADAUARhTKi5hVn63vtW6ycfOF/hoE83/+QlXfNfz2pLXWu6hwYAAFyOIIxJcdGCQt3/sUv1z+9Yrn1Hu7T+P/+kT/58s/Y3daV7aAAAwKVMuv5MvXr1artx48a0fDfSq6M3pjse26O7nt6naCKpJTOydfWyUl29vFQLiiPpHh4AAJhmjDGbrLWrj7tOEEa6HG7r0f1bDuvB1+q1aX+LJGlBcVhXLyvVlUtLtXRmNr2IAQDAGSMIY0praO/Vw1vr9eCr9Xp+X5OSVpqdn6mrlpXqqmWlWlWWKw/bNwMAgNNAEMZZo6mzT49ua9CDr9XrmT1HFUtYlWaHdNWyUr1xSYnOmZ2njADbOAMAgLEhCOOs1NYT0x+2O6H4yZ2N6osn5fMYLZ2ZrXPn5Om81GNGTka6hwoAAKYogjDOel19cT23t0mb9rdo0/4WvVLXqt5YUpI0Myekc+fkafWcPJ03J1+LZkTk9568KYq1Vp19cbX1xNTaHVNbT0xFkaCqSli0BwDAdHGiIOxLx2CA05EV9OnPFpfozxaXSJJiiaS2H27XxpoWbTrQoo01LfrdlsOSpAy/VyvLc3TO7DwFvB619cRSYTeq1p6Y2lKht7UnpkTy2H8MGiPdcPFcfebKhQr5KcEAAGC6YkYY08qh1p6BGeOXDrRo66F2JZJW2SGfcjL9ys0IKDfTr+wMv3Iz/MrN9Csnw7mekzq/f8th/ei5/ZpXlKV/ffdKnTM7L90/FgAAOAOURsCVovGkvB4j7yl2nHh611F99n+36HBbj25aO18fv7xSQd/0nh221tKuDgAwLZ0oCLOzHKa1gM9zyiFYki6pLNRDH79U7z6vXN95fI/e8q2n9Wpd2wSMMP0SSasv3bdVl3ztMR1o6k73cAAAmDQEYeAEIiG/vvauFfrB+1+ntp6Y3vbtP+nfHt2paDyZ7qGNm95YQjff85L+55kaNXb26YM/3KjOvni6hwUAwKQgCAMn8fpFxXrk42u1fuVM/ccfdultd/xJO+rb0z2sM9bWHdP77npBD22t1xfetFh3Xf867TrSoU/9fLOSyfSUTAEAMJkIwsAY5GT6dft7Vum//vI8Heno1Vu+9bTueGy34omzc3b4cFuP3v1fz+jlAy36jw3n6AOXztMllYX6uzct0cNbG/TNP+xK9xABAJhwBGHgFFy5tFSPfGKt3rikVF9/uFrv/O6z2n2kM93DOiW7Gjr0jm8/o0Otvbr7r9Zo/cqZA6/dcHGF3nlumb75h1166LXDaRwlAAATj64RwGn67SuH9MXfvKbuaEJrKvK1sjxHK8tytWp2roojoXQPb0Qv1jTrA3dvVMDn0f/81eu0dGbOcff0xhK69s7ntLOhQ7/6yEVaVJqdhpECADB+aJ8GTIAjHb369mN79GJNs3bUdwxszjEzJ6SV5blaWZ6rVeW5Wj4rR1nB9O5f8/DWen3spy9rVm6G7r5hjcrzM094b0O7U/4R9Hv0m5svUX5WYBJHCgDA+CIIAxOsN5bQ1kNt2lzbps21rXqltlUHmp12ZB4jVRZHtCoVjpfNylZlcUQZgcnpTfzj5/br1t+8phVlubrr/a8bU7DdXNuqa/7rWZ03O08//Os1Y9qyGgCAqYggDKRBc1dUr9S2OsG4zgnHLd0xSc5WznPyM1VVEtHC0ogqSyJaWBLR3MIsBXzjEzqttbr90Z36jz/u1hsWFes/33uOMgNjn5n+3011+tQvXtH7L6rQl9YvHZcxAQAw2U4UhNP7t1pgmsvPCuj1i4r1+kXFkpxgeqC5W9sPt6u6vlM7GzpU3dChP+w4MlBW4fMYzSvKcgJySURVpRFVFoc1IyfjlGaQ44mk/u7Xr+nejbW6ZnWZvvL25fKd4qzuO88r07bD7fr+0/u0eEZE73nd7FN6PwAAUxlBGJhExhjNKcjSnIIsXbVs8HpfPKG9jV1OMK7v0M6GDr1S16rfbTm2c0N2yKeS7JBKskMqzg4655HUMce5XhQOKpG0uuUnL+kPO47oY29YoE9cUXXa2yd//upF2tnQoS/832taUBzWeXPyz+RXAADAlEFpBDCFdfXFtetIp3Yf6VRDe6+OtPeqob1PDR29OtLepyMdvYoljv/fcIbfq754Qre9dZmuu2DOGY+jtTuqt97xJ3X1JfTbj16sGTkZZ/yZAABMFmqEgWkombRq6Y4OCcdOUG7s6NOfLS7WuoXF4/ZdOxs69PY7/qT5xWH9/KYLFfJPzkI/AADOFDXCwDTk8RgVhIMqCAe1RBPb77eqJKJ/v/YcffCHG/W5/92i29+z6rTLLQAAmAoIwgDG7IolJfrUFVX610d3asnMbN142fzj7onGk+qJJtQVjas7mlD3kKPP49Hs/EzNysugHRsAIO0IwgBOyS1vWKDt9e366oM7dN8rh9QdTTjBty+unlhixJrl4bweo1m5GZpTkOk88rM0uyBTFQVZmp2fOWn9lQEA7kYQBnBKjDH6xrtXKivg09HOPmUGfcr0e5UV9Ckj4FVWwKvMgE+ZAe/Aa5lB51pfLKH9zd060NSt/c3d2t/Upd++clhtPbFjvqM4ElRFQZbK8jOUHfIrM5D6fL/32M/tPw94B17LzQzI65mYko2mzj69dKBVRZGgls/KmbDvAQBMDhbLAUi7tu6Y9jd3qaapWweaurS/qVv7m7pV19Ktjj6ntKK/z/LJZAW8WlGWq1Wzne2tzynPVXF26LTGdai1Ry/WNOv5fc16YV+zdh/pHHgtPyugSysLtW5hkS6tLFJhOHha3wEAmHh0jQBw1rLWKppwao+PrTsedt4X176jXXq5tlXbDrUrngrPs3IztKo8FYxn52rZrJzjul5Ya1XT1K0X9jUNBN+6lh5JUiTo0+qKPK2ZW6Dz5uTpcFuPnqhu1BM7G9XUFZUx0vJZOVpXVaS1C4u0qjyP2WIAmEIIwgBcpTeW0NZDbXr5gLPF9csHWnWw1Qm2Po/RohkRrSrP1ez8TL1S16YX9jWrsaNPklSQFdCaufl6XUW+1szN1+IZ2SMG22TS6rVDbXqiulGP72zUywdalLRSToZfl1YWam0qGBdHTm9GGgAwPgjCAFyvsaMvFYpbtLm2VVvq2tTZF9eMnJDOn5uvNXMLtGZuvuYXZZ1Wa7jW7qie3n1Uj6dmi/uD9fyiLJXnZ2pGToZm5oQ0IzdDM3NDmpmTodKc0Cn1ZI4lkmrpiupoZ1TNXVE1dfWpqTMqK6kkO6jSITsPBn0sOgQAiSAMAMdJpDYkKcgKjHtP5GTSant9ux6vbtTm2lYdbuvR4dZeNXVFj7u3ICugGalgPDM3Q8XZQfVGE2rqiqopFXiPpgLv8IWFo8nPCqgkO6TS7KBKc0IqjoRUmhNSaXZIM3JDml8Upo0dAFdgQw0AGMbrMRO2yM3jMVo6M0dLZ+Ycc703ltDhtl4dbu3RobZeHWrt0eG2Hh1q7VVNU5ee2dOkzr64jJHyMwPKzwqoIBzQ4hnZKsgKqCArqPxwQIVZARWEg8rPCqgwHJAkNbT3qb69Vw1tvapv7z3m/NWD7Wrq6tPQuY+Q36MVZbk6d3aezp2dq3Pn5LHoD4CrMCMMAFNMV19cIb933BfcxRJJHenoU31br+paurW5tlUvHWjVtkNtA/2fy/MzUsHYeSyaEWHWGMBZj9IIAMCIemMJvXawTS8daNHLB1r10oEWNbQ79c39s8bnzM7V4tJszcrL0KzcDJVkh+iMAeCsQWkEAGBEIb9XqyvytboiX5LTSu5QW69e2t+ilw606KUDrbrr6X3H7Bro9RiVZoc0Ky9DZbkZAwG5/zgzN0Mhv1fWWvXFk2rvjamzN67Ovrg6e+PqSB07+5xHR29cnX0xBbxeZWf4lB3yKzvDr+yQT5GQ/5hrkaBPnmEhPJ5IqrUnppauqFq6Y2ruiqq1O6rm7qhahzxv74mrIBxQWV6GyvIyB46z8jIUDvJ/iYDb8L96AMAxjHG2wJ6Vm6G3rJwpyZk1rmvp0cHWHh1s6dHB1u7UsUfP7W1SfXuvhu95Egn61BNLDPRzHk3A51FWwKtYwqqzL36S8UnhoBOMfV6jlq6o2ntP/J6Az6P8zIDysgKKhHza2dChP+44or548pj78jL9Q8LxYFDOSv0c/X2se4b0ru6JDfay7n89aa1WlOXognkFWj0nny3DgSmM0ggAwBmLJZKqTy3+6w/LTV1RZQW9Cgf9Cod8igR9Cgd9CoecYyQ0+Hxoq7d4IqnOvrjae+Jq7405j/7znpjae+PqSF2LJZLKy/QrLyugvFTYzcv0H3Oe4fce1xXEWqujnVHVtXSrrqUn9eg+5jg8KI8k5PcoMzBk+++AVwlrtf1whxJJK7/XaGVZri6cX6AL5hXo3Nl5pxyM44mkapq6tP1wh6rrO7SjvkPNXX1aVZ6n8+fla01FvvKyAqf0mSNp6Yrq2b1NembPUT2zp0ntPXFdsqBA6xYW67KqIuWPw3fg1Oxt7NSdT+7VQ1vr9cYlJfrEFVWakZOR7mGdlagRBgBgjPqDcm1Lt3qiCWWkQm6m3zdwnuH3Hlei0a+rL66N+1v07J4mPbe3Sa8ebFMiaRXwerSqPFcXzMt3gvGcvIE+0tZaHeno0/bD7aquHwy9uxs7FU2Fcq/HaG5hlnIz/Hr1YNtAWF9UGjmmF3ZR5OTdPzr74nphX5Oe2d2kZ/Y0advhdknONuVr5uYrO8Ovp3cdHdg9ccWsHK1dWKx1C4u0six3yteI98UT2ne0S9khv2bkhMa9ReJEeqW2Vd99Yo8e2lqvgNejSysL9eTOozJGev9FFfrwuvnKzeQfJqeCIAwAQJp09sX1Yk2zntvbpOf2OME4aeUE49m5MpKqGzrU2j3YJ7okO6hFpdlaVBrRwtRjflF4IDj3xRPaUtem5/c624JvrGlRTywhydnE5fx5BTp/rhO4S7JD6o0l9NL+Fj2zx5n1faUuFc59Hp03O08XLyjQhfMLtaIsZ6BTSP/uiY9XN+rx6iPaXNuqpJVyM/26tLLI2T2xqmhMwXuiODPm3drZ0DHwqK7vUE1TtxKpspyiSHBgm/VV5blaUZajSMiftjGPxFqrp3cf1Xef2KM/7W5Sdsin911YofdfXKHCcFC1zd26/fc79euXDyoS9OlD6+brry6aS+nNGBGEAQCYIjp6Y9pY06JnUyHWY3RM6F1UGjnlGb9YIqlXDzrbhT+/t0kba1rUkaq3npWbocbOPkXjSXk9RivKcnTx/EJdNP/YWemTae2O6qldR/XEzmN3T1w2K1uXVRZp+awcVZZENKcgc9zb7iWSVodae5yg29ChnfUdqm7o1J4jnYomnJlxY6Q5+ZmqKnF+jwuKw2rpiuqVujZtrm3VvqNdA/fNLwprVXmuVpbn6pzyXC0sTU+rwETS6qHX6vWdJ3brtYPtKskO6gOXzNOG82ePuIBzR327/uWhav1xxxGVZAf18cur9O7zyuSjzeGoCMIAALhIImm17VC7nt/XpJcPtKo0J6SLFxTodRX54zIbmkxabTvc7oTi6kZtOtAyMAPr9xrNKwxrQUlYVcURVZWEVVkS1pyCrFHDZm8sodrmbu1v6tb+5m4daOpKHbtV29J9TOeSmTkhVZVGtLAkoqrUY0FxeNQZ0tZuJxS/UtuqzbWteqW2dWC3x6DPo2WzcrRkRrYKw87GNQWp2vOCsLO5TV5mYNxKQnpjCf3qpYO688k9qmnq1rzCLN20dp7eds6sMW2P/sK+Zn31we166UCr5hVl6TNvXKirlpWeVSUgk4kgDAAAJkx3NK49R7q0s6FDu450alfqWNvSPbCjod/r1DhXlkRUWexs8b2/qUv7m7p1oLlb9e29x+x+GA76NDs/U3MKMjW7IFMVBVmpUB1R9jiEeWut6lp6BkLx5tpWVTd0qOMEXUiMkXIy/M6Oj0NCcobfp6Dfo6DPo4DPo6DPO+R88HnQ51HQ79GLNS36/tP71NjRpxVlOfrIuvm6YknpKYdsa60e3dagrz9crV1HOrWyPFefu2qRLpxfcMa/m+mGIAwAACZdf0DedaRDOxs6tTt17A/IRZGg5uQ7QXdOftZA6J2Tn6n8rEBaZjij8aRau6Nq6oqqecjDed6nlq6Ymrr6Utdj6o0l1BdPHDNjfTKXVhbqw2vn68L5BWf8MyaSVv/7Up1uf3SnDrf1am1VkTasma1ZuRmakRtSfmbghAs73YIgDAAApoyeaEJWVpmB6bOlQTJpFU0k1RdLqi+RcI7xpKLxpPriidQxqZLskBaWRsb9+3tjCf3w2Rrd8dgetfUMLrwMeD0qyQlqRk6GZuSEVJoT0sycDJXmhDQjJ6QZORnKy/Srqy+htp7YqI/2/mNvTJGQM2Nfnp/pHPOcY26mf8qVaBCEAQAAXKCrL649jZ063Nbr9Pdu61F9W68Ot/XqcFuPGtr6BhYYjlXA61F2hl85GT7lZDi7PLZ2x1TX0q2jndFj7o0EfYPhOD/jmLBcUZCVltlptlgGAABwgaygTyvKcrWibOTXk0mr5u7owCY49e29au6KKhx0Qu7AI9Ov3IyAcjL8Cvk9J5zl7eqLq7bFWdR4oLlbtc3dqm3p0e7GTv2x+shAH2yPkar/8Wp5NHVmi8cUhI0xV0n6piSvpP+21n512OsfknSzpISkTkk3Wmu3jfNYAQAAcIY8HqPCcFCF4aCWzco548/LCvpS7f+yj3stmbRq7OzTgeZuHWnvS0uLutGcNAgbY7yS7pB0haQ6SS8aY+4bFnR/Yq39bur+9ZL+TdJVEzDeM9NRL237jRSMSIGwFAxLgYhzHLgWkTxTuDl1MiF1N0mJqBSZMbXHCgAAXM3jMSrJDqkkO5TuoYxoLDPCayTtttbulSRjzM8kvVXSQBC21rYPuT9LUnoKj0/m6E7pwb89+X2+DCcQB8NOOPZnSl6/5A2kHkPOfYGRr/szpUCmc/RnSv4MKZA1+DyQuubPct4X65Y6G6SOBufYeUTqrD/+WlejZJ2dg+QNSHkVUv4855E3N3U+V8qd7XwuAAAARjSWIDxLUu2Q53WSzh9+kzHmZkmflBSQ9IZxGd14m32R9Jk9Ul+H84h2Sn2dUrTDOQ5cG/JaX4cU75USMSnWJiX6nPNEdMhx2PmpMt7BcDv8erhYCpc4s78zVkqRUue5xyu11EjN+5zHvqekWNex780tHwzJ+fOkgkqpqErKmS15ptafJgAAACbbWILwSBXNx834WmvvkHSHMea9kr4g6frjPsiYGyXdKEmzZ88+tZGOB69Pyip0HhPFWicMx7qlaLcU63ECarTbuTZwfdh5MJIKucVSOBV2MwvGHlitdWaMm/cOPlr2Occtv5D62gbv9YWcUFxYKRUtdI6FC6WC+c4sNQAAgAuctH2aMeZCSV+y1l6Zev55SbLW/vMJ7vdIarHWjlp9Tfu0SWSt1N3slIYMfTRWS60HNPjvGiPlzZEKq5xH0SKpdJlzJCADAICz1Jm0T3tRUqUxZq6kg5KulfTeYR9eaa3dlXr6Jkm7hKnDGCmrQMq6UJpz4bGvxXqkpt2pYLxTOlotHd0l7X3CKQORJOORChZIJUtTj+XOMafM+ezJkIhJbbVOqUrRIskXnJzvBQAA09ZJg7C1Nm6MuUXSw3Lap91lrd1qjLlN0kZr7X2SbjHGXC4pJqlFI5RFYIryZ0ily53HUMmEU4Pc8JrUsFWqf006+JK09deD9wRzBsNx/8xxRt5g941A+NRqkXtanXKOlprB+uf+87a6wTpqj98Z76zzpLLVzjF/PnXPAADglLCzHE5Nb7t0ZPtgQO4/RjtHvt+fNdh9Y2i7uv7nPa2DYbe39dj3ZhamumLMdY55FU7njUObpYObpEMvD35vKEeaeW4qGKfCcbhown4NAADg7MEWy5g4yaTUdsAprehrH9aRozN1rfP4Lh3RTimYPRhyhwbevApnVnnU7004dc4HN0kHNzrHhm2DM8e5s51AnFfhhOqsotRiySLnkVngtL87mWi31H7QmZXuPw6cH3Ra2lVcLC2/Rqq6krINAACmGIIw3CHaJR3e4gTjuo3SoZek9sNSMjby/aGcY4NxVpHk8aVCbq0TdHuaj39fuETKnuXUSQezpV2PSF1HnHKRpW91QvGciynXAABgCiAIw72slXrbpK6jzuxtd+rYdYJjMiZll0k5qaCbPUvKKXeeZ8+SsmceP+ubiEv7npBe/YW0/bfObHf2LGn5u5xQXLosPT87AAAgCAOTJtotVT/ghOLdv5eScal4qbTi3dLydzvheiRDA3v30WOPsW6pdIU0+wKn3zQAABgzgjCQDl1N0tZfOaG49nnn2pxLnE4b3U2pGeqmVOhtOnEJh/FINumc586Wyi+QZp8vlZ8vFS9xdhoEAAAjIggD6da8T3r1l04o7jg8WJOcVZg6LxyyqK8gdZ66ZjxS/RbpwHNOoK59XupscD43mO10yyi/QCpf45yfbKEhAAAuQhAGphNrnZZz/aH4wPPSkW2SrBOaS5Y5M8e+oOTLSB1Dkj/kHPuf+4Y8D4SHdNUopPsFAGDaOJOd5QBMNcY47eby50orr3Wu9bQ63TIOpMJx8z4p3iPF+6R4r3OM9Qy2lzuZYI7Ti/mYtnPFx7agCxc7x1DO5O0yCADAOCEIA9NFRq604HLnMZpEPBWMe48NyNGuVPeMoZ00Uo+ju6T9z0jdzZJG+CuSNzgYisMlqQBdfPx5pNTZSAUAgCmAIAy4jdcnecOnF0gTcaevclej1Hlk8NjZMHjeXuf0b+46OvLsc1bRsRun5FVIeanNVCIzJqf3srXS0Z3S3iec3REzcgdnvAdmwYud2m0v/5kEgOmK/8IDGDuvz5n5DRc7nS9Gk0w6obmzYTA0t9UNbqld+4L02q+ODcvegJQ759idBgsXSkULnbZzZ1J+0VbnBN99T0j7nnQWLEpSRp4zG56IjvAmI2XmD5aEhIud88IF0sxznFrs8ailjnY7ZS37n5UOPONsH166QlpxjbR4vRPUAQDjjsVyANInEXN28OsPx837Bs9bapztufsFwk4gLlo05LHQ2exkpFnk7map5qnB8Nu027meWSjNvUyat1aau9YJ3AM9nBuPne0eft7/PNrpfJbHL5UskWascoLxzHOcdnYn27q7p8XpALL/GenAs9KhzanWecYJ1zNWOoG4ea/zj4PKNzqhuPJKZ8EjAOCU0DUCwNnFWqe3cmO11LhjyKN6sHWcJPmzpKKqwXDc3eQE38NbJFknQM+5eDD4Fi85s/ILa6XWA9Lhzc7Mbf+jt+3/t3evwXFWdRzHf/9c22STtmkuTS+0XIq0KLRcSi+IyDgMKiPOiDM6MoMzKCoMAzOgg75xdLjoG/WFvnGUkRfeuAgyqCMIreBMBSm0tKU3WtvSNk3Se0LSNMkeX/zPZnfTVFq6m033+X5mnnkuu9mc5MD2tyf/5xx/vLLGR8szwXjmYh913v1vD727Vmdn+KiolmZdIc1dLp233Ke/y4z+huAlJm8/KW14Oi7h3egjxB+71cM880cDwGkhCAMoH32HYkDelA3KXZul3v0eLucs8dB7wSekWVdKldXFbU9mOrvcYNyxLn9EW/LQPmeJB/O5y7xt1ZM/+PWHh6Sdr/g81O88J53o8ZsPP/oFX61w5uL8spF02j8QHNsrHduXs88c7/XfYdP5vuphW85W31zQXw0ATAQEYQDlr/+Ij8jW1JW6JR5GD//XQ3HfQWn21V73e7Y33w32S1v/7guzbHvBa5unX+Sv3bPfQ25Px8k1zxVVUsNMqTFuk6d66UXnRi/5yEi1+ah526VeptG20Ou0KckAcA4jCANAuek/7CPE65/0WuvGWdmgO/q4vvXUJSG9XR6IOzd62UbnBh9hHx7wx63Sw/bU8zxAT5oiTYr73PPc49rG8ZkBBB9s12pp9c/9rwAVlb7oTkWl9+vIvmLUeaXfuHrNN/yGUeAcRxAGAJy+4aE4YrwhhuONPtLcf8TroY8f/YDFWczLLNovl2Zd5WUgs6705cOLaaB37HKQng7f93b7h4PWhVLrJVLrAqllgV8rY3+x/AAACcJJREFUt0Vh3ntdWvmItGOlTwnYcomUHvZ+y9unx75+dI/X2C/9lrTsbmYvwTmNIAwAKJwQfPaMkWB8JP/4+FEPnnvf8lrukPavmzYvhuIYjtsvO7066fSwz03ds89LQHo6YinIvvxt4OjJX1s3PTsyXtfso+fdm/Nvuqydkh+MWxd4WE61FOTXNa72rJFWPSK9+w//ea+9T7rqjjMvGeraLK16VHrnWR/pX36PdM03pdqG4rQbKCKCMACgNAZ6fZaNvWukPW9Ie9/0hVckr11uuzQG4yskWU7YjUG3Z7+H1pNGoM3nds4rAxlVFtLQfuqg3XdI6trkI97dm7PH/Yezz6mb7lP0Vdd5nXR1nb9eVc7x6MeqJ/tIam1D3KekmgY/r6ot3sjz3jc9uG57QZrcJK24V1rydamm/uxet+Ntf90tf/Xfx4r7pKu/NjFq8YHTRBAGAEwcPftzgvEav6kwd5aNSVM9xDa2+75hRtznnKfaCr/yXwheM929KRuMe/b7TYqD/b4s+WBf9nywP1tLfToqqmI4bowBOQblKXOyM3e0LjyzutyOddLKR6Wtf/Op+pbfIy25s/Ajt3vWSCsflra/5L/7j98vXfnVwiwqAxQZQRgAMHGl09Kh7X6T1v8bxZ2I0sMxIPdnQ/JAr09zN9ATj3s96I8cx8dO9ErHj/kMI7kj0Q0zfcaOtkuzU9w1X5y/WMv+9dKqH0mbn/fShWX3+M1tkxqL+/PuWu2BeOerPvp+3belxbcVf5pC4CwQhAEAmKhC8JHnro3ZGTw6Y8lGetCfU1HlYbh1oYftLX/x2uZld/kNbZOmjG+bd/xTevkhac/rPsPEdQ/4NH419dmtuv7Dj9oPD/ry5yNbr5Qe8tHzhhnld3MjioogDADAuWZ40JcH79yYv53o9dHfpXeVdjaHEPymvJcf8jrwsVTWxmCc8rri3JBs5j/L6MB74v2T58LOVV0vNV3gi8JMv1BqujC7T7WeXkgeOiH1HchZQv2g74f6fZaN1Ax/rYYZfl6qEe8Q/K8FVbVeh84HgA+FIAwAQLkIYWIFohC83vv9bg+xg6ODbV9+yB3s8zIRKdZK54TjkRHlVP7ock3K50A+sks6uN1LaQ5u9/P0ULYtNQ35AdksBt0DcYvHY80wckrmNwqm2qSGNt+nWrNheepcnxGlvvns+mVkCfd1cVvr+8yiNxXVp567e/Rxw0ypeT7T3kWnCsIFvssAAAAU3UQKwZK3Z87Vpfnew0PS0d3SwR3ZcHxou7RvrS84o+DTyNU3Z+e2rm+J23Tf1zXH82avT+/tilucsaSn0/eZrXur7zNlKxk1KQ/Eedv5vp86J//GwhB8ru7RoTdTK26VPo3f/Bu9HCY9ePJ0hX2H/DUy18ea2zvV5iU1I9t8qeUjXt890f47KgFGhAEAQHkaHoor6RVhlcNMyULPfh/FPbzTb3o8vDO7DR3P+QKTpsz2UCz5tHSZUemKar85sv1yqX2Rb20Lz+ym0by5vY/4gijdW6QD26QDW6UDWzwsZ1TXS80X+RLqzRf78bR5Pro9eVrZhWRKIwAAAMZLOu2jxrnBOBOW08O+mEz7Ig+/rQuKPw1dCF5icWDrqIC81ReZyVXT4Euq527T5maPJ08rbluLgNIIAACA8VJR4fNgN7ZLc5eVujU+wptq9W3etfmPnXg/1lvvjtuu7PHOf/lUgLlqp3ipR00qjhzbqL3Gvm6V0m1PFf9nPQMEYQAAgCSrqY8j1Jed/FimBGSskDzY549nnqfgI+EK2fPcvRWhROUsEYQBAAAwNjNf6bCuSZq5qNStKbiJF80BAACAcUAQBgAAQCIRhAEAAJBIBGEAAAAkEkEYAAAAiUQQBgAAQCIRhAEAAJBIBGEAAAAkEkEYAAAAiUQQBgAAQCIRhAEAAJBIBGEAAAAkEkEYAAAAiUQQBgAAQCIRhAEAAJBIBGEAAAAkEkEYAAAAiUQQBgAAQCJZCKE039isW9KuknxzqVnSgRJ9b4w/+jtZ6O9kob+Thf5OnkL1+dwQQsvoiyULwqVkZm+EEK4qdTswPujvZKG/k4X+Thb6O3mK3eeURgAAACCRCMIAAABIpKQG4V+WugEYV/R3stDfyUJ/Jwv9nTxF7fNE1ggDAAAASR0RBgAAQMIlKgib2U1mtsXM3jWzB0vdHhSemT1mZl1mtiHnWpOZvWhm2+J+WinbiMIwszlmttLMNpnZRjO7N16nv8uUmU0ys9fNbF3s8x/E6+eb2Wuxz/9oZjWlbisKx8wqzewtM3s+ntPfZcrMdprZejNba2ZvxGtFfU9PTBA2s0pJv5D0aUkLJX3ZzBaWtlUogt9IumnUtQclvRRCmC/ppXiOc9+QpPtDCAskLZV0d/x/mv4uXwOSbgghXC5pkaSbzGyppB9L+mns88OS7ihhG1F490ralHNOf5e3T4YQFuVMmVbU9/TEBGFJSyS9G0LYEUI4IekPkm4pcZtQYCGEVyQdGnX5FkmPx+PHJX1+XBuFogghdIQQ3ozHPfJ/KGeJ/i5bwfXG0+q4BUk3SHoqXqfPy4iZzZb0WUm/iucm+jtpivqenqQgPEvSeznne+I1lL+2EEKH5OFJUmuJ24MCM7N5khZLek30d1mLfyZfK6lL0ouStks6EkIYik/hvb28/EzSdySl4/l00d/lLEh6wczWmNmd8VpR39OrCvliE5yNcY0pM4BznJmlJD0t6b4QwjEfMEK5CiEMS1pkZlMlPSNpwVhPG99WoRjM7GZJXSGENWZ2febyGE+lv8vHihDCPjNrlfSimW0u9jdM0ojwHklzcs5nS9pXorZgfHWaWbskxX1XiduDAjGzankI/m0I4U/xMv2dACGEI5JWyevDp5pZZmCH9/bysULS58xsp7yc8Qb5CDH9XaZCCPvivkv+QXeJivyenqQg/B9J8+PdpjWSviTpuRK3CePjOUm3x+PbJf25hG1BgcRawV9L2hRC+EnOQ/R3mTKzljgSLDObLOlT8trwlZJujU+jz8tECOG7IYTZIYR58n+zXw4hfEX0d1kys3oza8gcS7pR0gYV+T09UQtqmNln5J8mKyU9FkJ4uMRNQoGZ2e8lXS+pWVKnpO9LelbSE5LOk7Rb0hdDCKNvqMM5xsyulfSqpPXK1g9+T14nTH+XITO7TH6zTKV8IOeJEMIPzewC+Yhhk6S3JN0WQhgoXUtRaLE04oEQws30d3mK/fpMPK2S9LsQwsNmNl1FfE9PVBAGAAAAMpJUGgEAAACMIAgDAAAgkQjCAAAASCSCMAAAABKJIAwAAIBEIggDAAAgkQjCAAAASCSCMAAAABLpfzQg2H+9LGoeAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.plot(figsize=(12, 8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}