ML_application1_MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "24fb98fc",
   "metadata": {},
   "source": [
    "# Recognizing handwritten digits with an artificial neural network "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13f85c37",
   "metadata": {},
   "source": [
    "We will develop and train an artificial neural network to recognize handwritten digits using the KERAS API in Python in combination with tensorflow, which is an open source platform for machine learning. \n",
    "\n",
    "The MNIST database of handwritten digits is described here: http://yann.lecun.com/exdb/mnist/. It provides a training set of 60,000 examples and a test set of 10,000 examples. The images are normalized and of fixed size. Many methods have already been tested on this database: what will be the error rate of your neural network? \n",
    "\n",
    "Documentation about KERAS can be found here: https://keras.io"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d82f043",
   "metadata": {},
   "source": [
    "## Installation \n",
    "\n",
    "We first create a Python virtual environment to install KERAS, tensorflow, and their dependencies. \n",
    "\n",
    "### - On a local mac/linux computer\n",
    "\n",
    "````\n",
    "brew update\n",
    "brew install python3\n",
    "\n",
    "pip3 install virtualenv\n",
    "pip3 install −−user virtualenvwrapper\n",
    "export WORKON_HOME=./\n",
    "source ∼/.local/bin/virtualenvwrapper.sh -p $WORKON_HOME\n",
    "mkvirtualenv kerasenv -p python3\n",
    "workon kerasenv\n",
    "\n",
    "pip3 install jupyterlab\n",
    "pip3 install numpy scipy \n",
    "pip3 install scikit-learn\n",
    "pip3 install matplotlib \n",
    "pip3 install tensorflow\n",
    "pip3 install keras==2.3.1\n",
    "pip3 install seaborn\n",
    "````\n",
    "\n",
    "### - In a virtual environment on astromaster\n",
    "\n",
    "````\n",
    "python3 -m venv keras3 # To create a virtual environment keras3 with python3\n",
    "source keras3/bin/activate # To activate the virtual environment keras3\n",
    "\n",
    "pip3 install wheel\n",
    "pip3 install numpy scipy \n",
    "pip3 install matplotlib \n",
    "pip3 install scikit-learn\n",
    "pip3 install --upgrade tensorflow\n",
    "pip3 install keras==2.3.1\n",
    "pip3 install jupyterlab\n",
    "pip3 install seaborn\n",
    "````\n",
    "\n",
    "## Open a Jupyter Notebook\n",
    "\n",
    "### - On your local machine:\n",
    "\n",
    "````\n",
    "jupyter notebook\n",
    "````\n",
    "\n",
    "### - On astromaster\n",
    "\n",
    "Launch Jupyter Notebook on astromaster and access it from a remote machine using SSH, with a SSH tunnel:\n",
    "\n",
    "(1) On astromaster (change port number if needed)\n",
    "\n",
    "````\n",
    "jupyter notebook --no-browser --port=7777\n",
    "````\n",
    "\n",
    "(2) On your local machine (change username, requires a VPN to access astromaster)\n",
    "\n",
    "````\n",
    "ssh -L 7777:localhost:7777 jfreundlich@astromaster\n",
    "````\n",
    "\n",
    "(3) On your local brower, navigate to http://localhost:7777/\n",
    "Note: you will need the token number resulting from step (1)\n",
    "\n",
    "\n",
    "### - Create a new Python notebook: \n",
    "\n",
    "Create a new Python notebook from the menu at the top-right: \n",
    "\n",
    "<img src=\"newnotebook.png\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce139a34",
   "metadata": {},
   "source": [
    "## Importations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e6beba9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['figure.figsize'] = (7,7)\n",
    "\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Activation\n",
    "from keras.utils import to_categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db360daf",
   "metadata": {},
   "source": [
    "## Load training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "37c00c3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_classes = 10\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d97bb31",
   "metadata": {},
   "source": [
    "Check data shape: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6f26a160",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train original shape (60000, 28, 28)\n",
      "y_train original shape (60000,)\n",
      "X_test original shape (10000, 28, 28)\n",
      "y_test original shape (10000,)\n"
     ]
    }
   ],
   "source": [
    "print(\"X_train original shape\", X_train.shape)\n",
    "print(\"y_train original shape\", y_train.shape)\n",
    "print(\"X_test original shape\", X_test.shape)\n",
    "print(\"y_test original shape\", y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c93a7bd",
   "metadata": {},
   "source": [
    "Check some examples: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b3a9b66c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaUAAAGrCAYAAABg2IjeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA2q0lEQVR4nO3deZRU5bU28OcBcUAGQQ0SFVHBAYkjToQPzBUngoIxGo0DeL1inDVqJGoMxqioN64QxxBFUFkaE1TQyBWiKI5c0EsSBBQwIiiDggyCosD+/jgH0vtNdw1d01vdz2+tWvRT09ndtam3Tr1noJlBREQkBk0qXYCIiMgmGpRERCQaGpRERCQaGpRERCQaGpRERCQaGpRERCQaGpTqQHIIyccqXYfETX0iuVCf5K5RD0okf0xyGskvSC4iOZ5kjwrV8iHJL9NaviA5oRJ1yL+LrE86kpxEci3J2SR7V6IO+Xcx9UmNmnqRNJK/rmQd+Wi0gxLJnwL4LYBbAbQD0AHAfQD6VbCsE82sRXo5toJ1SCrCPnkcwP8B2B7A9QD+THLHCtUiqQj7BCSbARgGYEqlaqiPRjkokWwN4FcALjazp8xsjZl9Y2bPmtk1dTzmTyQXk1xJcjLJ/Wrc1ofkTJKrSX5M8ur0+h1IPkdyBcnlJF8l2Sj/5tUotj4huReAgwH80sy+NLMxAP4B4JRS/P6Sm9j6pIarAEwAMLuIv27JNdY3yCMBbA3g6TweMx5AZwDfAvAOgNE1bnsIwAVm1hJAVwAvpddfBWAhgB2RfHq6DkCm4zqNJvkpyQkkD8ijNimN2PpkPwAfmNnqGtf9Lb1eKie2PgHJ3QD8J5LBsqo01kFpewCfmdn6XB9gZiPMbLWZrQMwBMAB6SckAPgGQBeSrczsczN7p8b17QHsln5yetXqPtjgmQA6AtgNwCQAL5DcLt9fTIoqtj5pAWBlcN1KAC3z+J2k+GLrEwD4HYBfmNkX9fqNKqixDkrLAOxAcotc7kyyKcmhJOeRXAXgw/SmHdJ/TwHQB8B8kq+QPDK9/k4AcwFMIPkBycF1LcPMXk+/kllrZrcBWAHg/+X9m0kxxdYnXwBoFVzXCsDqWu4r5RNVn5A8EUBLM/tjPX+fyjKzRncB0BrAGgA/zHCfIQAeS38+G8AsALsDIIDtkKw2dwoe0wzAlQAW1PJ8XQEsBXB0jjXOAnBSpf9WjfkSW58A2AvAV0jecDZdNxnATyr9t2rMlwj75LcAVgFYnF6+RPKBZmyl/1a5XBrlmpKZrQRwI4B7SfYn2ZxkM5InkLyjloe0BLAOySei5ki2sAEAkNyS5JkkW5vZN0iaYWN6W1+SnUgSydcsGzbdVhPJDiS/mz7X1iSvQfKp6fXi/uaSj9j6xMzeBzAdwC/TPjkZwP4AxhTx15Y8xdYnAH6B5APMgellHIA/ADi3KL9wiTXKQQkAzOw3AH4K4AYAnwJYAOASAM/UcvdHAMwH8DGAmQDeCm4/G8CH6ar4T5DMDwHJROZfkXxKeRPAfWY2qZbnbwngfgCfp8s4HsAJZrasnr+eFElkfQIApwPohqRXhiL5dP5pfX43KZ6Y+sSSuarFmy5I1pTWmNnygn7JMmG6uiciIlJxjXZNSURE4qNBSUREoqFBSUREolHQoETyeJLvkZybaR8cadzUJ5IL9YkAqP9+SgCaApgHYA8AWyI53EmXLI8xXar68qn6RBf1iS7FuJRiP6XDAMw1sw/M7GsAT6CyR9iW0ptfj8eoTxof9YnUWyGD0s5ItsXfZGF6nUNyEJNzjEwrYFlSvdQnkgv1iQAAcjpWUyHMbDiA4QBA0kq9PKlO6hPJhfqk4StkTeljALvWyLuk14nUpD6RXKhPBEBhg9JUAJ1J7k5ySySHPxlXnLKkAVGfSC7UJwKggK/vzGw9yUsAvIBky5kRZvZu0SqTBkF9IrlQn8gmZT32nb4Drnpvm1m3Ui9EfVL11CeSlZmxtut1RAcREYmGBiUREYmGBiUREYmGBiUREYmGBiUREYmGBiUREYmGBiUREYmGBiUREYlGyQ/IKiKZHXLIIS5fcsklLp9zzjkuP/LIIy7ffffdLr/zzjtFrE6kvLSmJCIi0dCgJCIi0dCgJCIi0dABWfPQtGlTl1u3bp3X48O5gubNm7u89957u3zxxRe7/N///d8un3HGGS5/9dVXLg8dOtTlm266Kfdia6cDbRbBgQce6PJLL73kcqtWrfJ6vpUrV7q8/fbb16uuIlKfVIGjjz7a5dGjR7vcq1cvl997772iLl8HZBURkehpUBIRkWhoUBIRkWg0qv2UOnTo4PKWW27pcvfu3V3u0aOHy9ttt53Lp5xySvGKA7Bw4UKXf/e737l88sknu7x69WqX//a3v7n8yiuvFLE6qa/DDjvM5TFjxrgczk2G87zh6/z111+7HM4hHXHEES6H+y2Fj2+sevbs6XL4d3z66afLWU7ZHXrooS5PnTq1QpV4WlMSEZFoaFASEZFoaFASEZFoNOg5pWz7g+S7n1Gxbdy40eUbbrjB5S+++MLlcD+CRYsWufz555+7XOz9CqR24f5mBx98sMuPPfaYy+3bt8/r+efMmePyHXfc4fITTzzh8uuvv+5y2Fe33XZbXstvqI466iiXO3fu7HJDm1Nq0sSvg+y+++4u77bbbi6Tte5GVHJaUxIRkWhoUBIRkWhoUBIRkWg06Dmljz76yOVly5a5XOw5pSlTpri8YsUKl7/3ve+5HO4v8uijjxa1HimP3//+9y6HxyQsVDhH1aJFC5fD/dHCuZL999+/qPU0FOF5qt58880KVVIe4Vzm+eef73I49zl79uyS11QbrSmJiEg0NCiJiEg0NCiJiEg0GvSc0vLly12+5pprXO7bt6/L//d//+dyeOy50PTp010+5phjXF6zZo3L++23n8uXX355xueXOB1yyCEuf//733c52/4d4RzQs88+63J43qxPPvnE5bBPw/3T/uM//iOvehqrcL+dhu7BBx/MeHu4P1ylNK5XRUREoqZBSUREopF1UCI5guRSkjNqXNeW5ESSc9J/25S2TImd+kRyoT6RbBieu+Xf7kD2BPAFgEfMrGt63R0AlpvZUJKDAbQxs2uzLozMvLAya9WqlcvheWvC/U/OO+88l8866yyXH3/88SJWF6W3zaxbbTc05D7JdgzFsI9C48ePdzncj6lXr14uh/sVhXMBn376acblbdiwweW1a9dmXF54vqUiiLJPwr9ruF/SU0895fLZZ5+dz9NH74033nA5PO9WeD65t956q6T1mFmtk51Z15TMbDKA5cHV/QCMSn8eBaB/IcVJ9VOfSC7UJ5JNfbe+a2dmmw5RvRhAu7ruSHIQgEH1XI5UN/WJ5EJ9IpsVvEm4mVmm1WgzGw5gOBDf1zJSPuoTyYX6ROo7KC0h2d7MFpFsD2BpMYsql1WrVmW8feXKlRlvD48d9cc//tHl8HxJjVBV9slee+3lcrh/W3jMxM8++8zl8DxXo0aNcjk8T9Zf/vKXjLlQ22yzjctXXXWVy2eeeWZRl1cPZemTPn36uBz+XRqadu38Cmd4/qTQxx9/XMpyclbfTcLHARiQ/jwAwNjilCMNjPpEcqE+kc1y2ST8cQBvAtib5EKS5wEYCuAYknMA9E6zNGLqE8mF+kSyyfr1nZnVdRz+o4tci1Qx9YnkQn0i2TToY98VasiQIS6HxzwL9/fo3bu3yxMmTChJXVJcW221lcvhsefCuYhwf7bwvDzTpk1zOba5iw4dOlS6hIrYe++9M97+7rvvlqmS8gj7OJxjev/9910O+7pSdJghERGJhgYlERGJhgYlERGJhuaUMgjPhxTulxQeM+wPf/iDy5MmTXI5nGu49957Xc52HEIpjYMOOsjlcA4p1K9fP5fD8yNJdZo6dWqlS8goPMbi8ccf73J4LM5jjz024/PdfPPNLq9YsaL+xRWR1pRERCQaGpRERCQaGpRERCQamlPKw7x581weOHCgyw8//LDL4flYwrztttu6/Mgjj7gcHkNNSuOuu+5ymfSneQnnjGKfQ2rSxH/W1DEYc9O2bduCHn/AAQe4HPZRuB/jLrvs4vKWW27pcnhMwvB1/fLLL12eMmWKy+vWrXN5iy382/3bb7+NGGlNSUREoqFBSUREoqFBSUREoqE5pQI8/fTTLs+ZM8flcK7i6KP9MSdvvfVWl3fbbTeXb7nlFpdjOd9Jtevbt6/LBx54oMvh/mLjxo0rdUlFFc4hhb/P9OnTy1hNPMI5mPDv8sADD7h83XXX5fX8+++/v8vhnNL69etdXrt2rcszZ850ecSIES6H+zmGc5tLlixxeeHChS6Hx2CcPXs2YqQ1JRERiYYGJRERiYYGJRERiYbmlIpoxowZLp922mkun3jiiS6H+zVdcMEFLnfu3NnlY445ptASBf/+3Xq4f8jSpUtd/uMf/1jymvIRnv8pPO9X6KWXXnL55z//ebFLqgoXXXSRy/Pnz3e5e/fuBT3/Rx995PIzzzzj8qxZs1x+6623ClpeaNCgQS7vuOOOLn/wwQdFXV6paE1JRESioUFJRESioUFJRESioTmlEgrPT/Loo4+6/OCDD7ocHpuqZ8+eLh911FEuv/zyywXVJ7ULjxlW6WMQhnNIN9xwg8vXXHONy+H+Kb/5zW9c/uKLL4pYXfW6/fbbK11CUYX7QYbGjBlTpkoKozUlERGJhgYlERGJhgYlERGJhuaUiig89tUPf/hDlw899FCXwzmkUHgsrMmTJxdQneSq0se6C4/FF84Z/ehHP3J57NixLp9yyiklqUuqW3iszlhpTUlERKKhQUlERKKhQUlERKKhOaU87L333i5fcsklLv/gBz9weaeddsrr+Tds2OByuH9MeJ4cqZ/wPDdh7t+/v8uXX355Seu58sorXf7FL37hcuvWrV0ePXq0y+ecc05pChOpAK0piYhINDQoiYhINLIOSiR3JTmJ5EyS75K8PL2+LcmJJOek/7YpfbkSK/WJ5EJ9ItnkMqe0HsBVZvYOyZYA3iY5EcBAAC+a2VCSgwEMBnBt6UotvXAO6IwzznA5nEPq2LFjQcubNm2ay7fccovLld5fJk9V0ydmljGHffC73/3O5REjRri8bNkyl4844giXzz77bJcPOOAAl3fZZReXw/PyvPDCCy7fd999qGJV0yfVLpwr3WuvvVwu9vmciiXrmpKZLTKzd9KfVwOYBWBnAP0AjErvNgpA/xLVKFVAfSK5UJ9INnltfUeyI4CDAEwB0M7MNm0ethhAuzoeMwjAoNpuk4ZJfSK5UJ9IbXLe0IFkCwBjAFxhZqtq3mbJ9x9W2+PMbLiZdTOzbgVVKlVBfSK5UJ9IXXJaUyLZDEkDjTazp9Krl5Bsb2aLSLYHsLRURRZLu3b+w1eXLl1cvueee1zeZ599ClrelClTXL7zzjtdDo9ZVu37ITWUPmnatKnLF110kcvhseVWrXLvqejcuXNey3vjjTdcnjRpkss33nhjXs8Xu4bSJ7EL50qbNKmOja1z2fqOAB4CMMvM7qpx0zgAA9KfBwAYGz5WGg/1ieRCfSLZ5LKm9F0AZwP4B8np6XXXARgK4EmS5wGYD+C0klQo1UJ9IrlQn0hGWQclM3sNAOu4OfP5d6XRUJ9ILtQnkk2DOvZd27ZtXf7973/vcniemj322KOg5YVzAb/5zW9cDvcv+fLLLwtanhTHm2++6fLUqVNdDs97FQr3YwrnKkPhfkxPPPGEy6U+tp4IABx55JEujxw5sjKFZFEdM18iItIoaFASEZFoaFASEZFoVNWc0uGHH+7yNddc4/Jhhx3m8s4771zQ8tauXetyeAy0W2+91eU1a9YUtDwpj4ULF7ocngfrggsucPmGG27I6/mHDRvm8v333+/y3Llz83o+kfoIj31XLbSmJCIi0dCgJCIi0dCgJCIi0aiqOaWTTz45Y85m5syZLj/33HMur1+/3uVwv6MVK1bktTypDosWLXJ5yJAhGbNIjMaPH+/yqaeeWqFKCqM1JRERiYYGJRERiYYGJRERiQbDc26UdGFk+RYmpfB2OU6upj6peuoTycrMat2RSmtKIiISDQ1KIiISDQ1KIiISDQ1KIiISDQ1KIiISDQ1KIiISDQ1KIiISDQ1KIiISDQ1KIiISDQ1KIiISDQ1KIiISjXKfT+kzAPMB7JD+HCvVV7vdyrQc9UlxqE/iEHN90fVIWQ/Iunmh5LRyHLCxvlRfHGL/PVVfHGL/PWOuL8ba9PWdiIhEQ4OSiIhEo1KD0vAKLTdXqi8Osf+eqi8Osf+eMdcXXW0VmVMSERGpjb6+ExGRaGhQEhGRaJR1UCJ5PMn3SM4lObicy64LyREkl5KcUeO6tiTnkVxFciLJNhWqbVeSk0jOJPkuyctr1DeR5JxK1lcqsfVJhh6ZSHIZyUWVfA3UJ+qTHOurij4p26BEsimAewGcAKALgDNIdinX8uuo6ccAugNoAWBvkuNJ9gAwGMA/AYwD8GKaS13LzST/QXI9ySHp1esBXGVmXQAcAeDi9G82GMCLZta5XPWVS4x9AmAhkh0M903fWMYDuBvJ3/5uAItRpteAZHeS/0tyNcm/p/2qPlGfbEbyWyQfJ/kJyZUkXyd5OKqkT8q5pnQYgLlm9oGZfQ3gCQD9yrh8h+RPAfwWwHUADgHwHoD70pr6AZie3nUUgP5lKGkugJ8B+MumK8xskZm9k/68GsAsADun9Y0qc33lEmOf/ATAMACzAXRA0icn4F+vwTyU4TUg2RbAswDuBLAdgDvS/JX6RH1SQwsAU5G8r7VNl/8XAKuroU/KOSjtDGBBjbwwva7sSLYG8CsAF5vZUwC+BAAze9bMrgHQDsAX6d0XA2hH8k8kF6efPCaT3K/G8/VJV4lXk/yY5NXp9TuQfI7kCpLLSb5Ksta/uZmNMrPxAFbXUXNHAAcBmAKgnZktqllfYX+RqETZJwBeAGBm9o2ZPQugSY3X4Cukr0GJ+6Q7gMVm9icz22BmjwH4FMAPaiyjI9QnZRVbn6QD9V3ph9oNZjYcwJYA9q6xjI6ItE8a64YORwLYGsDT2e5oyTbzBmA8gM4AvgXgHQCja9ztIQAXmFlLAF0BvJRefxWS/yw7Inmhr0ufKy8kWwAYA+AKM1tVR31SfDn3Cf71GpS6T1hL7gqoTyooxj7ZjOSBSAaluWmOuk/KOSh9DGDXGnmX9LpK2B7AZ2a2vo7blyBZBQbJ9gCWmtkIM1ttZusADAFwQPoJCQC+AdCFZCsz+3zTKnJ6fXsAu6WfnF61PHcMI9kMSQONTtfqAGBJWtfm+vJ5zshVS59sfg2QvCEtBYAS98mbAL5N8gySzUgOALAngObqE/VJbUi2AvAogJvMbGU19Ek5B6WpADqT3J3klgBOR7IhQSUsA7ADybqOkj4OwIHpzwMAjCM5lOkWeQA+TG/bIf33FAB9AMwn+QrJI9Pr70Ty6WQCyQ+Y5xZCJInkU9MsM7srqG9AjfrG5vO8kauWPqn5GuwJYCzJpqXsEzNbhuT7/58i+eB0PIC/Ivn0rD5Rnzgkt0Ey5/iWmd1WNe8nZla2C5I/9PtIJvyuL+eygzpaA1gD4IcAHgewCMmnkIUAzkPyyecDAKuQ/Kf/CZJJwd2RfF2yHZJV3E7B8zYDcCWABbUssyuSTyBHZ6ntMQBD0p97pMv5O5INL6anf8PtkWwlMyetr22l/paNpE9eq6NHXkTyhrQIyYTy2eXqk/S+WwD4CMkgpT5Rn9S8fSsk81ujkcxrVc37SVnPp2RmzwN4vpzLrKOOlSRvRLJJ6QVImucbAL0BfM/MlpF8BEmTnEXyIgDrkDRWcwC3bnqu9FPaqQCeS593FYCN6W19kWyJMw/ASgAbNt0WSlermyJZe92C5NYA3jSzcA5hk6ML+RvELMI++RmSPpmAf/XJtWZ2NJPN9zuZ2XKSLVH6PjkIwAwA2yCZXF9gyafeu2q7P9Qn5agjqj5J30v+jGQDrgFmtjGt8zX8+5zkJvH0SaU/7VTyAuBMANOQfMpZjGSzye7pbUMAPJb+3ALJKu1qJCcVOwfpJxskE4j/A+BzJGtWUwH0SB93JZJV8zVIPjn9IkMtI9PnrHkZWOm/kS7R9cnjSN6QVgL4I4BvVfrvo0tcfQKgV/p8a5FsRbzp8v8q/TfK5aIDsoqISDQa6ybhIiISIQ1KIiISjYIGJUZ2QESJk/pEcqE+EaCAk/wxOSDi+wCOQTLpNhXAGWY2s3jlSbVTn0gu1CeySSGbhG8+ICIAkNx0QMQ6m4iktqqobp+Z2Y55PkZ90vioTyQrq2N3l0K+vovmgIhSNvPr8Rj1SeOjPpF6K/nOsyQHARhU6uVIdVOfSC7UJw1fIYNSTgdEtOSw6cMBrW43UuoTyYX6RAAU9vVdTAdElHipTyQX6hMBUMCakpmtJ3kJkoP+NQUwwszeLVpl0iCoTyQX6hPZpKyHGdLqdtV728y6lXoh6pOqpz6RrEqx9Z2IiEhRaVASEZFoaFASEZFoaFASEZFoaFASEZFoaFASEZFoaFASEZFoaFASEZFoaFASEZFoaFASEZFoaFASEZFoaFASEZFolPwkf5K7G264weWbbrrJ5SZN/GeIo446yuVXXnmlJHWJSOW1bNnS5RYtWrj8/e9/3+Udd/RnpL/rrrtcXrduXRGrKx6tKYmISDQ0KImISDQ0KImISDQ0p1RBAwcOdPnaa691eePGjRkfX84TNIpIaXXs2NHl8P3gyCOPdLlr1655PX/79u1dvuyyy/J6fLloTUlERKKhQUlERKKhQUlERKKhOaUK2m233VzeeuutK1SJFNPhhx/u8llnneVyr169XN5vv/0yPt/VV1/t8ieffOJyjx49XH7sscdcnjJlSsbnl/LYZ599XL7iiitcPvPMM13eZpttXCbp8oIFC1xevXq1y/vuu6/Lp512msv33Xefy7Nnz66l6vLTmpKIiERDg5KIiERDg5KIiERDc0pl1Lt3b5cvvfTSjPcPv+Pt27evy0uWLClOYVKQH/3oRy4PGzbM5R122MHlcG7g5Zdfdjk8Ztmdd96Zcfnh84WPP/300zM+XoqjdevWLt9+++0uh30SHssumzlz5rh83HHHudysWTOXw/ePsA/DHAutKYmISDQ0KImISDQ0KImISDQ0p1RC4f4jDz/8sMvhd9ChcC5h/vz5xSlM8rLFFv6/Sbdu3Vz+wx/+4HLz5s1dnjx5sss333yzy6+99prLW221lctPPvmky8cee2zGeqdNm5bxdimNk08+2eX/+q//Kuj55s2b5/IxxxzjcrifUqdOnQpaXiy0piQiItHQoCQiItHIOiiRHEFyKckZNa5rS3IiyTnpv21KW6bETn0iuVCfSDa5zCmNBHAPgEdqXDcYwItmNpTk4DRfW8tjG7UBAwa4/O1vfzvj/cP9VR555JHa7xinkWigfRIeu+7BBx/MeP+JEye6HO6fsmrVqoyPD++fbQ5p4cKFLo8aNSrj/StsJBpon5x66ql53f/DDz90eerUqS6H51MK55BC4bHuqlXWNSUzmwxgeXB1PwCbOn8UgP7FLUuqjfpEcqE+kWzqO6fUzswWpT8vBtCuSPVIw6I+kVyoT2SzgjcJNzMjWed5uUkOAjCo0OVIdVOfSC7UJ1LfQWkJyfZmtohkewBL67qjmQ0HMBwAMjVbQxAeS+o///M/Xd64caPLK1ascPnXv/51SeqqoKrsk3A/ouuuu85lM19eeF6aG264weVsc0ih66+/Pq/7X3bZZS5/+umneT0+AlXZJ6Hzzz/f5UGD/Ng5YcIEl+fOnevy0qV1/to5adeuYaxg1vfru3EANs3iDwAwtjjlSAOjPpFcqE9ks1w2CX8cwJsA9ia5kOR5AIYCOIbkHAC90yyNmPpEcqE+kWyyfn1nZmfUcdPRRa5Fqpj6RHKhPpFsdOy7AnTs2NHlMWPG5PX4u+++2+VJkyYVWpLUw4033uhyOIf09ddfu/zCCy+4HO5P8uWXX2Zc3tZbb+1yuB9Shw4dXA7PlxTOPY4dq2+7YvDJJ5+4PGTIkLIu/8gjjyzr8kpFhxkSEZFoaFASEZFoaFASEZFoaE6pAMcff7zL+++/f8b7v/jiiy4PGzas6DVJdtttt53LF110kcvhfkjhHFL//v3zWl54npvRo0e7fMghh2R8/J///GeX77jjjryWL9Uh3N9s2223zevx3/nOdzLe/sYbb7j85ptv5vX85aI1JRERiYYGJRERiYYGJRERiYbmlPIQziUMHZp5x/PXXnvN5fD8SitXrixKXZKfLbfc0uXwmIWh8Lv+b33rWy6fe+65Lp900kkud+3a1eUWLVq4HM5hhfmxxx5zec2aNRnrlTg0b97c5S5durj8y1/+0uU+ffpkfL4mTfw6RHgszVC431TYpxs2bMj4+ErRmpKIiERDg5KIiERDg5KIiERDc0oZFHpsuw8++MDlJUuWFFqSFEF4LLvw/EM77rijy//85z9dDud8sgm/2w/Pr9S+fXuXP/vsM5efffbZvJYn5dGsWTOXDzroIJfD94vwdQ6PkRj2SbgfUbhfZDhnFdpiC//2/oMf/MDlcD/J8P9FpWhNSUREoqFBSUREoqFBSUREoqE5pQzC8+Rk2y8glG0/JqmMFStWuBzuf/bcc8+53LZtW5fnzZvncng+o5EjR7q8fPlyl5944gmXw7mG8HaJQ7h/WzjH89RTT2V8/E033eTySy+95PLrr7/ucth34f3D/d9C4dzobbfd5vJHH33k8jPPPOPyunXrMj5/qWhNSUREoqFBSUREoqFBSUREoqE5pRoOPPBAl4899ti8Hh/OLbz33nuFliRlMGXKFJfD7+IL1bNnT5d79erlcjhXGe7fJpUR7ocUzgldc801GR8/fvx4l++++26Xw7nNsO+ef/55l8PzJYX7FYXn2QrnnPr16+dyeF6vv/71ry7ffvvtLn/++efIZPr06Rlvz5XWlEREJBoalEREJBoalEREJBqaU6phwoQJLrdp0ybj/d966y2XBw4cWOySpAHYZpttXA7nkMJj6Wk/pcpo2rSpyzfffLPLV199tcvhea0GDx7scvg6hnNI3bp1c/mee+5xOTyW3pw5c1y+8MILXZ40aZLLrVq1crl79+4un3nmmS6H5wGbOHEiMlmwYIHLu+++e8b750prSiIiEg0NSiIiEg0NSiIiEg3me26YghZGlm9h9RCesz7bse7OOecclx9//PGi1xSZt82sW/a7FSb2PilU2Gfh/8HwWHjh+Z6qQFX2SThHE+5XtHbtWpcHDRrkcjgnffjhh7t87rnnunzCCSe4HM49/upXv3L54Ycfdjmc0ynUGWec4fKPf/zjjPe/8sorXZ47d25eyzMz1na91pRERCQaGpRERCQaWQclkruSnERyJsl3SV6eXt+W5ESSc9J/M28/LQ2a+kRyoT6RbLLOKZFsD6C9mb1DsiWAtwH0BzAQwHIzG0pyMIA2ZnZt3c8U31xB+B1tuJ9RtjmlPfbYw+X58+cXpa6I1TlX0JD7pFDHHXecy+ExzRrTnFLMfbJo0SKXw2PRhecXmj17tsvbbruty506dcpr+UOGDHE5PP9ROBdZ7eo9p2Rmi8zsnfTn1QBmAdgZQD8Ao9K7jULSWNJIqU8kF+oTySavOSWSHQEcBGAKgHZmtumjxWIA7YpbmlQr9YnkQn0itcn5MEMkWwAYA+AKM1tF/mvNy8ysrlVpkoMADKrtNml41CeSC/WJ1CWnQYlkMyQNNNrMNp2IfgnJ9ma2KP2eeGltjzWz4QCGp89T0bmC8HxJvXv3djmcQwrPV3Lvvfe6vGTJkuIV1wA0lD4ptnDusbGLtU8WL17scjintNVWW7l8wAEHZHy+cO5w8uTJLj/zzDMuf/jhhy43tDmkXOWy9R0BPARglpndVeOmcQAGpD8PADA2fKw0HuoTyYX6RLLJZU3puwDOBvAPktPT664DMBTAkyTPAzAfwGklqVCqhfpEcqE+kYyyDkpm9hqAWjfdA3B0ccuRaqU+kVyoTySbRnU+pe22287lnXbaKeP9P/74Y5fD86mI5OLVV191uUkT/615tv3hpDx69uzpcv/+/V0++OCDXV661E97jRgxwuXPP//c5XCOWmqnwwyJiEg0NCiJiEg0NCiJiEg0GtWckkglzJgxw+U5c+a4HO7HtOeee7pchce+q0qrV692+dFHH82YpTS0piQiItHQoCQiItHQoCQiItFoVHNK4flP3njjDZd79OhRznKkkbr11ltdfvDBB12+5ZZbXL700ktdnjlzZmkKE4mA1pRERCQaGpRERCQaGpRERCQaNCvfqWsa2nlyGqG3zaxbqRfS0PukVatWLj/55JMuh+f5euqpp1w+99xzXV6zZk0RqysK9YlkZWa1HphXa0oiIhINDUoiIhINDUoiIhINzSlJPjRXUALhHFO4n9KFF17o8v777+9yhPstqU8kK80piYhI9DQoiYhINDQoiYhINDSnJPnQXIHkQn0iWWlOSUREoqdBSUREoqFBSUREolHu8yl9BmA+gB3Sn2Ol+mq3W5mWoz4pDvVJHGKuL7oeKeuGDpsXSk4rx0Rofam+OMT+e6q+OMT+e8ZcX4y16es7ERGJhgYlERGJRqUGpeEVWm6uVF8cYv89VV8cYv89Y64vutoqMqckIiJSG319JyIi0dCgJCIi0SjroETyeJLvkZxLcnA5l10XkiNILiU5o8Z1bUnOI7mK5ESSbSpU264kJ5GcSfJdkpfXqG8iyTmVrK9UYuuTDD0ykeQykosq+RqoT9QnOdZXFX1StkGJZFMA9wI4AUAXAGeQ7FKu5ddR048BdAfQAsDeJMeT7AFgMIB/AhgH4MU0l7qWm0n+g+R6kkPSq9cDuMrMugA4AsDF6d9sMIAXzaxzueorlxj7BMBCJDsY7pu+sYwHcDeSv/3dABajTK9B+qbyafqB6W8k+0F9oj4JVPP7STnXlA4DMNfMPjCzrwE8AaBfGZfvkPwpgN8CuA7AIQDeA3BfWlM/ANPTu44C0L8MJc0F8DMAf9l0hZktMrN30p9XA5gFYOe0vlFlrq9cYuyTnwAYBmA2gA5I+uQE/Os1mIfyvQaXA2hvZq0ADALwGACoT9Qngap9PynnoLQzgAU18sL0urIj2RrArwBcbGZPAfgSAMzsWTO7BkA7AF+kd18MoB3JP5FcTHIlyckk96vxfH3SVeLVJD8meXV6/Q4knyO5guRykq+SrPVvbmajzGw8gNV11NwRwEEApgBoZ2aLatZX2F8kKlH2CYAXAJiZfWNmzwJoUuM1+Arpa1CGPvm7ma3fFAE0A7BrjWV0hPqkrCLtk6p9P2msGzocCWBrAE9nu6Ml28wbgPEAOgP4FoB3AIyucbeHAFxgZi0BdAXwUnr9VUj+s+yI5IW+Ln2uvJBsAWAMgCvMbFUd9Unx5dwn+NdrUPI+Sd+YvkLyhvIygGnp9eqTyoiyT+oSe5+Uc1D6GDU+0QHYJb2uErYH8FmNT5yhJUjmmUCyPYClZjbCzFab2ToAQwAckH5CAoBvAHQh2crMPt+0ipxe3x7Abuknp1ctzx3DSDZD0kCj07U6AFiS1rW5vnyeM3LV0iebXwMkb0hLAaAcfWJmfQG0BNAHwAQz26g+UZ/kohr6pJyD0lQAnUnuTnJLAKcj2ZCgEpYB2IFkXUdJHwfgwPTnAQDGkRzKdIs8AB+mt+2Q/nsKkjeI+SRfIXlkev2dSL7bnUDyA+a5hRBJIvnUNMvM7grqG1CjvrH5PG/kqqVPar4GewIYS7JpufokfVMaD+BYkidBfaI+yaJq3k/MrGwXJH/o95FM+F1fzmUHdbQGsAbADwE8DmARkk8hCwGch+STzwcAVgH4K5IJzFkAdgdAANshWcXtFDxvMwBXAlhQyzK7IvkEcnSW2h4DMCT9uUe6nL8j2fBievo33B7JVjJz0vraVupv2Uj65LU6euRFJG9IiwC0BXB2ufqkxv3/CuB36hP1SR21Vd37SVnPp2RmzwN4vpzLrKOOlSRvRLJJ6QVImucbAL0BfM/MlpF8BEmTnEXyIgDrkDRWcwC3bnqu9FPaqQCeS593FYCN6W19kWyJMw/ASgAbNt0WSlermyJZe92C5NYA3rQ6zmMP4OhC/gYxi7BPfoakTybgX31yrZkdzWRz205mtpxkS5SwT0jug+SN7GUkm/f+CEBPAD8zs8vq+DXUJ6WvI6o+Se9bve8nlf60U8kLgDORTBKvQbLVyV8AdE9vGwLgsfTnFkhWaVcjOanYOUg/2QDYEsD/APgcyZrVVAA90sddiWTVfA2ST06/yFDLyPQ5a14GVvpvpEs8fQJgXyQbN6wGsCJ9jpMr/ffRJa4+Se9bte8nOiCriIhEo7FuEi4iIhHSoCQiItHQoCQiItEoaFBiZEfplTipTyQX6hMBUP+t75BsbjgPwB5Ithj5G4AuWR4Tbg2iS3VdPlWf6KI+0aUYl7pe10LWlKI6Sq+Uxfx6PEZ90vioT6TeChmUcjpKL8lBJKeRnFbAsqR6qU8kF+oTAYDSH9HBzIYDGA4AJK3Uy5PqpD6RXKhPGr5C1pRiOkqvxEt9IrlQnwiAwgalmI7SK/FSn0gu1CcCoICv78xsPclLkJxpsSmAEWb2btEqkwZBfSK5UJ/IJmU99p2+A656b5tZt1IvRH1S9dQnkpXVccRyHdFBRESioUFJRESioUFJRESioUFJRESioUFJRESioUFJRESioUFJRESioUFJRESiUfIDsjYkw4YNc/myyy5zecaMGS737dvX5fnz63NEfxGRxkNrSiIiEg0NSiIiEg0NSiIiEg3NKWXQsWNHl8866yyXN27c6PK+++7r8j777OOy5pQapr322svlZs2audyzZ0+X77vvPpfDPirU2LFjXT799NNd/vrrr4u6PKmfsE+6d+/u8q233uryd7/73ZLXFAOtKYmISDQ0KImISDQ0KImISDQ0p5TBp59+6vLkyZNdPumkk8pZjlTIfvvt5/LAgQNdPvXUU11u0sR/1vv2t7/tcjiHVOwTbYZ9+cADD7h8xRVXuLxq1aqiLl9y07p1a5cnTZrk8uLFi13eaaedMt7eUGhNSUREoqFBSUREoqFBSUREoqE5pQzWrFnjsvYzapxuu+02l/v06VOhSurnnHPOcfmhhx5y+fXXXy9nOZKjcA5Jc0oiIiJlpkFJRESioUFJRESioTmlDLbbbjuXDzjggMoUIhU1ceJEl7PNKS1dutTlcA4n3I8p27HvwmOi9erVK+P9pWEgWekSKkJrSiIiEg0NSiIiEg0NSiIiEg3NKWXQvHlzlzt06JDX4w899FCXZ8+e7bL2e6oO999/v8vPPPNMxvt/8803Lhe6P0mrVq1cnjFjhsvhsfVCYb3Tpk0rqB4pj/CYiFtvvXWFKikvrSmJiEg0NCiJiEg0sg5KJEeQXEpyRo3r2pKcSHJO+m+b0pYpsVOfSC7UJ5JNLnNKIwHcA+CRGtcNBvCimQ0lOTjN1xa/vMr65JNPXB45cqTLQ4YMyfj48PYVK1a4fM8999SzsiiNRAPtk/Xr17u8YMGCsi7/uOOOc7lNm/zesxcuXOjyunXrCq6pACPRQPuk1Lp16+byW2+9VaFKSivrmpKZTQawPLi6H4BR6c+jAPQvbllSbdQnkgv1iWRT363v2pnZovTnxQDa1XVHkoMADKrncqS6qU8kF+oT2azgTcLNzEjWeT5nMxsOYDgAZLqfNGzqE8mF+kTqOygtIdnezBaRbA9gadZHNAA333yzy9nmlKRx9kmhTj/9dJfPP/98l7fZZpu8nu/GG28suKYSa5R9Es5Vrly50uXWrVu7vOeee5a8phjUd5PwcQAGpD8PADC2OOVIA6M+kVyoT2SzXDYJfxzAmwD2JrmQ5HkAhgI4huQcAL3TLI2Y+kRyoT6RbLJ+fWdmZ9Rx09FFrkWqmPpEcqE+kWx07LsC5HteHBEAOPPMM10ePHiwy506dXK5WbNmeT3/9OnTXQ6PxSdxCPdbfPXVV13u27dvGauJhw4zJCIi0dCgJCIi0dCgJCIi0dCcUgHCOaTw/CfSMHTs2NHls88+2+XevXvn9Xw9evRwOd++WbVqlcvhnNTzzz/v8pdffpnX84tUktaUREQkGhqUREQkGhqUREQkGppTEgl07drV5XHjxrncoUOHcpbzb8L9WYYPH16hSqSctt9++0qXUBZaUxIRkWhoUBIRkWhoUBIRkWhoTkkkC5IZc74KPWZieEy0E044weXx48fXrzCJ2kknnVTpEspCa0oiIhINDUoiIhINDUoiIhINzSkVIN+5gZ49e7p8zz33FL0mKdyMGTNcPuqoo1w+66yzXH7hhRdc/uqrrwpa/nnnnefypZdeWtDzSXWYNGmSyzqfkoiISIVpUBIRkWhoUBIRkWiwnOcAItmgTji0YcMGl/P9W+6///4uz5w5s+CaSuxtM+tW6oU0tD7JV+vWrV1etmxZxvufeOKJLkewn5L6pB5OOeUUl//0pz+5HJ4Xq0uXLi7Pnz+/NIWViJnVusOf1pRERCQaGpRERCQaGpRERCQa2k+pAA888IDLF1xwQV6PHzRokMtXXHFFoSVJA3DcccdVugSpgPXr12e8PTzm4lZbbVXKcipGa0oiIhINDUoiIhINDUoiIhINzSkVYPbs2ZUuQeqhWbNmLh977LEuv/TSSy6H+4cU27nnnuvysGHDSro8idPYsWNdDt9f9tlnH5fDOeiLLrqoJHWVm9aUREQkGhqUREQkGlkHJZK7kpxEcibJd0lenl7fluREknPSf9uUvlyJlfpEcqE+kWyyHvuOZHsA7c3sHZItAbwNoD+AgQCWm9lQkoMBtDGza7M8V4M6VlXo/fffd3nPPffMeP/wfEydOnVyed68ecUprHjqPKZZzH3So0cPl6+//nqXjznmGJd33313lxcsWFDQ8tu2betynz59XL777rtdbtmyZcbnC+e4TjrpJJfD8/JUQFX2SWx++9vfuhzOPbZr187lQs/jVW71PvadmS0ys3fSn1cDmAVgZwD9AIxK7zYKSWNJI6U+kVyoTySbvLa+I9kRwEEApgBoZ2aL0psWA2hXx2MGARhU223SMKlPJBfqE6lNzhs6kGwBYAyAK8xsVc3bLPkOsNZVaTMbbmbdynEoe6k89YnkQn0idclpTYlkMyQNNNrMnkqvXkKyvZktSr8nXlqqIqvFu+++6/Iee+yR8f4bN24sZTllF2uf3HPPPS537do14/1/9rOfubx69eqClh/OWR188MEuZ5vXffnll12+//77XY5gDikvsfZJ7MI++frrrytUSWnlsvUdATwEYJaZ3VXjpnEABqQ/DwAwNnysNB7qE8mF+kSyyWVN6bsAzgbwD5LT0+uuAzAUwJMkzwMwH8BpJalQqoX6RHKhPpGMsg5KZvYagFo33QNwdHHLkWqlPpFcqE8kGx37roiGDx/u8oknnlihSqQQF154YVmXt3Spnz559tlnXb788stdrrb9UaQ4WrVq5XK/fv1cfvrpp8tZTsnoMEMiIhINDUoiIhINDUoiIhINzSkV0cyZM12eNWuWy/vuu285y5HUwIEDXb700ktdHjBgAIopPGbh2rVrXX711VddDuciZ8yYUdR6pDqddprfAHHdunUuh+8vDYXWlEREJBoalEREJBoalEREJBqaUyqi+fPnu/yd73ynQpVITdOnT3f5oosucvl///d/Xf71r3/tcps2/nxzzzzzjMsTJ050eexYf4ScxYsX51qqyGaTJ092OZyTDs+r1VBoTUlERKKhQUlERKKhQUlERKLBbOdyKerCyPItTErh7XKcXE19UvXUJ5KVmdV6YF6tKYmISDQ0KImISDQ0KImISDQ0KImISDQ0KImISDQ0KImISDQ0KImISDQ0KImISDQ0KImISDQ0KImISDQ0KImISDTKfT6lzwDMB7BD+nOsVF/tdivTctQnxaE+iUPM9UXXI2U9IOvmhZLTynHAxvpSfXGI/fdUfXGI/feMub4Ya9PXdyIiEg0NSiIiEo1KDUrDK7TcXKm+OMT+e6q+OMT+e8ZcX3S1VWROSUREpDb6+k5ERKKhQUlERKJR1kGJ5PEk3yM5l+Tgci67LiRHkFxKckaN69qSnEhyTvpvmwrVtivJSSRnknyX5OUx1VcqsfVJzD2S1qI+UZ/kUl9V9EnZBiWSTQHcC+AEAF0AnEGyS7mWn8FIAMcH1w0G8KKZdQbwYporYT2Aq8ysC4AjAFyc/s1iqa/oIu2TkYi3RwD1ifokN9XRJ2ZWlguAIwG8UCP/HMDPy7X8LLV1BDCjRn4PQPv05/YA3qt0jWktYwEcE2t9DblPqqVH1CcVr0t9UuClnF/f7QxgQY28ML0uRu3MbFH682IA7SpZDACQ7AjgIABTEGF9RVQtfRLla6A+iU6Ur0HMfaINHbKw5ONDRbebJ9kCwBgAV5jZqpq3xVBfYxfLa6A+iVssr0HsfVLOQeljALvWyLuk18VoCcn2AJD+u7RShZBshqSBRpvZU7HVVwLV0idRvQbqE/VJLqqhT8o5KE0F0Jnk7iS3BHA6gHFlXH4+xgEYkP48AMl3r2VHkgAeAjDLzO6qcVMU9ZVItfRJNK+B+kR9kouq6ZMyT6z1AfA+gHkArq/0RF9a0+MAFgH4Bsn30ucB2B7JVihzAPwVQNsK1dYDyar03wFMTy99YqmvsfRJzD2iPlGfNLQ+0WGGREQkGtrQQUREoqFBSUREoqFBSUREoqFBSUREoqFBSUREoqFBSUREoqFBSUREovH/AQJsNs9uJCe8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 504x504 with 9 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for i in range(9):\n",
    "    plt.subplot(3,3,i+1)\n",
    "    plt.imshow(X_train[i], cmap='gray', interpolation='none')\n",
    "    plt.title(\"Class {}\".format(y_train[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26c0b108",
   "metadata": {},
   "source": [
    "Check min, max:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "77089701",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train min= 0 ,max= 255\n",
      "y_train min= 0 ,max= 9\n"
     ]
    }
   ],
   "source": [
    "print(\"X_train min=\",X_train.min(),\",max=\",X_train.max())\n",
    "print(\"y_train min=\",y_train.min(),\",max=\",y_train.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ba5d22f",
   "metadata": {},
   "source": [
    "## Format the data for training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a495b97",
   "metadata": {},
   "source": [
    "The neural network is going to take a single vector for each training example, so we need to reshape the input so that each 28x28 image becomes a single 784 dimensional vector. We'll also scale the inputs to be in the range [0-1] rather than [0-255]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4bbd2c67",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training matrix shape (60000, 784)\n",
      "Testing matrix shape (10000, 784)\n"
     ]
    }
   ],
   "source": [
    "X_train = X_train.reshape(60000, 784)\n",
    "X_test = X_test.reshape(10000, 784)\n",
    "X_train = X_train.astype('float32')\n",
    "X_test = X_test.astype('float32')\n",
    "X_train /= 255\n",
    "X_test /= 255\n",
    "print(\"Training matrix shape\", X_train.shape)\n",
    "print(\"Testing matrix shape\", X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e85a6ed",
   "metadata": {},
   "source": [
    "Modify the target matrices to be in the one-hot format, i.e.\n",
    "\n",
    "```\n",
    "0 -> [1, 0, 0, 0, 0, 0, 0, 0, 0]\n",
    "1 -> [0, 1, 0, 0, 0, 0, 0, 0, 0]\n",
    "2 -> [0, 0, 1, 0, 0, 0, 0, 0, 0]\n",
    "etc.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1cfba90a",
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_train = to_categorical(y_train, nb_classes)\n",
    "Y_test = to_categorical(y_test, nb_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "fa01049d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training matrix shape (60000, 10)\n",
      "Testing matrix shape (10000, 10)\n"
     ]
    }
   ],
   "source": [
    "print(\"Training matrix shape\", Y_train.shape)\n",
    "print(\"Testing matrix shape\", Y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1225d69",
   "metadata": {},
   "source": [
    "## Building the neural network\n",
    "\n",
    "We define a sequential model with three layers incrementally. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7921026a",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(5, input_shape=(784,)))\n",
    "model.add(Activation('relu')) # An \"activation\" is just a non-linear function applied to the output\n",
    "                              # of the layer above. Here, with a \"rectified linear unit\",\n",
    "                              # we clamp all values below 0 to 0.\n",
    "                           \n",
    "#model.add(Dropout(0.2))   # Dropout helps protect the model from memorizing or \"overfitting\" the training data\n",
    "model.add(Dense(5))\n",
    "model.add(Activation('relu'))\n",
    "#model.add(Dropout(0.2))\n",
    "model.add(Dense(10))\n",
    "model.add(Activation('softmax')) # This special \"softmax\" activation among other things,\n",
    "                                 # ensures the output is a valid probaility distribution, that is\n",
    "                                 # that its values are all non-negative and sum to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5418b5d",
   "metadata": {},
   "source": [
    "Display the contents of the model:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c78c94e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " dense_6 (Dense)             (None, 5)                 3925      \n",
      "                                                                 \n",
      " activation_6 (Activation)   (None, 5)                 0         \n",
      "                                                                 \n",
      " dense_7 (Dense)             (None, 5)                 30        \n",
      "                                                                 \n",
      " activation_7 (Activation)   (None, 5)                 0         \n",
      "                                                                 \n",
      " dense_8 (Dense)             (None, 10)                60        \n",
      "                                                                 \n",
      " activation_8 (Activation)   (None, 10)                0         \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 4,015\n",
      "Trainable params: 4,015\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "459b2648",
   "metadata": {},
   "source": [
    "Compilation of the model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d8a2e943",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "296712c2",
   "metadata": {},
   "source": [
    "Other possible loss functions include: \n",
    "\n",
    "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "    \n",
    "    model.compile(loss='Poisson', optimizer='adam')\n",
    "    \n",
    "    def my_loss_fn(y_true, y_pred):\n",
    "      squared_difference = (y_true - y_pred)**2\n",
    "      return np.mean(squared_difference)   \n",
    "     \n",
    "    model.compile(loss='my_loss_fn', optimizer='adam')\n",
    "\n",
    "etc..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "db78e2db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((60000, 784), (60000, 10), (10000, 784), (10000, 10))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape, Y_train.shape, X_test.shape, Y_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "61bb7c30",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "600/600 [==============================] - 1s 1ms/step - loss: 1.4288 - val_loss: 0.8679\n",
      "Epoch 2/10\n",
      "600/600 [==============================] - 1s 992us/step - loss: 0.7294 - val_loss: 0.6292\n",
      "Epoch 3/10\n",
      "600/600 [==============================] - 1s 995us/step - loss: 0.5971 - val_loss: 0.5454\n",
      "Epoch 4/10\n",
      "600/600 [==============================] - 1s 1ms/step - loss: 0.5309 - val_loss: 0.5017\n",
      "Epoch 5/10\n",
      "600/600 [==============================] - 1s 1ms/step - loss: 0.4936 - val_loss: 0.4831\n",
      "Epoch 6/10\n",
      "600/600 [==============================] - 1s 1ms/step - loss: 0.4722 - val_loss: 0.4673\n",
      "Epoch 7/10\n",
      "600/600 [==============================] - 1s 1ms/step - loss: 0.4582 - val_loss: 0.4560\n",
      "Epoch 8/10\n",
      "600/600 [==============================] - 1s 1ms/step - loss: 0.4478 - val_loss: 0.4539\n",
      "Epoch 9/10\n",
      "600/600 [==============================] - 1s 1ms/step - loss: 0.4395 - val_loss: 0.4467\n",
      "Epoch 10/10\n",
      "600/600 [==============================] - 1s 1ms/step - loss: 0.4335 - val_loss: 0.4426\n"
     ]
    }
   ],
   "source": [
    "history=model.fit(X_train, Y_train,\n",
    "          batch_size=100, epochs=10,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43925092",
   "metadata": {},
   "source": [
    "## Evaluate performance "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "84c56fa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The predict_classes function outputs the highest probability class\n",
    "# according to the trained classifier for each input example.\n",
    "predict=model.predict(X_test) \n",
    "Ytest_predicted=np.argmax(predict,axis=1)\n",
    "Y_test_pred = np_utils.to_categorical(Ytest_predicted, nb_classes)\n",
    "\n",
    "# Check which items we got right / wrong\n",
    "correct_indices = np.nonzero(Ytest_predicted == y_test)[0]\n",
    "incorrect_indices = np.nonzero(Ytest_predicted != y_test)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14e31907",
   "metadata": {},
   "source": [
    "### Plot correctly or incorrectly classified examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff602233",
   "metadata": {},
   "source": [
    "### Calculate the global accuracy and the error rate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81df7f88",
   "metadata": {},
   "source": [
    "### Model training history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "94804125",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'val_loss'])\n"
     ]
    }
   ],
   "source": [
    "# list all data in history\n",
    "print(history.history.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa27e8a0",
   "metadata": {},
   "source": [
    "### ROC curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "00f9840c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve,auc\n",
    "\n",
    "# Compute ROC curve and ROC area for each class\n",
    "fpr = dict()\n",
    "tpr = dict()\n",
    "roc_auc = dict()\n",
    "for i in range(nb_classes):\n",
    "    fpr[i], tpr[i], _ = roc_curve(Y_test[:, i], Y_test_pred[:, i])\n",
    "    roc_auc[i] = auc(fpr[i], tpr[i])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f33ecdf",
   "metadata": {},
   "source": [
    "### Precision-Recall curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ff90fb69",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_curve\n",
    "\n",
    "# Compute precision and recall for each class\n",
    "precision = dict()\n",
    "recall = dict()\n",
    "for i in range(nb_classes):\n",
    "    precision[i], recall[i], _ = precision_recall_curve(Y_test[:, i], Y_test_pred[:, i])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f145300a",
   "metadata": {},
   "source": [
    "### Confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "35346b5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix, plot_confusion_matrix\n",
    "\n",
    "#The input to confusion_matrix must be an array of int not one hot encodings\n",
    "matrix = confusion_matrix(Y_test.argmax(axis=1), Y_test_pred.argmax(axis=1))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}