02-hyperparameter-tuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float: left; margin-bottom: 1em\" src=\"images/PRACE.png\" width=\"200\">\n",
    "<img style=\"float: right; margin-bottom: 1em\" src=\"images/surfsara.png\" width=\"150\">\n",
    "<hr style=\"clear: both\"/>\n",
    "\n",
    "# Hyperparameter tuning\n",
    "In this notebook you will learn how the different hyperparameters affect the performance of your network. You will start with a small network and extend and tweak it gradually to improve its performance. \n",
    "\n",
    "We have prepared a more complicated problem for you and turned the original two-class classification problem (healthy versus diseased leafs) into a ten-class classification problem. All pictures are taken from tomato plants. Let's load the data set, and display the class names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import lib\n",
    "\n",
    "X, Y, labels = lib.dataset_plant_village_tomato()\n",
    "\n",
    "for index, name in enumerate(labels):\n",
    "    print('{}: {}'.format(str(index).rjust(2), name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, we have one healthy class, and nine disease classes.\n",
    "\n",
    "Let's plot the first 32 examples in the data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "lib.plot_examples(X[:32], Y[:32], labels);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the size of the inputs `X` and `Y` using numpy's [`shape`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.shape.html) method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Shape of input images X: {}'.format(X.shape))\n",
    "print('Shape of input labels Y: {}'.format(Y.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1\n",
    "What is the size (width and height) of the examples, in pixels? How many examples do we have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2\n",
    "Implement a model with a single dense hidden layer with 16 units and ReLU activation by filling out the skeleton below.\n",
    "\n",
    "**Hints**:\n",
    "* Go back to the slides to find out the activation functions you will need in the hidden and output layers.\n",
    "* It may help to keep open the first notebook for reference purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Flatten(input_shape=X.shape[1:]))\n",
    "# <FILL IN>\n",
    "model.add(Dense(10, activation='softmax'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3\n",
    "Initialise the Adam optimizer with a learning rate of 0.1 and train the network for 10 epochs by filling out the skeleton below.\n",
    "\n",
    "Does the network train at all? Why or why not, do you think? Fill in your answer in the cell below.\n",
    "\n",
    "**Hint**: what would the accuracy be like when we always predict the same class in a ten-class classification problem?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.optimizers import Adam\n",
    "\n",
    "model.compile(Adam(lr=0.1), loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n",
    "history = model.fit(X, Y, epochs=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4\n",
    "Experiment with the learning rate by changing it in orders of magnitudes (e.g. 0.1, 0.01, 0.001) in the cell below. We have filled out the network definition and number of epochs for you.\n",
    "\n",
    "What seems to be a good learning rate for this problem? What happens if you make the learning rate too small? Why? Fill in your answer in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Flatten(input_shape=X.shape[1:]))\n",
    "model.add(Dense(16, activation='relu'))\n",
    "model.add(Dense(10, activation='softmax'))\n",
    "model.summary()\n",
    "\n",
    "model.compile(Adam(lr=<FILL IN>), loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n",
    "history = model.fit(X, Y, epochs=25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5\n",
    "Our model improves, but slowly. That is because it has low capacity. There is only one hidden layer with a small number of units.\n",
    "\n",
    "Increase the number of units in the hidden layer and retrain the model by filling out the skeleton below. Try to get the accuracy at epoch 25 greater than or equal to 0.7. How many units do you need? Motivate your answer in the cell below.\n",
    "\n",
    "**Hint**:\n",
    "* Use the learning rate that worked for you in exercise 4. Adapt it if you think the model does not converge well enough.\n",
    "* Increase the number of hidden units by factors of 2 or 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Flatten(input_shape=X.shape[1:]))\n",
    "model.add(Dense(<FILL IN>, activation='relu'))\n",
    "model.add(Dense(10, activation='softmax'))\n",
    "model.summary()\n",
    "\n",
    "model.compile(Adam(lr=<FILL IN>), loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n",
    "history = model.fit(X, Y, epochs=25)\n",
    "lib.plot_history(history);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validation set\n",
    "We will now introduce a **validation set** into our problem to verify our network and optimizer configuration. This is as simple as providing the Keras [`fit`](https://keras.io/models/sequential/#fit) function with an argument for its `validation_split` parameter. This parameter governs what proportion of the original data set is set aside for validation during training. After each epoch, Keras will run the validation set through the network to obtain the validation loss and accuracy for that epoch.\n",
    "\n",
    "## Exercise 6\n",
    "In the following cell we have set this parameter to 0.2, so that 20% of our original data is used for validation. Fill out the network by adding two hidden layers: the first with 512 units, the second with 256. Both should have ReLU activation.\n",
    "\n",
    "Train the network for 25 epochs with learning rate 0.00001 and batch size 256 (the `batch_size` parameter in Keras's `fit` function). Inspect the resulting accuracy and loss curves. How do you explain the divergence between the training and validation loss and accuracy?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Flatten(input_shape=X.shape[1:]))\n",
    "<FILL IN>\n",
    "model.add(Dense(10, activation='softmax'))\n",
    "model.summary()\n",
    "\n",
    "model.compile(Adam(lr=<FILL IN>), loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n",
    "history = model.fit(X, Y, epochs=<FILL IN>, batch_size=<FILL IN>, validation_split=0.2)\n",
    "lib.plot_history(history);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus exercise (hard)\n",
    "How many parameters do we have in the first dense layer of exercise 2, and why? Write down your answer in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus exercise (hard)\n",
    "Play around with the `batch_size` parameter of the network in the previous cell. What happens with the training speed and network convergence compared with the previous example? Why? Fill in your answer in the cell below.\n",
    "\n",
    "**Hint**: try to increase or decrease the batch size by factors of 4 or 8. What happens for very low batch sizes (single digits), and what happens with very high batch sizes (triple digits)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Flatten(input_shape=X.shape[1:]))\n",
    "model.add(Dense(512, activation='relu'))\n",
    "model.add(Dense(256, activation='relu'))\n",
    "model.add(Dense(10, activation='softmax'))\n",
    "model.summary()\n",
    "\n",
    "model.compile(Adam(lr=0.0001), loss=\"categorical_crossentropy\", metrics=[\"accuracy\"])\n",
    "history = model.fit(X, Y, epochs=25, batch_size=512, validation_split=0.2)\n",
    "lib.plot_history(history);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN YOUR ANSWER HERE"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}