05_transfer_learning_for_image_classification.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"//fonts.googleapis.com/css?family=Quicksand:300\" />\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"custom.css\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%HTML\n",
    "<link rel=\"stylesheet\" type=\"text/css\" href=\"//fonts.googleapis.com/css?family=Quicksand:300\" />\n",
    "<link rel=\"stylesheet\" type=\"text/css\" href=\"custom.css\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ['KERAS_BACKEND'] = 'tensorflow'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "\n",
    "from pprint import pprint\n",
    "\n",
    "import matplotlib.image as mpimg\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from keras import datasets\n",
    "from keras import layers\n",
    "from keras.models import Model\n",
    "from keras.preprocessing import image\n",
    "\n",
    "from utils import plot_training_summary\n",
    "from utils import TimeSummary\n",
    "from utils import save_keras_dataset_to_disk\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams['figure.figsize'] = 15, 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Transfer learning for image classification\n",
    "\n",
    "- Rodrigo Agundez\n",
    "- Amsterdam @ Booking.com\n",
    "- Friday May 25th, 2018\n",
    "\n",
    "![footer_logo](images/logo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Goal**\n",
    "\n",
    "- Use transfer learning model to classify images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-trained models available in Keras\n",
    "\n",
    "There are several pre-trained models for image classification in Keras. Here is a summary:\n",
    "\n",
    "![pre-trained models](images/pre_trained_models.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ### extras\n",
    "> \n",
    "Have a look at the documentation about the different pre-trained models https://keras.io/applications/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inception V3\n",
    "\n",
    "Trained for `imagenet` over 1000 classes. Check the classes it was trained on [here](https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a).\n",
    "\n",
    "![inception V3](images/inceptionV3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MobileNet\n",
    "\n",
    "Trained for `imagenet` over 1000 classes. Check the classes it was trained on [here](https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a).\n",
    "\n",
    "![MobileNet](images/mobilenet.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict image using [`MobileNet`](https://keras.io/applications/#mobilenet)\n",
    "\n",
    "The model has been pre-trained on [ImageNet 1000 classes](https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.applications.mobilenet import decode_predictions\n",
    "from keras.applications.mobilenet import MobileNet\n",
    "from keras.applications.mobilenet import preprocess_input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = MobileNet(weights='imagenet') # this can take a bit\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "try the default first and then add your own image to the `image` directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_path = 'images/scorpion.jpg'\n",
    "plt.imshow(mpimg.imread(img_path));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default it expects an image of input shape of `(224, 224, 3)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = image.load_img(img_path, target_size=(224, 224))\n",
    "x = image.img_to_array(img)\n",
    "x = np.expand_dims(x, axis=0)\n",
    "x = preprocess_input(x)\n",
    "preds = decode_predictions(model.predict(x), top=5)[0]\n",
    "preds = [(x[1], x[2]) for x in preds]\n",
    "pprint(preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# del model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data \n",
    "\n",
    "We will use the CIFAR10 dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train, y_train), (X_test, y_test) = datasets.cifar10.load_data()\n",
    "[ax.imshow(random.choice(X_train), cmap='gray') for ax in plt.subplots(1, 6)[1]];"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to simulate a more realistic scenario I'll save the images to disk and we will ingest them from there to the model input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# takes around 1.5 minutes and occupies ~50MB\n",
    "save_keras_dataset_to_disk(X_train, y_train, X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this only works if you have the tree command\n",
    "! tree -L 2 data/CIFAR10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use pre-trained `MobileNet` model as base model\n",
    "\n",
    "load model without last classification layer and set input_shape as closer as possible to original data. Not all input shapes work, check the [documentation](https://keras.io/applications/#mobilenet)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_model = MobileNet(weights='imagenet', include_top=False, input_shape=(128, 128, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "compare last layers with the full model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "freeze weights for these layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for layer in base_model.layers:\n",
    "    layer.trainable = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1: add custom layers for our CIFAR10 dataset problem (solution in `utils.py`)\n",
    "\n",
    "using the [`functional` API](https://keras.io/getting-started/functional-api-guide/) you will:\n",
    "\n",
    "- get the output from the base model\n",
    "- add `GlobalAveragePooling2D` layer\n",
    "- add `Dense` layer of 512 units\n",
    "- add output `Dense` layer with 10 units\n",
    "- put layers together into custom model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_inceptionV3_custom_model(base_model):\n",
    "    # get base model output\n",
    "    x = base_model.output\n",
    "    \n",
    "    # add GlobalAveragePooling2D layer\n",
    "\n",
    "    # add Dense layer of 512 units\n",
    "\n",
    "    # add output Dense layer with 10 units and softmax activation function\n",
    "#     predictions =\n",
    "    \n",
    "    # put layers together into custom model\n",
    "    model = Model(inputs=base_model.input, outputs=predictions)\n",
    "    return model\n",
    "\n",
    "model = make_inceptionV3_custom_model(base_model)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **extra**\n",
    ">\n",
    "> Check how many trainable parameters in the summary, does it make sense?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "trainable layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for l in model.layers:\n",
    "    if l.trainable:\n",
    "        print(l)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = make_inceptionV3_custom_model(base_model)\n",
    "model.compile(\n",
    "    optimizer='Adam',\n",
    "    loss='categorical_crossentropy',\n",
    "    metrics=['accuracy']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2: Ingest `CIFAR10` data from file\n",
    "\n",
    "Implement [`ImageDataGenerator`](https://keras.io/preprocessing/image/) such that:\n",
    "- pixel values are re-scaled in the interval 0 to 1\n",
    "- when changing the image size to (128X128) the missing pixels are filled with 0 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_datagen = image.ImageDataGenerator(\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement [`ImageDataGenerator.flow_from_directory`](https://keras.io/preprocessing/image/) such that:\n",
    "- we read the images from `data/CIFAR10/train`\n",
    "- the target size is set to 128x128\n",
    "- the class vairable is set to categorical values (10 classes)\n",
    "- ensure shuffle\n",
    "- and the betch size is set by us"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 50\n",
    "\n",
    "train_generator = train_datagen.flow_from_directory(\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use  [`Sequential.fit_generator`](https://keras.io/models/sequential/) such that:\n",
    "- we ingest images using the `train_generator`\n",
    "- train for 5 `epochs`\n",
    "- use 10 `steps_per_epoch`\n",
    "- use a suitable `max_queue_size` (check your memory consumption)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time_summary = TimeSummary()\n",
    "summary = model.fit_generator(\n",
    "    # fill here\n",
    "    verbose=1,\n",
    "    callbacks=[time_summary]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_training_summary(summary, time_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **extra**\n",
    ">\n",
    "> Make a validator data generator and pass it to the [`Sequential.fit_generator`](https://keras.io/models/sequential/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pydata]",
   "language": "python",
   "name": "conda-env-pydata-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}