line_model.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "line_model.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Machine-Learning-Tokyo/intro-to-DL/blob/master/line_model.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "-Jv0xHZ3BtKp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Line model\n",
        "\n",
        "This notebook has a very simple example of how to build a one layer-model in keras to find the parameters of a given line.\n",
        "\n",
        "First, we import some necessary packages, like keras for the model, pyplot for plotting the generated line segments and numpy for the array caclucations\n"
      ]
    },
    {
      "metadata": {
        "id": "S4W_uQ3ryGIx",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import keras\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Activation\n",
        "from keras.optimizers import Adam\n",
        "import keras.backend as K\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9H2hsqKODRCo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We start by creating a line segment\n",
        "\n",
        "We actually create some pairs of x, y numbers and plot them as points.\n",
        "If we connect these points we get a line segment."
      ]
    },
    {
      "metadata": {
        "id": "s56BcOXqChd7",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "x = np.linspace(0, 10, 11)\n",
        "y = np.linspace(-2, 3, 11)\n",
        "\n",
        "row_format =\"{:>6}\" * (len(x))\n",
        "print('y = a*x + b\\n')\n",
        "print('x:', row_format.format(*x))\n",
        "print('y:', row_format.format(*y), '\\n')\n",
        "print('y = 0.5*x - 2')\n",
        "\n",
        "plt.plot(x, y, '-')\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-VN0BqLtDm8P",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now let's build our first keras model. The model will have only one layer with one neuron.\n",
        "This means that the \"network\" will\n",
        "\n",
        "- take as input one number\n",
        "- pass the number through the neuron\n",
        "- apply a f(x) = a*x + b function inside the neuron\n",
        "- return the result of the neuron as output\n",
        "\n",
        "The summary of the model inform us that the model has one Dense layer with 2 parameters.\n",
        "\n",
        "These parameters are the \"a\" and \"b\" of the aforementioned equation"
      ]
    },
    {
      "metadata": {
        "id": "rQ3PCXKV80I7",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "K.clear_session()\n",
        "model =Sequential([Dense(1, input_shape=(1,))])\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "aR0E5rJpEf30",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "When building the model these parameters are randomly initialized.\n",
        "\n",
        "What we do in the next cell is that we pass the x values through the model and get the results (y_pred).\n",
        "\n",
        "Then we plot both the original points and the model's points."
      ]
    },
    {
      "metadata": {
        "id": "FjHTGecLDBUq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "a, b = model.get_weights()\n",
        "y_pred = model.predict(x)\n",
        "y_pred = np.round(y_pred.flatten().astype(float), 1)\n",
        "\n",
        "row_format =\"{:>6}\" * (len(x))\n",
        "\n",
        "print('y = 0.5*x - 2')\n",
        "print('x:     ', row_format.format(*x))\n",
        "print('y:     ', row_format.format(*y), '\\n')\n",
        "\n",
        "print('y_pred = %.1f*x + %.1f' % (a[0,0], b[0]))\n",
        "print('x:     ', row_format.format(*x))\n",
        "print('y_pred:', row_format.format(*y_pred), '\\n')\n",
        "\n",
        "plt.plot(x, y, '-')\n",
        "plt.plot(x, y_pred, '--')\n",
        "plt.yticks(ticks=range(-14, 15, 2))\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "gV6woA_uG3GK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As we can see the two line segments defined by the sets of points are not the same.\n",
        "\n",
        "Which means that the *y* values and the *y_pred* values are differnet.\n",
        "\n",
        "This happens because the *a* and *b* parameters that we defined are different from the *a* and *b* parameters of the neuron of the model.\n",
        "\n",
        "Thus we have to train the model to get these parameters as close as possible to ours.\n",
        "\n",
        "First of all we have to select some (or all) of the points to use them as training data.\n",
        "These points will be fed to the model as training data.\n",
        "We take three out of ten points as training data."
      ]
    },
    {
      "metadata": {
        "id": "lm3hdWagl8ff",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "x_train = x[0::5]\n",
        "y_train = y[0::5]\n",
        "\n",
        "\n",
        "row_format =\"{:>6}\" * (len(x_train))\n",
        "print('x_train:', row_format.format(*x_train))\n",
        "print('y_train:', row_format.format(*y_train))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Jz-krpSGIBd1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Before we start the training of the model we need to define the target and the way to achieve it.\n",
        "\n",
        "In our case we want the model's points to be as close as possible to the given points which means that the distance between y an y_pred for a given x shall be minimized.\n",
        "\n",
        "To ahcieve this we use a *loss function*. In our case the loss function will be the *mean squared error*\n",
        "\n",
        "Also we need to define a method based upon the model will try to minimize the loss function.\n",
        "\n",
        "The method (also called optimizer since it optimize the model's parameters) that we will use is Adam.\n",
        "\n",
        "We don't need to get into too much details for this one."
      ]
    },
    {
      "metadata": {
        "id": "k_BGgf7Vz5jl",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model.compile(optimizer=Adam(0.1), loss='mean_squared_error')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "IoyM1giGI3rp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now we will train our model to minimize the distance between the the given 3 points and the generated ones.\n",
        "\n",
        "We will feed the data once to the model and see how the parameters of the model change."
      ]
    },
    {
      "metadata": {
        "id": "kGnMaKbXmoOs",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model.train_on_batch(x_train, y_train)\n",
        "y_pred = model.predict(x)\n",
        "\n",
        "y_pred = np.round(y_pred.flatten().astype(float), 1)\n",
        "a, b = model.get_weights()\n",
        "\n",
        "row_format =\"{:>6}\" * (len(x))\n",
        "\n",
        "print('y =       0.5*x + -2')\n",
        "print('x:     ', row_format.format(*x))\n",
        "print('y:     ', row_format.format(*y), '\\n')\n",
        "\n",
        "print('y_pred = %.1f*x + %.1f' % (a[0,0], b[0]))\n",
        "print('x:     ', row_format.format(*x))\n",
        "print('y_pred:', row_format.format(*y_pred), '\\n')\n",
        "\n",
        "plt.plot(x, y, '-')\n",
        "plt.plot(x, y_pred, '--')\n",
        "\n",
        "plt.yticks(ticks=range(-14, 15, 2))\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4eA5DKYvJnhN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As we can see it takes more that one repetition for the model to find the right set of parameters for the given problem.\n",
        "\n",
        "Let's train the model on the given data a few times more"
      ]
    },
    {
      "metadata": {
        "id": "slZK3am5EfJd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for i in range(20):\n",
        "  model.train_on_batch(x_train, y_train)\n",
        "\n",
        "\n",
        "y_pred = model.predict(x)\n",
        "\n",
        "a, b = model.get_weights()\n",
        "y_pred = np.round(y_pred.flatten().astype(float), 1)\n",
        "\n",
        "row_format =\"{:>6}\" * (len(x))\n",
        "\n",
        "print('y =       0.5*x + -2')\n",
        "print('x:     ', row_format.format(*x))\n",
        "print('y:     ', row_format.format(*y), '\\n')\n",
        "\n",
        "print('y_pred = %.1f*x + %.1f' % (a[0,0], b[0]))\n",
        "print('x:     ', row_format.format(*x))\n",
        "print('y_pred:', row_format.format(*y_pred), '\\n')\n",
        "\n",
        "plt.plot(x, y, '-')\n",
        "plt.plot(x, y_pred, '--')\n",
        "\n",
        "plt.yticks(ticks=range(-20, 15, 2))\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PehY2r3QKAPS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "After a few more repetitions the model is able to generate new points on the line that we used to train it.\n",
        "\n",
        "That was it!\n",
        "\n",
        "We just trained a model to generate points on a given line.\n",
        "We did it not by giving to the model the function of the line but by giving it three points of the line and train it to learn the function of the line by itself.\n",
        "\n",
        "Not that astonishing but actually this is more or less the core of what is going on when training a model on a task.\n",
        "\n",
        "We change the internal parameters of the model so that it outputs the values that we want.\n",
        "\n",
        "The result is a (usually very complicated) function that takes some input and transforms it to the desired output.\n",
        "\n",
        "### The end"
      ]
    }
  ]
}