Exercise 06 - Binary Classification with 3 Artificial Neurons.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%config InlineBackend.figure_format = 'retina'\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "np.set_printoptions(precision=3)\n",
    "np.set_printoptions(suppress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Binary Classification with 3 Artificial Neurons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our binary classification task, we used the function  $F = f(\\mathbf{X},\\mathbf{Y})=A\\mathbf{X}+B\\mathbf{Y}+C$. This function corresponds to a simplified neuron without the activation function. Its inputs are $\\mathbf{X}$ and $\\mathbf{Y}$, their corresponding weights are $A$ and $B$, and $C$ represents the bias. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A neural network works by connecting multiple neurons in a network. Unlike a single neuron, a network of neurons is capable of creating smoother, non-linear decision boundaries between data points belonging to different classes we are interested in predicting. Our example network contains 5 neurons, distributed in 3 layers. The first layer contains two neurons which are our data points, and is called the __input layer__. The second layer, called a __hidden layer__, comprises of 2 identical neurons, $n_1$, and $n_2$, each multiplying our data points $\\mathbf{X}$ and $\\mathbf{Y}$ from the input layer by a set of randomly initialised parameters $A_i, B_i, C_i$, where $i=1,2$. The third layer, called the __output layer__ contains a single neuron $s$ that multiplies the outputs $N_1$, and $N_2$ of the previous neurons  by a new set of parameters $A_3$, $B_3$, and $C_3$, and outputs the result $S$, like shown on the following diagram."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/neural_networks_23.png\" alt=\"drawing\" width=\"650\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In machine learning this scheme is usually simplified as following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/neural_networks_24.png\" alt=\"drawing\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an activation function we will again be using sigmoid function $sig(x)$ defined as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "sig(x) = \\frac{1}{1+e^{-x}}\\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement the `sigmoid(x)` function in code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataset contains several 2D points:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.array([[ 1.2, 0.7],\n",
    "                 [-0.3,-0.5],\n",
    "                 [ 3.0, 0.1],\n",
    "                 [-0.1,-1.0],\n",
    "                 [-0.0, 1.1],\n",
    "                 [ 2.1,-1.3],\n",
    "                 [ 3.1,-1.8],\n",
    "                 [ 1.1,-0.1],\n",
    "                 [ 1.5,-2.2],\n",
    "                 [ 4.0,-1.0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each data point, there is an associated label `1` or `-1`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = np.array([  1,\n",
    "                    -1,\n",
    "                     1,\n",
    "                    -1,\n",
    "                    -1,\n",
    "                     1,\n",
    "                    -1,\n",
    "                     1,\n",
    "                    -1,\n",
    "                    -1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A python `plot_data` can be used to plot the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [],
   "source": [
    "def plot_data(data, labels):\n",
    "    fig = plt.figure(figsize=(5,5))\n",
    "    ax = fig.add_subplot(111)\n",
    "    ax.scatter(data[:,0], data[:,1], c=labels, s=50,  cmap=plt.cm.bwr,zorder=50)\n",
    "    nudge = 0.08\n",
    "    for i in range(data.shape[0]):\n",
    "        d = data[i]\n",
    "        ax.annotate(f'{i}',(d[0]+nudge,d[1]+nudge))\n",
    "    ax.set_aspect('equal', 'datalim')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the function `plot_data` to plot our data points and their labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`np.random.normal(size=x)` is a numpy function that can generate a number of normally distributed random values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the function to initialise the initial values for all the weights and biases of the neural network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print some of the values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Computing the forward pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take the first data point `[1.2, 0.7]` and store it in the variables `X` and `Y`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialise the variable `label` and store the corresponding label for the datapoint "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the forward pass (output of the whole network) and keep the intermediate steps `z1`, `z2`, `N1` and `N2` in separate variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialise the variable `sign` and set it to `1` or `-1` dependent on the output of the network and the label:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Computing the backward pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During the lecture we computed the partial derivatives for this network:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/neural_networks_23.png\" alt=\"drawing\" width=\"650\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The partial derivatives for the neuron $s$ are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align*}\n",
    "\\frac{\\partial S}{\\partial A_3} &=N_1; &\\frac{\\partial S}{\\partial B_3} &=N2; &\\frac{\\partial S}{\\partial C_3}&=1;\\\\\\\\\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialise the variables `dA3`, `dB3` and `dC3` and store the appropriate derivative values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The partial derivatives for the neuron $n_1$ are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align*}\n",
    "\\frac{\\partial S}{\\partial A_1} &=A_3\\cdot N_1(1-N_1)\\cdot X; &\\frac{\\partial S}{\\partial B_1} &=A_3\\cdot N_1(1-N_1)\\cdot Y; &\\frac{\\partial S}{\\partial C_1}&=A_3\\cdot N_1(1-N_1)\\\\\\\\\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialise the variables `dA1`, `dB1` and `dC1` and store the appropriate derivative values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The partial derivatives for the neuron $n_2$ are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align*}\n",
    "\\frac{\\partial S}{\\partial A_2} &=B_3\\cdot N_2(1-N_2)\\cdot X; &\\frac{\\partial S}{\\partial B_2} &=B_3\\cdot N_2(1-N_2)\\cdot Y; &\\frac{\\partial S}{\\partial C_2}&=B_3\\cdot N_2(1-N_2)\\\\\\\\\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialise the variables `dA2`, `dB2` and `dC2` and store the appropriate derivative values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Introduce the variable `step_size` to hold a very small number and minimise/maximise the weights and biases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the  function $f(\\mathbf{X},\\mathbf{Y})$, with the updated parameters $A_1—C_3$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This result should better then the original result!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 1:\n",
    "Write a simple algorithm for binary classification by using the previously written code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A simple neural network algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_meshgrid(data):\n",
    "    h = 0.02\n",
    "    x_min, x_max = data[:, 0].min() - 1, data[:, 0].max() + 1\n",
    "    y_min, y_max = data[:, 1].min() - 1, data[:, 1].max() + 1\n",
    "    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
    "    return (xx,yy,np.ones(xx.shape))\n",
    "\n",
    "def eval_accuracy_neural(params, data, labels):\n",
    "    A1, A2, A3, B1, B2, B3, C1, C2, C3 = params\n",
    "    num_correct = 0;\n",
    "    data_len = data.shape[0]\n",
    "    \n",
    "    for i in range(data_len):\n",
    "        X,Y = data[i]\n",
    "        true_label = labels[i]\n",
    "        output = A3*sigmoid(A1*X + B1*Y + C1) + B3*sigmoid(A2*X + B2*Y + C2) + C3\n",
    "        predicted_label = 1 if output > 1 else -1 if output < -1 else 0\n",
    "        \n",
    "        if (predicted_label == true_label):\n",
    "            num_correct += 1\n",
    "    return num_correct / data_len\n",
    "\n",
    "def plot_neural_simple(params, grid,data, labels, iteration, accuracy):\n",
    "    nudge = 0.06\n",
    "    A1, A2, A3, B1, B2, B3, C1, C2, C3 = params\n",
    "    xx,yy,Z = grid\n",
    "    \n",
    "    for i in range(xx.shape[0]): # row\n",
    "        for j in range(yy.shape[1]): #column\n",
    "            X, Y = xx[i][j],yy[i][j]\n",
    "            output = A3*sigmoid(A1*X + B1*Y + C1) + B3*sigmoid(A2*X + B2*Y + C2) + C3\n",
    "            predicted_label = 1 if output > 1 else -1 if output < -1 else 0\n",
    "            Z[i][j] = predicted_label\n",
    "\n",
    "    fig = plt.figure(figsize=(5,5))\n",
    "    ax = fig.add_subplot(111)\n",
    "    plt.title(f'accuracy at the iteration {iteration}: {accuracy}')\n",
    "    ax.contourf(xx, yy, Z, cmap=plt.cm.binary, alpha=0.1, zorder=15)\n",
    "    ax.scatter(data[:, 0], data[:, 1], c=labels, s=50,  cmap=plt.cm.bwr,zorder=50)\n",
    "    ax.set_aspect('equal')\n",
    "    for i in range(data.shape[0]):\n",
    "        d = data[i]\n",
    "        ax.annotate(f'{i}',(d[0]+nudge,d[1]+nudge))\n",
    "    plt.xlim(xx.min(), xx.max())\n",
    "    plt.ylim(yy.min(), yy.max())\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can put these elements together in a working algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     80,
     87,
     102
    ],
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def train_neural_network(data, labels, step_size, no_loops, iter_info):\n",
    "    rnd = np.random.normal(size=9)\n",
    "\n",
    "    # hidden layer neuron 1\n",
    "    A1 = rnd[0] #weight for X\n",
    "    B1 = rnd[1] #weight for Y\n",
    "    C1 = rnd[2] #bias\n",
    "\n",
    "    # hidden layer neuron 2\n",
    "    A2 = rnd[3] #weight for X\n",
    "    B2 = rnd[4] #weight for Y\n",
    "    C2 = rnd[5] #bias\n",
    "\n",
    "    # output layer neuron\n",
    "    A3 = rnd[6] #weight for n1\n",
    "    B3 = rnd[7] #weight for n2\n",
    "    C3 = rnd[8] # bias\n",
    "    \n",
    "    grid = create_meshgrid(data)\n",
    "\n",
    "    for i in range(no_loops):\n",
    "        # get a single random data point\n",
    "        index = np.random.randint(data.shape[0])\n",
    "        # get X, Y of that data point and its label\n",
    "        X,Y = data[index]\n",
    "        label = labels[index]\n",
    "\n",
    "        # forward pass\n",
    "        N1 = sigmoid(A1*X + B1*Y + C1) # 1st neuron\n",
    "        N2 = sigmoid(A2*X + B2*Y + C2) # 2nd neuron\n",
    "        S = A3*N1 + B3*N2 + C3 # final activation\n",
    "\n",
    "        sign = 1 if (label == 1 and S < 1) else -1 if (label == -1 and S > -1) else 0\n",
    "\n",
    "        # backpropagating through the network\n",
    "        # partial derivatives of the neuron s\n",
    "        dA3, dB3, dC3 = N1, N2, 1\n",
    "        # partial derivatives of the neuron n1\n",
    "        dz1 = A3*N1*(1-N1)\n",
    "        dA1, dB1, dC1 = dz1*X, dz1*Y, dz1\n",
    "        # partial derivatives of the neuron n2\n",
    "        dz2 = B3*N2*(1-N2)\n",
    "        dA2, dB2, dC2 = dz2*X, dz2*Y, dz2\n",
    "        \n",
    "        # finally, do the parameter update\n",
    "        step_size = 0.01\n",
    "        A1 += sign * dA1 * step_size\n",
    "        B1 += sign * dB1 * step_size\n",
    "        C1 += sign * dC1 * step_size\n",
    "        A2 += sign * dA2 * step_size\n",
    "        B2 += sign * dB2 * step_size\n",
    "        C2 += sign * dC2 * step_size\n",
    "        A3 += sign * dA3 * step_size\n",
    "        B3 += sign * dB3 * step_size\n",
    "        C3 += sign * dC3 * step_size\n",
    "        \n",
    "        params = A1, A2, A3, B1, B2, B3, C1, C2, C3\n",
    "        \n",
    "        if (i%iter_info==0):\n",
    "            accuracy = eval_accuracy_neural(params,data,labels)\n",
    "            plot_neural_simple(params,grid, data, labels, i, accuracy)\n",
    "        \n",
    "    return (A1, A2, A3, B1, B2, B3, C1, C2, C3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "train = train_neural_network(data, labels, 0.01, 30001, 2500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A1,A2,A3,B1,B2,B3,C1,C2,C3 = train\n",
    "for i, ((X,Y), label) in enumerate(zip(data, labels)):\n",
    "    output = A3*sigmoid(A1*X + B1*Y + C1) + B3*sigmoid(A2*X + B2*Y + C2) + C3\n",
    "    predicted_label = 1 if output > 1 else -1 if output < -1 else 0\n",
    "    print (f'''data point {i}: real label : {label}, pred. label: {predicted_label}, prediction: {(label==predicted_label)}''')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}