codes/01_gradient.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📝 **Author:** Amirhossein Heydari - 📧 **Email:** amirhosseinheydari78@gmail.com - 📍 **Linktree:** [linktr.ee/mr_pylin](https://linktr.ee/mr_pylin)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!DOCTYPE html>\n",
    "<html>\n",
    "<head></head>\n",
    "<body>\n",
    "    <h1>A Simple Neuron Structure (Perceptron)</h1>\n",
    "    <ul><li>In many contexts, the terms \"neuron\" and \"perceptron\" are used interchangeably</li></ul>\n",
    "    <div style=\"display: flex; margin-top: 50px;\">\n",
    "        <div style=\"width: 20%;\">\n",
    "            <table style=\"margin-left: auto; margin-right: auto;\">\n",
    "                <caption>Dataset</caption>\n",
    "                <tr>\n",
    "                    <th>#</th>\n",
    "                    <th><span style=\"color: cyan;\">x<sub>1</span></th>\n",
    "                    <th><span style=\"color: cyan;\">x<sub>2</span></th>\n",
    "                    <th><span style=\"color: #FF9999;\">y</span></th>\n",
    "                </tr>\n",
    "                <tr>\n",
    "                    <th><span style=\"color: #99DD00\">1</span></th>\n",
    "                    <td>1</td>\n",
    "                    <td>1</td>\n",
    "                    <td>2</td>\n",
    "                </tr>\n",
    "                <tr>\n",
    "                    <th><span style=\"color: #99DD00\">2</span></th>\n",
    "                    <td>2</td>\n",
    "                    <td>3</td>\n",
    "                    <td>5</td>\n",
    "                </tr>\n",
    "                <tr>\n",
    "                    <th><span style=\"color: #99DD00\">3</span></th>\n",
    "                    <td>1</td>\n",
    "                    <td>2</td>\n",
    "                    <td>3</td>\n",
    "                </tr>\n",
    "                <tr>\n",
    "                    <th><span style=\"color: #99DD00\">4</span></th>\n",
    "                    <td>3</td>\n",
    "                    <td>1</td>\n",
    "                    <td>4</td>\n",
    "                </tr>\n",
    "                <tr>\n",
    "                    <th><span style=\"color: #99DD00\">5</span></th>\n",
    "                    <td>2</td>\n",
    "                    <td>4</td>\n",
    "                    <td>6</td>\n",
    "                </tr>\n",
    "            </table>\n",
    "        </div>\n",
    "        <div style=\"width: 80%;\">\n",
    "            <figure style=\"text-align: center;\">\n",
    "                <img src=\"../resources/images/SVGs/perceptron-1.svg\" alt=\"perceptron-1.png\" style=\"width: 100%;\">\n",
    "                <figcaption style=\"text-align: center;\">A simple Neuron (Perceptron)</figcaption>\n",
    "            </figure>\n",
    "        </div>\n",
    "    </div>\n",
    "    <div style=\"margin-top: 50px;\">\n",
    "        <h2>How to estimate <span style=\"color: #FF9999;\">y</span> ?</h2>\n",
    "\n",
    "   1. <span style=\"color: orange;\">System of Equations</span>\n",
    "\n",
    "        $$\\left\\{\n",
    "        \\begin{aligned}\n",
    "        1w_1 + 1w_2 &= 2 \\\\\n",
    "        2w_1 + 3w_2 &= 5 \\\\\n",
    "        1w_1 + 2w_2 &= 3 \\\\\n",
    "        3w_1 + 1w_2 &= 4 \\\\\n",
    "        2w_1 + 4w_2 &= 6 \\\\\n",
    "        \\end{aligned}\n",
    "        \\right.$$\n",
    "\n",
    "      <ul>\n",
    "        <li>Disadvantages</li>\n",
    "            <ul>\n",
    "                <li><span style=\"font-family: Consolas;\">Complexity &nbsp;&nbsp;:</span> Neural networks are highly complex systems with millions of parameters <a href= \"https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/\">[GPT-4 has 1.76 trillion parameters]</a></li>\n",
    "                <li><span style=\"font-family: Consolas;\">Non-linearity:</span> Neural networks use <u>activation functions like Sigmoid</u>, which introduce non-linearity into the network</li>\n",
    "            </ul>\n",
    "        <li>Critical issue [Overdetermined system]</li>\n",
    "            <ul>\n",
    "                <li>the number of <u>equations</u> is more than the number of <u>unknowns</u></li>\n",
    "                <li>the system is inconsistent and cannot be solved exactly</li>\n",
    "                <li>it is possible to have either no solution or an infinite number of solutions</li>\n",
    "            </ul>\n",
    "      </ul>\n",
    "\n",
    "   2. <span style=\"color: lightgreen;\">Delta Rule</span>\n",
    "      - The delta rule, also known as the <u>Widrow-Hoff</u> rule or the <u>LMS</u> (least mean squares) rule\n",
    "      - The delta rule is commonly associated with the <u>Adaline</u> (Adaptive Linear Neuron) model\n",
    "      - It is a simple supervised learning rule used for training single-layer neural networks <u>(perceptrons)</u>\n",
    "   3. <span style=\"color: lightgreen;\">Backpropagation</span>\n",
    "      - Backpropagation is an extended version of Delta Rule for multi-layer neural networks.\n",
    "      - It allows the network to learn from its mistakes by updating the weights iteratively using <span style=\"color: tomato;\">Gradient Descent</span> (aka Steepest Descent).\n",
    "   4. ...\n",
    "\n",
    "</body>\n",
    "</html>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient\n",
    "   - the rate of change of the `output` with respect to the `input` at a particular point in the function.\n",
    "   - it can be seen as the generalization of the derivative for functions with multiple variables.\n",
    "   - it doesn't have a direction for one-dimensional functions.\n",
    "\n",
    "# autograd\n",
    "   - PyTorch’s automatic differentiation engine that powers neural network training\n",
    "   - `torch.Tensor.backward()` is used to compute gradients and update the grad attribute of the input tensors.\n",
    "   - `torch.Tensor.grad` is used to access the computed gradients stored in the grad attribute"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1: $f(x) = 2x + 3 \\rightarrow \\nabla f(x) = \\frac{\\partial f}{\\partial x} = 2$\n",
    "   - $\\nabla f(4) = 2$\n",
    "   - $\\nabla f(0) = 2$\n",
    "   - $\\nabla f(1) = 2$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x     : tensor(1., requires_grad=True)\n",
      "y     : tensor(5., grad_fn=<AddBackward0>)\n",
      "x.grad: tensor(2.)\n"
     ]
    }
   ],
   "source": [
    "def f(x: torch.Tensor):\n",
    "    return 2 * x + 3  # torch.add(torch.multiply(2, x), 3)\n",
    "\n",
    "\n",
    "# x: independent variable\n",
    "x = torch.tensor(1, dtype=torch.float32, requires_grad=True)\n",
    "\n",
    "# f(x) or y : dependent variable\n",
    "y = f(x)\n",
    "\n",
    "# compute the gradients with respect to all Tensors that have `requires_grad=True`\n",
    "y.backward()\n",
    "\n",
    "# access computed gradients\n",
    "# if x at 1 moves by ε, then y moves by 2ε\n",
    "gradients = x.grad\n",
    "\n",
    "# log\n",
    "print('x     :', x)\n",
    "print('y     :', y)\n",
    "print(\"x.grad:\", gradients)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot\n",
    "_ = np.linspace(-4, 6, 100)\n",
    "plt.figure(figsize=(6, 4))\n",
    "plt.title(f\"x.grad: {x.grad}\")\n",
    "plt.plot(_, f(_), label=\"f(x) = 2x + 3\", color='blue')\n",
    "plt.axvline(x=x.item(), color='red', linestyle='--', label=f\"x = {x}\")\n",
    "plt.axhline(y=f(x).item(), color='green', linestyle='--', label=f\"y = {f(x)}\")\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('f(x)')\n",
    "plt.xticks(range(-10, 16, 2))\n",
    "plt.yticks(range(-10, 16, 2))\n",
    "plt.grid(True)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: $f(x) = 3x^2 - 2x + 5 \\quad\\rightarrow\\quad \\nabla f(x) = \\frac{\\partial f}{\\partial x} = 6x - 2$\n",
    "   - $\\nabla f(3) = 16$\n",
    "   - $\\nabla f(0) = -2$\n",
    "   - $\\nabla f(1) = 4$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x     : tensor(3., requires_grad=True)\n",
      "y     : tensor(26., grad_fn=<AddBackward0>)\n",
      "x.grad: 16.0 [at x=3.0]\n"
     ]
    }
   ],
   "source": [
    "def f(x):\n",
    "    # torch.add(torch.sub(torch.mul(3, torch.pow(x, 2)), torch.mul(2, x)), 5)\n",
    "    return 3 * x ** 2 - 2 * x + 5\n",
    "\n",
    "\n",
    "x = torch.tensor(3, dtype=torch.float32, requires_grad=True)\n",
    "y = f(x)\n",
    "\n",
    "# compute the gradients with respect to all Tensors that have `requires_grad=True`\n",
    "y.backward()\n",
    "\n",
    "# access computed gradients\n",
    "# if x at 3 moves by ε, then y moves by (6 * 3 - 2)ε\n",
    "gradients = x.grad\n",
    "\n",
    "# log\n",
    "print('x     :', x)\n",
    "print('y     :', y)\n",
    "print(f\"x.grad: {gradients} [at x={x}]\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot\n",
    "_ = np.linspace(-5, 5, 100)\n",
    "plt.figure(figsize=(6, 4))\n",
    "plt.title(f\"x.grad: {x.grad}\")\n",
    "plt.plot(_, f(_), label=\"f(x) = 3x^2 - 2x + 5\", color='blue')\n",
    "plt.axvline(x=x.item(), color='red', linestyle='--', label=f\"x = {x}\")\n",
    "plt.axhline(y=f(x).item(), color='green', linestyle='--', label=f\"y = {f(x).item()}\")\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('f(x)')\n",
    "plt.xticks(range(-5, 6))\n",
    "plt.yticks(range(0, 101, 10))\n",
    "plt.grid(True)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3: $f(w_1, w_2) = w_1x_1 + w_2x_2 \\quad\\rightarrow\\quad \\nabla f(W) = \\left( \\frac{\\partial f}{\\partial w_1}, \\frac{\\partial f}{\\partial w_2} \\right) = (x_1, x_2)$\n",
    "   - `magnitude:` $|\\nabla f(w_1, w_2)| = \\sqrt{x_1^2 + x_2^2}$\n",
    "\n",
    "   - `direction:` $\\nabla f(w_1, w_2) = \\frac{x_1}{\\sqrt{x_1^2 + x_2^2}} \\hat{i} + \\frac{x_2}{\\sqrt{x_1^2 + x_2^2}} \\hat{j}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W: tensor([1., 2.], requires_grad=True)\n",
      "X: tensor([2., 3.])\n",
      "y: tensor(8., grad_fn=<DotBackward0>)\n",
      "--------------------------------------------------\n",
      "magnitude of gradients: 3.605551242828369\n",
      "direction of gradients: tensor([0.5547, 0.8321])\n"
     ]
    }
   ],
   "source": [
    "def f(X, W):\n",
    "    return torch.dot(X, W)\n",
    "\n",
    "\n",
    "W = torch.tensor([1, 2], dtype=torch.float32, requires_grad=True)\n",
    "X = torch.tensor([2, 3], dtype=torch.float32)\n",
    "y = f(X, W)\n",
    "\n",
    "# compute the gradients\n",
    "y.backward()\n",
    "\n",
    "# access the gradients\n",
    "gradients = W.grad\n",
    "\n",
    "magnitude_grad = torch.norm(gradients)      # same as (grad ** 2).sum().sqrt()\n",
    "direction_grad = gradients / magnitude_grad  # normalized (unit vector)\n",
    "\n",
    "# log\n",
    "print('W:', W)\n",
    "print('X:', X)\n",
    "print('y:', y)\n",
    "print('-' * 50)\n",
    "print('magnitude of gradients:', magnitude_grad.item())\n",
    "print('direction of gradients:', direction_grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot\n",
    "w1 = np.linspace(-10, 10, 100)\n",
    "w2 = np.linspace(-10, 10, 100)\n",
    "X1, X2 = np.meshgrid(w1, w2)\n",
    "_ = X1 * W[0].detach().numpy() + X2 * W[1].detach().numpy()\n",
    "\n",
    "fig = plt.figure(figsize=(12, 4), layout='compressed')\n",
    "\n",
    "ax1 = fig.add_subplot(121, projection='3d')\n",
    "ax1.plot_surface(X1, X2, _, cmap='viridis')\n",
    "ax1.set_xlabel('w1')\n",
    "ax1.set_ylabel('w2')\n",
    "ax1.set_zlabel('f(w1, w2)')\n",
    "ax1.set_title(\"f(w1, w2) = 2w1 + 3w2\")\n",
    "ax2 = fig.add_subplot(122)\n",
    "ax2.quiver(0, 0, direction_grad[0], direction_grad[1], angles='xy', scale_units='xy', scale=1, color='red')\n",
    "ax2.set_xlim(-2, 2)\n",
    "ax2.set_ylim(-2, 2)\n",
    "ax2.set_xlabel('w1')\n",
    "ax2.set_ylabel('w2')\n",
    "ax2.set_title(\"Direction of gradients\")\n",
    "ax2.grid('on')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Descent\n",
    "   - The gradient direction is indeed the direction in which a function increases most rapidly\n",
    "   - To minimize the loss function, we shall move in the opposite of the gradient direction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 4: $f(w_1, w_2, b) = w_1x_1 + w_2x_2 + b \\quad\\rightarrow\\quad \\nabla f(W) = \\left( \\frac{\\partial f}{\\partial w_1}, \\frac{\\partial f}{\\partial w_2}, \\frac{\\partial f}{\\partial b} \\right) = (x_1, x_2, 1)$\n",
    "\n",
    "![Image Alt Text](../resources/images/SVGs/logistic-regression.svg)\n",
    "\n",
    "$\n",
    "    W = \\begin{bmatrix} w_0 \\\\ w_1 \\\\ w_2 \\end{bmatrix}\\quad,\\quad\n",
    "    X = \\begin{bmatrix} 1 \\\\ x_1 \\\\ x_2 \\end{bmatrix}\\quad,\\quad\n",
    "    output = W^T X = \\begin{bmatrix} w_0 \\ w_1 \\ w_2 \\end{bmatrix}.\\begin{bmatrix} 1 \\\\ x_1 \\\\ x_2 \\end{bmatrix}= w_0 + w_1x_1 + w_2x_2\n",
    "$\n",
    "\n",
    "#### Chain Rule\n",
    "   - Activation function must be differentiable\n",
    "   - Loss (error) function must be differentiable\n",
    "$$\n",
    "\\nabla L(W) = (\\frac{\\partial \\text{loss}}{\\partial y_{\\text{pred}}} \\cdot \\frac{\\partial y_{\\text{pred}}}{\\partial \\text{output}} \\cdot \\frac{\\partial \\text{output}}{\\partial W})\n",
    "$$\n",
    "\n",
    "#### Updating Weights\n",
    "$$\n",
    "W_{new} = W_{old} - \\alpha \\nabla L(W_{old})\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Binary Classification [0, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### in-place operation with `requires_grad=True` on a leaf_node\n",
    "   - you can't perform in-place operations on tensors that require gradients `[e.g. updating weights]`\n",
    "   - It disrupts tracking the operations on tensors to compute gradients during backpropagation\n",
    "   - operations that end with an underscore (e.g., add_()) are considered as \"in-place\" operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x1: tensor(1., dtype=torch.float64)\n",
      "x2: tensor(1., dtype=torch.float64, grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "x1 = torch.tensor(0, dtype=torch.float64)\n",
    "x2 = torch.tensor(0, dtype=torch.float64, requires_grad=True)\n",
    "\n",
    "# regular assignment\n",
    "x1 = x1 + 1  # x1 = x1.add(1)\n",
    "x2 = x2 + 1  # x2 = x2.add(1)\n",
    "\n",
    "# log\n",
    "print('x1:', x1)\n",
    "print('x2:', x2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a leaf Variable that requires grad is being used in an in-place operation.\n",
      "x1: tensor(1., dtype=torch.float64)\n",
      "x2: tensor(0., dtype=torch.float64, requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "x1 = torch.tensor(0, dtype=torch.float64)\n",
    "x2 = torch.tensor(0, dtype=torch.float64, requires_grad=True)\n",
    "\n",
    "# in-place assignment\n",
    "x1 += 1  # x1.add_(1)\n",
    "\n",
    "try:\n",
    "    x2 += 1  # x2.add_(1)\n",
    "except RuntimeError as e:\n",
    "    print(e)\n",
    "\n",
    "# log\n",
    "print('x1:', x1)\n",
    "print('x2:', x2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Dataset\n",
    "   - $x = [2, 3] \\quad,\\quad y = 0$\n",
    "   - Note: $x$ is a single sample with two features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0\n",
      "y_true    : 0 (label)\n",
      "y_pred    : 0.960834264755249\n",
      "prediction: 1 (label)\n",
      "loss: 0.9232024550437927\n",
      "grad: tensor([0.0723, 0.1446, 0.2169])\n",
      "step: tensor([0.0362, 0.0723, 0.1085])\n",
      "W_new tensor([0.2638, 0.6277, 0.3915], requires_grad=True)\n",
      "--------------------------------------------------\n",
      "epoch: 1\n",
      "y_true    : 0 (label)\n",
      "y_pred    : 0.9366591572761536\n",
      "prediction: 1 (label)\n",
      "loss: 0.8773303627967834\n",
      "grad: tensor([0.1111, 0.2223, 0.3334])\n",
      "step: tensor([0.0556, 0.1111, 0.1667])\n",
      "W_new tensor([0.2083, 0.5165, 0.2248], requires_grad=True)\n",
      "--------------------------------------------------\n",
      "epoch: 2\n",
      "y_true    : 0 (label)\n",
      "y_pred    : 0.8716690540313721\n",
      "prediction: 1 (label)\n",
      "loss: 0.7598069310188293\n",
      "grad: tensor([0.1950, 0.3900, 0.5850])\n",
      "step: tensor([0.0975, 0.1950, 0.2925])\n",
      "W_new tensor([ 0.1108,  0.3215, -0.0677], requires_grad=True)\n",
      "--------------------------------------------------\n",
      "epoch: 3\n",
      "y_true    : 0 (label)\n",
      "y_pred    : 0.6342986822128296\n",
      "prediction: 1 (label)\n",
      "loss: 0.4023348093032837\n",
      "grad: tensor([0.2943, 0.5885, 0.8828])\n",
      "step: tensor([0.1471, 0.2943, 0.4414])\n",
      "W_new tensor([-0.0364,  0.0273, -0.5091], requires_grad=True)\n",
      "--------------------------------------------------\n",
      "epoch: 4\n",
      "y_true    : 0 (label)\n",
      "y_pred    : 0.18106062710285187\n",
      "prediction: 0 (label)\n",
      "loss: 0.03278294950723648\n",
      "grad: tensor([0.0537, 0.1074, 0.1611])\n",
      "step: tensor([0.0268, 0.0537, 0.0805])\n",
      "W_new tensor([-0.0632, -0.0264, -0.5897], requires_grad=True)\n",
      "--------------------------------------------------\n",
      "epoch: 5\n",
      "y_true    : 0 (label)\n",
      "y_pred    : 0.13181111216545105\n",
      "prediction: 0 (label)\n",
      "loss: 0.01737416908144951\n",
      "grad: tensor([0.0302, 0.0603, 0.0905])\n",
      "step: tensor([0.0151, 0.0302, 0.0453])\n",
      "W_new tensor([-0.0783, -0.0566, -0.6349], requires_grad=True)\n",
      "--------------------------------------------------\n",
      "epoch: 6\n",
      "y_true    : 0 (label)\n",
      "y_pred    : 0.1094651073217392\n",
      "prediction: 0 (label)\n",
      "loss: 0.011982609517872334\n",
      "grad: tensor([0.0213, 0.0427, 0.0640])\n",
      "step: tensor([0.0107, 0.0213, 0.0320])\n",
      "W_new tensor([-0.0890, -0.0779, -0.6669], requires_grad=True)\n",
      "--------------------------------------------------\n",
      "epoch: 7\n",
      "y_true    : 0 (label)\n",
      "y_pred    : 0.09572887420654297\n",
      "prediction: 0 (label)\n",
      "loss: 0.009164017625153065\n",
      "grad: tensor([0.0166, 0.0331, 0.0497])\n",
      "step: tensor([0.0083, 0.0166, 0.0249])\n",
      "W_new tensor([-0.0973, -0.0945, -0.6918], requires_grad=True)\n",
      "--------------------------------------------------\n",
      "epoch: 8\n",
      "y_true    : 0 (label)\n",
      "y_pred    : 0.0861462652683258\n",
      "prediction: 0 (label)\n",
      "loss: 0.007421179208904505\n",
      "grad: tensor([0.0136, 0.0271, 0.0407])\n",
      "step: tensor([0.0068, 0.0136, 0.0203])\n",
      "W_new tensor([-0.1040, -0.1081, -0.7121], requires_grad=True)\n",
      "--------------------------------------------------\n",
      "epoch: 9\n",
      "y_true    : 0 (label)\n",
      "y_pred    : 0.07895941287279129\n",
      "prediction: 0 (label)\n",
      "loss: 0.006234589032828808\n",
      "grad: tensor([0.0115, 0.0230, 0.0345])\n",
      "step: tensor([0.0057, 0.0115, 0.0172])\n",
      "W_new tensor([-0.1098, -0.1196, -0.7293], requires_grad=True)\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# y = 0\n",
    "y_true = torch.tensor(0, dtype=torch.int64)\n",
    "\n",
    "# 1 is the multiplication for bias\n",
    "X = torch.tensor([1, 2, 3], dtype=torch.float32)\n",
    "\n",
    "# initial weights [bias = .3]\n",
    "W = torch.tensor([.3, .7, .5], dtype=torch.float32, requires_grad=True)\n",
    "\n",
    "# hyper parameters\n",
    "epochs = 10\n",
    "learning_rate = .5\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    print('epoch:', epoch)\n",
    "\n",
    "    # feed-forward\n",
    "    output = torch.dot(X, W)\n",
    "    y_pred = torch.sigmoid(output)\n",
    "    print(f\"y_true    : {y_true.item()} (label)\")\n",
    "    print(f\"y_pred    : {y_pred.item()}\")\n",
    "    print(f\"prediction: {torch.where(y_pred < .5, 0, 1)} (label)\")\n",
    "\n",
    "    # loss\n",
    "    loss = (y_pred - y_true) ** 2\n",
    "    print('loss:', loss.item())\n",
    "\n",
    "    # backward\n",
    "    loss.backward()\n",
    "    dW = W.grad\n",
    "    step = learning_rate * dW\n",
    "    print('grad:', dW)\n",
    "    print('step:', step)\n",
    "\n",
    "    # update weights [method 1]\n",
    "    # W.requires_grad_(False)\n",
    "    # W -= step\n",
    "    # W.grad.zero_()\n",
    "    # W.requires_grad_(True)\n",
    "\n",
    "    # update weights [method 2]\n",
    "    # W = W.detach() - step\n",
    "    # W.requires_grad_(True)\n",
    "\n",
    "    # update weights [method 3] : preferred\n",
    "    with torch.no_grad():\n",
    "        W -= step\n",
    "        W.grad.zero_()\n",
    "\n",
    "    print('W_new', W)\n",
    "    print('-' * 50)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}