code/utils/activation-functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📝 **Author:** Amirhossein Heydari - 📧 **Email:** <amirhosseinheydari78@gmail.com> - 📍 **Origin:** [mr-pylin/pytorch-workshop](https://github.com/mr-pylin/pytorch-workshop)\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Table of contents**<a id='toc0_'></a>    \n",
    "- [Dependencies](#toc1_)    \n",
    "  - [`torch`](#toc1_1_)    \n",
    "  - [`torch.nn`](#toc1_2_)    \n",
    "  - [`torch.nn.functional`](#toc1_3_)    \n",
    "- [Activation Functions](#toc2_)    \n",
    "  - [Linear](#toc2_1_)    \n",
    "  - [Sigmoid](#toc2_2_)    \n",
    "  - [Hyperbolic Tangent (Tanh)](#toc2_3_)    \n",
    "  - [Softplus](#toc2_4_)    \n",
    "  - [LogSigmoid](#toc2_5_)    \n",
    "  - [Rectified Linear Unit (ReLU)](#toc2_6_)    \n",
    "  - [LeakyReLU](#toc2_7_)    \n",
    "  - [Exponential Linear Unit (ELU)](#toc2_8_)    \n",
    "  - [Sigmoid Linear Unit (SiLU)](#toc2_9_)    \n",
    "  - [Mish](#toc2_10_)    \n",
    "  - [Softmax](#toc2_11_)    \n",
    "  - [LogSoftmax](#toc2_12_)    \n",
    "  - [Gaussian Error Linear Units (GeLU)](#toc2_13_)    \n",
    "  - [Plot Activation Functions](#toc2_14_)    \n",
    "- [Threshold Functions](#toc3_)    \n",
    "  - [Step](#toc3_1_)    \n",
    "  - [Sign](#toc3_2_)    \n",
    "  - [Plot Threshold Functions](#toc3_3_)    \n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=1\n",
    "\tmaxLevel=6\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc1_'></a>[Dependencies](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc1_1_'></a>[`torch`](#toc0_)\n",
    "\n",
    "- Not commonly used directly in user code.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import relu, sigmoid, sign, softmax, tanh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc1_2_'></a>[`torch.nn`](#toc0_)\n",
    "\n",
    "- Creates a module.\n",
    "- Can be used as a layer in a neural network.\n",
    "- Suitable for building complex models.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.nn import ELU, GELU, LeakyReLU, LogSigmoid, LogSoftmax, Mish, ReLU, Sigmoid, SiLU, Softmax, Softplus, Tanh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc1_3_'></a>[`torch.nn.functional`](#toc0_)\n",
    "\n",
    "- Functional API for applying activation functions.\n",
    "- More flexible than `torch.nn` for custom operations.\n",
    "- Often used directly in model forward passes.\n",
    "- Provides more control over the computation graph.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.nn.functional import (\n",
    "    elu,\n",
    "    gelu,\n",
    "    leaky_relu,\n",
    "    log_softmax,\n",
    "    logsigmoid,\n",
    "    mish,\n",
    "    relu,\n",
    "    sigmoid,\n",
    "    silu,\n",
    "    softmax,\n",
    "    softplus,\n",
    "    tanh,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc2_'></a>[Activation Functions](#toc0_)\n",
    "\n",
    "- Activation functions are used to introduce non-linearity into the neural network.\n",
    "- Without an activation function, a neural network would behave like a linear regression model, no matter how many layers it has!\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "  <img src=\"../../assets/images/original/mlp/no-activation-network.svg\" alt=\"no-activation-network.svg\" style=\"width: 100%;\">\n",
    "  <figcaption style=\"text-align: center;\">Neural Network without Any Activation Functions is just a Linear Transformation of Input to the Output</figcaption>\n",
    "</figure>\n",
    "\n",
    "📝 Docs:\n",
    "\n",
    "- Non-linear Activations (weighted sum, nonlinearity): [pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity](https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity)\n",
    "- Non-linear Activations (other): [pytorch.org/docs/stable/nn.html#non-linear-activations-other](https://pytorch.org/docs/stable/nn.html#non-linear-activations-other)\n",
    "- Non-linear activation functions: [pytorch.org/docs/stable/nn.functional.html#non-linear-activation-functions](https://pytorch.org/docs/stable/nn.functional.html#non-linear-activation-functions)\n",
    "\n",
    "✍️ **Notes**:\n",
    "\n",
    "- Using Python functions is not a correct implementation of an activation function for Pytorch\n",
    "- The correct implementation is covered in the future notebooks\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# domain [-10, +10]\n",
    "x = torch.linspace(-10, +10, 1001)\n",
    "\n",
    "# log\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_1_'></a>[Linear](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def linear_func(x: torch.Tensor) -> torch.Tensor:\n",
    "    return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, linear_func(x))\n",
    "plt.title(\"Linear\")\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_2_'></a>[Sigmoid](#toc0_)\n",
    "\n",
    "- Historically used for `binary classification`, but less common now due to [vanishing gradient](https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484) issues.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid_func(x: torch.Tensor) -> torch.Tensor:\n",
    "    return 1 / (1 + torch.exp(-x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, sigmoid_func(x))\n",
    "plt.title(\"Sigmoid\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-4, 4)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_3_'></a>[Hyperbolic Tangent (Tanh)](#toc0_)\n",
    "\n",
    "- Similar to `sigmoid` but centered around 0, used in [recurrent neural networks (RNNs)](https://karpathy.github.io/2015/05/21/rnn-effectiveness/) and older architectures.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tanh_func(x: torch.Tensor) -> torch.Tensor:\n",
    "    exp_x = torch.exp(x)\n",
    "    exp_neg_x = torch.exp(-x)\n",
    "    return (exp_x - exp_neg_x) / (exp_x + exp_neg_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, tanh_func(x))\n",
    "plt.title(\"Hyperbolic Tangent (Tanh)\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-4, 4)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_4_'></a>[Softplus](#toc0_)\n",
    "\n",
    "- Smooth approximation of `ReLU`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softplus_func(x: torch.Tensor) -> torch.Tensor:\n",
    "    return torch.log(1 + torch.exp(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, softplus_func(x))\n",
    "plt.title(\"Softplus\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-10, 10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_5_'></a>[LogSigmoid](#toc0_)\n",
    "\n",
    "- Logarithm of `sigmoid`, less common but used in specific applications.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logsigmoid_func(x: torch.Tensor) -> torch.Tensor:\n",
    "    return torch.log(1 / (1 + torch.exp(-x)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, logsigmoid_func(x))\n",
    "plt.title(\"LogSigmoid\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-10, 10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_6_'></a>[Rectified Linear Unit (ReLU)](#toc0_)\n",
    "\n",
    "- Most commonly used, computationally efficient, but suffers from the [dying ReLU](https://datascience.stackexchange.com/questions/5706/what-is-the-dying-relu-problem-in-neural-networks) ([vanishing gradient](https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484)) problem.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def relu_func(x: torch.Tensor) -> torch.Tensor:\n",
    "    return torch.max(x, torch.tensor(0.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, relu_func(x))\n",
    "plt.title(\"Rectified Linear Unit (ReLU)\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-10, 10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_7_'></a>[LeakyReLU](#toc0_)\n",
    "\n",
    "- Addresses the `dying ReLU` problem by allowing a small, non-zero gradient for negative inputs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def leaky_relu_func(x: torch.Tensor, negative_slope: float = 0.2) -> torch.Tensor:\n",
    "    return torch.max(x, negative_slope * x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, leaky_relu_func(x))\n",
    "plt.title(\"LeakyReLU\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-10, 10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_8_'></a>[Exponential Linear Unit (ELU)](#toc0_)\n",
    "\n",
    "- Similar to `LeakyReLU` but uses an exponential function for negative inputs, often providing better performance than `ReLU`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def elu_func(x: torch.Tensor, alpha: int = 1.0) -> torch.Tensor:\n",
    "    return torch.where(x > 0, x, alpha * (torch.exp(x) - 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, elu_func(x))\n",
    "plt.title(\"Exponential Linear Unit (ELU)\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-10, 10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_9_'></a>[Sigmoid Linear Unit (SiLU)](#toc0_)\n",
    "\n",
    "- Combines ReLU-like behavior with a smooth curve, often yielding better results than `ReLU`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def silu_func(x: torch.Tensor) -> torch.Tensor:\n",
    "    return x * torch.sigmoid(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, silu_func(x))\n",
    "plt.title(\"Sigmoid Linear Unit (SiLU)\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-10, 10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_10_'></a>[Mish](#toc0_)\n",
    "\n",
    "- Self-regularized activation function, generally performs better than `ReLU` and its variants.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mish_func(x: torch.Tensor) -> torch.Tensor:\n",
    "    return x * torch.tanh(torch.nn.functional.softplus(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, mish_func(x))\n",
    "plt.title(\"Mish\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-10, 10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_11_'></a>[Softmax](#toc0_)\n",
    "\n",
    "- Used for `multi-class classification`, outputs probabilities [[mutually exclusive](https://en.wikipedia.org/wiki/Softmax_function)] for each class, often used `internally` in `CrossEntropyLoss`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax_func(x: torch.Tensor, dim: int | None = None) -> torch.Tensor:\n",
    "    if dim is None:\n",
    "        dim = len(x.shape) - 1\n",
    "    exp_x = torch.exp(x - x.max(dim=dim, keepdim=True).values)\n",
    "    return exp_x / exp_x.sum(dim=dim, keepdim=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, softmax_func(x))\n",
    "plt.title(\"Softmax\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-0.05, 0.05)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_12_'></a>[LogSoftmax](#toc0_)\n",
    "\n",
    "- Logarithm of softmax, often used in `NLLLoss`.\n",
    "- Reducing the risk of numerical issues and ensuring more reliable calculations rather than `Softmax`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logsoftmax_func(x: torch.Tensor, dim: int | None = None) -> torch.Tensor:\n",
    "    if dim is None:\n",
    "        dim = len(x.shape) - 1\n",
    "    softmax_x = torch.nn.functional.softmax(x, dim=dim)\n",
    "    return torch.log(softmax_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, logsoftmax_func(x))\n",
    "plt.title(\"LogSoftmax\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-25, 0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_13_'></a>[Gaussian Error Linear Units (GeLU)](#toc0_)\n",
    "\n",
    "- Approximates the expected value of `ReLU` with a Gaussian input, often used in `transformer-based` models.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gelu_func(x: torch.Tensor) -> torch.Tensor:\n",
    "    return x * 0.5 * (1.0 + torch.erf(x / 2.0**0.5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, gelu_func(x))\n",
    "plt.title(\"Gaussian Error Linear Units (GeLU)\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-10, 10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_14_'></a>[Plot Activation Functions](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(nrows=3, ncols=4, figsize=(12, 8), layout=\"compressed\")\n",
    "fig.suptitle(\"Activation Functions\")\n",
    "axs[0, 0].plot(x, relu_func(x))\n",
    "axs[0, 0].set(title=\"Rectified Linear Unit (ReLU)\", xlim=[-10, 10], ylim=[-10, 10])\n",
    "axs[0, 1].plot(x, leaky_relu(x))\n",
    "axs[0, 1].set(title=\"LeakyReLU\", xlim=[-10, 10], ylim=[-10, 10])\n",
    "axs[0, 2].plot(x, elu_func(x))\n",
    "axs[0, 2].set(title=\"Exponential Linear Unit (ELU)\", xlim=[-10, 10], ylim=[-10, 10])\n",
    "axs[0, 3].plot(x, silu_func(x))\n",
    "axs[0, 3].set(title=\"Sigmoid Linear Unit (SiLU)\", xlim=[-10, 10], ylim=[-10, 10])\n",
    "axs[1, 0].plot(x, mish_func(x))\n",
    "axs[1, 0].set(title=\"Mish\", xlim=[-10, 10], ylim=[-10, 10])\n",
    "axs[1, 1].plot(x, sigmoid_func(x))\n",
    "axs[1, 1].set(title=\"Sigmoid\", xlim=[-10, 10], ylim=[-4, 4])\n",
    "axs[1, 2].plot(x, tanh_func(x))\n",
    "axs[1, 2].set(title=\"Hyperbolic Tangent (Tanh)\", xlim=[-10, 10], ylim=[-4, 4])\n",
    "axs[1, 3].plot(x, softplus_func(x))\n",
    "axs[1, 3].set(title=\"Softplus\", xlim=[-10, 10], ylim=[-10, 10])\n",
    "axs[2, 0].plot(x, logsigmoid_func(x))\n",
    "axs[2, 0].set(title=\"LogSigmoid\", xlim=[-10, 10], ylim=[-10, 10])\n",
    "axs[2, 1].plot(x, softmax_func(x))\n",
    "axs[2, 1].set(title=\"Softmax\", xlim=[-10, 10], ylim=[-0.05, 0.05])\n",
    "axs[2, 2].plot(x, logsoftmax_func(x))\n",
    "axs[2, 2].set(title=\"LogSoftmax\", xlim=[-10, 10], ylim=[-25, 0])\n",
    "axs[2, 3].plot(x, gelu_func(x))\n",
    "axs[2, 3].set(title=\"Gaussian Error Linear Units (GeLU)\", xlim=[-10, 10], ylim=[-10, 10])\n",
    "for ax in fig.axes:\n",
    "    ax.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc3_'></a>[Threshold Functions](#toc0_)\n",
    "\n",
    "- Threshold functions are a simpler type of activation function primarily used in the early development of neural networks\n",
    "- These functions decide whether a neuron should be activated or not based on whether the input surpasses a certain threshold\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_1_'></a>[Step](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def step_func(x: torch.Tensor) -> torch.Tensor:\n",
    "    return torch.where(x >= 0, torch.ones_like(x), torch.zeros_like(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, step_func(x))\n",
    "plt.title(\"Step\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-2, 2)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_2_'></a>[Sign](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sign_func(x: torch.Tensor) -> torch.Tensor:\n",
    "    return torch.where(x > 0, torch.ones_like(x), torch.where(x < 0, torch.ones_like(x) * -1, torch.zeros_like(x)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4, 4))\n",
    "plt.plot(x, sign_func(x))\n",
    "plt.title(\"Sign\")\n",
    "plt.grid(True)\n",
    "plt.xlim(-10, 10)\n",
    "plt.ylim(-2, 2)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_3_'></a>[Plot Threshold Functions](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot\n",
    "fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(8, 4), layout=\"compressed\")\n",
    "fig.suptitle(\"Threshold Functions\")\n",
    "axs[0].plot(x, step_func(x))\n",
    "axs[0].grid(True)\n",
    "axs[0].set(title=\"step\", xlim=[-10, 10], ylim=[-2, 2])\n",
    "axs[1].plot(x, sign_func(x))\n",
    "axs[1].grid(True)\n",
    "axs[1].set(title=\"sign\", xlim=[-10, 10], ylim=[-2, 2])\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "author_email": "AmirhosseinHeydari78@gmail.com",
  "author_github": "https://github.com/mr-pylin",
  "author_name": "Amirhossein Heydari",
  "kernelspec": {
   "display_name": "pytorch-workshop-U_zYfVTd-py3.12",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  },
  "origin_repo": "https://github.com/mr-pylin/pytorch-workshop"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}