NumPy_Fall_2020.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of NumPy_Fall_2020.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/acm-wwu/numpy_workshop/blob/main/NumPy_Fall_2020.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x7PobKY92nen"
      },
      "source": [
        "![link text](https://upload.wikimedia.org/wikipedia/commons/thumb/3/31/NumPy_logo_2020.svg/512px-NumPy_logo_2020.svg.png)\n",
        "\n",
        "# WWU ACM Fall 2020 Intro to NumPy Workshop\n",
        "\n",
        "*Authors: Alex Gavin (He/Him), Dani Sprague (She/Her)*\n",
        "\n",
        "---\n",
        "\n",
        "### Suggested Prerequisites:\n",
        "\n",
        "* Some experience with a programming language, preferably Python\n",
        "* A basic understanding of matrices (experience w/ Linear Algebra suggested)\n",
        "\n",
        "### Things You Will Learn:\n",
        "* Why and when you should use NumPy\n",
        "* NumPy Arrays, Basic operations\n",
        "* A practical application of NumPy\n",
        "* Resources for where to learn more about NumPy\n",
        "* Data science/ML research at WWU\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZbKx-ymdoYMP"
      },
      "source": [
        "### 0 --- Motivation\n",
        "\n",
        "From a quick glance, NumPy's main utility appears to be its addition of arrays to Python. With lists already being a data structure built in to Python, this may seem redundant. However, **NumPy arrays are both faster and less memory intensive than Python lists**.\n",
        "\n",
        "NumPy arrays only support data of a single type, which allows type checks to run far less often. As well, **vectorized operations allow for repetitive calculations to be sped up by over 100x in some instances** (see Section 3 - NumPy Speed). NumPy also has plenty of useful utilities built-in.\n",
        "\n",
        "NumPy has been picked as a base to the vast majority of the data science tooling and libraries in the Python community (including Matplotlib, pandas, TensorFlow, and PyTorch), making it essential knowledge for anyone interested in data science. It is also widely utilized in a variety of scientific computing simulations.\n",
        "\n",
        "[Nature recently published a paper on NumPy](https://www.nature.com/articles/s41586-020-2649-2), which we recommend for those more interested in a quick overview of the implementation and ecosystem of NumPy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "He-bbKNF5KIQ"
      },
      "source": [
        "### 1 --- NumPy Array Basics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YzV8hND3qi7n"
      },
      "source": [
        "#### 1.1 --- Creation and Indexing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a3AeVuNpJFFv"
      },
      "source": [
        "NumPy arrays can be created from Python lists, although this is far from the only way."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5f_3OXN4JQNy"
      },
      "source": [
        "# It is standard to import NumPy as \"np\"\n",
        "import numpy as np\n",
        "\n",
        "nums_list = [0, 1, 2, 3, 4]\n",
        "print(f'nums_list: {nums_list}')\n",
        "\n",
        "nums_np = np.array(nums_list)\n",
        "print(f'nums_np: {nums_np}\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h2PggLu5JXE3"
      },
      "source": [
        "NumPy arrays share very similar indexing, including zero-indexing, slicing, and strides. There is also some more advanced syntax sugar for indexing, detailed [in the NumPy docs](https://numpy.org/doc/stable/reference/arrays.indexing.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZRDQrFvRJeuF"
      },
      "source": [
        "# Zero-indexing\n",
        "print(f'nums_list[0]: {nums_list[0]}')\n",
        "print(f'nums_np[0]: {nums_np[0]}\\n')\n",
        "\n",
        "# Slicing\n",
        "print(f'nums_list, first three elements: {nums_list[:3]}')\n",
        "print(f'nums_np, first three elements: {nums_np[:3]}\\n')\n",
        "\n",
        "#Strides\n",
        "print(f'nums_list, all even elements: {nums_list[::2]}')\n",
        "print(f'nums_np, all even elements: {nums_np[::2]}\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rP5Djx0YKUyN"
      },
      "source": [
        "NumPy supports ranges, similar to Python ranges that are common in for-loops."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c2t27b04KxDM"
      },
      "source": [
        "# This will have the same values as nums_np\n",
        "range_nums = np.arange(5)\n",
        "print(f'range_nums: {range_nums}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oN8mdmvwK3iG"
      },
      "source": [
        "NumPy has a few more ways to instantiate arrays. In this example, we are trying to create a 3x3 array of all 8's."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yAdkKmWA2mbC"
      },
      "source": [
        "# Method 1:\n",
        "#   Create array of all 1's, multiply each element by 8\n",
        "arr = np.ones([3, 3])\n",
        "# NOTE: np.multiply is an example of a vectorized function. We will see more of this later.\n",
        "arr = np.multiply(8, arr)\n",
        "print(f'arr of eights, m1:\\n {arr}\\n')\n",
        "\n",
        "# Method 2:\n",
        "#   Use numpy full to create array of eights\n",
        "arr = np.full([3, 3], 8)\n",
        "print(f'arr of eights, m2:\\n {arr}\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U_aUBe0UqVKg"
      },
      "source": [
        "#### 1.2 --- Basic Operations and Multiplication"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ykLAOWpelWoQ"
      },
      "source": [
        "NumPy allows for some operations on the entirety of a matrix, including many basic math operations. If wanting to change all values of a matrix by a constant, passing in either just the constant or another matrix consisting entirely of the contant works. The former works thanks to broadcasting (convered in Section 1.5). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GwVJnJLemAbq"
      },
      "source": [
        "arr_add = np.add(arr, np.full([3, 3], 10))\n",
        "print(f'arr add ten:\\n {arr_add}\\n')\n",
        "\n",
        "arr_sub = np.subtract(arr, np.full([3, 3], 5))\n",
        "print(f'arr minus five:\\n {arr_sub}\\n')\n",
        "\n",
        "arr_div = np.divide(arr, 2)\n",
        "print(f'arr divided by two:\\n {arr_div}\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hV8AnGFWmJr9"
      },
      "source": [
        "In numpy, multiplication is element-wise, meaning each element is multiplied by its corresponding element in the other array. To compute proper matrix multiplication or dot products, you must use the [matmul](https://numpy.org/doc/stable/reference/generated/numpy.matmul.html) or [dot](https://numpy.org/doc/stable/reference/generated/numpy.dot.html) methods, respectively. To get the transpose of an array, simply do [arr.T](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.T.html).\n",
        "\n",
        "These matrix concepts are mainly taught in MATH 204. If you are unfamiliar with these concepts, here are some useful links:\n",
        "- Matrix multiplications:\n",
        "https://www.mathsisfun.com/algebra/matrix-multiplying.html\n",
        "- Dot product:\n",
        "https://betterexplained.com/articles/vector-calculus-understanding-the-dot-product/\n",
        "- Transpose: https://mathinsight.org/matrix_transpose"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kHvH2LUgqO2O"
      },
      "source": [
        "# Creating random vectors for the dot product.\n",
        "v1 = np.random.randint(-10, 10, size=[3, 1])\n",
        "v2 = np.random.randint(-10, 10, size=[3, 1])\n",
        "\n",
        "print(f'v1:\\n {v1}\\n')\n",
        "print(f'v2:\\n {v2}\\n')\n",
        "\n",
        "# Outputs a scalar value wrapped in a matrix\n",
        "dot_prod = np.dot(v1.T, v2)\n",
        "print(f'v1 • v2 = {dot_prod[0, 0]}\\n')\n",
        "\n",
        "# Multiplication examples\n",
        "identity_matrix = np.array([[1, 0], [0, 1]])\n",
        "twos = np.full([2,2], 2)\n",
        "print(f'identity_matrix × twos element-wise:\\n{identity_matrix * twos}\\n')\n",
        "print(f'identity_matrix × twos as matrices:\\n{np.matmul(identity_matrix, twos)}\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lzksrYHSsmgx"
      },
      "source": [
        "#### 1.3 - Array Shape and Slicing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "85s4ipQRnp_V"
      },
      "source": [
        "Arrays have properties, including the shape (dimensions of the array) and data type. Remember that all members of a NumPy array must have the same datatype."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n6MGPWnfn1TF"
      },
      "source": [
        "print(type(range_nums))\n",
        "print(f'range_nums dtype: {range_nums.dtype}')\n",
        "print(f'range_nums astype np.int8: {range_nums.astype(np.int8)}\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mpIYtNfGn4Ry"
      },
      "source": [
        "An array and its values can be recast to a different type."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "njW1BLsyoPT3"
      },
      "source": [
        "# Recasting 64-bit floats to 8-bit integers\n",
        "float_nums = np.array([0.98, 0.84, 0.931, 0.5247])\n",
        "print(f'float_nums: {float_nums}')\n",
        "print(f'float_nums dtype: {float_nums.dtype}')\n",
        "print(f'float_nums astype np.int8: {float_nums.astype(np.int8)}\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tTDVj9BspL4T"
      },
      "source": [
        "Arrays can be reshaped, or changed in dimension. NumPy automatically tries to figure out how to shift the data. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RDQ3NNdnpyQe"
      },
      "source": [
        "print(f'nums_np shape: {nums_np.shape}')\n",
        "\n",
        "nums_np_2d = nums_np.reshape([1, 5])\n",
        "print(f'nums_np_2d shape: {nums_np_2d.shape}')\n",
        "print(f'nums_np_2d: {nums_np_2d}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KqUzYY8vp2i9"
      },
      "source": [
        "Multi-dimensional arrays are indexed in multiple dimensions. There are two ways main ways of doing this: ```arr[x,y,z]``` and ```arr[x][y][z]```. The former is much preferred where possible. The latter is more intensive, due to needing to copy multiple arrays instead of leaping directly to the needed value.\n",
        "\n",
        "More slicing examples can be found here: https://github.com/rougier/numpy-tutorial#slicing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o0XJGel99hJO"
      },
      "source": [
        "NumPy array reshaping and slicing (done correctly) is inexpensive. Reshaping mainly changes an equation in memory, not requiring any copying of the array data in most cases. Slicing returns a view by default, which again avoids copying array data, but means that modifying the data in the slice will modify the original array."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oZ8UORLfsw9w"
      },
      "source": [
        "print(f'nums_np_2d first element (nums_np_2d[0, 0]): {nums_np_2d[0, 0]}')\n",
        "\n",
        "print(f'nums_np_2d first element different indexing (nums_np_2d[0][0]): {nums_np_2d[0][0]}\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LLewuw8nPzT8"
      },
      "source": [
        "You can also concatenate two arrays together! This combines the two arrays together along a given axis. This is done via [np.concatenate](https://numpy.org/doc/stable/reference/generated/numpy.concatenate.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wl5zq6nHk3bM"
      },
      "source": [
        "cat_arr1 = np.array([[1, 2, 3], [5, 6, 7], [9, 10 ,11]])\n",
        "cat_arr2 = np.array([[4], [8], [12]])\n",
        "catted_arr = np.concatenate((cat_arr1, cat_arr2), axis=1)\n",
        "\n",
        "print(f\"cat_arr1:\\n{cat_arr1}\")\n",
        "print(f\"cat_arr2:\\n{cat_arr2}\")\n",
        "print(f\"After first concatenation:\\n{catted_arr}\\n\\n\")\n",
        "\n",
        "cat_arr3 = np.array([[13, 14, 15, 16], [17, 18, 19, 20]])\n",
        "# The axis defaults to 0 when not specified\n",
        "catted_arr = np.concatenate((catted_arr, cat_arr3))\n",
        "\n",
        "print(f\"cat_arr3:\\n{cat_arr3}\")\n",
        "print(f\"After second concatenation:\\n{catted_arr}\\n\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fK1c8b_iv9rY"
      },
      "source": [
        "#### 1.4 --- Random Number Sampling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HFZ8j-OawmZE"
      },
      "source": [
        "NumPy includes functions that allows for random number sampling, either into a scalar value or into an array. This utility comes in use for quite a few data science tasks."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1FxawzczXamJ"
      },
      "source": [
        "One useful method is [randint](https://numpy.org/doc/stable/reference/random/generated/numpy.random.randint.html?highlight=randint#numpy.random.randint), which returns a random integer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4jGtpkX8wFLK"
      },
      "source": [
        "print(f\"10 random integers in the interval [0, 100): {np.random.randint(low=0, high=100, size=10)}\\n\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IznTaxsTYWqp"
      },
      "source": [
        "NumPy allows for sampling from a [normal distribution](https://simple.wikipedia.org/wiki/Normal_distribution) via the [normal method](https://numpy.org/doc/stable/reference/random/generated/numpy.random.normal.html). Normal distributions are taught in MATH 341, but essentially, any time the same continuous random event happens over-and-over again, the output will form a normal distribution."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BesYjl7oYLKn"
      },
      "source": [
        "# A Z-distribution is a normal distribution with a mean of 0 and a standard deviation of 1.\n",
        "print(f\"10 random samplings from a Z-distribution: {np.random.normal(loc=0, scale=1, size=10)}\\n\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WhE6xzv7ZkvG"
      },
      "source": [
        "If a sampling is desired from only a select few discrete choices, then the [choice method](https://numpy.org/doc/stable/reference/random/generated/numpy.random.choice.html) can be used. This can optionally be used without replacement to prevent the same value from occuring twice."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hRkR_BdwaXxG"
      },
      "source": [
        "options = (5, 435, 99,  234857, 314, 271828)\n",
        "print(f\"10 random choices from options: {np.random.choice(options, size=10)}\\n\")\n",
        "print(f\"5 random choices from options without replacement: {np.random.choice(options, size=5, replace=False)}\\n\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2GIMHVRlc8MH"
      },
      "source": [
        "An array's element can be randomly shuffled via the [shuffle method](https://numpy.org/doc/stable/reference/random/generated/numpy.random.shuffle.html). This operates in-place and returns None."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F8L59HLTce_E"
      },
      "source": [
        "data = np.array([5, 10, 15, 20, 25])\n",
        "np.random.shuffle(data)\n",
        "print(f\"Shuffled data: {data}\\n\")\n",
        "np.random.shuffle(data)\n",
        "print(f\"Re-shuffled data: {data}\\n\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mj-X8zaacHXe"
      },
      "source": [
        "These are just a few of the options in the [random](https://numpy.org/doc/stable/reference/random/) component of NumPy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DjmovgAdrXEK"
      },
      "source": [
        "#### 1.5 --- Broadcasting: a note\n",
        "\n",
        "Simply put, broadcasting allows for operations on arrays of otherwise incompatible shapes.\n",
        "\n",
        "Here are some visuals of what broadcasting does: https://github.com/rougier/numpy-tutorial#broadcasting\n",
        "      \n",
        "Normally, numpy requires the shapes of arrays to be exactly compatible. Broadcasting relaxes this rule, while imposing some constraints. These constraints are:\n",
        "- In order to broadcast, the size of the trailing axes for both arrays in an operation must either be the same size or one of them must be one.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nts304SEPkpE"
      },
      "source": [
        "### 2 --- Logistic Regression Problem"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GzNDX0blDvKX"
      },
      "source": [
        "#### 2.1 --- What is Logistic Regression?\n",
        "\n",
        "To be clear, we are only using part of logistic regression *as an example*. We don't expect you to know how to implement the entire algorithm from scratch. **It's okay if you don't understand the math!**\n",
        "\n",
        "> *Logistic regression is a supervised learning algorithm for binary classification.*\n",
        "\n",
        "In regular English, given *input data* **x** and *corresponding labels* **y** of two categories, you can use logistic regression to train a model that will make predictions on new, unseen data! (i.e. given new input data **x'** and new labels **y'**, you can predict **y'** from only **x'**!)\n",
        "\n",
        "We aren't going to get into too much detail here, but assuming you have trained a model with weights **w** and bias **b**, the formula for evaluating this trained model is:\n",
        "\n",
        "> *y'* = sigmoid(**x'^T** * **w** + *b*)\n",
        ">\n",
        "> where **x'^T** is the transponse of **x'**.\n",
        "\n",
        "Dimensions of parameters (evaluating one prediction at a time):\n",
        "> **x**, **x'**: 1 x d (vector)\n",
        ">\n",
        "> *y*, *y'*: 1 x 1 (scalar)\n",
        ">\n",
        "> **w**: d x 1 (vector)\n",
        ">\n",
        "> *b*: 1 x 1 (scalar)\n",
        "\n",
        "The sigmoid function takes any real number (-inf, +inf) and outputs some number between 0 and 1. Thus, if you have some decent weights and bias values, you can predict the class (category) of your input!\n",
        "\n",
        "As a visual aid, here's a graph of the sigmoid function:\n",
        "![Sigmoid function](https://upload.wikimedia.org/wikipedia/commons/thumb/8/88/Logistic-curve.svg/1920px-Logistic-curve.svg.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_tF19D2iJiwL"
      },
      "source": [
        "#### 2.2 -- Let's write some code!\n",
        "So, let's do some logistic regression! One commonly used dataset is MNIST, which is comprised of images of digits with corresponding labels.\n",
        "\n",
        "Here are a few examples:\n",
        "![](https://3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com/wp-content/uploads/2019/02/Plot-of-a-Subset-of-Images-from-the-MNIST-Dataset.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sulpNtQJW-j8"
      },
      "source": [
        "# Load data\n",
        "# Reshape test data into something compatible for logistic regression\n",
        "from tensorflow.keras.datasets import mnist\n",
        "(train_x, train_y), (test_x, test_y) = mnist.load_data()\n",
        "\n",
        "test_x = test_x.reshape([10000, -1])\n",
        "test_y = test_y.reshape([-1, 1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IiPXzoXiIrUr"
      },
      "source": [
        "Normally, when doing machine learning you shuffle the data to ensure that the model learns on the features of the data and not the order you fed it the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U2-Pt8GFd3UG"
      },
      "source": [
        "# Shuffle data to randomize our choice!\n",
        "data = np.concatenate((test_x, test_y), axis=1)\n",
        "np.random.shuffle(data)  # This method operates in place, returns None"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TJiUtDn0JN0P"
      },
      "source": [
        "Since the logistic regression is a **binary** classifier (i.e. it can only classify data into two categories), we can't attempt to classify all the digits in MNIST.\n",
        "\n",
        "So instead, let's try to classify the 0's!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qH7BOQQHeMR3"
      },
      "source": [
        "# Let's get all of the 0's from the dataset\n",
        "indices = data[:, -1] == 0\n",
        "data = data[indices]\n",
        "\n",
        "# Split data back into x and y\n",
        "x_zeros = data[:, :-1]\n",
        "y_zeros = data[:, -1]\n",
        "\n",
        "print(f\"Number of 0's in dataset: {data.shape[0]}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J0q3c6qJ-PV1"
      },
      "source": [
        "Now that we have our data loaded into NumPy arrays, let's generate some random weights and see how good they are!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A-XRW3HNhHU6"
      },
      "source": [
        "from scipy.special import expit\n",
        "\n",
        "# Generate random weights\n",
        "random_weights = np.random.normal(0, 10, size=[784, 1])\n",
        "random_bias = np.random.normal(-10, 10)\n",
        "\n",
        "# Get random sample\n",
        "rand_index = np.random.randint(0, x_zeros.shape[0])\n",
        "sample_x = x_zeros[rand_index, :]\n",
        "sample_y = y_zeros[rand_index]\n",
        "\n",
        "# Your turn to implement! \n",
        "# Reference the logistic equation at the beginning of the section\n",
        "def apply_logistic(x):\n",
        "    y = x  # Implement the logistic equation here.\n",
        "    return expit(y)  # This is the sigmoid function\n",
        "\n",
        "# Try your model!\n",
        "y = apply_logistic(sample_x)\n",
        "\n",
        "if y.size != 1:\n",
        "  print(\"Oops! You didn't implement apply_logistic() correctly!\")\n",
        "else:\n",
        "  print(f\"Prediction: {y.item()}, True value: {sample_y}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a5nIEb-AqWmL"
      },
      "source": [
        "### 3 - NumPy Speed"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PxhvfY6fLtMW"
      },
      "source": [
        "#### 3.1 -- NumPy vs. Vanilla Python Speed\n",
        "As previously mentioned, one of the main benefits of NumPy, in addition to providing functions for complex mathematical operations, is its speed. Largely implemented in C, NumPy arrays are both faster and less memory intensive than corresponding Python lists.\n",
        "\n",
        "Here, we are going to explore the speed difference in evaluating our logistic regression model between vanilla Python and NumPy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LK4MnIJdKepG"
      },
      "source": [
        "import time\n",
        "\n",
        "# Iterative vanilla Python testing function\n",
        "def test_vanilla(x_zeros, y_zeros):\n",
        "  # x_zeros: n x d matrix of mnist images\n",
        "  # y_zeros: n x 1 vector of corresponding labels\n",
        "  true_preds = 0\n",
        "\n",
        "  # Iterate through and evaluate all data pairs\n",
        "  for ix in range(x_zeros.shape[0]):\n",
        "    sample_x = x_zeros[ix, :]\n",
        "    sample_y = y_zeros[ix]\n",
        "\n",
        "    pred_y = apply_logistic(sample_x)\n",
        "\n",
        "    if pred_y == sample_y:\n",
        "      true_preds += 1\n",
        "\n",
        "  accuracy = true_preds / x_zeros.shape[0]\n",
        "  return accuracy"
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AODcPVELMPtG"
      },
      "source": [
        "# Vectorized NumPy\n",
        "def test_vectorized(x_zeros, y_zeros):\n",
        "  # Use matrix of inputs to evaluate all at once!\n",
        "  # Instead of vector-vector multiplication, do matrix-vector multiplication\n",
        "  pred_y = apply_logistic(x_zeros.T)\n",
        "\n",
        "  # Calculate a boolean array where\n",
        "  # all trues are where the predictions are correct\n",
        "  # all falses are where the predictions are incorrect\n",
        "  cmp = np.equal(pred_y, y_zeros)\n",
        "\n",
        "  # Sum of all equal \n",
        "  accuracy = cmp.sum() / x_zeros.shape[0]\n",
        "  return accuracy"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8ZucfP37N0oo"
      },
      "source": [
        "Now that we have implemented the test functions, let's run them and compare results!\n",
        "\n",
        "**Note:** Neither of the tests will run correctly if *apply_logistic()* is not implemented correctly!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "crkDHQiyqZCo"
      },
      "source": [
        "# Test Vanilla\n",
        "start = time.time()\n",
        "accuracy = test_vanilla(x_zeros, y_zeros)\n",
        "vanilla_time = time.time() - start\n",
        "\n",
        "print(f\"Accuracy vanilla: {accuracy * 100:.2f}%\")\n",
        "print(f\"Vanilla time: {vanilla_time:.4f} sec\\n\")\n",
        "\n",
        "\n",
        "# Test Vectorized\n",
        "start = time.time()\n",
        "accuracy = test_vectorized(x_zeros, y_zeros)\n",
        "vectorized_time = time.time() - start\n",
        "\n",
        "print(f\"Accuracy vectorized: {accuracy * 100:.2f}%\")\n",
        "print(f\"Vectorized time: {vectorized_time:.4f} sec\\n\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jBF_uY6CQb-c"
      },
      "source": [
        "### 4 --- Extras"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B0wFz92iQzd1"
      },
      "source": [
        "#### 4.1 --- Department Research\n",
        "\n",
        "Here is a non-exhaustive list of professors working with machine learning. If you don't see a professor on this list, it is possible that they also use machine learning in their work. **It never hurts to ask!**\n",
        "- **Ahmed** - Applied machine learning to HCI problems.**\n",
        "\n",
        "- **Harris** - Computational neuroscience, networks, graph theory, applied mathematics.\n",
        "- **Hearne** - Data mining, computational linguistics, artificial intelligence, computer music.\n",
        "- **Hutchinson** - Speech and language processing, machine learning and optimization.\n",
        "- **Idriss** - Lightweight security, machine learning, internet of things.\n",
        "- **Jagodzinski** - Computational structural biology, big data, intelligent information systems.\n",
        "- **Liu** - Natural language processing, information extraction, application of eye-tracking data.\n",
        "- **Sharmin** - Applied machine learning to HCI problems.**\n",
        "- **Tsikerdekis** - Applied machine learning to cyber security problems.\n",
        "- **Wehrwein** - Computer vision, computational photography, computer graphics\n",
        "\n",
        "\\*\\* *Not entirely sure about this*\n",
        "\n",
        "#### 4.2 --- Other Tutorial Links\n",
        "1. [Official NumPy Quickstart Tutorial](https://numpy.org/devdocs/user/quickstart.html)\n",
        "2. [NumPy Tutorial Through Game of Life](https://github.com/rougier/numpy-tutorial) with intuitive [slicing/broadcasting visualizations](https://github.com/rougier/numpy-tutorial#quick-references)\n",
        "3. [Stanford CS231n NumPy/MatPlotLib Tutorial](https://cs231n.github.io/python-numpy-tutorial/#numpy\n",
        ")\n",
        "4. [Intro to NumPy video, SciPy 2019](https://www.youtube.com/watch?v=ZB7BZMhfPgk)\n",
        "\n",
        "#### 4.3 --- Advanced NumPy\n",
        "1. [General Advanced NumPy, SciPy Japan 2019](https://www.youtube.com/watch?v=cYugp9IN1-Q) -- broadcasting rules, strides / stride tricks, and advanced indexing.\n",
        "2. Einsums [text tutorial](https://ajcr.net/Basic-guide-to-einsum/) and [video tutorial](https://www.youtube.com/watch?v=pkVwUVEHmfI) -- optimized shorthand notation for multiplications and summations of vectors, matrices, and tensors.\n",
        "\n",
        "#### 4.4 --- Alex's Favorite Functions\n",
        "* [*np.where()*](https://numpy.org/doc/stable/reference/generated/numpy.where.html)\n",
        "  * Check where a condition is true in the array (think SQL)\n",
        "* [*np.vectorize()*](https://numpy.org/doc/stable/reference/generated/numpy.vectorize.html)\n",
        "  * Doesn't increase performance, but can make complex array operations like convolutions easier to deal with.\n",
        "* [*np.stack()*](https://numpy.org/doc/stable/reference/generated/numpy.stack.html), [*np.concatenate()*](https://numpy.org/doc/stable/reference/generated/numpy.concatenate.html), [*np.append()*](https://numpy.org/doc/stable/reference/generated/numpy.append.html)\n",
        "  * Useful for extending existing arrays. Context varies.\n",
        "* [*np.moveaxis()*](https://numpy.org/doc/stable/reference/generated/numpy.moveaxis.html)\n",
        "  * Useful for iterating. Changes shape ordering of array\n",
        "  * **Ex:** *for vector in np.moveaxis(array, -1, 0):*\n",
        "  * Move last dimension to front, iterate through formerly last dimension\n",
        "* [*np.apply_along_axis()*](https://numpy.org/doc/stable/reference/generated/numpy.apply_along_axis.html)\n",
        "  * \"Apply a function to 1-D slices along the given axis.\" \n",
        "  * Slight performance increase over other methods, but must still use Python function. \n",
        "  * Makes for some cute one-liners.\n",
        "* These aren't functions, but are also useful:\n",
        "  * [Boolean indexing](https://numpy.org/doc/stable/user/basics.indexing.html?highlight=boolean#boolean-or-mask-index-arrays)\n",
        "  * [Broadcasting](https://numpy.org/doc/stable/user/basics.broadcasting.html)\n",
        "    * [Here](https://github.com/rougier/numpy-tutorial#broadcasting) is a broadcasting visual (also linked above)\n",
        "\n",
        "#### 4.5 --- **\\~\\~ Join ACM! \\~\\~**\n",
        "- You get unlimited online access to O'Reilly--books on practically anything CS!\n",
        "- Participate in the CS community, collaborate with other clubs.\n",
        "- Plan and host events that **you** are interested in.\n",
        "- In normal times, we host the annual dept picnic as well as a 24 hour hackathon.\n",
        "\n",
        " **Email us at acm.wwu@gmail.com or ping Alex if you're interested!**\n",
        "\n"
      ]
    }
  ]
}