09. Introduction to Vectors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Linear Algebra "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the introduction to calculus, we made a distinction between vector and scalar quantities. Vector quantities involve both direction and magnitude (in our case distance), thus we represented them geometrically, with arrows. For two vectors to be equal, their arrows must point in the same direction and have the same length.  For that reason, the vectors on the image below are all different from each other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/derivatives-02.png\" alt=\"drawing\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conventions of Representation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cartesian Representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In linear algebra, we deal extensively with vectors, and we need a convention to represent them. We will start by rooting every vector's tail to the coordinate origin of the space. In the case of two-dimensional space, we will represent vectors as arrows whose tail is fixed at the coordinate $(0,0)$. Their tip will then land at a certain point in space, which we can capture it with a pair of coordinates $(x,y)$, as shown in the illustration below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/linear-algebra01.png\" alt=\"drawing\" width=\"600\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have an analogous situation in the 3-dimensional space. A vector's tail is fixed at the coordinate centre $(0,0,0)$ and we note where its tip lands using 3 coordinates $(x,y,z)$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/linear-algebra02.png\" alt=\"drawing\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Symbolic Representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By using this convention, each vector becomes uniquely associated with a list of numbers containing its tip's coordinates. Thus, we can deduce that two vectors (rooted at the origin) are equal if their their tip lands on the same coordinate in space. To represent a vector numerically, we will introduce another convention and write them as a vertical arrangement of numbers (coordinates) within square brackets. To each spatial dimension corresponds a single row. The convention is that rows go from the top to bottom, and dimensions from lower to higher, thus $x$-coordinate is represented with the first row, $y$ with the second, $z$ with the third, and so on. The five vectors from the illustrations above can be represented as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\vec{a}=\\begin{bmatrix} 3\\\\4\\end{bmatrix}; \\quad \n",
    "\\vec{b}=\\begin{bmatrix} 2\\\\-1\\end{bmatrix}; \\quad \n",
    "\\vec{c}=\\begin{bmatrix} -3\\\\-3\\end{bmatrix}; \\quad \n",
    "\\vec{d}=\\begin{bmatrix} -2\\\\3\\end{bmatrix}; \\quad\n",
    "\\vec{e}=\\begin{bmatrix} 2\\\\3\\\\5\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Representing vectors in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To represent vectors in Python programming language, we will use the linear algebra library called NumPy. To import it we write the following line of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use NumPy's data structure called array to represent vectors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([[3],\n",
    "              [4]])\n",
    "\n",
    "b = np.array([[2],\n",
    "              [-1]])\n",
    "\n",
    "c = np.array([[-3],\n",
    "              [-3]])\n",
    "\n",
    "d = np.array([[-2],\n",
    "              [3]])\n",
    "\n",
    "e = np.array([[2],\n",
    "              [3],\n",
    "              [5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With Python's print command, we can display our variables and confirm that the vectors we defined are indeed represented as a vertical arrangement of numbers—coordinates. Let us print the vector $\\vec{a}$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[3]\n",
      " [4]]\n"
     ]
    }
   ],
   "source": [
    "print (a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[2]\n",
      " [3]\n",
      " [5]]\n"
     ]
    }
   ],
   "source": [
    "print (e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Computing the magnitude"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the coordinates, we can easily derive a vector's direction and magnitude (length). For example, to find the length of the vector $\\vec{a}$, we can take the square root of the sum of its coordinates squared.  This, of course, comes from applying the Pythagorean theorem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/linear-algebra03.png\" alt=\"drawing\" width=\"350\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\|\\vec{a}\\|=\\sqrt{3^2+4^2}=\\sqrt{25}=5$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same procedure applies to 3-dimensional vectors such is $\\vec{e}$, but also scales seamlessly into higher dimensions!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\|\\vec{e}\\|=\\sqrt{2^2+3^2+5^2}=\\sqrt{38}\\approx6.16$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute this in NumPy, we can use NumPy's built-in function `linalg.norm`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.norm(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6.164414002968976"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.norm(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vector Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two elementary vector operations are adding (and subtracting) vectors, and multiplying a vector by a constant."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Vector addition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's examine the vector addition from the geometric point of view. The animation below shows how we can add the vector $\\vec{b}$ to $\\vec{a}$. Both vectors are first rooted at the origin. Then we translate the vector $\\vec{b}$ such that its tail lands at the tip of the vector $\\vec{a}$. The resulting vector $\\vec{a+b}$ has its tail at the origin, while its tip points at the same place as the translated vector $\\vec{b}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](img/linear-algebra04a.gif \"segment\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is interesting is that the same result can be achieved if we carried out the addition the other way around—if we translated $\\vec{a}$ such that its tail landed on the tip of the vector $\\vec{b}$. This tells us that the vector addition is __commutative__ or in other words: $\\vec{a}+\\vec{b} = \\vec{b}+\\vec{a}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Multiplying a vector by a scalar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multiplying a vector by a scalar does not affect vector's direction, rather its magnitude and orientation. The magnitude of the scalar will change the vector's magnitude. If we multiply a vector  $\\vec{u}$ by the number 2, the resulting vector will be twice as long. If we multiply it by 1/2 or 0.5, it will be half as long. The sign of the scalar (negative or positive number) will by itself not change its magnitude but it will change its direction. If we multiply the vector $\\vec{u}$ by -1, the resulting vector will be of the same length but it will point in the opposite direction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/linear-algebra05.png\" alt=\"drawing\" width=\"450\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the animation below, we are multiplying the vector $\\vec{u}$ by the scalar $a$, whose value is oscillating between -2 and 2. This, in turn, scales and flips the resulting vector $a\\vec{u}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](img/linear-algebra06.gif \"segment\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Vector subtraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can think of vector subtraction in terms of two operations we already know how to do: vector addition and multiplication of a vector by a (negative) scalar. Instead of subtracting $\\vec{b}$ from $\\vec{a}$, we can simply add $\\vec{a}$ to $-\\vec{b}$, which is $\\vec{b}$ multiplied by -1 . In geometric terms, we would be adding the reversed vector $\\vec{b}$ to the vector $\\vec{a}$, and call this subtraction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/linear-algebra07.png\" alt=\"drawing\" width=\"900\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Adding vectors numerically"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding and subtracting vectors numerically is quite similar to adding and subtracting numbers. What changes is that we have more than one number representing each vector that we need to add/subtract. In the case of two-dimensional vectors, we simply need to add or subtract vectors' respective $x$ and $y$ components. In case of n-dimensional vectors, we have $n$ numbers to add/subtract. To perform these operations, the vectors need to match: one can only add/subtract vectors of the same size (dimensionality). The consequence of this rule is that the final vector will preserve the dimensionality of the vectors we are adding or subtracting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To add the vectors $\\vec{a}$ and $\\vec{b}$, we simply add $\\vec{a}$'s $x$ coordinate 3 to $\\vec{b}$'s $x$ coordinate 2. The resulting vector's $x$ coordinate will be $3+2=5$. The same procedure is then applied to all vector's dimensions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\\vec{a}+\\vec{b}&=\\begin{bmatrix} 3\\\\4\\end{bmatrix}+\\begin{bmatrix} 2\\\\-1\\end{bmatrix}=\\begin{bmatrix} 3+2\\\\4+(-1)\\end{bmatrix}\\\\\\\\\n",
    "\\vec{a}+\\vec{b}&=\\begin{bmatrix} 5\\\\3\\end{bmatrix}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can express this in a general form (for 2-dimensional vectors) as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\\vec{a}+\\vec{b}&=\\begin{bmatrix} a_x+b_x\\\\a_y+b_y\\end{bmatrix}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  Multiplying a vector by a scalar numerically"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To multiply a vector by a scalar, we simply multiply each of the vector's components by that scalar. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "3\\vec{a}&=3\\cdot\\begin{bmatrix} 3\\\\4\\end{bmatrix}=\\begin{bmatrix} 3\\cdot3\\\\3\\cdot4\\end{bmatrix}\\\\\\\\\n",
    "3\\vec{a}&=\\begin{bmatrix} 9\\\\12\\end{bmatrix}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can express this in a general form (for two dimensional vectors) as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "p\\vec{a}&=\\begin{bmatrix} p a_x\\\\p a_y\\end{bmatrix}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Subtracting vectors numerically"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To subtract the vector $\\vec{b}$ from $\\vec{a}$, we can do the same as we did geometrically: reverse the vector $\\vec{b}$'s direction and add it to $\\vec{a}$. To achieve this numerically, we need to multiply the vector $\\vec{b}$'s components by -1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "\\vec{a}-\\vec{b}&=\\vec{a}+(-1\\vec{b})\\\\\\\\\n",
    "\\vec{a}-\\vec{b}&=\\begin{bmatrix} 3\\\\4\\end{bmatrix}+(-1)\\begin{bmatrix} 2\\\\-1\\end{bmatrix}=\\begin{bmatrix} 3\\\\4\\end{bmatrix}+\\begin{bmatrix} -2\\\\1\\end{bmatrix}\\\\\\\\\n",
    "\\vec{a}-\\vec{b}&=\\begin{bmatrix} 3-2\\\\4+1\\end{bmatrix}\\\\\\\\\n",
    "\\vec{a}-\\vec{b}&=\\begin{bmatrix} 1\\\\5\\end{bmatrix}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can express this in a general form (for two dimensional vectors) as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\\vec{a}-\\vec{b}&=\\begin{bmatrix} a_x-b_x\\\\a_y-b_y\\end{bmatrix}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same principle is not specific to vector's dimensionality and it applies to vectors of any size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  Vector dot product"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Geometrically, the dot product of vectors $\\vec{a}$ and $\\vec{b}$ is defined as a product of vectors' lengths and the cosine of an angle between them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\vec{a} \\cdot \\vec{b}= \\|\\vec{a}\\| \\;\\|\\vec{b}\\| \\; cos(\\theta)$$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/linear-algebra08.png\" alt=\"drawing\" width=\"300\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dot product is useful because it can be used to find an angle between any two vectors or determine the length of the projection of one vector onto another. Those methods, however,  will not be covered in this presentation. <br> By looking at the dot product formula we can deduce that it will be a scalar quantity, since the lengths and the cosine of the angle $\\theta$ are all scalars. Luckily, to compute this number, there is a very convenient numerical procedure. We need to multiply the vectors' respective components elementwise, and sum them together. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "\\vec{a}\\cdot\\vec{b}&=\\begin{bmatrix} 3\\\\4\\end{bmatrix}\\cdot\\begin{bmatrix} 2\\\\-1\\end{bmatrix} = 3\\cdot2 + 4\\cdot(-1)\\\\\\\\\n",
    "\\vec{a}\\cdot\\vec{b}&=2\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can express this in a general form (for 2-dimensional vectors) as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\\vec{a}\\cdot\\vec{b}&=a_x b_x+a_y b_y\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multiplying vector components elementwise and adding them takes part in the Pythagorean theorem. To find the length of the vector $\\vec{e}$ we squared each of its components and took the square root of their sum."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\|\\vec{e}\\|=\\sqrt{2^2+3^2+5^2}=\\sqrt{38}\\approx6.16$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can generalise this procedure by using dot product:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\|\\vec{e}\\|^2=\\vec{e}\\cdot\\vec{e}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "\\|\\vec{e}\\|^2&=\\begin{bmatrix} 2\\\\3\\\\5\\end{bmatrix}\\cdot \\begin{bmatrix} 2\\\\3\\\\5 \\end{bmatrix} \\\\\\\\\n",
    "\\|\\vec{e}\\|^2&= 2^2+3^3+5^5 = 38 \\\\\\\\\n",
    "\\|\\vec{e}\\|&= \\sqrt{38}\\approx6.16\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Adding, subtracting and multiplying vectors by using code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of these operations are very easy to perform by using code. To add the vectors $\\vec{a}$ and $\\vec{b}$, we simply type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[5],\n",
       "       [3]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a+b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make the output look somewhat nicer, we can use the `print` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[5]\n",
      " [3]]\n"
     ]
    }
   ],
   "source": [
    "print (a+b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To subtract $\\vec{b}$ from $\\vec{a}$, we can write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1]\n",
      " [5]]\n"
     ]
    }
   ],
   "source": [
    "print (a-b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To multiply the vector $\\vec{a}$ by a scalar, we simply write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[15]\n",
      " [20]]\n"
     ]
    }
   ],
   "source": [
    "print (5*a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Multiplying vectors elementwise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the light of vector addition and scalar multiplication, we can ask, what would happen if we typed the command `a*b`? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6],\n",
       "       [-4]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a*b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This command multiplies vectors' respective coordinates, similar to vector addition, thus produces the product vector of the same size as $\\vec{a}$ and $\\vec{b}$. Although this operation is very useful in computer science, it is somewhat obscured in elementary mathematics. It is known as the __Hadamard product__, and denoted as $\\vec{a}\\odot \\vec{b}$. In our case:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "\\vec{a}\\odot \\vec{b}&=\\begin{bmatrix}3\\\\4\\end{bmatrix}\\odot\\begin{bmatrix}2\\\\-1\\end{bmatrix}=\\begin{bmatrix}3\\cdot 2\\\\4\\cdot (-1)\\end{bmatrix}\\\\\\\\\n",
    "\\vec{a}\\odot \\vec{b}&=\\begin{bmatrix}6\\\\-4\\end{bmatrix}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can express this in a general form (for two dimensional vectors) as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\\vec{a}\\odot \\vec{b}&=\\begin{bmatrix} a_x \\, b_x\\\\a_y \\, b_y\\end{bmatrix}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Computing the dot product by using code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute the dot product of $\\vec{a}$ and $\\vec{b}$ we can multiply two vectors elementwise and then sum the multiples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(a*b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This also gives us an alternative way to compute the length of a vector. We simply multiply the vector by itself elementwise, sum the multiples and compute the square root of the sum. To compute the length of the vector $\\vec{a}$ instead of using the function `linalg.norm` we can type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sqrt(np.sum(a*a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basis Vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amid all the possible vectors that can be represented in a space of given dimensionality, some vectors are more unique than the others. They are called __basis vectors__, and we can think of them as the elementary units constituting the given vector space. They emerge from the vector's scaling ability. Let's illustrate this with an example. Let's start with the vector $\\vec{a}$ whose coordinates are (3,4). Now, let's introduce two new vectors: $\\boldsymbol{\\hat{\\imath}}$, parallel with the $x$-axis and $\\boldsymbol{\\hat{\\jmath}}$ parallel to the $y$-axis. The most important property of $\\boldsymbol{\\hat{\\imath}}$ (i-hat) and $\\boldsymbol{\\hat{\\jmath}}$ (j-hat) is that their magnitude (length) is equal to one. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/linear-algebra09.png\" alt=\"drawing\" width=\"350\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can represent them numerically as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "\\boldsymbol{\\hat{\\imath}}=\\begin{bmatrix} 1\\\\0\\end{bmatrix};  \\quad \\boldsymbol{\\hat{\\jmath}}=\\begin{bmatrix} 0\\\\1\\end{bmatrix}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vector $\\vec{a}$'s horizontal projection is 3 times as long as $\\boldsymbol{\\hat{\\imath}}$'s and they point in the same direction. Vector $\\vec{a}$'s vertical projection is 4 times as long as $\\boldsymbol{\\hat{\\jmath}}$'s and they point in the same direction. By scaling the basis vectors $\\boldsymbol{\\hat{\\imath}}$ and $\\boldsymbol{\\hat{\\jmath}}$ (multiplying them by 3 and 4 respectively) and then adding them, we can recreate the vector $\\vec{a}$! This is shown in the illustration below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/linear-algebra10.png\" alt=\"drawing\" width=\"350\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the vectors $\\boldsymbol{\\hat{\\imath}}$ and $\\boldsymbol{\\hat{\\jmath}}$ are orthogonal to each other, their dot product will be equal to zero. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "\\boldsymbol{\\hat{\\imath}} \\cdot \\boldsymbol{\\hat{\\jmath}} &= \\begin{bmatrix} 1\\\\0 \\end{bmatrix} \\cdot \\begin{bmatrix} 0\\\\1 \\end{bmatrix} \\\\\\\\\n",
    "\\boldsymbol{\\hat{\\imath}} \\cdot \\boldsymbol{\\hat{\\jmath}} &= 1 \\cdot 0 + 0 \\cdot 1 = 0\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear combination of Vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can write the vector $\\vec{a}$ symbolically as $\\vec{a}=3\\boldsymbol{\\hat{\\imath}}+4\\boldsymbol{\\hat{\\jmath}}$. Not only can we represent the vector $\\vec{a}$ by means of $\\boldsymbol{\\hat{\\imath}}$ and $\\boldsymbol{\\hat{\\jmath}}$, but also any possible two-dimensional vector!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\vec{a}=3\\boldsymbol{\\hat{\\imath}}+4\\boldsymbol{\\hat{\\jmath}}\\\\\n",
    "\\vec{b}=-2\\boldsymbol{\\hat{\\imath}}+1\\boldsymbol{\\hat{\\jmath}}\\\\\n",
    "\\vec{c}=-3\\boldsymbol{\\hat{\\imath}}-3\\boldsymbol{\\hat{\\jmath}}\\\\\n",
    "\\vec{d}=-2\\boldsymbol{\\hat{\\imath}}+3\\boldsymbol{\\hat{\\jmath}}\\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In doing so, we are describing these vectors as a __linear combination__ of vectors $\\boldsymbol{\\hat{\\imath}}$ and $\\boldsymbol{\\hat{\\jmath}}$.This idea seamlessly extends into higher dimensions. For three dimensions, we need another basis vector parallel to the Z-axis, which is usually denoted as $\\boldsymbol{\\hat{k}}$ (k-hat). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/linear-algebra11.png\" alt=\"drawing\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can describe $\\vec{e}$ as a linear combination of $\\boldsymbol{\\hat{\\imath}}$, $\\boldsymbol{\\hat{\\jmath}}$ and $\\boldsymbol{\\hat{k}}$:\n",
    "$$\\vec{e}=2\\boldsymbol{\\hat{\\imath}}+3\\boldsymbol{\\hat{\\jmath}}+5\\boldsymbol{\\hat{k}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vectors $\\boldsymbol{\\hat{\\imath}}$, $\\boldsymbol{\\hat{\\jmath}}$ and $\\boldsymbol{\\hat{k}}$ are __orthogonal__ to each other and their linear combinations can describe any vector in their respective space. For example, the linear combination of $\\boldsymbol{\\hat{\\imath}}$ and $\\boldsymbol{\\hat{\\jmath}}$ can describe any two dimensional vector, and the linear combination of $\\boldsymbol{\\hat{\\imath}}$, $\\boldsymbol{\\hat{\\jmath}}$ and $\\boldsymbol{\\hat{k}}$ can describe any 3-dimensional vector.  This is possible precisely because neither one of the vectors themselves can be described by combining the other two. This makes sense, as $\\boldsymbol{\\hat{\\imath}}$ and $\\boldsymbol{\\hat{\\jmath}}$ alone simply cannot describe anything stretching in the $z$-direction. We call the vectors that have this property, __linearly independent vectors__. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the other side, __linearly dependent__ vectors in a set of vectors would be those that can be described by a linear combination of the others. Let's see an example. The image below contains 3 __unit vectors__—vectors whose length is equal to one, located on a plane."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/linear-algebra12.png\" alt=\"drawing\" width=\"270\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To describe any vector in this plane, one vector is redundant! A linear combination of the vectors $\\boldsymbol{\\hat{\\imath}}$ and $\\boldsymbol{\\hat{\\jmath}}$ can describe the vector $\\boldsymbol{\\hat{p}}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\boldsymbol{\\hat{p}} = -0.88\\boldsymbol{\\hat{\\imath}} - 0.47\\boldsymbol{\\hat{\\jmath}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/linear-algebra13.png\" alt=\"drawing\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To describe any point in 2-dimensional space, we do need exactly two vectors, but they need not necessarily be orthogonal to each other! They only must not be parallel (If two vectors are parallel, then each of them can be used to describe the other). For example, we can describe $\\boldsymbol{\\hat{\\jmath}}$ as a linear combination of $\\boldsymbol{\\hat{\\imath}}$ and $\\boldsymbol{\\hat{p}}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\boldsymbol{\\hat{\\jmath}} = -1.87\\boldsymbol{\\hat{\\imath}} - 2.13\\boldsymbol{\\hat{p}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/linear-algebra14.png\" alt=\"drawing\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can also describe $\\boldsymbol{\\hat{\\imath}}$ as a linear combination of $\\boldsymbol{\\hat{\\jmath}}$ and $\\boldsymbol{\\hat{p}}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\boldsymbol{\\hat{\\imath}} = -1.14\\boldsymbol{\\hat{p}} - 0.52\\boldsymbol{\\hat{\\jmath}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/linear-algebra15.png\" alt=\"drawing\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we have two vectors $\\vec{u}$ and $\\vec{v}$, a set of all vectors that can be reached with this linear combination is called the __span__ of the vectors $\\vec{u}$ and $\\vec{v}$. To fully span two-dimensional vector space requires two-linearly independent vectors. If these two vectors are linearly dependent, they do not fully span the two-dimensional space, rather only a single line in this space. The animation below aims to illustrate how a linear combination of two linearly independent non-orthogonal vectors $a\\vec{u}+b\\vec{v}$ can be used to reach any 2-dimensional vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](img/linear-algebra16.gif \"segment\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same happens in higher dimensions. Fully spaning a 3-dimensional vector space requires 3 linearly independent vectors. If 2 out of 3 vectors are linearly dependent, their span will be a 2-D plane in 3-D space, like in the animation below. If all three are linearly dependent, their span will be a line in 3-dimensional space. Now we can give a technical definition of a __vector basis__: It is a set of linearly independent vectors that span the full space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](img/linear-algebra17.gif \"segment\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}