15.Principal Component Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An important machine learning method for dimensionality reduction is called Principal Component Analysis. It is a method that uses simple matrix operations from linear algebra and statistics to calculate a projection of the original data into the same number or fewer dimensions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. What is Principal Component Analysis?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Principal Component Analysis, or PCA for short, is a method for reducing the dimensionality of data.\n",
    "* It can be thought of as a projection method where data with m-columns (features) is projected into a subspace with m or fewer columns, whilst retaining the essence of the original data.\n",
    "* The PCA method can be described and implemented using the tools of linear algebra.\n",
    "* PCA is an operation applied to a dataset, represented by an $n \\times m$ matrix A that results in a projection of $A$ which we will call $B$. Let's walk through the steps of this operation.\n",
    "<br><br>\n",
    "\\begin{equation*}\n",
    "A =  \\begin{vmatrix}\n",
    "a_{1,1} & a_{1,2} \\\\\n",
    "a_{2,1} & a_{2,2} \\\\\n",
    "a_{3,1} & a_{3,2} \n",
    "\\end{vmatrix}\n",
    "\\end{equation*}\n",
    "<br><br>\n",
    "\\begin{equation*}\n",
    "B =  PCA(A)\n",
    "\\end{equation*}\n",
    "<br><br>\n",
    "* The first step is to calculate the mean values of each column.\n",
    "<br><br>\n",
    "\\begin{equation*}\n",
    "M =  mean(A)\n",
    "\\end{equation*}\n",
    "<br><br>\n",
    "* Next, we need to center the values in each column by subtracting the mean column value.\n",
    "<br><br>\n",
    "\\begin{equation*}\n",
    "C =  A - M\n",
    "\\end{equation*}\n",
    "<br><br>\n",
    "* The next step is to calculate the covariance matrix of the centered matrix $C$. Correlation is a normalized measure of the amount and direction (positive or negative) that two columns change together.\n",
    "* Covariance is a generalized and unnormalized version of correlation across multiple columns. A covariance matrix is a calculation of covariance of a given matrix with covariance scores for every column with every other column, including itself.\n",
    "<br><br>\n",
    "\\begin{equation*}\n",
    "V =  cov(C)\n",
    "\\end{equation*}\n",
    "<br><br>\n",
    "* Finally, we calculate the eigendecomposition of the covariance matrix $V$ . This results in a list of eigenvalues and a list of eigenvectors.\n",
    "<br><br>\n",
    "\\begin{equation*}\n",
    "values, vectors =  eig(V)\n",
    "\\end{equation*}\n",
    "<br><br>\n",
    "* The eigenvectors represent the directions or components for the reduced subspace of $B$, whereas the eigenvalues represent the magnitudes for the directions.\n",
    "* The eigenvectors can be sorted by the eigenvalues in descending order to provide a ranking of the components or axes of the new subspace for $A$.\n",
    "* If all eigenvalues have a similar value, then we know that the existing representation may already be reasonably compressed or dense and that the projection may offer little.\n",
    "* If there are eigenvalues close to zero, they represent components or axes of $B$ that may be discarded.\n",
    "* A total of $m$ or less components must be selected to comprise the chosen subspace. Ideally, we would select $k$ eigenvectors, called principal components, that have the $k$ largest eigenvalues.\n",
    "<br><br>\n",
    "\\begin{equation*}\n",
    "B = select(values, vectors)\n",
    "\\end{equation*}\n",
    "<br><br>\n",
    "Other matrix decomposition methods can be used such as Singular-Value Decomposition, or SVD. As such, generally the values are referred to as singular values and the vectors of the subspace are referred to as principal components. Once chosen, data can be projected into the subspace via matrix multiplication.\n",
    "<br><br>\n",
    "\\begin{equation*}\n",
    "P = B^{T} \\cdot A\n",
    "\\end{equation*}\n",
    "<br><br>\n",
    "Where $A$ is the original data that we wish to project, $B^{T}$ is the transpose of the chosen principal components and $P$ is the projection of $A$. This is called the covariance method for calculating the PCA, although there are alternative ways to calculate it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 2. Calculate Principal Component Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* There is no $pca()$ function in NumPy, but we can easily calculate the Principal Component Analysis step-by-step using NumPy functions.\n",
    "* The example below defines a small $3 \\times 2$ matrix, centers the data in the matrix, calculates the covariance matrix of the centered data, and then the eigendecomposition of the covariance matrix.\n",
    "* The eigenvectors and eigenvalues are taken as the principal components and singular values and used to project the original data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2]\n",
      " [3 4]\n",
      " [5 6]]\n",
      "[[ 0.70710678 -0.70710678]\n",
      " [ 0.70710678  0.70710678]]\n",
      "[8. 0.]\n",
      "[[-2.82842712  0.        ]\n",
      " [ 0.          0.        ]\n",
      " [ 2.82842712  0.        ]]\n"
     ]
    }
   ],
   "source": [
    "# principal component analysis\n",
    "from numpy import array\n",
    "from numpy import mean\n",
    "from numpy import cov\n",
    "from numpy.linalg import eig\n",
    "# define matrix\n",
    "A = array([\n",
    "[1, 2],\n",
    "[3, 4],\n",
    "[5, 6]])\n",
    "print(A)\n",
    "# column means\n",
    "M = mean(A.T, axis=1)\n",
    "# center columns by subtracting column means\n",
    "C = A - M\n",
    "# calculate covariance matrix of centered matrix\n",
    "V = cov(C.T)\n",
    "# factorize covariance matrix\n",
    "values, vectors = eig(V)\n",
    "print(vectors)\n",
    "print(values)\n",
    "# project data\n",
    "P = vectors.T.dot(C.T)\n",
    "print(P.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Principal Component Analysis in scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We can calculate a Principal Component Analysis on a dataset using the PCA() class in the scikit-learn library.\n",
    "* The benefit of this approach is that once the projection is calculated, it can be applied to new data again and again quite easily. \n",
    "* When creating the class, the number of components can be specified as a parameter.\n",
    "* The class is first fit on a dataset by calling the fit() function, and then the original dataset or other data can be projected into a subspace with the chosen number of dimensions by calling the transform() function.\n",
    "* Once fit, the singular values and principal components can be accessed on the PCA class via the explained variance and components attributes.\n",
    "* The example below demonstrates using this class by first creating an instance, fitting it on a $3 \\times 2$ matrix, accessing the values and vectors of the projection, and transforming the original data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2]\n",
      " [3 4]\n",
      " [5 6]]\n",
      "[[ 0.70710678  0.70710678]\n",
      " [-0.70710678  0.70710678]]\n",
      "[5.33333333 0.        ]\n",
      "[[-2.82842712e+00 -2.22044605e-16]\n",
      " [ 0.00000000e+00  0.00000000e+00]\n",
      " [ 2.82842712e+00  2.22044605e-16]]\n"
     ]
    }
   ],
   "source": [
    "# principal component analysis with scikit-learn\n",
    "from numpy import array\n",
    "from sklearn.decomposition import PCA\n",
    "# define matrix\n",
    "A = array([\n",
    "[1, 2],\n",
    "[3, 4],\n",
    "[5, 6]])\n",
    "print(A)\n",
    "# create the transform\n",
    "pca = PCA(2)\n",
    "# fit transform\n",
    "pca.fit(A)\n",
    "# access values and vectors\n",
    "print(pca.components_)\n",
    "print(pca.explained_variance_)\n",
    "# transform data\n",
    "B = pca.transform(A)\n",
    "print(B)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}