Bird PCA-complete.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Principal Component Analysis (PCA) for Data Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.cm as cm\n",
    "import matplotlib.colors as colors\n",
    "import random\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Bird Dataset into Pandas DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://alexandra-sandbox-bucket.s3.amazonaws.com/birdData_reduced.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create Pandas DataFrame using read_csv\n",
    "df = pd.read_csv(url, index_col=0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop rows with null values from our DataFrame using dropna\n",
    "df.dropna(axis='rows', how='any', inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's peek at our DataFrame head\n",
    "df.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of our attributes may be too highly correlated to add value to our Principal Component Analysis. Let's look at the correlation between our variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check DataFrame correlation\n",
    "df.corr()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standardize the Data\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PCA yields a feature subspaces that maximizes variance along axes. Therefore, when working with data that was measured on different scales, we need to standardize it. This dataset has lengths in centimeters, mass, and time periods of days and years. In this next step, we will transform the data onto unit scale with `mean=0` and `variance=1`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# features will hold all our numeric columns' labels\n",
    "# Let's exclude WingU_MEAN, EggL_MEAN and EggW_MEAN due to their high correlation with other attributes\n",
    "features = ['LengthU_MEAN', 'TailU_MEAN', 'BillU_MEAN', 'TarsusU_MEAN', 'WeightU_MEAN', 'Clutch_MEAN', 'Broods per year', 'Egg_MASS', 'Incubation period', 'Fledging period', 'Age of first breeding', 'Life span']\n",
    "\n",
    "# set x as features values\n",
    "x = df.loc[:, features].values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set y to Order values\n",
    "y = df.loc[:,['Order']].values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now we will use StandardScaler to transform x (our features)\n",
    "x = StandardScaler().fit_transform(x)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print new dataframe of standardized values\n",
    "pd.DataFrame(data = x, columns = features).head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PCA Projection to 2-Dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize an instance of PCA with 2 components\n",
    "pca = PCA(n_components=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# transform x using pca\n",
    "principalComponents = pca.fit_transform(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how our principal components have been condensed into two columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print principalComponents\n",
    "print(principalComponents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a new DataFrame with these principal components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new DataFrame with these 2 principal components\n",
    "pcaDF = pd.DataFrame(data = principalComponents\n",
    "        , columns = ['Principal Component 1', 'Principal Component 2'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's see the head of that DataFrame\n",
    "pcaDF.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the unique values from Order to see what they are\n",
    "print(df['Order'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All our specimens will belong to one of these orders\n",
    "Now we'll combine the principal components DataFrame with the Order column to create our final DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# concatenate the Order column with our pca DataFrame\n",
    "finalDF = pd.concat([pcaDF, df[['Order']]], axis = 1)\n",
    "\n",
    "# drop NA vallues again, just to be safe \n",
    "finalDF.dropna(axis='rows', how='any', inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#let's display the head again \n",
    "finalDF.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize 2D projection\n",
    "Now we will use your PCA projection to create a 2-Dimensional visualization of our dataset. We will plot the different orders as different colors to see if we find any differences. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (10,10))\n",
    "ax = fig.add_subplot(1,1,1) \n",
    "\n",
    "ax.set_xlabel('Principal Component 1', fontsize = 15)\n",
    "ax.set_ylabel('Principal Component 2', fontsize = 15)\n",
    "ax.set_title('2 Component PCA', fontsize = 20)\n",
    "\n",
    "targets = ['Accipitriformes', 'Anseriformes', 'Caprimulgiformes', 'Ciconiiformes',\n",
    " 'Columbiformes', 'Coraciiformes', 'Falconiformes', 'Galliformes',\n",
    " 'Gaviiformes', 'Gruiformes', 'Charadriiformes', 'Otidiformes',\n",
    " 'Passeriformes', 'Pelecaniformes', 'Phoenicopteriformes', 'Piciformes',\n",
    " 'Podicipediformes', 'Procellariiformes', 'Psittaciformes', 'Strigiformes',\n",
    " 'Suliformes']\n",
    "\n",
    "\n",
    "for target in targets:\n",
    "    indicesToKeep = finalDF['Order'] == target\n",
    "    ax.scatter(finalDF.loc[indicesToKeep, 'Principal Component 1']\n",
    "               , finalDF.loc[indicesToKeep, 'Principal Component 2'])\n",
    "ax.legend(targets)\n",
    "ax.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that even with our Principal Component Analysis, there are a lot of similarities and overlaps between these orders of birds. \n",
    "\n",
    "Now it's your turn to customize this visualization. Do you think it's possible to get more separation out of our Orders? Perhaps you may remove a few Orders to simplify the visualization. You might even consider refactoring to use Family instead of Order, if you have time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize = (10,10))\n",
    "ax = fig.add_subplot(1,1,1) \n",
    "\n",
    "ax.set_xlabel('Principal Component 1', fontsize = 15)\n",
    "ax.set_ylabel('Principal Component 2', fontsize = 15)\n",
    "ax.set_title('2 Component PCA', fontsize = 20)\n",
    "\n",
    "targets = ['Falconiformes', 'Gaviiformes', 'Gruiformes']\n",
    "\n",
    "\n",
    "for target in targets:\n",
    "    indicesToKeep = finalDF['Order'] == target\n",
    "    ax.scatter(finalDF.loc[indicesToKeep, 'Principal Component 1']\n",
    "               , finalDF.loc[indicesToKeep, 'Principal Component 2'])\n",
    "ax.legend(targets)\n",
    "ax.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example we see that Falconiformes (falcons) are pretty distinct from Gaviifomres (loons) and Gruiformes (crane-like birds). "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}