F0C-03-CS-EDA1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FLIP (00): Data Science \n",
    "**(Module 02: Data Visualization)**\n",
    "\n",
    "---\n",
    "- Materials in this module include resources collected from various open-source online repositories.\n",
    "- You are free to use,but NOT allowed to change and distribute this package.\n",
    "\n",
    "Prepared by and for \n",
    "**Student Members** |\n",
    "2006-2023 [TULIP Lab](http://www.tulip.org.au), Australia\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Session D -  Case Study (Exploratory Data Analysis 1)\n",
    "\n",
    "In this lesson, we'll get a taste of what matplotlib can do by using it to perform a bit of Exploratory Data Analysis on a well known data set. \n",
    "\n",
    "\n",
    "\n",
    "[Exploratory Data Analysis (EDA)][1] is an approach to analyzing data that focuses on summarizing the main characteristics of a data set. One of EDA's major proponents, the venerable \n",
    "statistician [John Tukey][2], was a major force behind the acceptance of EDA into every modern day data scientist's toolbox, and his efforts to popularize the practice encouraged the development of statistical computing packages such as the [S][3] and [R][4] programming languages. The matplotlib library can itself be seen as being directly influenced by the work of Tukey. \n",
    "\n",
    "[1]: https://en.wikipedia.org/wiki/Exploratory_data_analysis\n",
    "[2]: https://en.wikipedia.org/wiki/John_Tukey\n",
    "[3]: https://en.wikipedia.org/wiki/S_(programming_language)\n",
    "[4]: https://www.r-project.org/\n",
    "\n",
    "<figure>\n",
    "  <img src=\"https://github.com/tuliplab/mds/blob/master/Jupyter/image/John_Tukey.jpg?raw=true\" alt=\"John Tukey\" width=\"268\" height=\"326\">\n",
    "  <figcaption style=\"text-align:center;padding-top:10px\">By Source, <a href=\"//en.wikipedia.org/wiki/File:John_Tukey.jpg\" title=\"<a href=&quot;//en.wikipedia.org/wiki/Wikipedia:Non-free_use_rationale_guideline&quot; title=&quot;Wikipedia:Non-free use rationale guideline&quot;>Fair use</a> of copyrighted material in the context of <a href=&quot;//en.wikipedia.org/wiki/John_Tukey&quot; title=&quot;John Tukey&quot;>John Tukey</a>\">Fair use</a>, https://en.wikipedia.org/w/index.php?curid=17099473</figcaption>\n",
    "</figure>\n",
    "\n",
    "Exploratory data analysis can encompass many different techniques, but key amongst them is the act of visualizing the data set in question, and this is where matplotlib comes in. The first step, however, before we can do anything, is to import the matplotlib library.\n",
    "\n",
    "<!---\n",
    "In this lesson, we'll get a sneak peak into the power that matplotlib possesses by using the library to perform a few simple visualizations on a well known data set to see what we can learn about the data.\n",
    "\n",
    "## Interacting With matplotlib\n",
    "\n",
    "The matplotlib library has exactly three different interfaces to choose from when deciding how you want to interact with the library. We'll go over each in a bit more detail in a later lesson, but for this lesson, we'll be using the `pyplot` interface, since it's the recommended way for doing interactive plotting with matplotlib. Also, since this lesson is really just meant to showcase some of the things you can expect to learn in this course, rather than being instructional, I wouldn't get too worried if you don't understand everything that we do in this lesson; I promise that by the end of this course, you'll have everything you need to understand everything we'll be doing in this lesson, and more so. With that in mind, let's get started...\n",
    "\n",
    "--->\n",
    "\n",
    "## Importing the Libraries\n",
    "\n",
    "The matplotlib library has exactly three different interfaces to choose from when deciding how you want to interact with the library. We'll go over each in a bit more detail in a later lesson, but for this lesson, we'll be using the `pyplot` interface, since it's the recommended way for doing interactive plotting with matplotlib. Also, since this lesson is really just meant to showcase some of the things you can expect to learn in this course, rather than being instructional, I wouldn't get too worried if you don't understand everything that we do in this lesson; I promise that by the end of this course, you'll have everything you need to understand every last detail of what we'll be doing in this lesson, and more so. With that in mind, let's get started...\n",
    "\n",
    "The `import` line below (line 2) is the standard way to import the `pyplot` module, and you'll generally see it imported just like this everywhere it's used. Incidentally, you'll see that pretty much every module that you regularly use from matplotlib, and the rest of the python scientific stack for that matter, generally have a commonly accepted way in which they are imported, and stickler for the rules that I am, I will endeavor to always use the standard way throughout this course---in order to foster good coding practices in all of us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from IPython.display import set_matplotlib_formats\n",
    "set_matplotlib_formats('retina')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Census Income Data Set\n",
    "\n",
    "Now that we have our notebook properly set up to work with matplotlib, it's time to pull in some data and get our hands dirty.\n",
    "\n",
    "The University of California at Irvine maintains a repository of over 300 real world data sets for use in testing machine learning algorithms. This repository is a fantastic resource since it allows us to play around with a relatively small, real world dataset while ignoring all of the cumbersome pre-processing steps you'd normally have to perform before getting a chance to explore the data.\n",
    "\n",
    "In this quick example of matplotlib in action, we'll be using the [\"Census Income\"][1] data set, also known as the [\"Adult\"][2] data set. The stated purpose of this data set is to predict whether or not an individual makes more than \\$50k a year based on data gathered during the 1994 Census. While we won't be using this data set to test a predictive algorithm, we can still make use of the data to find some interesting insights. \n",
    "\n",
    "[1]: http://archive.ics.uci.edu/ml/datasets/Census+Income\n",
    "[2]: http://archive.ics.uci.edu/ml/datasets/Adult\n",
    "\n",
    "The first step is to grab the data from the [UCI Machine Learning Repository][3]. To do that, we'll take advantage of python's \"batteries included\" philosophy and make use of a few modules from the standard library to get this bit of our work done. In the code below, we use the [`urllib2`][4] module to download the data set from the repository, the [`csv`][5] module to read the data in, and [`namedtuples`][6] from the `collections` module to transform that data into a format that's easy to work with. Incidentally, I've saved the data file locally, so if for some reason you find yourself in a situation where you don't have access to a fast network connection, you can simply replace the download code on lines 25 and 26 below with some code to pull the data in from the `./data/adult_data.csv` file instead.\n",
    "\n",
    "[3]: http://archive.ics.uci.edu/ml/index.html\n",
    "[4]: https://docs.python.org/2/library/urllib2.html\n",
    "[5]: https://docs.python.org/2/library/csv.html\n",
    "[6]: https://docs.python.org/2/library/collections.html#collections.namedtuple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "#import urllib2   python 3 no longer use\n",
    "import urllib.request  \n",
    "from collections import namedtuple\n",
    "\n",
    "\n",
    "# Create a namedtuple constructor for each record in the Census data\n",
    "fields = ('age', \n",
    "          'workclass', \n",
    "          'fnlwgt', \n",
    "          'education', \n",
    "          'education_num', \n",
    "          'marital_status', \n",
    "          'occupation', \n",
    "          'relationship', \n",
    "          'race', \n",
    "          'sex', \n",
    "          'capital_gain', \n",
    "          'capital_loss', \n",
    "          'hours_per_week', \n",
    "          'native_country', \n",
    "          'target')\n",
    "CensusRecord = namedtuple('CensusRecord', fields)\n",
    "\n",
    "# Download and read in the data from the UCI Machine Learning Repository\n",
    "response = urllib.request.urlopen('http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data')\n",
    "adult_data_csv = response.read().strip()\n",
    "\n",
    "# Convert each record into a format that's easier to work with (i.e.,\n",
    "# wrap each record in the namedtuple that we created).\n",
    "data = []\n",
    "for row in csv.reader(adult_data_csv.splitlines()):\n",
    "    data.append(CensusRecord(\n",
    "        age              = int(row[0]),\n",
    "        workclass        = row[1].strip(),\n",
    "        fnlwgt           = float(row[2].strip()),\n",
    "        education        = row[3].strip(),\n",
    "        education_num    = int(row[4]),\n",
    "        marital_status   = row[5].strip(),\n",
    "        occupation       = row[6].strip(),\n",
    "        relationship     = row[7].strip(),\n",
    "        race             = row[7].strip(),\n",
    "        sex              = row[9].strip(),\n",
    "        capital_gain     = int(row[10]),\n",
    "        capital_loss     = int(row[11]),\n",
    "        hours_per_week   = int(row[12]),\n",
    "        native_country   = row[13].strip(),\n",
    "        target           = row[14].strip()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the Data Set\n",
    "\n",
    "Now that we've read the data in and have it in a format that we can easily work with, we're ready to explore our data set a bit. \n",
    "\n",
    "### Histograms and Bar Charts\n",
    "\n",
    "A [histogram][1] is a fantastic visualization for getting a feel for the distribution of your data set. Creating a histogram with matplotlib is relatively easy, you simply call the [`pyplot.hist`][2] function and pass in a list of values, and matplotlib will take care of calculating the frequency distribution and plotting the data for you. We could do that here, but a histogram is really meant to be used with continuous data, and in our case, we're dealing with discrete values, so a [bar chart is really a more fitting tool][3] for this particular situation, and we can dress it up to look just like a typical matplotlib histogram, so that's what we'll do here.\n",
    "\n",
    "[1]: https://en.wikipedia.org/wiki/Histogram\n",
    "[2]: http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.hist\n",
    "[3]: http://www.forbes.com/sites/naomirobbins/2012/01/04/a-histogram-is-not-a-bar-chart/#570a8dda28af\n",
    "\n",
    "Let's take a first look at our data by creating a histogram using the [`pyplot.bar`][4] function. Since we aren't using the `pyplot.hist` function, we'll have to do the frequency count ourselves, but luckily, the python standard library provides a collection class that will do that for us. \n",
    "\n",
    "[4]: http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.bar\n",
    "\n",
    "The code below, uses the [`collections.Counter`][5] class to create a dictionary-like object whose keys will be the education level from our data set, and values the number of times we see each of those levels occurring in the data. Then, we simply plot the keys and values in our `Counter` object using the `pyplot.bar` function to get a simple \"histogram\".\n",
    "\n",
    "[5]: https://docs.python.org/2/library/collections.html#collections.Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "import numpy as np\n",
    "\n",
    "# Calculate the frequency count for each education level.\n",
    "freqs = Counter(r.education_num for r in data)\n",
    "# Draw the bar chart/histogram\n",
    "plt.figure(figsize=(10, 4))\n",
    "plt.bar(freqs.keys(), freqs.values(), width=1);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is the chart above, and it's showing us a couple of interesting spikes in the number of people with a particular level of education. These spikes occurred around levels 9, 10, and 13. While this is interesting, it'd be better if the education levels actually meant something to us. Fortunately, the census data has another feature called `education` that contains the human readable counterpart to the education level. So, all we need to do is replace the education levels along the x-axis with their corresponding names from the `education` field. \n",
    "\n",
    "To match up the human readable education levels with their numerical equivalent, we'll need to first figure out the order in which the names should be displayed along the x-axis. We can do that by matching each name up to its corresponding education level, and sorting the names based on the level. The easiest way to do that is to just change the keys in our `Counter` object from a scalar value, the education level, to a tuple containing both the education level and the human-readable name of that level---e.g., `(9, 'HS-Grad')`. Then, we can simply sort our list of keys by the first element in the tuple, and extract the name (i.e., the second element in the tuple) to get a list of names ordered according to their education level.\n",
    "\n",
    "Once we have our list of ordered names, we can recreate the histogram from the last section and replace the tick marks along the x-axis with those names using the `pyplot.xticks` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "# Calculate the frequency count for each education level. Our \n",
    "# keys in the Counter object will be a tuple of the form \n",
    "# (number, name) so it will be possible to sort the keys.\n",
    "freqs = Counter((r.education_num, r.education) for r in data)\n",
    "# Create a list of names sorted by the education level number\n",
    "names = [name for _, name in sorted(freqs.keys())]\n",
    "# Create a list of counts in the same order as the names\n",
    "counts = [freqs[key] for key in sorted(freqs.keys())]\n",
    "# An array containing the x coordinates of the left sides of the bars in the chart\n",
    "left = range(len(names))\n",
    "\n",
    "# Draw the bar chart/histogram\n",
    "plt.figure(figsize=(10, 4))\n",
    "# Plot the counts at the given x\n",
    "plt.bar(left, counts, width=1)\n",
    "# Change the x-axis ticks to the education level name. To make it\n",
    "# easier to read, we rotate the labels 90 degrees and move them to\n",
    "# the center of the bar by adding 0.5 to each value in the left \n",
    "# array (we chose 0.5 because we set the width of each bar to 1 earlier).\n",
    "plt.xticks(left, names);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our chart now shows us the information we want, but unfortunately, the labels along the x-axis are impossible to read in many cases. Fortunately, there's an easy fix for that. \n",
    "\n",
    "First, we need to rotate the names by 90 degrees using the `rotation` parameter of the `pyplot.xticks` function. We could leave it at that, but the labels would look nicer if they were in the center of each bar in our graph, instead of being lined up with the left side of the bar. Since we set the bar width to 1 earlier, we can simply add 0.5 to each value in the `left` array before passing it into the `pyplot.bar` function. That will move each label to the right by half the width of the bar which will put each label right in the middle of its corresponding bar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Calculate the frequency count for each education level. Our \n",
    "# keys in the Counter object will be a tuple of the form \n",
    "# (number, name) so it will be possible to sort the keys.\n",
    "freqs = Counter((r.education_num, r.education) for r in data)\n",
    "# Create a list of names sorted by the education level number\n",
    "names = [name for _, name in sorted(freqs.keys())]\n",
    "# Create a list of counts in the same order as the names\n",
    "counts = [freqs[key] for key in sorted(freqs.keys())]\n",
    "# An array containing the x coordinates of the left sides of the bars in the chart\n",
    "left = np.arange(len(names))\n",
    "\n",
    "# Draw the bar chart/histogram\n",
    "plt.figure(figsize=(10, 4))\n",
    "# Plot the counts at the given x\n",
    "plt.bar(left, counts, width=1)\n",
    "# Change the x-axis ticks to the education level name. To make it\n",
    "# easier to read, we rotate the labels 90 degrees and move them to\n",
    "# the center of the bar by adding 0.5 to each value in the left \n",
    "# array (we chose 0.5 because we set the width of each bar to 1 earlier).\n",
    "plt.xticks(left + 0.5, names, rotation=90);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that our chart is complete, we can finally see that the majority of participants in the 1994 Census, had at least a high school education, followed by some college, and then a Bachelor's degree. The other interesting tidbit that we can gather from this simple visualization is just how much more popular a four year college is than a traditional trade school, or a two year associates degree. In many countries, apprenticeships are seen as a popular route for getting the education and training one needs for a secure future, but here in the US, it would seem that we still highly favor the four year institution over these other options.\n",
    "\n",
    "### Scatter Plots\n",
    "\n",
    "The next bit of data we're going to investigate are the number of hours worked per week. If we use a [scatter plot][1] we can plot two different variables in our data set against one another. Doing so allows us to visually see the relationship, should one exist, between the two variables. In our example below, we're going to plot the average number of hours worked per week for each level of education. \n",
    "\n",
    "[1]: https://en.wikipedia.org/wiki/Scatter_plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a list of the average hours/week worked order by education level\n",
    "avgs = []\n",
    "for name in names:\n",
    "    # Get the total number of hours/week for the current education level\n",
    "    hours_per_week = [r.hours_per_week for r in data if r.education == name]\n",
    "    avgs.append(np.average(hours_per_week))\n",
    "\n",
    "# Create a list of x values\n",
    "xs = np.arange(1, len(names) + 1)\n",
    "\n",
    "# Create a scatter plot of the data\n",
    "plt.figure(figsize=(10, 4))\n",
    "plt.scatter(xs, avgs)\n",
    "\n",
    "# Tighten the graph up by reducing the margins on the x-axis\n",
    "plt.margins(x=0.05)\n",
    "\n",
    "# Adding the education level labels to the graph\n",
    "plt.xticks(xs, names, rotation=90)\n",
    "\n",
    "# Add a title and labels to each of the axes\n",
    "plt.ylabel('Avg. Hours of Work per Week');\n",
    "plt.xlabel('Education Level')\n",
    "plt.title('Avg. Hours of Work per Week by Education Level', y=1.05);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our scatter plot above seems to show a very distinct positive correlation between the number of hours worked per week on average and the highest level of education that a person has attained. We do see a dip in the number of hours worked if the person dropped out of school after the 8th grade, but before finishing high school, but aside from that rather strange dip in the data, the overall shape suggests a positive relationship between the two variables. \n",
    "\n",
    "We can go one step further though and verify our intuition by performing a [simple linear regression][1] on the data, and drawing the line that it produces onto our graph. If you're not familiar with linear regressions, it's actually a fairly easy concept to grasp. Essentially, you're trying to find a line that fits the data in the best possible way. So, imagine you had the scatter plot above on a piece of graph paper, and you took a ruler and drew a line through that scatter plot in such a way that the sum of the distances between your line and every point in the data set was as small as possible. The reason for doing this is two fold: first, the line we produce gives us a fairly intuitive view into the relationship of the variables in our data, and second, it gives us an equation that we can use to roughly predict the value of the dependent variable, the variable on the y-axis, given a value for the independent variable.\n",
    "\n",
    "[1]: https://en.wikipedia.org/wiki/Simple_linear_regression\n",
    "\n",
    "Since matplotlib is dependent on the [NumPy][2] library, we already have everything installed that we need to run a linear regression on our data set. To do so, we simply call the [`numpy.polyfit`][3] function and pass in our data. The `polyfit` function takes two arrays---x values in the first, and y values in the second---and the degree of the equation that we're trying to fit, and returns the slope and y-intercept for the best fit line. The [degree of an equation][4] is simply the largest exponent in that equation, and since we are fitting a [linear equation][5], our largest exponent, and hence our degree, will be 1. Since we already have a list of x values, all we have to do then, is create a list of y values by multiplying each x value in our list by the slope of our line and adding the y-intercept to get the corresponding y value.\n",
    "\n",
    "[2]: http://www.numpy.org/\n",
    "[3]: http://docs.scipy.org/doc/numpy/reference/generated/numpy.polyfit.html\n",
    "[4]: https://en.wikipedia.org/wiki/Degree_of_a_polynomial\n",
    "[5]: https://en.wikipedia.org/wiki/Linear_equation\n",
    "\n",
    "Now, let's redraw our scatter plot from above, and add two more lines of code (31 and 32) to our previous example: the first will perform the linear regression, and the second will plot the resultant line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a list of the average hours/week worked order by education level\n",
    "avgs = []\n",
    "for name in names:\n",
    "    # Get the total number of hours/week for the current education level\n",
    "    hours_per_week = [r.hours_per_week for r in data if r.education == name]\n",
    "    avgs.append(np.average(hours_per_week))\n",
    "\n",
    "# Create a list of x values\n",
    "xs = np.arange(1, len(names) + 1)\n",
    "\n",
    "# Create a scatter plot of the data\n",
    "plt.figure(figsize=(10, 4))\n",
    "plt.scatter(xs, avgs)\n",
    "\n",
    "# Adding the education level labels to the graph\n",
    "plt.xticks(xs, names, rotation=90)\n",
    "\n",
    "# Tighten the graph up by reducing the margins on the x-axis\n",
    "plt.margins(x=0.05)\n",
    "\n",
    "# Add a title and labels to each of the axes\n",
    "plt.ylabel('Avg. Hours of Work per Week')\n",
    "plt.xlabel('Education Level')\n",
    "plt.title('Avg. Hours of Work per Week by Education Level', y=1.05)\n",
    "\n",
    "# Plot a linear regression line to show the trend in the data. \n",
    "# The ployfit function takes the x and y data and a degree which\n",
    "# is just the largest exponent in the polynomial function we are\n",
    "# trying to fit. Since we are fitting a simple line, the highest\n",
    "# exponent is just 1, so we pass that in for the degree.\n",
    "m, b = np.polyfit(xs, avgs, 1)\n",
    "plt.plot(xs, m*xs+b, 'r--');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the best fit line drawn on our scatter plot, we can definitely see a positive trend in the data. So, as it stands now, it would appear that the more education you have, the more you work, on average. The question then becomes, is this true because, as your level of education increases, so too does the availability of work? Or, is it because a higher level of education is indicative of a \"workaholic\" personality, and so individuals with higher levels of education just tend to work more? Or, is it due to some other factor that we haven't yet considered?\n",
    "\n",
    "The key here is that visualizing our data has led us to ask new questions and to form new hypotheses. This, in turn, could lead to further data collection and experimentation. The whole cycle is a giant feedback loop, where data exploration feeds inquiry and further insight. It's easy to see why this was one of the main reasons that John Tukey encouraged statisticians to participate in the practice of Exploratory Data Analysis.\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "In this lesson, we got our first peak at just how useful matplotlib can be, and in the process, we learned a little bit about a technique, popularized by the famous statistician John Tukey, called Exploratory Data Analysis. Along the way, we saw just how useful visualizing our data can be with respect to uncovering new questions and new ways of thinking about our data. We also saw how, even though our first attempt at visualizing data may not be exactly what we want, matplotlib gets us most of the way to a proper visualization right out of the box, and with just a little tweaking, it's possible to take the initial result and turn it into the perfect visualization. \n",
    "\n",
    "I hope this lesson has served to whet your appetite a bit for what's to come in future lessons. It is my hope, that by the end of this course, you will be properly equipped to interact with your data the way we just did in this lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}