04.2_data_wrangling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Practicum AI Logo image](https://github.com/PracticumAI/practicumai.github.io/blob/main/images/logo/PracticumAI_logo_250x50.png?raw=true)  <img src='https://github.com/PracticumAI/practicumai.github.io/blob/main/images/icons/practicumai_python.png?raw=true' align='right' width=50>\n",
    "\n",
    "# *Practicum AI Python*: Data Wrangling - Part 2\n",
    "\n",
    "This exercise adapted from Lipp et al. (2020) <i>The Data Wrangling Workshop</i> from <a href=\"https://www.packtpub.com/product/the-data-wrangling-workshop-second-edition/9781839215001\">Packt Publishers</a> and the <a href=\"https://github.com/swcarpentry/python-novice-gapminder\">Software Carpentries</a>\n",
    "\n",
    "\n",
    "***\n",
    "\n",
    "## 1. Use the Pandas library to do statistics on tabular data\n",
    "\n",
    "Pandas has a lot of built in functionality in terms of statistics and visualization. Many of these have been highly optimized and run very quickly. \n",
    "\n",
    "`matplotlib` is another common visualization library that we will explore some too.\n",
    "\n",
    "First, let's make a dataframe for this exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "# Used in Jupyter to make graphs display in notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "people = ['Ann', 'Brandon', 'Chen', 'David', 'Emily',\n",
    "         'Farook', 'Gagen', 'Hamish', 'Imran', 'Joseph',\n",
    "         'Katherine', 'Lily']\n",
    "\n",
    "age    = [21, 12, 32, 45, 37, 18, 28, 52, 5 ,40, 48, 15]\n",
    "weight = [55, 35, 77, 68, 70, 60, 72, 69, 18, 65, 82, 48]\n",
    "height = [160, 135, 170, 165, 173, 168, 175, 159, 105, 171, 155, 158]\n",
    "\n",
    "people_dict = { 'People':people, 'Age':age, 'Weight':weight, 'Height':height}\n",
    "df = pd.DataFrame(data = people_dict)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Explore dataframe `.shape`, `.count()`, `.sum()`, `.describe()`\n",
    "\n",
    "As noted, there are a lot of methods that can be used with dataframes. Let's explore some here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.shape # Note .shape does not have (), shape is an attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Can everyone go in the elevator at once?\n",
    "\n",
    "df['Weight'].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Height'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are also `.max()`, `.min()`, `.std()`, `.median()` and others, but `.describe()` shows lots of information about data distributions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Visualizing data with Pandas\n",
    "\n",
    "Pandas has a number of built in visualization functions\n",
    "\n",
    "<div style=\"padding: 10px;margin-bottom: 20px;border: thin solid #30335D;border-left-width: 10px;background-color: #fff\"><strong>Note:</strong> Matplotlib is but one Python plotting option.  If you are already familiar with ggplot and the grammar of graphics, the plotnine library is the Python equivalent of this popular R package.  Seaborn is also a great Python graphing library.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Weight'].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.plot.scatter('Weight', 'Height')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can also modify the output using matplotlib options\n",
    "df.plot.scatter('Weight', 'Height', s=150, c='orange')\n",
    "\n",
    "plt.grid(True)\n",
    "plt.title('Weight vs. Heights scatter plot', fontsize=15)\n",
    "plt.xlabel('Weight (in kg)', fontsize=10)\n",
    "plt.ylabel('Height (in cm)', fontsize=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pair plots also, known as scatter matrices, are a great way to visualize all your data\n",
    "\n",
    "pd.plotting.scatter_matrix(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Saving plots\n",
    "\n",
    "We may want to save the output of a plot. This can be done using the  `plt.savefig()` function. By default, the format will be determined from the filename extension you provide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.savefig('plot.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Use `DataFrame.info` to find out more about a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. The `DataFrame.columns` variable stores information about the dataframe's columns\n",
    "\n",
    "* Note that this is an attribute, *not* a method.\n",
    "    * Like `math.pi`.\n",
    "    * So do not use `()` to try to call it.\n",
    "* Called a *member variable*, or just *member*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Use `DataFrame.T` to transpose a dataframe\n",
    "\n",
    "* Sometimes we want to treat columns as rows and vice versa.\n",
    "* Transpose (written `.T`) doesn't copy the data, just changes the program's view of it.\n",
    "* Like `columns`, it is a member variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "## Bonus Questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q1: Reading Other Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the data in `gapminder_gdp_americas.csv` (which should be the same directory as `gapminder_gdp_oceania.csv`) into the variable called `americas` and display its summary statistics. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**\n",
    "\n",
    "Click on the '...' below to show the solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# To read in a CSV, we use `pandas.read_csv`and pass the filename \n",
    "# 'data/gapminder_gdp_americas.csv' to it. We also once again pass the column \n",
    "# name 'country' to the parameter `index_col` in order to index by country: \n",
    "\n",
    "americas = pandas.read_csv('data/gapminder_gdp_americas.csv', index_col='country')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q2: Inspecting Data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After reading the data for the AMericans, use `help(americas.head)` and `help(americas.tal)` to find out what `DataFrame.head` and `DataFrame.tail` do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. What method call will display the first three rows fo this data?\n",
    "2. What method call will display the last three columns of this data? (Hint: you may need to change your view of the data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**\n",
    "\n",
    "Click on the '...' below to show the solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# 1. We can check out the first five rows of `americas` by executing \n",
    "# `americas.head()` (allowing us to view the head of the DataFrame). We can \n",
    "# specify the number of rows we wish to see by specifying the parameter `n` \n",
    "# in our call to `americas.head()`. To view the first three rows, execute:\n",
    "\n",
    "americas.head(n=3)\n",
    "\n",
    "# The output is then\n",
    "\n",
    "         continent  gdpPercap_1952  gdpPercap_1957  gdpPercap_1962  \\\n",
    "country                                                               \n",
    "Argentina  Americas     5911.315053     6856.856212     7133.166023   \n",
    "Bolivia    Americas     2677.326347     2127.686326     2180.972546   \n",
    "Brazil     Americas     2108.944355     2487.365989     3336.585802`   \n",
    "\n",
    "          gdpPercap_1967  gdpPercap_1972  gdpPercap_1977  gdpPercap_1982  \\\n",
    "country                                                                     \n",
    "Argentina     8052.953021     9443.038526    10079.026740     8997.897412   \n",
    "Bolivia       2586.886053     2980.331339     3548.097832     3156.510452   \n",
    "Brazil        3429.864357     4985.711467     6660.118654     7030.835878`   \n",
    "\n",
    "          gdpPercap_1987  gdpPercap_1992  gdpPercap_1997  gdpPercap_2002  \\\n",
    "country                                                                     \n",
    "Argentina     9139.671389     9308.418710    10967.281950     8797.640716   \n",
    "Bolivia       2753.691490     2961.699694     3326.143191     3413.262690   \n",
    "Brazil        7807.095818     6950.283021     7957.980824     8131.212843`   \n",
    "\n",
    "          gdpPercap_2007  \n",
    "country                    \n",
    "Argentina    12779.379640  \n",
    "Bolivia       3822.137084  \n",
    "Brazil        9065.800825\n",
    "\n",
    "# 2. To check out the last three rows of `americas`, we would use the command, \n",
    "# `americas.tail(n=3)`, analogous to `head()` used above. However, here we want \n",
    "# to look at the last three columns so we need to change our view and then use \n",
    "# `tail()`. To do so, we create a new DataFrame in which rows and columns are \n",
    "# switched\n",
    "\n",
    "americas_flipped = americas.T\n",
    "\n",
    "# We can then view the last three columns of `americas` by viewing the last \n",
    "# three rows of `americas_flipped`:\n",
    "\n",
    "americas_flipped.tail(n = 3)\n",
    "\n",
    "# The output is then:\n",
    "\n",
    "country        Argentina  Bolivia   Brazil   Canada    Chile Colombia  \\\n",
    "gdpPercap_1997   10967.3  3326.14  7957.98  28954.9  10118.1  6117.36   \n",
    "gdpPercap_2002   8797.64  3413.26  8131.21    33329  10778.8  5755.26   \n",
    "gdpPercap_2007   12779.4  3822.14   9065.8  36319.2  13171.6  7006.58   \n",
    "\n",
    "country        Costa Rica     Cuba Dominican Republic  Ecuador    ...     \\\n",
    "gdpPercap_1997    6677.05  5431.99             3614.1  7429.46    ...      \n",
    "gdpPercap_2002    7723.45  6340.65            4563.81  5773.04    ...      \n",
    "gdpPercap_2007    9645.06   8948.1            6025.37  6873.26    ...      \n",
    "\n",
    "country          Mexico Nicaragua   Panama Paraguay     Peru Puerto Rico  \\\n",
    "gdpPercap_1997   9767.3   2253.02  7113.69   4247.4  5838.35     16999.4   \n",
    "gdpPercap_2002  10742.4   2474.55  7356.03  3783.67  5909.02     18855.6   \n",
    "gdpPercap_2007  11977.6   2749.32  9809.19  4172.84  7408.91     19328.7   \n",
    "\n",
    "country        Trinidad and Tobago United States  Uruguay Venezuela  \n",
    "gdpPercap_1997             8792.57       35767.4  9230.24   10165.5  \n",
    "gdpPercap_2002             11460.6       39097.1     7727   8605.05  \n",
    "gdpPercap_2007             18008.5       42951.7  10611.5   11415.8  \n",
    "\n",
    "Note: we could have done the above in a single line of code by 'chaining' the commands:\n",
    "\n",
    "americas.T.tail(n=3)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q3: Reading Files in Other Directories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data for your current project is stored in a file called `microbes.csv`, which is located in a folder called `field_data`. Your are doing analysis in a notebook called `analysis.ipynb` in a sibliong folder called `thesis`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "your home directory\n",
    "+-- field data/\n",
    "|    +-- microbes.csv\n",
    "+-- thesis/\n",
    "     +-- analysis.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What value(s) should you pass to `read.csv` to read `microbes.csv` in `analysis.ipynb`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**\n",
    "\n",
    "Click on the '...' below to show the solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# We need to specify the path to the file of interest in the call to \n",
    "# `pandas.read_csv`. We first need to `jump` out of the folder `thesis` using\n",
    "# `../` and then into the folder `field_data` using `field_data/`. Then we \n",
    "# can specify the filename `microbes.csv`.\n",
    "#\n",
    "# The result is as follows:\n",
    "\n",
    "data_microbes = pandas.read_csv('../field_data/microbes.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q4: Writing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as the `read_csv` function for reafing data from a  file, Pandas provides a `to_csv` function to write dataframes to files. Applying what you've learned about reading from files, write one of your dataframes to a file calles `processed.csv`. You can use `help` to get information on how to use `to_csv`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution**\n",
    "\n",
    "Click on the '...' below to show the solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# In order to write the DataFrame `americas` to a file called `processed.csv`, \n",
    "# execute the following command:\n",
    "\n",
    "americas.to_csv('processed.csv')\n",
    "\n",
    "# For help on `to_csv`, you could execute, for example,\n",
    "\n",
    "help(americas.to_csv)\n",
    "\n",
    "# Note that `help(to_csv)` throws an error! This is a subtlety and is due to \n",
    "# the fact that `to_csv` is NOT a function in and of itself and the actual \n",
    "# call is `americas.to_csv`. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "UFRC Python-3.8",
   "language": "python",
   "name": "python3-3.8-ufrc"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}