Jupyter/M04-DataManipulation/M04B-EDA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9u1NbbJcpyW1"
   },
   "source": [
    "![Cloud-First](../image/CloudFirst.png) \n",
    "\n",
    "\n",
    "# SIT742: Modern Data Science\n",
    "**(Module: Big Data Manipulation)**\n",
    "\n",
    "---\n",
    "- Materials in this module include resources collected from various open-source online repositories.\n",
    "- You are free to use, change and distribute this package.\n",
    "- If you found any issue/bug for this document, please submit an issue at [tulip-lab/sit742](https://github.com/tulip-lab/sit742/issues)\n",
    "\n",
    "\n",
    "Prepared by **SIT742 Teaching Team**\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "\n",
    "## Session 4B - Exploratory Data Analysis\n",
    "\n",
    "\n",
    "This practical session will show you how to use packages for data exploration.\n",
    "\n",
    "\n",
    "### Content\n",
    "\n",
    "#### Part 1 Matplotlib Module\n",
    "\n",
    "\n",
    "#### Part 2 Plotting a Histogram\n",
    "\n",
    "2.1 [Dataset](#ds)\n",
    "\n",
    "2.2 [Histogram](#hist)\n",
    "\n",
    "2.3 [Boxplot](#boxplot)\n",
    "\n",
    "\n",
    "#### Part 3: Data Understanding\n",
    "\n",
    "1.1 [Pie Chart](#pie)\n",
    "\n",
    "1.2 [Bar Chart](#bar)\n",
    "\n",
    "1.3 [Word Cloud](#wordcloud)\n",
    "\n",
    "1.4 [Step Plot](#stepplot)\n",
    "\n",
    "1.5 [Histogram](#histogram)\n",
    "\n",
    "1.6 [Box Plot](#box)\n",
    "\n",
    "1.7 [Scatter Plot](#scatter)\n",
    "\n",
    "#### Part 4: Exercise\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oU-CnH_apyW-"
   },
   "source": [
    "## <span style=\"color:#0b486b\">1. Matplotlib</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "utN7rWHOpyXC"
   },
   "source": [
    "matplotlib is a python plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. You can generate plots, histograms, power spectra, bar charts, errorcharts, scatterplots, etc, with just a few lines of code. \n",
    "\n",
    "For simple plotting the pyplot interface provides a MATLAB-like interface, particularly when combined with IPython. You have full control of line styles, font properties, axes properties, etc, via an object oriented interface or via a set of functions familiar to MATLAB users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yZaJRz0rpyXF"
   },
   "source": [
    "### <span style=\"color:#0b486b\">1.1 Get started</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l-v2Z-VLpyXI"
   },
   "source": [
    "To get started with `'matplotlib'` you can either execute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VmFefDFjpyXL"
   },
   "outputs": [],
   "source": [
    "from pylab import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qLSjDHzfpyXb"
   },
   "source": [
    "or"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KGOhGQK9pyXe"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "m4Ci8kw7pyXn"
   },
   "source": [
    "In fact it is a convention to import it under the name of `'plt'`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0W-_0YxZpyXq"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tDu5t0rgpyXy"
   },
   "source": [
    "**note: The method, 'import matplotlib.pyplot as plt', is preferred.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LNVf6kikpyX1"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vnlb6elHpyX_"
   },
   "source": [
    "Regardless of the method you use, it is better to configure matplotlib to embed figures in the notebook instead of opening them in a new window for each figure. To do this use the magic function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yVo566kIpyYC"
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-jiecSpnpyYv"
   },
   "source": [
    "### <span style=\"color:#0b486b\">1.2 `plot`</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KavGkG5_pyYz"
   },
   "source": [
    "By using `'subplots()'` you have access to both figure and axes objects. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Bz16q1AbpyZH"
   },
   "outputs": [],
   "source": [
    "#To define the x-Axis and y-Axis\n",
    "x = np.linspace(0, 10)\n",
    "y = np.sin(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YMrNHNxIpyZe"
   },
   "outputs": [],
   "source": [
    "#fig, ax = plt.subplots() is more consice the below code\n",
    "#If you use it,you unpack this tuple into the variables fig and ax\n",
    "#Actually, it equals the below 2 lines code. \n",
    "#fig = plt.figure()\n",
    "#ax = fig.add_subplot(111) #the 111 menas 1x1 grid, first subplot\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.plot(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "prBrY4HapyZy"
   },
   "source": [
    "### <span style=\"color:#0b486b\">1.3 title and labels</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yjZBIAH7pyZ5"
   },
   "outputs": [],
   "source": [
    "#To set the title, xlabel and ylabel \n",
    "ax.set_title('title here!')\n",
    "ax.set_xlabel('x')\n",
    "ax.set_ylabel('sin(x)')\n",
    "fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dfzKo93zpyaL"
   },
   "source": [
    "You can also use [$\\LaTeX$](https://www.latex-project.org/about/) in title or labels, or change the font size or font family."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "X0neGgt_pyaX"
   },
   "outputs": [],
   "source": [
    "#To set the domain of X-Axis from -10 to 10\n",
    "x = np.linspace(-10, 10)\n",
    "\n",
    "#Parameter figsize : (float, float), optional, default: None\n",
    "#width, height in inches. If not provided, defaults to rcParams[\"figure.figsize\"] = [6.4, 4.8].\n",
    "\n",
    "#Parameter dpi : integer, optional, default: None\n",
    "#resolution of the figure. If not provided, defaults to rcParams[\"figure.dpi\"] = 100.\n",
    "fig, ax = plt.subplots(figsize=(6, 4), dpi=100)\n",
    "\n",
    "#x, x**3-x**2 are used to define the horizontal / vertical coordinates of the data points. x values are optional. If not given, they default to [0, ..., N-1].\n",
    "#'bo-' means blue circle with the solid line style. Its name pattern follow the '[color][marker][line]' structure.\n",
    "#linewidth defines the line width\n",
    "#markersize defines the circle size\n",
    "ax.plot(x, x**3-x**2, 'bo-', linewidth=1, markersize=5)\n",
    "\n",
    "\n",
    "#To set title, xlabel, ylabel and their fontsize\n",
    "ax.set_title('$x^3-x^2$', fontsize=18)\n",
    "ax.set_xlabel('$x$', fontsize=18)\n",
    "ax.set_ylabel('$y$', fontsize=18)\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aQDy21Jupyan"
   },
   "source": [
    "### <span style=\"color:#0b486b\">1.4 Subplots</span>\n",
    "\n",
    "You can pass the number of subplots to `'subplots()'`. In this case, `'axes'` will be an array that each of its elements associates with one of the subgraphs. You can set properties of each `'ax'` object separately like the cell below. \n",
    "\n",
    "Obviously you caould use a loop to iterate over `'axes'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jkUu1MVapyar"
   },
   "outputs": [],
   "source": [
    "#nrows, ncols : int, optional, default: 1\n",
    "#They are used to define the number of rows/columns of the subplot grid.\n",
    "#You can try to comment the below code out and run the 'fig, axes = plt.subplots(nrows=1, ncols=2)', then you will find the difference.\n",
    "fig, axes = plt.subplots(nrows=2, ncols=1)\n",
    "\n",
    "#To define the domain of the x-Axis from 0 to 10\n",
    "x = np.linspace(0, 10)\n",
    "\n",
    "#x, np.sin(x) are used to define the horizontal / vertical coordinates of the data points. x values are optional. If not given, they default to [0, ..., N-1]. \n",
    "axes[0].plot(x, np.sin(x))\n",
    "axes[0].set_xlabel('x for sin(x)')\n",
    "axes[0].set_ylabel('sin(x)')\n",
    "\n",
    "#x, np.cos(x) are used to define the horizontal / vertical coordinates of the data points. x values are optional. If not given, they default to [0, ..., N-1]. \n",
    "axes[1].plot(x, np.cos(x))\n",
    "axes[1].set_xlabel('x for cos(x)')\n",
    "axes[1].set_ylabel('cos(x)')\n",
    "\n",
    "axes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LXa44TYxpybB"
   },
   "source": [
    "`'cos(x)'` label is overlapping with the `'sin'` graph. You can adjust the size of the graph or space between the subplots to fix it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_e1atSpkpybK"
   },
   "outputs": [],
   "source": [
    "#nrows, ncols : int, optional, default: 1\n",
    "#They are used to define the number of rows/columns of the subplot grid.\n",
    "#You can try to comment the below code out and run the 'fig, axes = plt.subplots(nrows=1, ncols=2)', then you will find the difference.\n",
    "#You also can use the fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(10,5)) to resize the graph\n",
    "fig, axes = plt.subplots(nrows=2, ncols=1)\n",
    "\n",
    "#To adjust the space between the subpots\n",
    "fig.subplots_adjust(hspace=1)\n",
    "\n",
    "#To define the domain of the x-Axis from 0 to 10\n",
    "x = np.linspace(0, 10)\n",
    "\n",
    "#x, np.sin(x) are used to define the horizontal / vertical coordinates of the data points. x values are optional. If not given, they default to [0, ..., N-1]. \n",
    "axes[0].plot(x, np.sin(x))\n",
    "axes[0].set_xlabel('x for sin(x)')\n",
    "axes[0].set_ylabel('sin(x)')\n",
    "\n",
    "#x, np.cos(x) are used to define the horizontal / vertical coordinates of the data points. x values are optional. If not given, they default to [0, ..., N-1]. \n",
    "axes[1].plot(x, np.cos(x))\n",
    "axes[1].set_xlabel('x for cos(x)')\n",
    "axes[1].set_ylabel('cos(x)')\n",
    "\n",
    "axes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9z7Wb5zlpybh"
   },
   "source": [
    "### <span style=\"color:#0b486b\">1.5 Legend</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Pbjzw44npybl"
   },
   "outputs": [],
   "source": [
    "##To define the domain of the x-Axis from 0 to 10\n",
    "x = np.linspace(0, 10)\n",
    "\n",
    "#To define the figsize\n",
    "fig, ax = plt.subplots(figsize=(7, 5))\n",
    "\n",
    "#To define the x-Axis and y-Axis and label\n",
    "ax.plot(x, np.sin(x), label='$sin(x)$')\n",
    "ax.plot(x, np.cos(x), label='$cos(x)$')\n",
    "\n",
    "#To place the legend for the two plot and define their fontsize and the location of the legend. \n",
    "#Possible codes for the 'loc' are from 0 to 10:\n",
    "ax.legend(fontsize=16, loc=3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RYSeqkgipyb_"
   },
   "source": [
    "### <span style=\"color:#0b486b\">1.6 Customizing ticks</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zAHSBylTpycI"
   },
   "source": [
    "In many cases you want to customize the ticks and their labels on x or y axis. First draw a simple graph and look at the ticks on x-axis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KhUY3CpypycO"
   },
   "outputs": [],
   "source": [
    "x = np.linspace(0, 10, num=100)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 5))\n",
    "ax.plot(x, np.sin(x), x, np.cos(x), linewidth=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XNp5E4xEpycx"
   },
   "source": [
    "You can change the ticks easily with passing a list (or array) to `'set_xticks()'` or `'set_yticks()'`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dhAkpOQEpyc0"
   },
   "outputs": [],
   "source": [
    "xticks = [0, 1, 2, 5, 8, 8.5]\n",
    "ax.set_xticks(xticks)\n",
    "fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0bpSjRm7pyc8"
   },
   "source": [
    "Or even you can change the labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pptHCz7PpydK"
   },
   "outputs": [],
   "source": [
    "xticklabels = ['$\\gamma$', '$\\delta$', 'apple', 'b', '', 'c'] \n",
    "ax.set_xticklabels(xticklabels, fontsize=18)\n",
    "fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "27yPvRUXpydb"
   },
   "source": [
    "### <span style=\"color:#0b486b\">1.7 Saving figures</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RnhUQ6lmpyde"
   },
   "outputs": [],
   "source": [
    "x = np.linspace(0, 10)\n",
    "fig, ax = plt.subplots(figsize=(7, 5))\n",
    "ax.plot(x, np.sin(x), label='$sin(x)$')\n",
    "ax.plot(x, np.cos(x), label='$cos(x)$')\n",
    "ax.legend(fontsize=16, loc=3)\n",
    "fig.savefig('P03Saved.pdf', format='PDF', dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZWrKOZHHpydt"
   },
   "source": [
    "### <span style=\"color:#0b486b\">1.8 Other plot styles</span>\n",
    "\n",
    "There are many other plot types in addition to simple `'plot'` supported by `'matplotlib'`. You will find a complete list of them on [matplotlib gallery](http://matplotlib.org/gallery.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ocLXzN5bpydy"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">1.8.1 Scatter plot</span>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WpHcbHAqpyd1"
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10,5))\n",
    "\n",
    "x = np.linspace(-0.75, 1., 100)\n",
    "\n",
    "\n",
    "#s is the marker size in points**2. Default is rcParams['lines.markersize'] ** 2.\n",
    "#alpha is the alpha blending value, between 0 (transparent) and 1 (opaque).\n",
    "\n",
    "#edgecolor it the edge color of the marker. Possible values:\n",
    "# 'face': The edge color will always be the same as the face color.\n",
    "# 'none': No patch boundary will be drawn.\n",
    "# A matplotib color.\n",
    "ax.scatter(x, np.random.randn(x.shape[0]), \n",
    "                   s = 250*np.abs(np.random.randn(x.shape[0])), \n",
    "                   alpha=0.5,\n",
    "                   facecolor='green',\n",
    "                   edgecolor='face')\n",
    "ax.set_title('scatter')\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TL_K2wTkpyd7"
   },
   "source": [
    "#### <span style=\"color:#0b486b\">1.8.2 Bar plot</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SEpoTfWtpyd-"
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "x = np.arange(1, 6)\n",
    "ax.bar(x, x**2, align=\"center\")\n",
    "ax.set_title('bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5MLBgUudpyeG"
   },
   "source": [
    "---\n",
    "## <span style=\"color:#0b486b\"> 2. Plotting a histogram</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xI0Zho3LpyeK"
   },
   "source": [
    "<a id = \"ds\"></a>\n",
    "\n",
    "\n",
    "### <span style=\"color:#0b486b\">2.1 Dataset</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kcR288dGpyeM"
   },
   "source": [
    "You are provided with a dataset of percentage of body fat and 10 simple body measurements recoreded for 252 men (courtesy of Journal of Statistics Education - JSE). You can read about this and other [JSE datasets here](http://www.amstat.org/publications/jse/jse_data_archive.htm).\n",
    "\n",
    "First load the data set into an array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pDgH5i4lpyeQ"
   },
   "outputs": [],
   "source": [
    "!pip install wget\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3FV2-2okpyeY"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import wget\n",
    "\n",
    "link_to_data = 'https://github.com/tulip-lab/sit742/raw/master/Jupyter/data/fat.dat.txt'\n",
    "DataSet = wget.download(link_to_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uCJbCbkRpyef"
   },
   "outputs": [],
   "source": [
    "data = np.genfromtxt(\"fat.dat.txt\")\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xeWy_uQMpyeo"
   },
   "source": [
    "Based on the [dataset description](http://www.amstat.org/publications/jse/datasets/fat.txt), 5th column represents the weight in lbs. Index the weight column and call it `'weights'`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OouARGANpyeq"
   },
   "outputs": [],
   "source": [
    "weights = data[:, 5]\n",
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hOuIEmIqpye0"
   },
   "source": [
    "Use array operators to convert the weigts into kg. 1 lb equals to 0.453592 kg."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "m1YQPaQJpye-"
   },
   "outputs": [],
   "source": [
    "# weights *= 0.453592 is equivalent to weights = weights * 0.453592\n",
    "#It multiplies right operand with the left operand and assign the result to left operand\n",
    "weights *= 0.453592\n",
    "\n",
    "#Round the converted weights to only two decimals:\n",
    "weights = weights.round(2)\n",
    "\n",
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hCZ0ReSApyfK"
   },
   "source": [
    "<a id = \"hist\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">2.2 Histogram</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aFNh0hE4pyfN"
   },
   "source": [
    "A histogtram is a bar plot that shows you the statistical distribution of the data over a variable. The bars represent the frequency of occurenve by classess of data. We use the package `'matplotlib'` and the function `'hist()'` for plotting the histogram. To learn more about `'matplotlib'` make sure you have read tutorial.\n",
    "\n",
    "The first line of the cell below if for showing the figure in the notebook and not opening it in a separate window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SOxUQjYqpyfP"
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "m4x1jJgqpyfb"
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.hist(weights)\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mXv9V1vspyfk"
   },
   "source": [
    "The `'hist()'` functions automatically group the data over 10 bins. Usually you need to tweek the number of bins to obtain a more expressive histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Qta5Meycpyfn"
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(7, 5))\n",
    "ax.hist(weights, bins=20)\n",
    "\n",
    "# title\n",
    "ax.set_title('Weight Distribution Diagram ')\n",
    "\n",
    "# label\n",
    "ax.set_xlabel('Weight (kg)  ')\n",
    "ax.set_ylabel('The number of people')\n",
    "\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "km8tH1yLpyft"
   },
   "source": [
    "<a id = \"boxplot\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">2.3 Boxplot</span>\n",
    "\n",
    "A `Boxplot` is a convenient way to graphically display numerical data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "duCfVmGOpyfw"
   },
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "\n",
    "fig, ax = matplotlib.pyplot.subplots(figsize=(7, 5))\n",
    "\n",
    "matplotlib.rcParams.update({'font.size': 14})\n",
    "\n",
    "ax.boxplot(weights, 0, labels=['group1'])\n",
    "\n",
    "ax.set_ylabel('weight (kg)', fontsize=16)\n",
    "ax.set_title('Weights BoxPlot', fontsize=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "T4EZsDglpygB"
   },
   "source": [
    "You have already been thought about different sorts of plots, how they help to get a better understanding of the data, and when to use which. In this practical session we will work with `matplotlib` package to learn more about plotting in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DMuQGhispygE"
   },
   "source": [
    "---\n",
    "## <span style=\"color:#0b486b\">3. Data Understanding </span>\n",
    "\n",
    "\n",
    "You have already been thought about different sorts of plots, how they help to get a better understanding of the data, and when to use which. In this practical session we will work with `matplotlib` package to learn more about plotting in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "32Pnk4ikpygK"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import csv\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5dAYd9kGpygg"
   },
   "source": [
    "<a id = \"pie\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.1 Pie Chart</span>\n",
    "\n",
    "Suppose you have the frequency count of a variable (e.g. hair_colour). Draw a pie chart to explain it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "y3fU1ip6pygm"
   },
   "outputs": [],
   "source": [
    "labels = 'Black', 'Red', 'Brown'\n",
    "\n",
    "# frequency count\n",
    "hair_colour_freq = [5, 3, 2]  # Black, Red, Brown\n",
    "\n",
    "# colors\n",
    "colors = ['yellowgreen', 'gold', 'lightskyblue']\n",
    "\n",
    "# explode the third one\n",
    "explode = (0, 0, 0.1)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(5, 5))\n",
    "ax.pie(hair_colour_freq, labels=labels, explode=explode, colors=colors, \n",
    "       autopct='%1.1f', shadow=True, startangle=90);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tkSG3FiYpygy"
   },
   "source": [
    "What if we have too many tags and sectors?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5BmfbTfwpyg1"
   },
   "outputs": [],
   "source": [
    "# Excellence in Reasearch Australia\n",
    "labels = ['HEALTH', 'ENGINEERING', 'COMPUTER SCIENCES', 'HUMAN SOCIETY', \n",
    "          'TOURISM SERVICES', 'EDUCATION', 'CHEMISTRY', 'BIOLOGY', 'PSYCHOLOGY', \n",
    "          'CREATIVE ARTS', 'LINGUISTICS', 'BUILT ENVIRONMENT', 'HISTORY', \n",
    "          'ECONOMICS', 'PHILOSOPHY', 'AGRICULTURE', 'ENVIRONMENT', 'TECHNOLOGY', \n",
    "          'LAW', 'MATHS', 'EARTH SCIENCES', 'PHYSICS']\n",
    "\n",
    "\n",
    "# frequency count\n",
    "xx = [2625.179999, 1306.259999, 1187.039999, 1166.04, 980.8599997, 810.5999998,\n",
    "      725.6399996, 678.7899998, 436.5999997, 404.3299999, 348.01, 304.33, 294.19, \n",
    "      293.02, 282.31, 228.21, 197.3399999, 164.0599998, 157, 50.49999998, 49.60999999, 48.08000005]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 10))\n",
    "ax.pie(xx, labels=labels, autopct=\"%1.1f\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "u8nJOaGJpyg-"
   },
   "source": [
    "<a id = \"bar\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.2 Bar Chart</span>\n",
    "\n",
    "Use the hair colour data to draw a bar chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yrDmcSwhpyhC"
   },
   "outputs": [],
   "source": [
    "labels = ['Black', 'Red', 'Brown']\n",
    "hair_colour_freq = [5, 3, 2]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=100)\n",
    "\n",
    "x_pos = np.arange(len(hair_colour_freq))\n",
    "colors = ['black', 'red', 'brown']\n",
    "\n",
    "ax.bar(x_pos, hair_colour_freq, align='center', color=colors)\n",
    "\n",
    "ax.set_xlabel(\"Hair Colour\")\n",
    "ax.set_ylabel(\"Number of participants\")\n",
    "ax.set_title(\"Hair Colour Distribution\")\n",
    "\n",
    "ax.set_xticks(x_pos)\n",
    "ax.set_xticklabels(labels)\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7lrq8Gj6pyhP"
   },
   "source": [
    "Now suppose we have the hair colour distribution across genders, so we can plot grouped bar charts. Plot a grouped bar chart to show the distribution of colours acros genders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "d5-PYAdRpyhY"
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "        black  red  brown\n",
    "Male      4     1     3\n",
    "Female    1     2     2\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "data = np.array([[4, 1, 3], \n",
    "                 [1, 2, 3]])\n",
    "\n",
    "x_pos = np.arange(2)\n",
    "width = 0.2\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=100)\n",
    "ax.bar(x_pos, data[:, 0], width=width, color='black', label='Black', align='center')\n",
    "ax.bar(x_pos+width, data[:, 1], width=width, color='red', label='Red', align='center')\n",
    "ax.bar(x_pos+2*width, data[:, 2], width=width, color='brown', label='Brown', align='center')\n",
    "\n",
    "ax.legend()\n",
    "\n",
    "ax.set_xlabel(\"Gender\")\n",
    "ax.set_ylabel(\"Frequency\")\n",
    "ax.set_title(\"Distribution of hair colour amongst genders\")\n",
    "\n",
    "ax.set_xticks(x_pos+width)\n",
    "ax.set_xticklabels(['Male', 'Female'])\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FNWfodMPpyhi"
   },
   "source": [
    "Can we plot it more intelligently? We are doing the same thing multiple times! Is it a good idea to use a loop?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "w5g41dPgpyhl"
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "        black  red  brown\n",
    "Male      4     1     3\n",
    "Female    1     2     2\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "data = np.array([[4, 1, 3], \n",
    "                 [1, 2, 3]])\n",
    "\n",
    "n_groups, n_colours = data.shape\n",
    "\n",
    "x_pos = np.arange(n_groups)\n",
    "width = 0.2\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=100)\n",
    "\n",
    "colours = ['black', 'red', 'brown']\n",
    "labels = ['Black', 'Red', 'Brown']\n",
    "for i in range(n_colours):\n",
    "    ax.bar(x_pos + i*width, data[:, i], width=width, color=colours[i], label=labels[i], align='center')\n",
    "    \n",
    "ax.legend()\n",
    "\n",
    "ax.set_xlabel(\"Gender\")\n",
    "ax.set_ylabel(\"Frequency\")\n",
    "ax.set_title(\"Distribution of hair colour amongst genders\")\n",
    "\n",
    "ax.set_xticks(x_pos+width)\n",
    "ax.set_xticklabels(['Male', 'Female'])\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4gO6fuM3pyhr"
   },
   "source": [
    "What if we want to group the bar charts based on the hair colour?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NH0Xk0O0pyht"
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "        black  red  brown\n",
    "Male      4     1     3\n",
    "Female    1     2     2\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "labels1 = ['Black', 'Red', 'Brown']\n",
    "labels2 = ['Male','Female']\n",
    "colours = ['r', 'y']\n",
    "data = np.array([[4, 1, 3], \n",
    "                 [1, 2, 3]])\n",
    "\n",
    "n_groups, n_colours = data.shape\n",
    "width = 0.2\n",
    "x_pos = np.arange(n_colours)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=100)\n",
    "for i in range(n_groups):\n",
    "    ax.bar(x_pos + i*width, data[i, :], width, align='center', label=labels2[i], color=colours[i])  #出错在这里\n",
    "ax.set_xlabel(\"Hair Colour\")\n",
    "ax.set_ylabel(\"Frequency\")\n",
    "ax.set_title(\"Distribution of gender amongst hair colours\")\n",
    "\n",
    "ax.set_xticks(x_pos+width/2)\n",
    "ax.set_xticklabels(labels1)\n",
    "\n",
    "ax.legend()\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "O-qBYciCpyh6"
   },
   "source": [
    "#### Stacked bar chart\n",
    "\n",
    "The other type of bar chart is stacked bar chart. draw a stacked bar plot of the hair colour data grouped on hair colours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tUZpO1xdpyh-"
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "        black  red  brown\n",
    "Male      4     1     3\n",
    "Female    1     2     2\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "labels = ['Black', 'Red', 'Brown']\n",
    "data = np.array([[4, 1, 3], \n",
    "                 [1, 2, 3]])\n",
    "\n",
    "male_freq = data[0,:]\n",
    "\n",
    "width = 0.4\n",
    "x_pos = np.arange(n_colours)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=100)\n",
    "ax.bar(x_pos, data[0, :], width, align='center', label='Male', color='r')\n",
    "ax.bar(x_pos, data[1, :], width, bottom=male_freq, align='center', label='Female', color='y')\n",
    "\n",
    "ax.set_xlabel(\"Hair Colour\")\n",
    "ax.set_ylabel(\"Frequency\")\n",
    "ax.set_title(\"Distribution of gender amongst hair colours\")\n",
    "\n",
    "ax.set_xticks(x_pos)\n",
    "ax.set_xticklabels(labels)\n",
    "\n",
    "ax.legend(loc=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z_l3qULSpyiE"
   },
   "source": [
    "draw a stacked bar plot grouped on the gender."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yoTEehVIpyiG"
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "        black  red  brown\n",
    "Male      4     1     3\n",
    "Female    1     2     2\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "labels = ['Black', 'Red', 'Brown']\n",
    "data = np.array([[4, 1, 3], \n",
    "                 [1, 2, 3]])\n",
    "\n",
    "black = data[:,0]\n",
    "red = data[:,1]\n",
    "brown = data[:,2]\n",
    "\n",
    "\n",
    "x_pos = np.arange(2)\n",
    "width = 0.4\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=100)\n",
    "ax.bar(x_pos, data[:, 0], width=width, color='black', label='Black', align='center')\n",
    "ax.bar(x_pos, data[:, 1], width=width, bottom=black, color='red', label='Red', align='center')\n",
    "ax.bar(x_pos, data[:, 2], width=width, color='brown', bottom=black+red, label='Brown', align='center')\n",
    "\n",
    "ax.legend(loc=0)\n",
    "\n",
    "ax.set_xlabel(\"Gender\")\n",
    "ax.set_ylabel(\"Frequency\")\n",
    "ax.set_title(\"Distribution of hair colour amongst genders\")\n",
    "\n",
    "ax.set_xticks(x_pos)\n",
    "ax.set_xticklabels(['Male', 'Female'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6qZP7nu3pyiO"
   },
   "outputs": [],
   "source": [
    "labels = ['Black', 'Red', 'Brown']\n",
    "male_freq = [4, 1, 3]\n",
    "female_freq = [1, 2, 2]\n",
    "\n",
    "x_pos = np.arange(3)\n",
    "\n",
    "fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(14, 5), dpi=100)\n",
    "\n",
    "w1 = 0.2\n",
    "w2 = 0.4\n",
    "\n",
    "ax[0].bar(x_pos, male_freq, width=w1, align='center', label='Male', color='r')\n",
    "ax[0].bar(x_pos+width, female_freq, width=w1, align='center', label='Female', color='y')\n",
    "ax[1].bar(x_pos, male_freq, width=w2, align='center', label='Male', color='r')\n",
    "ax[1].bar(x_pos, female_freq, width=w2, bottom=male_freq, align='center', label='Female', color='y')\n",
    "\n",
    "\n",
    "ax[0].set_xlabel(\"Hair Colour\")\n",
    "ax[0].set_ylabel(\"Frequency\")\n",
    "ax[0].set_title(\"Distribution of gender amongst hair colours\")\n",
    "ax[1].set_xlabel(\"Hair Colour\")\n",
    "ax[1].set_ylabel(\"Frequency\")\n",
    "ax[1].set_title(\"Distribution of gender amongst hair colours\")\n",
    "\n",
    "ax[0].set_xticks(x_pos+width/2)\n",
    "ax[0].set_xticklabels(labels)\n",
    "ax[1].set_xticks(x_pos)\n",
    "ax[1].set_xticklabels(labels)\n",
    "\n",
    "ax[0].legend()\n",
    "ax[1].legend(loc=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5BdqASAypyiX"
   },
   "source": [
    "What if we have too many groups? Draw a bar chart for the Excellence in Research Australia data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xt-_ukeypyic"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "# Excellence in Research Australia\n",
    "labels = ['HEALTH', 'ENGINEERING', 'COMPUTER SCIENCES', 'HUMAN SOCIETY', \n",
    "          'TOURISM SERVICES', 'EDUCATION', 'CHEMISTRY', 'BIOLOGY', 'PSYCHOLOGY', \n",
    "          'CREATIVE ARTS', 'LINGUISTICS', 'BUILT ENVIRONMENT', 'HISTORY', \n",
    "          'ECONOMICS', 'PHILOSOPHY', 'AGRICULTURE', 'ENVIRONMENT', 'TECHNOLOGY', \n",
    "          'LAW', 'MATHS', 'EARTH SCIENCES', 'PHYSICS']\n",
    "\n",
    "\n",
    "# frequency count\n",
    "xx = [2625.179999, 1306.259999, 1187.039999, 1166.04, 980.8599997, 810.5999998,\n",
    "      725.6399996, 678.7899998, 436.5999997, 404.3299999, 348.01, 304.33, 294.19, \n",
    "      293.02, 282.31, 228.21, 197.3399999, 164.0599998, 157, 50.49999998, 49.60999999, 48.08000005]\n",
    "\n",
    "xx_pos = np.arange(len(xx))\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15, 5))\n",
    "ax.bar(xx_pos, xx, align='center')\n",
    "ax.set_xlabel(\"research subject\")\n",
    "ax.set_ylabel(\"score\")\n",
    "ax.set_xticks(xx_pos)\n",
    "ax.set_xticklabels(labels, rotation=90)\n",
    "ax.set_xlim(-1, len(xx))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also refer to one online [post](http://queirozf.com/entries/add-labels-and-text-to-matplotlib-plots-annotation-examples) for different customization of those plots."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ecuMR8BspyjX"
   },
   "source": [
    "<a id = \"wordcloud\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.3 Wordcloud</span>\n",
    "\n",
    "As you saw, pie-chart is not very helpful when we have too many sectors. It is hard to read and visually ugly. Instead we can use wordcloud representation. A useful tool is [wordle.net](http://wordle.net). Go to [wordle.net](http://wordle.net) and use it to create a wordcloud for the previous data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lTuM-Bj2pyjd"
   },
   "outputs": [],
   "source": [
    "for i in range(len(labels)):\n",
    "    print(\"{}:{}\".format(labels[i], xx[i]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "get2r23bpyjm"
   },
   "outputs": [],
   "source": [
    "!pip install wordcloud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yXHCY7Snpyjr"
   },
   "outputs": [],
   "source": [
    "import wget\n",
    "\n",
    "link_to_data = 'https://github.com/tulip-lab/sit742/raw/master/Jupyter/data/constitution.txt'\n",
    "DataSet = wget.download(link_to_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RAM3U6cApyj0"
   },
   "outputs": [],
   "source": [
    "from os import path\n",
    "from wordcloud import WordCloud\n",
    "\n",
    "# Read the whole text.\n",
    "text = open('constitution.txt').read()\n",
    "\n",
    "# Generate a word cloud image\n",
    "wordcloud = WordCloud().generate(text)\n",
    "\n",
    "# Display the generated image:\n",
    "# the matplotlib way:\n",
    "import matplotlib.pyplot as plt\n",
    "plt.imshow(wordcloud, interpolation='bilinear')\n",
    "plt.axis(\"off\")\n",
    "plt.show()\n",
    "\n",
    "# lower max_font_size\n",
    "wordcloud = WordCloud(max_font_size=30).generate(text)\n",
    "plt.figure()\n",
    "plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "plt.axis(\"off\")\n",
    "plt.show()\n",
    "\n",
    "# The PIL way (if you don't have matplotlib)\n",
    "#image = wordcloud.to_image()\n",
    "#image.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ihjOiuc4pyj6"
   },
   "source": [
    "<a id = \"stepplot\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.4 Step plot</span>\n",
    "\n",
    "Draw a step plot for the seatbelt data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wyn8gQ_bpyj7"
   },
   "outputs": [],
   "source": [
    "freq = np.array([0, 2, 1, 5, 7])\n",
    "labels = ['Never', 'Rarely', 'Sometimes', 'Most-times', 'Always']\n",
    "freq_cumsum = np.cumsum(freq)\n",
    "x_pos = np.arange(len(freq))\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.step(x_pos, freq_cumsum, where='mid')\n",
    "ax.set_xlabel(\"Fastening seatbelt behaviour\")\n",
    "ax.set_ylabel(\"Cumulative frequency\")\n",
    "ax.set_xticks(x_pos)\n",
    "ax.set_xticklabels(labels)\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aw65PxMEpykC"
   },
   "source": [
    "<a id = \"histogram\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.5 Histogram</span>\n",
    "\n",
    "Google for this paper:\n",
    "\n",
    "``Johnson, Roger W. \"Fitting percentage of body fat to simple body measurements.\" Journal of Statistics Education 4.1 (1996): 265-266.``\n",
    "\n",
    "Download the dataset and read the dataset description. Draw a histogram of male weights and female weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6DtjpNt0pykE"
   },
   "outputs": [],
   "source": [
    "import wget\n",
    "\n",
    "link_to_data = 'https://github.com/tulip-lab/sit742/raw/master/Jupyter/data/body.dat.txt'\n",
    "DataSet = wget.download(link_to_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XqwzvUYjpyki"
   },
   "outputs": [],
   "source": [
    "data = np.genfromtxt('body.dat.txt')\n",
    "m_w = data[data[:, -1] == 1][:, -3]\n",
    "f_w = data[data[:, -1] == 0][:, -3]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=100)\n",
    "ax.hist(m_w, bins=15, alpha=0.6, label='male')\n",
    "ax.hist(f_w, bins=15, alpha=0.6, label='female')\n",
    "ax.set_xlabel(\"weight (kg)\")\n",
    "ax.set_title(\"Weight Distribution amongst gXenders\")\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9mzL5EqZpykt"
   },
   "source": [
    "<a id = \"box\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.6 Boxplot</span>\n",
    "Draw a box plot for male and female weights of the previous dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tdb1qUC8pykv"
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=100)\n",
    "\n",
    "#To define the style of the fliers.\n",
    "#red_square = dict(markerfacecolor='r', marker='s') is another example, you can try define you own style.\n",
    "green_diamond = dict(markerfacecolor='g', marker='D')\n",
    "\n",
    "#Set the value of showfliers with True to show the outliers beyond the caps.\n",
    "ax.boxplot([m_w, f_w], labels=['male', 'female'],showfliers=True, flierprops=green_diamond)\n",
    "\n",
    "ax.set_title(\"weight distribution amongst genders\")\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xdjEAYeOpyk7"
   },
   "source": [
    "<a id = \"scatter\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.7 Scatter plot</span>\n",
    "\n",
    "Draw a scatter plot of the car weights and their fuel consumption as displayed in the lecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GWEfJmRxpyk9"
   },
   "outputs": [],
   "source": [
    "import wget\n",
    "\n",
    "link_to_data = 'https://github.com/tulip-lab/sit742/raw/master/Jupyter/data/Auto.csv'\n",
    "DataSet = wget.download(link_to_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dfB9QgM2pylD"
   },
   "outputs": [],
   "source": [
    "datafile = 'Auto.csv'\n",
    "data = np.genfromtxt(datafile, delimiter=',')\n",
    "data = []\n",
    "with open(datafile, 'r') as fp:\n",
    "    reader = csv.reader(fp, delimiter=',')\n",
    "    for row in reader:\n",
    "        data.append(row)\n",
    "miles = [dd[1] for dd in data[1:]]\n",
    "weights = [dd[5] for dd in data[1:]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3QVAHhZ2pylK"
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15, 5), dpi=100)\n",
    "ax.scatter(weights,miles, alpha=0.6, edgecolor='none', s=100)\n",
    "ax.set_xlabel('Car Weight (tons)')\n",
    "ax.set_ylabel('Miles Per Gallon')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "m5FLFWk2pylP"
   },
   "source": [
    "Can I also show the number of cylinders on this graph? In other words use the scatter plot to show three variable?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZY5r0HA1pylR"
   },
   "outputs": [],
   "source": [
    "cylinder = 75 * np.array([int(dd[2]) for dd in data[1:]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jOFszTknpylY"
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15, 5), dpi=100)\n",
    "ax.scatter(weights,miles, alpha=0.6, edgecolor='none', s=cylinder)\n",
    "ax.set_xlabel('Car Weight (tons)')\n",
    "ax.set_ylabel('Miles Per Gallon')\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZmpBymmMpyld"
   },
   "source": [
    "---\n",
    "## 4. Mini Exercise\n",
    "\n",
    "\n",
    "In 1970, US Congress instituted a random selection process for the military draft. All 366 possible birth dates were placed in plastic capsules in a rotating drum and were selected one by one. The first date drawn from the drum received draft number one and eligible men born on that date were drafted first. The data is provided in a text file with a structure like:\n",
    "\n",
    "```\n",
    "Day    Month    MO.NUMBER    DAY_OF_YEAR    DRAFT_NO.\n",
    "1      JAN      1            1              305\n",
    "2      JAN      1            2              159\n",
    ".\n",
    "31     JAN      1            31             221\n",
    "1      FEB      2            32             86\n",
    ".\n",
    "31     Dec      12           366            100\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q-I109Mdpylg"
   },
   "source": [
    "Using what you have learnt by now, can you tell if it was a fair lottary or not?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cEVOF9fSpylk"
   },
   "source": [
    "Read the data file and save the values in a 2D array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TH5263_Vpyln"
   },
   "outputs": [],
   "source": [
    "import wget\n",
    "\n",
    "link_to_data = 'https://github.com/tulip-lab/sit742/raw/master/Jupyter/data/DraftLottery.txt'\n",
    "DataSet = wget.download(link_to_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qGS4EaR6pylt"
   },
   "outputs": [],
   "source": [
    "data = []\n",
    "with open('DraftLottery.txt', 'r') as fp:\n",
    "    reader = csv.reader(fp, delimiter='\\t')\n",
    "    for row in reader:\n",
    "        data.append(row)\n",
    "        \n",
    "birthdays = np.array([int(row[3]) for row in data[1:]])\n",
    "draft_no = np.array([int(row[4]) for row in data[1:]])\n",
    "months = np.array([int(row[2]) for row in data[1:]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pmJi6HFppyly"
   },
   "source": [
    "Plot a `'scatter plot'` of the draft priority vs birthdays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KiYy_t9vpyl1"
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 7), dpi=100)\n",
    "ax.scatter(birthdays, draft_no, alpha=0.7, s = 100, edgecolor='none')\n",
    "ax.set_xlabel(\"Birthday (day of the year)\", fontsize=12)\n",
    "ax.set_ylabel(\"Draft priority value\", fontsize=12)\n",
    "ax.set_title(\"USA Draft Lottery Data\", fontsize=14)\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "p24hHUDopymC"
   },
   "source": [
    "In a truly random lottery there should be no relationship between the date and the draft number. To investigate this further we draw boxplots by months and compare them together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HmVT3Sz8pymG"
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 7), dpi=100)\n",
    "months_range = range(1, 13)\n",
    "\n",
    "# boxplot data\n",
    "boxplot_data = [draft_no[months == mm] for mm in months_range]\n",
    "ax.boxplot(boxplot_data)\n",
    "\n",
    "# medians\n",
    "medians = [np.median(dd) for dd in boxplot_data]\n",
    "ax.plot(months_range, medians, \"g--\", lw=2)\n",
    "\n",
    "# means\n",
    "means = [dd.mean() for dd in boxplot_data]\n",
    "ax.plot(months_range, means, \"k--\", lw=2)\n",
    "\n",
    "month_labels = [\"Jan\", \"Feb\", \"Mar\", \"Apr\", \"May\", \"Jun\",\n",
    "                \"Jul\", \"Aug\", \"Sep\", \"Oct\", \"Nov\", \"Dec\"]\n",
    "ax.set_xlabel(\"Month\", fontsize=12)\n",
    "ax.set_xticklabels(month_labels)\n",
    "ax.set_ylabel(\"Draft priority value\", fontsize=12)\n",
    "ax.set_title(\"USA Draft Lottery Data\", fontsize=14)\n",
    "ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JcPjlN1dpymW"
   },
   "source": [
    "While it is impossible to view this trend in a scatterplot of draft number vs. birth date, a series of side-by-side boxplots by month illustrate it clearly. A further investigation of the lottery revealed that the birthdates were placed in the drum by month and were not thoroughly mixed."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "M04B-EDA.ipynb",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}