F0C-05-CS-EDA2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FLIP (00): Data Science \n",
    "**(Module 02: Data Visualization)**\n",
    "\n",
    "---\n",
    "- Materials in this module include resources collected from various open-source online repositories.\n",
    "- You are free to use,but NOT allowed to change and distribute this package.\n",
    "\n",
    "Prepared by and for \n",
    "**Student Members** |\n",
    "2006-2023 [TULIP Lab](http://www.tulip.org.au), Australia\n",
    "\n",
    "---\n",
    "\n",
    "# Session F - Case Study  (Exploratory Data Analysis 2)\n",
    " \n",
    "In this lesson,we will learn EDA with real world data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import json\n",
    "import operator\n",
    "\n",
    "from collections import namedtuple, Counter, OrderedDict\n",
    "\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from IPython.display import set_matplotlib_formats\n",
    "\n",
    "set_matplotlib_formats('retina')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import wget\n",
    "\n",
    "link_to_data = 'https://github.com/tuliplab/mds/blob/master/Jupyter/data/data_first%20step/sfpd.json?raw=true'\n",
    "DataSet = wget.download(link_to_data)\n",
    "#下载失败  手动将sfep加入文件夹"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!ls ./"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('sfpd.json') as fin:\n",
    "    data = json.load(fin)\n",
    "columns = [obj['name'] for obj in data['meta']['view']['columns']]\n",
    "Incident = namedtuple('Incident', columns)\n",
    "incidents = [Incident(*row) for row in data['data']]    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def dhist(d):\n",
    "    \"\"\"Plots a dict object in a bar graph to match the histogram layout.\n",
    "    \"\"\"\n",
    "    # For a bar graph, the first argument is the values at which the \n",
    "    # bars will be place. We set the width to 1 (default is 0.8) to \n",
    "    # remove any whitespace between bars and make it look more like the \n",
    "    # histogram. We set the align to 'edge' (default is 'center') to make\n",
    "    # sure that the left edge of the bar is aligned with its value.\n",
    "    plt.bar(range(len(d)), d.values(), width=1, align='edge')\n",
    "    # Since the bars' left edges are aligned with their values, we have to \n",
    "    # move the x-tick labels in a bit to make them centered within the bar.\n",
    "    # Since the bars are all a width of 1, we simply need to add 0.5 to the\n",
    "    # left values of every tick. Finally, we rotate the labels 90 degrees \n",
    "    # so they are readable.\n",
    "    plt.xticks([i + 0.5 for i in range(len(d))], d.keys(), rotation=90)\n",
    "    # Making the graph as tight as possible by limiting the X-axis values.\n",
    "    plt.xlim([0, len(d)]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where do Crimes Happen?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist = Counter(i.PdDistrict for i in incidents)\n",
    "hist = OrderedDict(sorted(hist.items(), key=operator.itemgetter(0)))\n",
    "dhist(hist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set(i.Category for i in incidents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "violent_crimes = ['ARSON', 'ASSAULT','KIDNAPPING', 'SEX OFFENSES, FORCIBLE']\n",
    "assaults = [i for i in incidents if i.Category in violent_crimes]\n",
    "hist = Counter(i.PdDistrict for i in assaults)\n",
    "hist = OrderedDict(sorted(hist.items(), key=operator.itemgetter(0)))\n",
    "dhist(hist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What Types of Crime?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "hist = Counter(i.Category for i in incidents)\n",
    "hist = OrderedDict(sorted(hist.items(), key=operator.itemgetter(1)))\n",
    "plt.figure(figsize=(20, 8))\n",
    "dhist(hist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## When do Crimes Occur?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at crime on each day of the week. It looks like crime overall is roughly the same on a daily basis with a slight bump on Friday and Saturday (which you would imagine to be true). But, what happens if we look at just violent crime? Now we see a little more \"flavor\" in the distribution of crime. It seems to be at its lowest on Wednesday, and it slowly builds up to it peak on Sunday, and then comes down again by Wednesday. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "violent_crimes = ['ARSON', 'ASSAULT','KIDNAPPING', 'SEX OFFENSES, FORCIBLE']\n",
    "hist = Counter(int(i.Time[:2]) for i in incidents if i.Category in violent_crimes)\n",
    "morning = range(6, 12)\n",
    "daytime =  [12] +  range(1, 13)\n",
    "late_night = range(1, 6)\n",
    "keys = ['%s AM' % h for h in morning] + \\\n",
    "       ['%s PM' % h for h in daytime] + \\\n",
    "       ['%s AM' % h for h in late_night]\n",
    "values = [ hist[k] for k in range(6, 24) + range(0, 6)]\n",
    "hist = OrderedDict(zip(keys, values))\n",
    "plt.figure(figsize=(12, 6))\n",
    "dhist(hist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the number of total crimes per day and the number of violent crimes per day side by side. \n",
    "\n",
    "If you notice, there is an order of magnitude difference between the two which is normally not a good thing to have when making a comparison, but since we're only interested in the shape of the distribution, that shouldn't be a problem for us. \n",
    "\n",
    "The other change we made in this example, is we adjusted the range of values that we display on the y-axes. The reason for this again is that we are just interested in the shape of the distribution so we can see what days have the most crime. Overall, there isn't a huge difference in crime per day, and that is kind of disguised here by the adjustment we made in the axes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keys = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']\n",
    "\n",
    "fig = plt.figure(figsize=(8, 2))\n",
    "\n",
    "fig.add_subplot(1, 2, 1)\n",
    "# Adjust the y limits for the current axes.\n",
    "plt.ylim([14000, 20000])\n",
    "hist = Counter(i.DayOfWeek for i in incidents)\n",
    "values = [hist[k] for k in keys]\n",
    "dhist(OrderedDict(zip(keys, values)))\n",
    "\n",
    "fig.add_subplot(1, 2, 2)\n",
    "plt.ylim([1400, 2000])\n",
    "hist = Counter(i.DayOfWeek for i in incidents if i.Category in violent_crimes)\n",
    "values = [hist[k] for k in keys]\n",
    "dhist(OrderedDict(zip(keys, values)))\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}