Headline Cleaning Part 1 - Solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Headline Cleaning Part 1\n",
    "## Solution\n",
    "\n",
    "Our goal in this phase is to read in one of the headline files, produce a list of the dates, turn them into dates that look like `YYYY-MM-DD`, and get a count of words in the file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Note, you have to change this to represent the path on your machine. Or make it \n",
    "# empty if the file you're reading in is in the same folder. \n",
    "working_dir = \"C:\\\\Users\\\\jchan\\\\Dropbox\\\\Teaching\\\\AppliedDataAnalytics\\\\Code\\\\headline-cleaning\\\\\"\n",
    "# working_dir = \"\"\n",
    "\n",
    "input_file = \"missoula.txt\" # Let's work with Missoula"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's open up one of these files and see what we have. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with open(working_dir + input_file,'r',encoding=\"Latin-1\") as infile :\n",
    "    for idx,row in enumerate(infile.readlines()) :\n",
    "        print(row)\n",
    "        if idx == 1 :\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, this is a mess. The first row has all the dates, the subsequent rows have headlines with many blanks put in. Let's start by getting all the dates, which are on the first row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with open(working_dir + input_file) as infile :\n",
    "    dates = infile.readline().strip().split(\"\\t\")\n",
    "\n",
    "dates[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An irksome formulation of dates. There are fancy ways to solve this, but let's use a simple way. I'm going to just make a lookup dictionary to map the month to the month number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "month_abbr = [\"Jan\",\"Feb\",\"Mar\",\"Apr\",\"May\",\"Jun\",\"Jul\",\"Aug\",\"Sep\",\"Oct\",\"Nov\",\"Dec\"]\n",
    "month_num = [n+1 for n in range(12)]\n",
    "month_str = [\"{:02d}\".format(n) for n in month_num]\n",
    "\n",
    "month_lu = dict(zip(month_abbr,month_str))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "clean_dates = []\n",
    "for date in dates :\n",
    "    d,m = date.split(\"-\")\n",
    "    m_num = month_lu[m]\n",
    "    d_str = \"{:02d}\".format(int(d))\n",
    "    \n",
    "    if ((m == \"Sep\" and int(d) > 22) or\n",
    "        m in [\"Oct\",\"Nov\",\"Dec\"]) :\n",
    "        y = \"2015\"\n",
    "    else :\n",
    "        y = \"2016\" \n",
    "\n",
    "    clean_dates.append(\"-\".join([y,m_num,d_str]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I could test this with something like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dict(zip(dates,clean_dates))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following goes above and beyond, but it'll be useful for what follows. Let's make a function that takes as input one of these dates and returns the correctly formatted date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def reformat_missoula_date(ugly_date) :\n",
    "    '''\n",
    "        Takes as input a date of the form \"D-MMM\" and returns a date of the form\n",
    "        \"YYYY-MM-DD\". Note that we have to do some work on years. Dates in Oct, Nov,\n",
    "        Dec are all 2015. Also, September dates _after_ 22-Sep are 2015. Note that \n",
    "        this cutoff only works for Missoula--we'd have to have come up with other \n",
    "        cutoffs for other papers. \n",
    "    '''\n",
    "    \n",
    "    month_abbr = [\"Jan\",\"Feb\",\"Mar\",\"Apr\",\"May\",\"Jun\",\"Jul\",\"Aug\",\"Sep\",\"Oct\",\"Nov\",\"Dec\"]\n",
    "    month_num = [n+1 for n in range(12)]\n",
    "    month_str = [\"{:02d}\".format(n) for n in month_num]\n",
    "    month_lu = dict(zip(month_abbr,month_str))\n",
    "    \n",
    "    d,m = ugly_date.split(\"-\")\n",
    "    m_num = month_lu[m]\n",
    "    d_str = \"{:02d}\".format(int(d))\n",
    "    \n",
    "    if ((m == \"Sep\" and int(d) > 22) or\n",
    "        m in [\"Oct\",\"Nov\",\"Dec\"]) :\n",
    "        y = \"2015\"\n",
    "    else :\n",
    "        y = \"2016\" \n",
    "\n",
    "    pretty_date = \"-\".join([y,m_num,d_str])\n",
    "    \n",
    "    return(pretty_date)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Let's test it\n",
    "assert(reformat_missoula_date(\"21-Sep\")==\"2016-09-21\")\n",
    "assert(reformat_missoula_date(\"23-Sep\")==\"2015-09-23\")\n",
    "assert(reformat_missoula_date(\"8-Jun\")==\"2016-06-08\")\n",
    "assert(reformat_missoula_date(\"30-Mar\")==\"2016-03-30\")\n",
    "\n",
    "# This one fails, just so you can see it.\n",
    "assert(reformat_missoula_date(\"27-Sep\")==\"2016-09-27\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'd like to count all the words in file. I'm going to do this three ways. The first one will use really basic techniques; the second will speed things up a bit with `defaultdict`; the third will use the `Counter` data type which is meant for this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dict_counter = {}\n",
    "\n",
    "with open(working_dir + input_file) as infile :\n",
    "    next(infile) # skip the first row--the dates\n",
    "\n",
    "    for line in infile.readlines() :\n",
    "        split_line = line.strip().split()\n",
    "        \n",
    "        for word in split_line :\n",
    "            if word not in dict_counter :\n",
    "                dict_counter[word] = 1\n",
    "            else :\n",
    "                dict_counter[word] += 1\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Let's do a little testing of this one\n",
    "print(dict_counter[\"the\"])\n",
    "print(dict_counter[\"The\"])\n",
    "print(dict_counter[\"Griz\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Now with default dict\n",
    "from collections import defaultdict\n",
    "\n",
    "ddict_counter = defaultdict(int)\n",
    "\n",
    "with open(working_dir + input_file) as infile :\n",
    "    next(infile) # skip the first row--the dates\n",
    "\n",
    "    for line in infile.readlines() :\n",
    "        split_line = line.strip().split()\n",
    "        \n",
    "        for word in split_line :\n",
    "            ddict_counter[word] += 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Let's do a little testing of this one\n",
    "print(ddict_counter[\"the\"])\n",
    "print(ddict_counter[\"The\"])\n",
    "print(ddict_counter[\"Griz\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Now let's use counter. Easiest way is to make a list of all the words.\n",
    "from collections import Counter\n",
    "all_words = []\n",
    "\n",
    "with open(working_dir + input_file) as infile :\n",
    "    next(infile) # skip the first row--the dates\n",
    "\n",
    "    for line in infile.readlines() :\n",
    "        split_line = line.strip().split()\n",
    "        all_words.extend(split_line)\n",
    "\n",
    "word_cnt = Counter(all_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "word_cnt.most_common(10)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}