PythonIntro_Dennis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A gentle introduction to Python, File IO, Parallel Programming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Python?\n",
    "\n",
    "![Image of Python](https://www.andreabacciu.com/upload/2015/02/Python-Logo-PNG-Image.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python is a high level programming language created by Guido van Rossum in 1991. \n",
    "\n",
    "It is a popular language because it is easy to read, write, and understand. Python abstracts a lot of underlying computational details allowing the programmer to focus on the overall idea of what the program should do.\n",
    "\n",
    "Python is used extensively in data analytics and machine learning due to the the massive number of libraries supporting these endeavors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lesson 1 - Basic Programming & Python Syntax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I did tell you that Python is easy to read right? Lets put that to the test - what do you think this line of code does? \n",
    "\n",
    "(Select the code snippet by clicking on it with your mouse and then hit \"Run\" on the menu above.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello World!\n"
     ]
    }
   ],
   "source": [
    "print(\"Hello World!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 2 things happening here:\n",
    "\n",
    "    1) A function called \"print\" is called.\n",
    "    2) The function is supplied with input data, that is, the text in quotations: \"Hello World!\"\n",
    "    \n",
    "As a result, Hello World! is printed on to the screen. Go figure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables\n",
    "\n",
    "In the previous coding example, you saw that a \"function\" could be supplied with data to produce a result. This \"data\" is more commonly refered to as \"variables\" in programming.\n",
    "\n",
    "The code snippet above could easily be re-written as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello World!\n"
     ]
    }
   ],
   "source": [
    "greeting = \"Hello World!\"\n",
    "\n",
    "print(greeting)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what is the difference? In this scenario, we store the text to be printed as a variable called \"message\". This is then put in to the function, print.\n",
    "\n",
    "We can manipulate that piece of data now that we have stored it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello World. I am alive!\n"
     ]
    }
   ],
   "source": [
    "greeting = \"Hello World.\"\n",
    "\n",
    "message = greeting + \" I am alive!\"\n",
    "\n",
    "print(message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functions\n",
    "\n",
    "*Skippable!*\n",
    "\n",
    "print() is a built in function provided by default in Python. \n",
    "\n",
    "In programming, we often find ourselves doing repetitive tasks. Functions can store the steps of these tasks, so that they can be executed via a single line of code. \n",
    "\n",
    "As a result, we have the ability write our own functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello World. This is a function call.\n"
     ]
    }
   ],
   "source": [
    "def greeting(message):\n",
    "    greeting = \"Hello World.\"\n",
    "    message = greeting + \" \" + message\n",
    "    print(message)\n",
    "\n",
    "message = \"This is a function call.\"\n",
    "greeting(message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To declare a function, simply follow this format:\n",
    "\n",
    "    def function_name(variable1, variable2):\n",
    "\n",
    "Lines of code belonging to a function should be indented 1 tab character. For example:\n",
    "\n",
    "    def function_name(variable1, variable2):\n",
    "        this_is_apart_of_the_function = True\n",
    "\n",
    "    this_is_not_apart_of_the_function = False\n",
    "\n",
    "A function can even return a piece of data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello World. This is a function call.\n"
     ]
    }
   ],
   "source": [
    "def make_greeting(message):\n",
    "    greeting = \"Hello World.\"\n",
    "    message = greeting + \" \" + message\n",
    "    return message\n",
    "\n",
    "message = \"This is a function call.\"\n",
    "greeting = make_greeting(message)\n",
    "\n",
    "print(greeting)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Types\n",
    "\n",
    "Variables can be of different data types. \n",
    "\n",
    "In the previous coding examples, I have shown you the String datatype:\n",
    "    \n",
    "    String = \"This is a string\"\n",
    "    \n",
    "And the Boolean data type:\n",
    "    \n",
    "    is_it_raining = False\n",
    "    are_we_hacking = True\n",
    "\n",
    "There are many others to store different types of data:\n",
    "\n",
    "    Integer = 1234\n",
    "    3+3\n",
    "    \n",
    "    Float = 12.34\n",
    "    2 * 12.34"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data structures\n",
    "\n",
    "Variables alone can only do so much. By placing variables within data structures, we can begin to preform more interesting tasks.\n",
    "\n",
    "For this Hackathon, we will only be looking at 1 data structure: Dictionaries.\n",
    "\n",
    "#### Dictionaries\n",
    "\n",
    "Dictionaries are known as \"key-value\" pairs. This means for each \"key\" there is a value associated with it.\n",
    "\n",
    "Here I have pre-initialized a dictionary with some key value pairs:\n",
    "\n",
    "    phone_dict = { \"Police\" : 911,\n",
    "                  \"City of Toronto Archives\": 4163970778,\n",
    "                  \"Pizza-Pizza\" : 4169671111 }\n",
    "\n",
    "This dictionary contains 3 key-value pairs. Each key is a String containing the name of a business. Each value is an Integer representing the phone number of the business. \n",
    "\n",
    "Note that, keys and values can be of any data type. We could have easily assigned the phone number to be the key and the name of the business to be the value.\n",
    "\n",
    "#### Common operations on dictionaries\n",
    "\n",
    "To add/edit a key-value pair inside a dictionary:\n",
    "\n",
    "    phone_dict = {}\n",
    "\n",
    "    # to add    \n",
    "    phone_dict[\"Dominos\"] = 4166407777\n",
    "    \n",
    "    # now to edit an existing value\n",
    "    phone_dict[\"Dominos\"] = 1234567890\n",
    "    \n",
    "To access the value of a key:\n",
    "\n",
    "    print(phone_dict[\"Police\"])\n",
    "    \n",
    "And finally to check if a key is in a dictionary:\n",
    "\n",
    "    if \"Pizza-Pizza\" in phone_dict:\n",
    "        print('This is Pizza Pizza's number: ' + phone_dict[\"Pizza-Pizza\"])\n",
    "    elif \"Pizza-Pizza\" not in phone_dict:\n",
    "        print('We don't have the number')\n",
    "\n",
    "Note that if you were to try to get the value for a key which doesn't exist in the dictionary, you would get an error message. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lesson 2 - File I/O (Input & Output)\n",
    "\n",
    "As a member of the archive team, you are aware that there are thousands of individual files containing important information. Using Python you can access the text within these files and manipulate, sort and transform this data.\n",
    "\n",
    "For this section, we will only be looking at how to access data within a plain-text file (.txt). Matthew will show you how to access other file types (like PDF's) in a future section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Opening and reading from a file\n",
    "\n",
    "You can open a file in Python using a single line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We opened it\n"
     ]
    }
   ],
   "source": [
    "with open(\"~$HF 2010-11 annual report final from MOHLTC.txt\") as f:\n",
    "    print('We opened it')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In line 1, we open a file named \"filename.txt\" and stored a reference to this file in a variable called 'f'.\n",
    "\n",
    "To begin reading this file, we need to operate on this 'f'. So, we write:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Since I assumed this position five years ago, much has changed at the Foundation:  the Foundation moved offices from College Street to Jarvis Street; Ms. Sandy Hengeveld and Ms. Emmanuelle Fontaine joined the staff; the Foundation moved from a paper-based system to a web-based system for applications and reports; Drs. Keith Jarvie and Vivian Rakoff began and completed their terms as, respectively, Chair and Vice-Chair;  six new members have joined the Board of the Foundation – Drs. Mary Seeman, Gregory Brown, George Tolomizencko, Mr. Herman Gill, Ms. Clare Sullivan, and Ms. Jeanette Lewis; Drs. William Avison and Harriet MacMillan were appointed Chair and Vice-Chair of the Foundation, and the membership of both the Grants and Fellowships Committees has changed considerably  since 2005. The Foundation has, however, continued – as it has for nearly 45 years – to support excellent research to promote the mental health of Ontarians, to prevent mental illness, and to improve diagnosis, treatment, and rehabilitation.  \n",
      "\n",
      "Perhaps the greatest change in my tenure as Executive Director has been the challenge of meeting the rising demand for mental health research funding in Ontario. In 2005, the Foundation received 65 fellowship/studentship applications and 60 grant applications; the Foundation was able to fund 33% of the fellowship/studentship applications and 25% of the grant applications.  In 2009, the number of fellowship/studentship applications had risen to 93 and the number of grant applications rose to 78; the Foundation was able to fund only 19% of the grant applications, but 34% of the fellowships/studentships.  \n",
      "\n",
      "It is through the OMHF Board’s commitment to sustaining and enlarging the pool of mental health researchers in Ontario and its prudence in managing the funds provided through the generosity of the Government and taxpayers of Ontario that the Foundation has been able to continue to fulfill its mission.  The Board’s decision, for but one example, to allocate a greater portion of the available funds to support of students will ensure that mental health research in Ontario remains the most energetic in Canada. \tThere are sure to be challenges ahead; I am also sure, however, that the Foundation with the counsel and support of the Board, the staff, the Ministry, and the volunteer scientists who serve on the Foundation’s peer review Committees will meet them and continue to deserve its reputation for excellence in mental health research.\n",
      "\n",
      "I am grateful for the support of many people over the past year. The Board of the Foundation, both those whose terms have expired and the current members, has offered wise counsel and encouragement. Ms. Fontaine and Ms. Hengeveld keep the Foundation running smoothly and are responsible for the web-based application system. Thanks also to the staff of the Ministry of Health and Long-Term Care – Vasanthi Srinivasan, Anne Bowlby, Brian Davidson, Danielle Laymen-Pleet, Christy Pentland, and Amy Herskowitz; their commitment to the mental health of Ontarians is evident in the work of the Foundation.\n",
      "\n",
      "The members of the Foundation’s peer review Committees – Grant and Fellowships – are the backbone of the Foundation’s mission. The Chairs of the Committees – Drs. Patricia Rosebush (McMaster) and Steffany Bennett (Ottawa) have completed their terms. Both have been energetic and wise in their leadership and have been of inestimable help to me. Dr. Michael Kawaja (Queen’s) will serve as Chair of the Grants Committee and Dr. John Sylvestre (Ottawa) as the Chair of Fellowships. As usual, there are arrivals and departures to acknowledge. Drs. Walter Rushlow and Tracy Vaillancourt have left the Grants Committee; we welcome Dr. Steven Laviolette to that Committee. Drs. Michael Mazurek and Ayal Schaffer have left the Fellowships Committee. My thanks to all of the panels’ members for their commitment to mental health in Ontario.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Dr. Alexander Greer\n",
      "\n",
      "Executive Director\n",
      "\n",
      "July 2010\n"
     ]
    }
   ],
   "source": [
    "with open(\"~$HF 2010-11 annual report final from MOHLTC.txt\") as f:\n",
    "    data = f.read()\n",
    "    print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we store all text within the file in a String variable called data. When we print data, we print the contents of an entire file out the screen. \n",
    "\n",
    "And it only took 3 lines of code. Nice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Breaking the data\n",
    "\n",
    "What if you wanted to access the words in the data? We can do this by using split(' ').\n",
    "\n",
    "    with open(\"~$HF 2010-11 annual report final from MOHLTC.txt\") as f:\n",
    "        for line in f:\n",
    "            words = line.split(' ')\n",
    "            for word in words:\n",
    "                print('One word within the file: ' + word)\n",
    "                \n",
    "(*Hint*)\n",
    "Something interesting that you can do now, is that you can now count how many words there are in a file:\n",
    "\n",
    "    word_count = 0\n",
    "    \n",
    "    with open(\"~$HF 2010-11 annual report final from MOHLTC.txt\") as f:\n",
    "        for line in f:\n",
    "            words = line.split(' ')\n",
    "            for word in words:\n",
    "                word_count = word_count + 1\n",
    "    \n",
    "    print(word_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"~$HF 2010-11 annual report final from MOHLTC.txt\") as f:\n",
    "    for line in f:\n",
    "        words = line.split(' ')\n",
    "        print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In most written languages around the world, words are seperated by whitespace. split(' ') stores each block of text in between spaces in to a List.\n",
    "\n",
    "We can also go through each element within a List by doing this:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "One word within the file: Since\n",
      "One word within the file: I\n",
      "One word within the file: assumed\n",
      "One word within the file: this\n",
      "One word within the file: position\n",
      "One word within the file: five\n",
      "One word within the file: years\n",
      "One word within the file: ago,\n",
      "One word within the file: much\n",
      "One word within the file: has\n",
      "One word within the file: changed\n",
      "One word within the file: at\n",
      "One word within the file: the\n",
      "One word within the file: Foundation:\n",
      "One word within the file: \n",
      "One word within the file: the\n",
      "One word within the file: Foundation\n",
      "One word within the file: moved\n",
      "One word within the file: offices\n",
      "One word within the file: from\n",
      "One word within the file: College\n",
      "One word within the file: Street\n",
      "One word within the file: to\n",
      "One word within the file: Jarvis\n",
      "One word within the file: Street;\n",
      "One word within the file: Ms.\n",
      "One word within the file: Sandy\n",
      "One word within the file: Hengeveld\n",
      "One word within the file: and\n",
      "One word within the file: Ms.\n",
      "One word within the file: Emmanuelle\n",
      "One word within the file: Fontaine\n",
      "One word within the file: joined\n",
      "One word within the file: the\n",
      "One word within the file: staff;\n",
      "One word within the file: the\n",
      "One word within the file: Foundation\n",
      "One word within the file: moved\n",
      "One word within the file: from\n",
      "One word within the file: a\n",
      "One word within the file: paper-based\n",
      "One word within the file: system\n",
      "One word within the file: to\n",
      "One word within the file: a\n",
      "One word within the file: web-based\n",
      "One word within the file: system\n",
      "One word within the file: for\n",
      "One word within the file: applications\n",
      "One word within the file: and\n",
      "One word within the file: reports;\n",
      "One word within the file: Drs.\n",
      "One word within the file: Keith\n",
      "One word within the file: Jarvie\n",
      "One word within the file: and\n",
      "One word within the file: Vivian\n",
      "One word within the file: Rakoff\n",
      "One word within the file: began\n",
      "One word within the file: and\n",
      "One word within the file: completed\n",
      "One word within the file: their\n",
      "One word within the file: terms\n",
      "One word within the file: as,\n",
      "One word within the file: respectively,\n",
      "One word within the file: Chair\n",
      "One word within the file: and\n",
      "One word within the file: Vice-Chair;\n",
      "One word within the file: \n",
      "One word within the file: six\n",
      "One word within the file: new\n",
      "One word within the file: members\n",
      "One word within the file: have\n",
      "One word within the file: joined\n",
      "One word within the file: the\n",
      "One word within the file: Board\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: Foundation\n",
      "One word within the file: –\n",
      "One word within the file: Drs.\n",
      "One word within the file: Mary\n",
      "One word within the file: Seeman,\n",
      "One word within the file: Gregory\n",
      "One word within the file: Brown,\n",
      "One word within the file: George\n",
      "One word within the file: Tolomizencko,\n",
      "One word within the file: Mr.\n",
      "One word within the file: Herman\n",
      "One word within the file: Gill,\n",
      "One word within the file: Ms.\n",
      "One word within the file: Clare\n",
      "One word within the file: Sullivan,\n",
      "One word within the file: and\n",
      "One word within the file: Ms.\n",
      "One word within the file: Jeanette\n",
      "One word within the file: Lewis;\n",
      "One word within the file: Drs.\n",
      "One word within the file: William\n",
      "One word within the file: Avison\n",
      "One word within the file: and\n",
      "One word within the file: Harriet\n",
      "One word within the file: MacMillan\n",
      "One word within the file: were\n",
      "One word within the file: appointed\n",
      "One word within the file: Chair\n",
      "One word within the file: and\n",
      "One word within the file: Vice-Chair\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: Foundation,\n",
      "One word within the file: and\n",
      "One word within the file: the\n",
      "One word within the file: membership\n",
      "One word within the file: of\n",
      "One word within the file: both\n",
      "One word within the file: the\n",
      "One word within the file: Grants\n",
      "One word within the file: and\n",
      "One word within the file: Fellowships\n",
      "One word within the file: Committees\n",
      "One word within the file: has\n",
      "One word within the file: changed\n",
      "One word within the file: considerably\n",
      "One word within the file: \n",
      "One word within the file: since\n",
      "One word within the file: 2005.\n",
      "One word within the file: The\n",
      "One word within the file: Foundation\n",
      "One word within the file: has,\n",
      "One word within the file: however,\n",
      "One word within the file: continued\n",
      "One word within the file: –\n",
      "One word within the file: as\n",
      "One word within the file: it\n",
      "One word within the file: has\n",
      "One word within the file: for\n",
      "One word within the file: nearly\n",
      "One word within the file: 45\n",
      "One word within the file: years\n",
      "One word within the file: –\n",
      "One word within the file: to\n",
      "One word within the file: support\n",
      "One word within the file: excellent\n",
      "One word within the file: research\n",
      "One word within the file: to\n",
      "One word within the file: promote\n",
      "One word within the file: the\n",
      "One word within the file: mental\n",
      "One word within the file: health\n",
      "One word within the file: of\n",
      "One word within the file: Ontarians,\n",
      "One word within the file: to\n",
      "One word within the file: prevent\n",
      "One word within the file: mental\n",
      "One word within the file: illness,\n",
      "One word within the file: and\n",
      "One word within the file: to\n",
      "One word within the file: improve\n",
      "One word within the file: diagnosis,\n",
      "One word within the file: treatment,\n",
      "One word within the file: and\n",
      "One word within the file: rehabilitation.\n",
      "One word within the file: \n",
      "One word within the file: \n",
      "\n",
      "One word within the file: \n",
      "\n",
      "One word within the file: Perhaps\n",
      "One word within the file: the\n",
      "One word within the file: greatest\n",
      "One word within the file: change\n",
      "One word within the file: in\n",
      "One word within the file: my\n",
      "One word within the file: tenure\n",
      "One word within the file: as\n",
      "One word within the file: Executive\n",
      "One word within the file: Director\n",
      "One word within the file: has\n",
      "One word within the file: been\n",
      "One word within the file: the\n",
      "One word within the file: challenge\n",
      "One word within the file: of\n",
      "One word within the file: meeting\n",
      "One word within the file: the\n",
      "One word within the file: rising\n",
      "One word within the file: demand\n",
      "One word within the file: for\n",
      "One word within the file: mental\n",
      "One word within the file: health\n",
      "One word within the file: research\n",
      "One word within the file: funding\n",
      "One word within the file: in\n",
      "One word within the file: Ontario.\n",
      "One word within the file: In\n",
      "One word within the file: 2005,\n",
      "One word within the file: the\n",
      "One word within the file: Foundation\n",
      "One word within the file: received\n",
      "One word within the file: 65\n",
      "One word within the file: fellowship/studentship\n",
      "One word within the file: applications\n",
      "One word within the file: and\n",
      "One word within the file: 60\n",
      "One word within the file: grant\n",
      "One word within the file: applications;\n",
      "One word within the file: the\n",
      "One word within the file: Foundation\n",
      "One word within the file: was\n",
      "One word within the file: able\n",
      "One word within the file: to\n",
      "One word within the file: fund\n",
      "One word within the file: 33%\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: fellowship/studentship\n",
      "One word within the file: applications\n",
      "One word within the file: and\n",
      "One word within the file: 25%\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: grant\n",
      "One word within the file: applications.\n",
      "One word within the file: \n",
      "One word within the file: In\n",
      "One word within the file: 2009,\n",
      "One word within the file: the\n",
      "One word within the file: number\n",
      "One word within the file: of\n",
      "One word within the file: fellowship/studentship\n",
      "One word within the file: applications\n",
      "One word within the file: had\n",
      "One word within the file: risen\n",
      "One word within the file: to\n",
      "One word within the file: 93\n",
      "One word within the file: and\n",
      "One word within the file: the\n",
      "One word within the file: number\n",
      "One word within the file: of\n",
      "One word within the file: grant\n",
      "One word within the file: applications\n",
      "One word within the file: rose\n",
      "One word within the file: to\n",
      "One word within the file: 78;\n",
      "One word within the file: the\n",
      "One word within the file: Foundation\n",
      "One word within the file: was\n",
      "One word within the file: able\n",
      "One word within the file: to\n",
      "One word within the file: fund\n",
      "One word within the file: only\n",
      "One word within the file: 19%\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: grant\n",
      "One word within the file: applications,\n",
      "One word within the file: but\n",
      "One word within the file: 34%\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: fellowships/studentships.\n",
      "One word within the file: \n",
      "One word within the file: \n",
      "\n",
      "One word within the file: \n",
      "\n",
      "One word within the file: It\n",
      "One word within the file: is\n",
      "One word within the file: through\n",
      "One word within the file: the\n",
      "One word within the file: OMHF\n",
      "One word within the file: Board’s\n",
      "One word within the file: commitment\n",
      "One word within the file: to\n",
      "One word within the file: sustaining\n",
      "One word within the file: and\n",
      "One word within the file: enlarging\n",
      "One word within the file: the\n",
      "One word within the file: pool\n",
      "One word within the file: of\n",
      "One word within the file: mental\n",
      "One word within the file: health\n",
      "One word within the file: researchers\n",
      "One word within the file: in\n",
      "One word within the file: Ontario\n",
      "One word within the file: and\n",
      "One word within the file: its\n",
      "One word within the file: prudence\n",
      "One word within the file: in\n",
      "One word within the file: managing\n",
      "One word within the file: the\n",
      "One word within the file: funds\n",
      "One word within the file: provided\n",
      "One word within the file: through\n",
      "One word within the file: the\n",
      "One word within the file: generosity\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: Government\n",
      "One word within the file: and\n",
      "One word within the file: taxpayers\n",
      "One word within the file: of\n",
      "One word within the file: Ontario\n",
      "One word within the file: that\n",
      "One word within the file: the\n",
      "One word within the file: Foundation\n",
      "One word within the file: has\n",
      "One word within the file: been\n",
      "One word within the file: able\n",
      "One word within the file: to\n",
      "One word within the file: continue\n",
      "One word within the file: to\n",
      "One word within the file: fulfill\n",
      "One word within the file: its\n",
      "One word within the file: mission.\n",
      "One word within the file: \n",
      "One word within the file: The\n",
      "One word within the file: Board’s\n",
      "One word within the file: decision,\n",
      "One word within the file: for\n",
      "One word within the file: but\n",
      "One word within the file: one\n",
      "One word within the file: example,\n",
      "One word within the file: to\n",
      "One word within the file: allocate\n",
      "One word within the file: a\n",
      "One word within the file: greater\n",
      "One word within the file: portion\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: available\n",
      "One word within the file: funds\n",
      "One word within the file: to\n",
      "One word within the file: support\n",
      "One word within the file: of\n",
      "One word within the file: students\n",
      "One word within the file: will\n",
      "One word within the file: ensure\n",
      "One word within the file: that\n",
      "One word within the file: mental\n",
      "One word within the file: health\n",
      "One word within the file: research\n",
      "One word within the file: in\n",
      "One word within the file: Ontario\n",
      "One word within the file: remains\n",
      "One word within the file: the\n",
      "One word within the file: most\n",
      "One word within the file: energetic\n",
      "One word within the file: in\n",
      "One word within the file: Canada.\n",
      "One word within the file: \tThere\n",
      "One word within the file: are\n",
      "One word within the file: sure\n",
      "One word within the file: to\n",
      "One word within the file: be\n",
      "One word within the file: challenges\n",
      "One word within the file: ahead;\n",
      "One word within the file: I\n",
      "One word within the file: am\n",
      "One word within the file: also\n",
      "One word within the file: sure,\n",
      "One word within the file: however,\n",
      "One word within the file: that\n",
      "One word within the file: the\n",
      "One word within the file: Foundation\n",
      "One word within the file: with\n",
      "One word within the file: the\n",
      "One word within the file: counsel\n",
      "One word within the file: and\n",
      "One word within the file: support\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: Board,\n",
      "One word within the file: the\n",
      "One word within the file: staff,\n",
      "One word within the file: the\n",
      "One word within the file: Ministry,\n",
      "One word within the file: and\n",
      "One word within the file: the\n",
      "One word within the file: volunteer\n",
      "One word within the file: scientists\n",
      "One word within the file: who\n",
      "One word within the file: serve\n",
      "One word within the file: on\n",
      "One word within the file: the\n",
      "One word within the file: Foundation’s\n",
      "One word within the file: peer\n",
      "One word within the file: review\n",
      "One word within the file: Committees\n",
      "One word within the file: will\n",
      "One word within the file: meet\n",
      "One word within the file: them\n",
      "One word within the file: and\n",
      "One word within the file: continue\n",
      "One word within the file: to\n",
      "One word within the file: deserve\n",
      "One word within the file: its\n",
      "One word within the file: reputation\n",
      "One word within the file: for\n",
      "One word within the file: excellence\n",
      "One word within the file: in\n",
      "One word within the file: mental\n",
      "One word within the file: health\n",
      "One word within the file: research.\n",
      "\n",
      "One word within the file: \n",
      "\n",
      "One word within the file: I\n",
      "One word within the file: am\n",
      "One word within the file: grateful\n",
      "One word within the file: for\n",
      "One word within the file: the\n",
      "One word within the file: support\n",
      "One word within the file: of\n",
      "One word within the file: many\n",
      "One word within the file: people\n",
      "One word within the file: over\n",
      "One word within the file: the\n",
      "One word within the file: past\n",
      "One word within the file: year.\n",
      "One word within the file: The\n",
      "One word within the file: Board\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: Foundation,\n",
      "One word within the file: both\n",
      "One word within the file: those\n",
      "One word within the file: whose\n",
      "One word within the file: terms\n",
      "One word within the file: have\n",
      "One word within the file: expired\n",
      "One word within the file: and\n",
      "One word within the file: the\n",
      "One word within the file: current\n",
      "One word within the file: members,\n",
      "One word within the file: has\n",
      "One word within the file: offered\n",
      "One word within the file: wise\n",
      "One word within the file: counsel\n",
      "One word within the file: and\n",
      "One word within the file: encouragement.\n",
      "One word within the file: Ms.\n",
      "One word within the file: Fontaine\n",
      "One word within the file: and\n",
      "One word within the file: Ms.\n",
      "One word within the file: Hengeveld\n",
      "One word within the file: keep\n",
      "One word within the file: the\n",
      "One word within the file: Foundation\n",
      "One word within the file: running\n",
      "One word within the file: smoothly\n",
      "One word within the file: and\n",
      "One word within the file: are\n",
      "One word within the file: responsible\n",
      "One word within the file: for\n",
      "One word within the file: the\n",
      "One word within the file: web-based\n",
      "One word within the file: application\n",
      "One word within the file: system.\n",
      "One word within the file: Thanks\n",
      "One word within the file: also\n",
      "One word within the file: to\n",
      "One word within the file: the\n",
      "One word within the file: staff\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: Ministry\n",
      "One word within the file: of\n",
      "One word within the file: Health\n",
      "One word within the file: and\n",
      "One word within the file: Long-Term\n",
      "One word within the file: Care\n",
      "One word within the file: –\n",
      "One word within the file: Vasanthi\n",
      "One word within the file: Srinivasan,\n",
      "One word within the file: Anne\n",
      "One word within the file: Bowlby,\n",
      "One word within the file: Brian\n",
      "One word within the file: Davidson,\n",
      "One word within the file: Danielle\n",
      "One word within the file: Laymen-Pleet,\n",
      "One word within the file: Christy\n",
      "One word within the file: Pentland,\n",
      "One word within the file: and\n",
      "One word within the file: Amy\n",
      "One word within the file: Herskowitz;\n",
      "One word within the file: their\n",
      "One word within the file: commitment\n",
      "One word within the file: to\n",
      "One word within the file: the\n",
      "One word within the file: mental\n",
      "One word within the file: health\n",
      "One word within the file: of\n",
      "One word within the file: Ontarians\n",
      "One word within the file: is\n",
      "One word within the file: evident\n",
      "One word within the file: in\n",
      "One word within the file: the\n",
      "One word within the file: work\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: Foundation.\n",
      "\n",
      "One word within the file: \n",
      "\n",
      "One word within the file: The\n",
      "One word within the file: members\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: Foundation’s\n",
      "One word within the file: peer\n",
      "One word within the file: review\n",
      "One word within the file: Committees\n",
      "One word within the file: –\n",
      "One word within the file: Grant\n",
      "One word within the file: and\n",
      "One word within the file: Fellowships\n",
      "One word within the file: –\n",
      "One word within the file: are\n",
      "One word within the file: the\n",
      "One word within the file: backbone\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: Foundation’s\n",
      "One word within the file: mission.\n",
      "One word within the file: The\n",
      "One word within the file: Chairs\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: Committees\n",
      "One word within the file: –\n",
      "One word within the file: Drs.\n",
      "One word within the file: Patricia\n",
      "One word within the file: Rosebush\n",
      "One word within the file: (McMaster)\n",
      "One word within the file: and\n",
      "One word within the file: Steffany\n",
      "One word within the file: Bennett\n",
      "One word within the file: (Ottawa)\n",
      "One word within the file: have\n",
      "One word within the file: completed\n",
      "One word within the file: their\n",
      "One word within the file: terms.\n",
      "One word within the file: Both\n",
      "One word within the file: have\n",
      "One word within the file: been\n",
      "One word within the file: energetic\n",
      "One word within the file: and\n",
      "One word within the file: wise\n",
      "One word within the file: in\n",
      "One word within the file: their\n",
      "One word within the file: leadership\n",
      "One word within the file: and\n",
      "One word within the file: have\n",
      "One word within the file: been\n",
      "One word within the file: of\n",
      "One word within the file: inestimable\n",
      "One word within the file: help\n",
      "One word within the file: to\n",
      "One word within the file: me.\n",
      "One word within the file: Dr.\n",
      "One word within the file: Michael\n",
      "One word within the file: Kawaja\n",
      "One word within the file: (Queen’s)\n",
      "One word within the file: will\n",
      "One word within the file: serve\n",
      "One word within the file: as\n",
      "One word within the file: Chair\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: Grants\n",
      "One word within the file: Committee\n",
      "One word within the file: and\n",
      "One word within the file: Dr.\n",
      "One word within the file: John\n",
      "One word within the file: Sylvestre\n",
      "One word within the file: (Ottawa)\n",
      "One word within the file: as\n",
      "One word within the file: the\n",
      "One word within the file: Chair\n",
      "One word within the file: of\n",
      "One word within the file: Fellowships.\n",
      "One word within the file: As\n",
      "One word within the file: usual,\n",
      "One word within the file: there\n",
      "One word within the file: are\n",
      "One word within the file: arrivals\n",
      "One word within the file: and\n",
      "One word within the file: departures\n",
      "One word within the file: to\n",
      "One word within the file: acknowledge.\n",
      "One word within the file: Drs.\n",
      "One word within the file: Walter\n",
      "One word within the file: Rushlow\n",
      "One word within the file: and\n",
      "One word within the file: Tracy\n",
      "One word within the file: Vaillancourt\n",
      "One word within the file: have\n",
      "One word within the file: left\n",
      "One word within the file: the\n",
      "One word within the file: Grants\n",
      "One word within the file: Committee;\n",
      "One word within the file: we\n",
      "One word within the file: welcome\n",
      "One word within the file: Dr.\n",
      "One word within the file: Steven\n",
      "One word within the file: Laviolette\n",
      "One word within the file: to\n",
      "One word within the file: that\n",
      "One word within the file: Committee.\n",
      "One word within the file: Drs.\n",
      "One word within the file: Michael\n",
      "One word within the file: Mazurek\n",
      "One word within the file: and\n",
      "One word within the file: Ayal\n",
      "One word within the file: Schaffer\n",
      "One word within the file: have\n",
      "One word within the file: left\n",
      "One word within the file: the\n",
      "One word within the file: Fellowships\n",
      "One word within the file: Committee.\n",
      "One word within the file: My\n",
      "One word within the file: thanks\n",
      "One word within the file: to\n",
      "One word within the file: all\n",
      "One word within the file: of\n",
      "One word within the file: the\n",
      "One word within the file: panels’\n",
      "One word within the file: members\n",
      "One word within the file: for\n",
      "One word within the file: their\n",
      "One word within the file: commitment\n",
      "One word within the file: to\n",
      "One word within the file: mental\n",
      "One word within the file: health\n",
      "One word within the file: in\n",
      "One word within the file: Ontario.\n",
      "\n",
      "One word within the file: \n",
      "\n",
      "One word within the file: \n",
      "\n",
      "One word within the file: \n",
      "\n",
      "One word within the file: \n",
      "\n",
      "One word within the file: \n",
      "\n",
      "One word within the file: \n",
      "\n",
      "One word within the file: \n",
      "\n",
      "One word within the file: Dr.\n",
      "One word within the file: Alexander\n",
      "One word within the file: Greer\n",
      "\n",
      "One word within the file: \n",
      "\n",
      "One word within the file: Executive\n",
      "One word within the file: Director\n",
      "\n",
      "One word within the file: \n",
      "\n",
      "One word within the file: July\n",
      "One word within the file: 2010\n"
     ]
    }
   ],
   "source": [
    "with open(\"~$HF 2010-11 annual report final from MOHLTC.txt\") as f:\n",
    "    for line in f:\n",
    "        words = line.split(' ')\n",
    "        for word in words:\n",
    "            print('One word within the file: ' + word)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(*Hint*)\n",
    "Something interesting that you can do now, is that you can now count how many words there are in a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "643\n"
     ]
    }
   ],
   "source": [
    "word_count = 0\n",
    "    \n",
    "with open(\"~$HF 2010-11 annual report final from MOHLTC.txt\") as f:\n",
    "    for line in f:\n",
    "        words = line.split(' ')\n",
    "        for word in words:\n",
    "            word_count = word_count + 1\n",
    "    \n",
    "print(word_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing to files\n",
    "\n",
    "*Skippable!*\n",
    "\n",
    "It is possible to write lines of text to a file. We use similar code as reading:\n",
    "\n",
    "    with open(\"filename.txt\", \"w\") as f:\n",
    "       f.write(\"I love programming in Python.\")\n",
    "\n",
    "Note a few differences:\n",
    "    1. We open \"filename.txt\" in 'w' mode - write mode.\n",
    "    2. We still store a reference to this file in a variable called 'f'\n",
    "    3. But we can now use the 'write' operation. If you were to open a file in read mode and run this line of code, your script would crash\n",
    "\n",
    "Writing to files is important, but is not as relevant to this Hackathon. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Programming challenge #1\n",
    "\n",
    "Section requirements:\n",
    "    1. Data structures\n",
    "    2. Opening and reading a file\n",
    "    3. Breaking down each line\n",
    "\n",
    "Now that you can read the contents of a file and store data inside a data structure, can you do both by storing the count of each word in to a Dictionary?\n",
    "\n",
    "(*Hint*) Your program will first need to have a completely empty dictionary. This can be initialized by using:\n",
    "\n",
    "    word_count_dict = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lesson 3 - Parallel Programming\n",
    "\n",
    "In the world of big data analytics, it not unusual that a computational job takes a single computer several days to complete. A common way to lower the time a program's runs for is to distribute certain portions of the job across many computers running simultaneously.\n",
    "\n",
    "The field of using simultaneous computional processes is known as parallel computing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Parallelism\n",
    "\n",
    "![Image of Data Parallelism](https://computing.llnl.gov/tutorials/parallel_comp/images/domain_decomp.gif)\n",
    "\n",
    "Data parallelism is the most well known method to achieve parallel computing. In this method, the input data set is divided equally and sent to multiple processes. \n",
    "\n",
    "Here is an analogy. Lets say that I want to make an entire bag of flour in to pizza dough. If I were to make a set amount alone, it would take me a while, lets say 4 hours. If 3 of my buddies joined in and we were making this dough all at the same time, it would take me 4 / 4 = 1 hour. Each buddy would take 1/4th of the flour to make the dough.\n",
    "\n",
    "Data paralleism works in a similar way. Each \"worker\" process preforms the exactly the same task, but is given a different set of inputs to work on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding process 0, which runs on the word # 0 - 155.\n",
      "Adding process 1, which runs on the word # 155 - 311.\n",
      "Adding process 2, which runs on the word # 311 - 467.\n",
      "Adding process 3, which runs on the word # 467 - 623.\n",
      "4\n",
      "Process 0 found the following word counts in its section:\n",
      "{'Since': 1, 'I': 1, 'assumed': 1, 'this': 1, 'position': 1, 'five': 1, 'years': 2, 'ago,': 1, 'much': 1, 'has': 3, 'changed': 2, 'at': 1, 'the': 10, 'Foundation:': 1, '': 3, 'Foundation': 4, 'moved': 2, 'offices': 1, 'from': 2, 'College': 1, 'Street': 1, 'to': 5, 'Jarvis': 1, 'Street;': 1, 'Ms.': 4, 'Sandy': 1, 'Hengeveld': 1, 'and': 11, 'Emmanuelle': 1, 'Fontaine': 1, 'joined': 2, 'staff;': 1, 'a': 2, 'paper-based': 1, 'system': 2, 'web-based': 1, 'for': 2, 'applications': 1, 'reports;': 1, 'Drs.': 3, 'Keith': 1, 'Jarvie': 1, 'Vivian': 1, 'Rakoff': 1, 'began': 1, 'completed': 1, 'their': 1, 'terms': 1, 'as,': 1, 'respectively,': 1, 'Chair': 2, 'Vice-Chair;': 1, 'six': 1, 'new': 1, 'members': 1, 'have': 1, 'Board': 1, 'of': 4, '–': 3, 'Mary': 1, 'Seeman,': 1, 'Gregory': 1, 'Brown,': 1, 'George': 1, 'Tolomizencko,': 1, 'Mr.': 1, 'Herman': 1, 'Gill,': 1, 'Clare': 1, 'Sullivan,': 1, 'Jeanette': 1, 'Lewis;': 1, 'William': 1, 'Avison': 1, 'Harriet': 1, 'MacMillan': 1, 'were': 1, 'appointed': 1, 'Vice-Chair': 1, 'Foundation,': 1, 'membership': 1, 'both': 1, 'Grants': 1, 'Fellowships': 1, 'Committees': 1, 'considerably': 1, 'since': 1, '2005.': 1, 'The': 1, 'has,': 1, 'however,': 1, 'continued': 1, 'as': 1, 'it': 1, 'nearly': 1, '45': 1, 'support': 1, 'excellent': 1, 'research': 1, 'promote': 1, 'mental': 2, 'health': 1, 'Ontarians,': 1, 'prevent': 1, 'illness,': 1}\n",
      "Process 1 found the following word counts in its section:\n",
      "{'to': 8, 'improve': 1, 'diagnosis,': 1, 'treatment,': 1, 'and': 7, 'rehabilitation.': 1, '': 4, '\\n\\nPerhaps': 1, 'the': 18, 'greatest': 1, 'change': 1, 'in': 4, 'my': 1, 'tenure': 1, 'as': 1, 'Executive': 1, 'Director': 1, 'has': 2, 'been': 2, 'challenge': 1, 'of': 10, 'meeting': 1, 'rising': 1, 'demand': 1, 'for': 2, 'mental': 2, 'health': 2, 'research': 1, 'funding': 1, 'Ontario.': 1, 'In': 2, '2005,': 1, 'Foundation': 4, 'received': 1, '65': 1, 'fellowship/studentship': 3, 'applications': 4, '60': 1, 'grant': 4, 'applications;': 1, 'was': 2, 'able': 3, 'fund': 2, '33%': 1, '25%': 1, 'applications.': 1, '2009,': 1, 'number': 2, 'had': 1, 'risen': 1, '93': 1, 'rose': 1, '78;': 1, 'only': 1, '19%': 1, 'applications,': 1, 'but': 2, '34%': 1, 'fellowships/studentships.': 1, '\\n\\nIt': 1, 'is': 1, 'through': 2, 'OMHF': 1, 'Board’s': 2, 'commitment': 1, 'sustaining': 1, 'enlarging': 1, 'pool': 1, 'researchers': 1, 'Ontario': 2, 'its': 2, 'prudence': 1, 'managing': 1, 'funds': 1, 'provided': 1, 'generosity': 1, 'Government': 1, 'taxpayers': 1, 'that': 1, 'continue': 1, 'fulfill': 1, 'mission.': 1, 'The': 1, 'decision,': 1}\n",
      "Process 2 found the following word counts in its section:\n",
      "{'one': 1, 'example,': 1, 'to': 5, 'allocate': 1, 'a': 1, 'greater': 1, 'portion': 1, 'of': 7, 'the': 17, 'available': 1, 'funds': 1, 'support': 3, 'students': 1, 'will': 2, 'ensure': 1, 'that': 2, 'mental': 2, 'health': 2, 'research': 1, 'in': 3, 'Ontario': 1, 'remains': 1, 'most': 1, 'energetic': 1, 'Canada.': 1, '\\tThere': 1, 'are': 2, 'sure': 1, 'be': 1, 'challenges': 1, 'ahead;': 1, 'I': 1, 'am': 2, 'also': 2, 'sure,': 1, 'however,': 1, 'Foundation': 2, 'with': 1, 'counsel': 2, 'and': 8, 'Board,': 1, 'staff,': 1, 'Ministry,': 1, 'volunteer': 1, 'scientists': 1, 'who': 1, 'serve': 1, 'on': 1, 'Foundation’s': 1, 'peer': 1, 'review': 1, 'Committees': 1, 'meet': 1, 'them': 1, 'continue': 1, 'deserve': 1, 'its': 1, 'reputation': 1, 'for': 3, 'excellence': 1, 'research.\\n\\nI': 1, 'grateful': 1, 'many': 1, 'people': 1, 'over': 1, 'past': 1, 'year.': 1, 'The': 1, 'Board': 1, 'Foundation,': 1, 'both': 1, 'those': 1, 'whose': 1, 'terms': 1, 'have': 1, 'expired': 1, 'current': 1, 'members,': 1, 'has': 1, 'offered': 1, 'wise': 1, 'encouragement.': 1, 'Ms.': 2, 'Fontaine': 1, 'Hengeveld': 1, 'keep': 1, 'running': 1, 'smoothly': 1, 'responsible': 1, 'web-based': 1, 'application': 1, 'system.': 1, 'Thanks': 1, 'staff': 1, 'Ministry': 1, 'Health': 1, 'Long-Term': 1, 'Care': 1, '–': 1, 'Vasanthi': 1, 'Srinivasan,': 1, 'Anne': 1, 'Bowlby,': 1, 'Brian': 1, 'Davidson,': 1, 'Danielle': 1, 'Laymen-Pleet,': 1}\n",
      "Process 3 found the following word counts in its section:\n",
      "{'Christy': 1, 'Pentland,': 1, 'and': 9, 'Amy': 1, 'Herskowitz;': 1, 'their': 4, 'commitment': 2, 'to': 6, 'the': 12, 'mental': 2, 'health': 2, 'of': 9, 'Ontarians': 1, 'is': 1, 'evident': 1, 'in': 3, 'work': 1, 'Foundation.\\n\\nThe': 1, 'members': 2, 'Foundation’s': 2, 'peer': 1, 'review': 1, 'Committees': 2, '–': 3, 'Grant': 1, 'Fellowships': 2, 'are': 2, 'backbone': 1, 'mission.': 1, 'The': 1, 'Chairs': 1, 'Drs.': 3, 'Patricia': 1, 'Rosebush': 1, '(McMaster)': 1, 'Steffany': 1, 'Bennett': 1, '(Ottawa)': 2, 'have': 5, 'completed': 1, 'terms.': 1, 'Both': 1, 'been': 2, 'energetic': 1, 'wise': 1, 'leadership': 1, 'inestimable': 1, 'help': 1, 'me.': 1, 'Dr.': 3, 'Michael': 2, 'Kawaja': 1, '(Queen’s)': 1, 'will': 1, 'serve': 1, 'as': 2, 'Chair': 2, 'Grants': 2, 'Committee': 1, 'John': 1, 'Sylvestre': 1, 'Fellowships.': 1, 'As': 1, 'usual,': 1, 'there': 1, 'arrivals': 1, 'departures': 1, 'acknowledge.': 1, 'Walter': 1, 'Rushlow': 1, 'Tracy': 1, 'Vaillancourt': 1, 'left': 2, 'Committee;': 1, 'we': 1, 'welcome': 1, 'Steven': 1, 'Laviolette': 1, 'that': 1, 'Committee.': 2, 'Mazurek': 1, 'Ayal': 1, 'Schaffer': 1, 'My': 1, 'thanks': 1, 'all': 1, 'panels’': 1, 'for': 1, 'Ontario.\\n\\n\\n\\n\\n\\n\\n\\nDr.': 1, 'Alexander': 1, 'Greer\\n\\nExecutive': 1, 'Director\\n\\nJuly': 1, '2010': 1}\n"
     ]
    }
   ],
   "source": [
    "#### Data parallelism coding example\n",
    "\n",
    "import multiprocessing as mp\n",
    "import string\n",
    "\n",
    "output = mp.Queue()\n",
    "\n",
    "def load_data(file):\n",
    "    with open(file) as f:\n",
    "        data = f.read()\n",
    "        for f in string.punctuation:\n",
    "            data = data.strip(f)\n",
    "        return data.split(' ')\n",
    "\n",
    "def count(output, data, begin, end):\n",
    "    unique_words = {}\n",
    "    for word in data[int(begin):int(end)]:\n",
    "        if word in unique_words:\n",
    "            unique_words[word] = unique_words[word] + 1\n",
    "        else:\n",
    "            unique_words[word] = 1\n",
    "    output.put(unique_words)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    data = load_data(\"~$HF 2010-11 annual report final from MOHLTC.txt\")\n",
    "    number_of_words_in_data = len(data)\n",
    "    set_size = number_of_words_in_data / 4\n",
    "        \n",
    "    processes = []\n",
    "    \n",
    "    # Specify which input data each \"worker\" process gets\n",
    "    for x in range(4):\n",
    "        begin = set_size * x\n",
    "        end = min(set_size * (x+1), number_of_words_in_data)\n",
    "        p = mp.Process(target=count, args=(output, data, begin, end))\n",
    "        \n",
    "        processes.append(p)\n",
    "        \n",
    "        print('Adding process %i, which runs on the word # %i - %i.' % (x, begin, end))\n",
    "        \n",
    "    # Run processes\n",
    "    for p in processes:\n",
    "        p.start()\n",
    "\n",
    "    # Exit the completed processes\n",
    "    for p in processes:\n",
    "        p.join()\n",
    "\n",
    "    # Get process results\n",
    "    results = [output.get() for p in processes]\n",
    "    \n",
    "    # Print the results to the screen\n",
    "    for process_id, result in enumerate(results):\n",
    "        print(\"Process %i found the following word counts in its section:\" % process_id)\n",
    "        print(str(result))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A note\n",
    "\n",
    "In reality, you will most likely be using tools which already implement parallel processing. In Shaikh's upcoming section, he will touch a little on Hadoop and how it can run on multiple computers. oooo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Programming challenge #2\n",
    "\n",
    "This is one for experienced programmers.\n",
    "\n",
    "In the above coding example, I showed you how to print a list of unique words corresponding to sections of the a body of text. Can you modify the program to show word counts across the whole document? You will need to combine the results of each process and store them in a central dictionary.\n",
    "\n",
    "*Hint*: To iterate across all key-value pairs within a dictionary:\n",
    "    \n",
    "    for key, value in dictionary.values():\n",
    "        print(key, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What's next?\n",
    "\n",
    "Congratulations! You have finished the first section of this Hackathon. You did it. You are the programmer.\n",
    "\n",
    "![Image of Hacker](https://ak1.picdn.net/shutterstock/videos/3365171/thumb/1.jpg)\n",
    "\n",
    "By the end of this section you will have touched upon how to read and store data in addition to having done a basic word count analysis. You will have touched on the topic of parallel computing as well.\n",
    "\n",
    "In future sections, you will learn how to read data in from other file extensions (.pdf, .doc, .xls etc.) and advanced analysis using machine learning models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}