SHARE_API_Skills_Training_Query_Institution_Data-In progress.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Objectives of this Webinar:**\n",
    "\n",
    "You will be able to query SHARE for records that are known to be tied to your institution, and might be tied to your institution.  You will then be able to export them to a csv file for review.\n",
    "\n",
    "Let's start in the Search Interface: http://share.osf.io"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Replicating this process in code**\n",
    "\n",
    "First we will set some values for Jupyter to use.\n",
    "\n",
    "Why is this useful?\n",
    "\n",
    "Remember Drew Barrymore in '50 First Dates'?\n",
    "\n",
    "http://www.imdb.com/title/tt0343660/\n",
    "\n",
    "**Special Tip:**  Put your cursor in the Cell (i.e., box) below and hit Shift+Enter on your keyboard to run it "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "SHARE_API = 'https://staging-share.osf.io/api/search/abstractcreativework/_search'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is now in memory!  However, beware.  Just like Drew Barrymore if you close this notebook and reopen it, jupyter has completely forgotten about it, and you will need to rerun it for Jupyter to know about it.\n",
    "\n",
    "**Special Tip:** Why all CAPS for 'SHARE_API'? It is convention (i.e., common practice) to use all capital letters when defining something that does not change?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's move onto to an example simple query to SHARE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import furl\n",
    "import requests\n",
    "\n",
    "search_url = furl.furl(SHARE_API)\n",
    "search_url.args['size'] = 3\n",
    "recent_results = requests.get(search_url.url).json()\n",
    "\n",
    "recent_results = recent_results['hits']['hits']\n",
    "\n",
    "print('The request URL is {}'.format(search_url.url))\n",
    "print('----------')\n",
    "for result in recent_results:\n",
    "    print(\n",
    "        '{} -- from {}'.format(\n",
    "            result['_source']['title'],\n",
    "            result['_source']['sources']\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add a search string\n",
    "\n",
    "**Special Tip:** Sometimes you can learn a lot from a url and let's look at it after we search using the search box on SHARE (http://share.osf.io)\n",
    "    \n",
    "https://share.osf.io/discover?q=university%20of%20oregon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Special Tip:**  To print text and variables together use the format function as below with '{}' anywhere you want to insert a value and then include them in order to the format method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Looking for exact matches use our facet filter**\n",
    "\n",
    "Let's go back to SHARE, apply a facet and see what it is doing\n",
    "\n",
    "http://share.osf.io\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "affiliation_query = {\n",
    "    \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's input a method into memory so we can use it later.  Do not worry too much about the specifics yet.\n",
    "\n",
    "**Special Tip:** Just like variables you can define methods that do a job and you can then use them later without rewriting what it does"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "\n",
    "def query_share(url, query):\n",
    "    # A helper function that will use the requests library,\n",
    "    # pass along the correct headers,\n",
    "    # and make the query we want\n",
    "    headers = {'Content-Type': 'application/json'}\n",
    "    data = json.dumps(query)\n",
    "    return requests.post(url, headers=headers, data=data, verify=False).json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Special Tip:** Use '#' as first character in a line if you want to put in a comment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "affiliation_results = query_share(search_url.url, affiliation_query)\n",
    "\n",
    "total_results = affiliation_results['hits']['total']\n",
    "print('total results found: {}'.format(total_results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where are my results?  Things can differently from staging to production environments.\n",
    "\n",
    "Let's create a new variable and change our search url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "affiliation_results = query_share(search_url.url, affiliation_query)\n",
    "\n",
    "total_results = affiliation_results['hits']['total']\n",
    "print('total results found: {}'.format(total_results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's dive into the results to see what is being returned and how to then push it into a csv file\n",
    "\n",
    "**Special Tip:** The built-in 'print' is a very easy way to explore what is happening"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "affiliation_results = query_share(search_url.url, affiliation_query)\n",
    "print('------------')\n",
    "print(affiliation_results)\n",
    "print('------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a lot of stuff and we need to dive in fartheR\n",
    "\n",
    "**Special Tip**: SHARE returns this in what is called a Dictionary in Python (i.e. map, hash) and you supply the name of 'key' to then get the 'value'\n",
    "\n",
    "**Special Tip**: You can also get the list of keys for a dictionary by calling the dict method.\n",
    "\n",
    "So, we will print out our list of keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like what we need is 'hits'\n",
    "\n",
    "**Special Tip:** Make sure you put '' (single quotes) around the key name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "affiliation_results = query_share(search_url.url, affiliation_query)\n",
    "print(affiliation_results['hits'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is better but we are still not quite there.  Let's call keys again one level down"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, it looks like we need to add another hits\n",
    "\n",
    "**Special Tip:** You can access multiple levels of nested dictionaries by adding additional '[]' to the variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check our next set of nested keys..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did you get an error?\n",
    "\n",
    "What just happened here?  We have now encountered 'list' and that does not have a notion of keys.  List is essentially just a pile of records that has an order (first, second,...,last) but no keys assigned.  So, how do we interact with that?\n",
    "\n",
    "Let's take a look at the documentation by doing a google search for python list..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Interacting with lists**\n",
    "\n",
    "It looks like maybe we have an item embedded in a list, and this list is actually contains an item per record returned by our search.\n",
    "\n",
    "What we really want though for outputting to csv is the field names to use as column headers\n",
    "\n",
    "So, let's see if we can get to that based on the 'pop' method we found in the documentation here: https://docs.python.org/3/tutorial/datastructures.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Close again.  It looks like what we need is the key '_source', and then maybe we can get our column headers..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "record = affiliation_results['hits']['hits'].pop()\n",
    "#add print statement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are REALLY close now, but not quite there.  Let's keeping digging and add one additional thing, a for loop\n",
    "\n",
    "**Special Tip:** You can move through a list of elements within a dictionary or list with a for loop.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "record = affiliation_results['hits']['hits'].pop()\n",
    "\n",
    "for key in record['_source'].keys(): \n",
    "    print(key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happened here?\n",
    "\n",
    "Why is the list empty?  When you pop something off the list, it is gone once you leave a 'cell'.  So we need to rerun our search and recreate our list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "affiliation_results = query_share(search_url.url, affiliation_query)\n",
    "#paste in the above code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Success!**    We have it, our list of fields and column headers!\n",
    "\n",
    "**Special Tip:** An advanced move would be to create a method that gives you your list of keys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at a record and print it to the output..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#put keys and values together\n",
    "for key in record['_source'].keys():\n",
    "    print('{}:{}'.format(key,record['_source'][key]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's further clean this up a little bit to make it easier to see what is happening...\n",
    "\n",
    "**Special Tip:** By breaking steps apart it makes code much easier to understand for everyone\n",
    "\n",
    "In this case we will add a new variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#put keys and values together\n",
    "record_source = record['_source']\n",
    "\n",
    "for key in record_source.keys():\n",
    "    print('{}:{}'.format(key,record_source[key]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Writing output to CSV**\n",
    "\n",
    "We are now ready to move to CSV.\n",
    "\n",
    "**Special Tip:** Copy, Paste, and Adapt...It is always easier to start from an example.\n",
    "\n",
    "Let's try writing to a simple csv file as an example (example pulled from https://docs.python.org/3/library/csv.html)\n",
    "\n",
    "**Special Tip:** Add print statements to show something started and finished when there is no output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "print('---begin writing file---')\n",
    "\n",
    "with open('names.csv', 'w') as csvfile:\n",
    "    fieldnames = ['first_name', 'last_name']\n",
    "    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n",
    "\n",
    "    writer.writeheader()\n",
    "    writer.writerow({'first_name': 'Baked', 'last_name': 'Beans'})\n",
    "    writer.writerow({'first_name': 'Lovely', 'last_name': 'Spam'})\n",
    "    writer.writerow({'first_name': 'Wonderful', 'last_name': 'Spam'})\n",
    "    \n",
    "print('---done writing file----')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did you find the file?  It should put it in whatever is the home directory for Jupyter.  This is likely your user folder.\n",
    "\n",
    "Let's adapt this now with our data from SHARE..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "affiliation_results = query_share(search_url.url, affiliation_query)\n",
    "records = affiliation_results['hits']['hits']\n",
    "\n",
    "print('---begin writing file---')\n",
    "\n",
    "#set our filenames\n",
    "SHARE_MATCHING_INSTITUTION_RECORDS = 'share_matching.csv'\n",
    "\n",
    "with open(SHARE_MATCHING_INSTITUTION_RECORDS, 'w') as csvfile:\n",
    "    #instead of pop use a for loop\n",
    "    \n",
    "    #add loop of records\n",
    "        #writer header row\n",
    "        #write each row    \n",
    "        \n",
    "print('---done writing file----')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happened here?  \n",
    "\n",
    "It was confused by a nested dictionary and fieldnames it did not expect.\n",
    "\n",
    "**Special Tip:** 'extrasaction' value of 'ignore' will not throw an error if there are keys in this method such as a nested dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "affiliation_results = query_share(search_url.url, affiliation_query)\n",
    "records = affiliation_results['hits']['hits']\n",
    "\n",
    "print('---begin writing file---')\n",
    "\n",
    "#set our filenames\n",
    "SHARE_MATCHING_INSTITUTION_RECORDS = 'share_matching.csv'\n",
    "\n",
    "with open(SHARE_MATCHING_INSTITUTION_RECORDS, 'w') as csvfile:\n",
    "    #instead of pop us a for loop\n",
    "    i = 0\n",
    "    for row in records:\n",
    "        if i == 0:\n",
    "            fieldnames = row['_source'].keys()\n",
    "            writer = csv.DictWriter(csvfile, fieldnames=fieldnames, extrasaction='ignore')\n",
    "            writer.writeheader()\n",
    "            i = i + 1\n",
    "        writer.writerow(row['_source'])    \n",
    "        \n",
    "print('---done writing file---')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Wrapping Up**\n",
    "\n",
    "You have now successfully queried share for records tied to your institution and have pushed them to a csv file.\n",
    "\n",
    "**Future Steps**\n",
    "\n",
    "-There is more cleanup to be done to get other values like if something has more than one link by adding nested keys to the field names list as well will capture these values\n",
    "\n",
    "-Once you have the data, update and then push it back to SHARE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Other tips...**\n",
    "\n",
    "You can use the os module as shown below to change directories to a certain path if you want to put the csv file in a particular directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "#os.getcwd()\n",
    "#os.chdir('..')\n",
    "#os.getcwd()\n",
    "#os.chdir('share_tutorials')\n",
    "#os.getcwd()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}